llama.cpp is great. It started off as CPU-only solution and now looks like it wants to support any computation device it can.
I find it interesting that it's an example of an ML software that's totally detached from Python ML ecosystem and also popular.
Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts.
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts.
For me it's less about the language and more about the dependencies. When I want to run a Python ML program, I need to go through the hassle of figuring out what the minimum version is, which package manager/distribution I need to use, and what system libraries those dependencies need to function properly. If I want to build a package for my distribution, these problems are dialed up to 11 and make it difficult to integrate (especially when using Nix). On top of that, those dependencies typically hide the juicy details of the program I actually care about.
For something like C or C++? Usually the most complicated part is running `make` or `cmake` with `pkgconfig` somewhere in my path. Maybe install some missing system libraries if necessary.
I just don't want to install yet another hundred copies of dependencies in a virtualenv and just hope it's set up correctly.
> For me it's less about the language and more about the dependencies. When I want to run a Python ML program, I need to go through the hassle of figuring out what the minimum version is, which package manager/distribution I need to use, and what system libraries those dependencies need to function properly.
This is exactly why I hate Python. They even have a pseudo-package.json style dependencies file that you should supposedly be able to just run "install" with, but it NEVER works. Not once have I ever downloaded someone's Python project from github and tried to install the dependencies and run it has it ever gone smoothly and without issue.
The Python language itself may be great, I don't know, but I'm forever put off from learning or using it because clearly in all the years it's been around they have yet to figure out reproducibility of builds. And it's obviously possible! JavaScript manages to accomplish it just fine with npm and package.json. But for some reason Python and its community cannot figure it out.
Would the problem "how can I run this cool ML project from GitHub" be solved if developers would publish their container images on dockerhub? The only downside I see is enormous image sizes
"and by the way, inside the image I slightly modified one file of the tensors library (and this is undocumented) and this totally changes the output if the fix is not there"
I use pypoetry for dependency management in Python projects. It helps a lot, but doesn't resolve the issue of pip packages to fail installing because you're missing system libraries. At least it specified the Python version to use. With many open source ML repos I have to guess what Python version to use.
I'd really like to see more Docker images (images, not Dockerfiles that fail to build). Maybe flatpack or snap packages do the trick, too.
> Not once have I ever downloaded someone's Python project from github and tried to install the dependencies and run it has it ever gone smoothly and without issue.
Same here. If it can't resolve dependencies or whatever, then there will almost certainly be some kind of showstopping runtime error (probably because of API changes or something). I avoid Python programs at all cost nowadays.
Strong agree. I'll willingly install a handful dependencies from my distro package manager, where the dependencies are battle-hardened Unixy tools and I can clearly see what they do and how they do it.
I'm not going to install thousands of dodgy-looking packages from pip, the only documentation for which is a Discord channel full of children exchanging 'dank memes'.
I like Python, but I simply do not trust the pip ecosystem at this point (same for npm, etc.).
> I'm not going to install thousands of dodgy-looking packages from pip, the only documentation for which is a Discord channel full of children exchanging 'dank memes'.
This made me laugh. It’s true, isn’t it? That’s really what we deal with day to day (for me in the js world, the create react app dependencies make my head spin)
> For something like C or C++? Usually the most complicated part is running `make` or `cmake` with `pkgconfig` somewhere in my path. Maybe install some missing system libraries if necessary.
I don't mean to detract from your main point re Python dependencies, but I find this about C to be rarely true. `make` etc build flows usually result in dependency-related compile errors, or may require a certain operating system. I notice this frequently with OSS or academic software that doens't ship binaries.
Yep. I've given up on any C or C++ projects because I find they almost never work and waste hours of my time. Part of the issue might be the fact that I'm often using Windows or MacOS but I've had bad experiences on Linux also.
> `make` etc build flows usually result in dependency-related compile errors,
Which are displayed to me during the `./configure` step before the `make`, and usually require me to type "apt-get install [blah] [blah] [blah]", and to run configure again.
So much this. As someone whose bread and butter is systems programming for things that run on end-user devices, every time I dig into a Python project I feel like I've been teleported into the darkest timeline, where everything is environment management hell.
Even the more complex and annoying scenarios in native-land for dependency management still feels positively idyllic in comparison to Python venvs.
It was fine, back in the early days (I started with 1.4-ish). I just downloaded the tarball, unpacked, configured, make, installed into /usr/local on my workstation, then downloaded and stuck any packages into site-packages. Numeric was sometime tricky to compile right, but ye olde "configure && make && make install" worked fine.
Of course, that worked because 1) I was really only doing one project, not juggling multiple ones, 2) there weren't all that many dependencies (Numeric, plotting, etc.), and 3) I was already up to my eyeballs in the build system with SWIG and linking to the actual compute code, so I knew my way around the system.
But every now and then I just shake my fist at the clouds then mutter darkly about just installing the dang thing and maybe not taking on so many dependencies. :-)
I’m newish to python, I’ve only used it for machine learning projects and some web scraping. Could somebody elaborate on venv? I just started using it but now everyone in this thread is saying how much they hate it. Is there an alternative?
venv is fine. Remember that this is a self-selected sample of people. You’re going to bump into the flaws and gotchas but it’s a perfectly usable tool.
Uhm so I was professionally setting up ML distros and ML containers for cloud deployments. Venv is not fine, especially if you've seen how other langs do it.
IMHO venvs are fine as an implementation detail, a building block for a slicker tool.
The annoyance with venvs is you have to create and activate them. In contrast for cargo (or stack or dotnet or yarn or pipenv or poetry), you just run the build tool in the project directory.
Another limitation of venv is it doesn't solve the problem of pinning a versions of Python, so you need another tool.
I had the same issue... Turned out it was because I used the flat pack version of intellij idea and it had problems with paths. Running from a plain terminal worked fine.
This is also the reason I like when I see a project in C or C++. It's often a ./configure && make or something. Sometimes running a Python project even if dependencies install, there might be some mystery crash because package dependencies were not set correctly or something similar (I had a lot of trouble with AUTOMATIC1111 StableDiffusion UI when using some extensions that installed their own requirements that might be in conflict with the main project).
With a boring C project, if it compiles it probably works without hassle.
Feels validating that other people have these thoughts too and I'm not just some old fart.
I recently hit the "classic" case. Saw a CLI tool for an API I'd like to use, written in Python. Tried it and found out it didn't work on my machine. I later found out it was a bug in a dependency of that tool. 100 lines of shell script later, I had the functionality I needed, and a codebase which was actually free of unexpected surprises. I know, this is an extreme example, but as personal anecotes go, Python has lost a lot of trust from my side. I also wonder how people can write >10k codebases without static types, but that is just me ....
It's not that Python is bad. It's the people who want to just hack something quick together go to Python, so any time you pick up some software written in Python it's marred with all kinds of compatibility issues and bugs where you can't just run it
The answer is "yeah use this other software to make it work in an isolated way because the whole ecosystem is actually broken" and that's somehow acceptable
I think it is the opposite for me but I am also a fan of system independent package mangers, provided they support easy package configuration.
Otherwise you not only bind to system architecture and OS, you also bind yourself to a distribution.
I find that Automatic1111 plugins tend to not share dependencies and instead redownloads them for their own use. Can make your hdd cry because some of these are larger models. Advantages and disadvantages probably...
There are package managers for C and some are quite good. But for most projects you are quite dependent on the package manager of your distro to supply you a fitting foundation. Sometimes it is easy, but if there is a problem, I think handling C is far harder than python. And I write quite a bit of C while I can only perhaps read python code.
No code is completely platform independent, especially a stable diffusion project, but Python is still more flexible as C by a long shot here.
Of course Llama is great. Time to get those LLMs on our devices for our personal dystopian AIs running amok.
> which package manager/distribution I need to use, and what system libraries those dependencies need to function properly
I don't understand why things are so complicated in Python+ML world.
Normally, when I have a Python project, I just pick the latest Python version - unless documentation specifically tells me otherwise (like if it's still Python 2 or if 3.11 is not yet supported). If the project maintainer had some sense, it will have a requirements list with exact locked versions, so I run `pip install -r requirements.txt` (if there is a requirements.txt), `pipenv sync` (if there is a Pipfile), or `poetry install` (if there's pyproject.toml). That's three commands to remember, and that's not one just because pip (the one de-facto package manager) has its limitations but community hadn't really decided on the successor. Kinda like `make` vs automake vs `cmake` (vs `bazel` and other less common stuff; same with Python).
External libraries are typically not needed - because they'll be either provided in binary form with wheels (prebuilt for all most common system types), or automatically built during the installation process, assuming that `gcc`, `pkgconfig` and essential headers are available.
Although, I guess, maybe binary wheels aren't covering all those Nvidia driver/CUDA variations? I'm not a ML guy, so I'm sure how this is handled - I've heard there are binary wheels for CUDA libraries, but never used that.
Ideally, there's Nix (and poetry2nix) that could take care of everything, but only a few folks write Flakes for their projects.
> Usually the most complicated part is running `make` or `cmake` with `pkgconfig` somewhere in my path
Getting the correct version of all the dependencies is the trickiest part as there is no universal package managers - so it's all highly OS/distro specific. Some projects vendor their dependencies just to avoid this (and risk getting stuck with awfully out-of-date stuff).
> Maybe install some missing system libraries if necessary.
And hope their ABIs (if they're just dynamically loaded)/headers (if linked with) are still compatible with what the project expects. At least that is my primary frustration when I try to build something and it says it doesn't work anymore with whatever OS provides (mostly, Debian stable fault lol). It is not exactly fun to backport a Debian package (twice so if doing this properly and not handwaving it with checkinstall).
The right combo for Nvidia/CUDA/RandomPythonML library is a nightmare at times. This is especially true if you want to use older hardware like a Tesla M40 (dirt cheap, still capable). And your maker hopefully be with you if you if you tried to use your distro's native drivers first.
It's fair to say part of the blame is on Nvidia, but wow is it frustrating when you have to find eclectic mixes.
My personal recipe (on NixOS) is pip-ed virtual environment for quick tests, or conda inside a nix-shell, on top of a dedicated zfs pool/conda mounted in ~/.conda with dedup=on so nothing nixified and nothing that last a nixos-rebuild...
Many pythonic projects not only in ML world tend to be just developers experiments, so to be run as an experiment, not worth to be packaged as a stable, released program...
Oh, BTW projects like home-assistant fell in the same bucket...
I totally agree with you the irony being python and languages like were built in part to reduce the complexity not only of the language but also to build and run the code… I feel machine learning is a low enough level thing that it should not be tied to a high level language like python… so I can use node, ruby, php or whatever by adding a c binding etc that to me is why this is most interesting
The problem is that python is designed assuming people want to use system-wide packages. In hindsight, that has turned out to be a mistake. Conda / venv try to bridge that gap but they’re kludgy, complex hacks compared to something like cargo or even npm.
Worse, because Python is a dynamic language, you also have to deal with all of that complexity at deployment time. (Vs C/C++/Zig/Rust where you can just ship the compiled binary).
> The problem is that python is designed assuming people want to use system-wide packages.
This wasn't true for decades, `virtualenv` was de-facto standard isolation solution (now baked in as `python -m venv`, still de-facto standard), and `pip` is the package manager (we don't talk about setuptools/distutils, ssh!). If someone still used system-wide packages that was either because a) they were building a container or some single-purpose system; or b) they were sloppy or had no idea what they're doing (most likely, following some crappy tutorial). Or it was distro people creating packages to satisfy dependencies for Python programs - but that's a whole different story (and one's virtualenv shouldn't inherit system packages unless it is really really necessary and iif it makes sense to do so).
The problem started when one needed some external non-Python dependencies. Python had invented binary wheels and they're around for a while (completely solving issues with e.g. PostgreSQL drivers, no one needs to worry about libpq), but I suppose depending on specific versions of kernel drivers and CUDA libraries is a more complex and nuanced subject.
> Vs C/C++/Zig/Rust where you can just ship the compiled binary
Only assuming that you can either statically link, or if all libraries' ABIs are stable (or if you're targeting a very specific ABI, but I've had my share of "version `GLIBC_2.xx' not found"s and not fond of those).
In a similar spirit, any Python project can be distributed as one binary (Python interpreter and a ZIP archive, bundled together) plus a set of zero or more .so files.
> This wasn't true for decades, `virtualenv` was de-facto standard isolation solution (now baked in as `python -m venv`, still de-facto standard)
Right; but python itself doesn’t check your local virtual environment unless you “activate” it (ugh what). And it can’t handle transitive dependency conflicts, like node and cargo can. Both of those problems stem from python assuming that a simple, flat set of dependencies are passed in from its environment variables.
Virtual envs are actually quite simple -- they contain a bin/ directory with a linked python binary. When the python binary runs, it checks it sibling directories (it knows it was executed as e.g. /home/user/.venv/bin/python) for what to load. You don't need the activate shell scripts or anything, just running that binary within your venv is enough; the shell script is just for convenient of inserting the bin directory into the $PATH so just "python" or "pip" runs the right thing.
> the shell script is just for convenient of inserting the bin directory into the $PATH so just "python" or "pip" runs the right thing.
Or so any reference in the program you run that launches another binary or loads a DLL relying onnthe environment gets the right one, etc. There are some binaries you can run without activating a venv with no problem, and others will crash hard, and others will just subtly do the wrong thing if the conditions are “right” in your normal system environment.
Another implication of this is that its impossible for 2 mutually incompatible copies of the same package to exist in the same environment. If packageA needs numpy 1.20 and packageB needs numpy 1.21, you're stuck.
You have never trashed your system from virtualenv?
Also, there is a problem when wheels assume they can have everything like tensorflow from years ago -- I don't know about now, since tf used to be tied to cuda versions you could get into trouble installing tf versions, even with venv, conda, etc.
> You have never trashed your system from virtualenv?
Unless one have done something they shouldn't have done (in particular, using sudo while working with virtualenv), this shouldn't be possible.
Due to limitations of most commonplace system-wide package managers (like, dpkg, rpm or ebuild, not modern stuff like nix) system packages exist to support other system packages. One installs some program, it needs libraries, dependencies get pulled. And then its distro package managers' job to ensure compatibility and deal with multiple version conflicts (not fun).
But if you start or check out some project, common knowledge was that you shouldn't be using on system packages, even if they're available and could work. With some obligatory exceptions like when you're working on a distribution packaging, or developing something meant to be tightly integrated with a particular distro (like a corporate standard stuff).
That is, unless we're talking about some system libraries/drivers needed for CUDA in particular (which is system stuff) rather than virtualenv itself.
> That is, unless we're talking about some system libraries/drivers needed for CUDA in particular (which is system stuff) rather than virtualenv itself.
Sir, this is an ML thread.
Venv interacts with that poorly, though to be fair it could be googles fault. Still it shouldn't be even possible.
I mean, virtualenv is not supposed to interact with that at all. System libraries are systems' package manager responsibility. Doubly so, as - as I get it - all this stuff is directly tied to the kernel driver.
What Python's package manager (pip in virtualenv) should do is build (or download prebuilt binaries) the relevant bindings and that's the extent of it. If others say it works this way with C (that comment about cmake and pkgconfig), then it must work this way with Python too.
> If someone still used system-wide packages that was either because a) [...] b) [...]
Or simply because they are packagers for some distro and they user want a simple way to pull-in some software by it's name, while the upstream devs imaging people cloning their public repo and run the software from the checkout in their own home, with regular pull, regularly rebuilding the needed surroundings...
Not to talking about modern systems/distro with not-really-posix vision like NixOS or Guix System...
> In a similar spirit, any Python project can be distributed as one binary
> For something like C or C++? Usually the most complicated part is running `make` or `cmake` with `pkgconfig` somewhere in my path. Maybe install some missing system libraries if necessary.
If you don't use virtual environments in python, isn't it basically the same in python? Just run `pip install` and maybe install some missing system libraries if necessary. In practice, it's not that simple in either language, and "maybe install some missing dependencies" sweeps a lot of pain under the rug.
Wow, so interesting to see the "depth" of anti-python feeling in some quarters. I guess that is the backlash from all the hordes of Python-bros.
Having used both C++ and Python for some time, the idea that managing C++ dependencies is easier than venv and pip install is one of the moments you wonder how credible is HN opinion on anything.
> a compelling non-Python solution appears
Confusing a large ML framework like pytorch that allows you to experiment and develop any type of model with a particular optimized implementation in a low level language suggests people are not even aware of basic workflows in this space.
> also popular
Ofcourse its popular. As in: People are delirious with LLM FOMO but can't fork gazillions to cloud gatekeepers or NVIDIA so anybody who can alleviate that pain is like a deus-ex-machina.
Ofcourse llama.cpp and its creator are great. But the exercise primarily points out that there isn't a unified platfrom to both develop and deploy ML type models in a flexible and hardware agnostic way.
p.s. For julia lovers that usually jump at the "two-language problem" of Python: here is your chance to shine. There is a Llama.jl that wraps Llama.cpp. You want to develop a native one.
Managing C++ dependencies _is_ much easier! It's either "run this setup exe" or "extract this zip file/tarball/dmg and run".
This is because most people don't care about developing the project, just using it. So they don't care what the dependencies are, just that things work. C++ might be more difficult to handle dependencies to build things, but few people will look into hacking on the code before checking to see if it's even relevant.
> Managing C++ dependencies _is_ much easier! It's either "run this setup exe" or "extract this zip file/tarball/dmg and run".
Not sure if that was half sarcastic, but from experience, anything touching c++ & CUDA has the potential to devolve into a nightmare on a developer-friendly platform (hello kernel updates!), or worse, into a hair-pulling experience if you happen to use the malware from Redmond (to compound your misery, throw boost and Qt dependencies in the mix).
Then again, some of the configurations steps required will be the same for Python or C++. And if the C++ developer was kind enough to provide a one-stop compilation step only depending on an up-to-date compiler, it might, indeed, be better than a Python solution with strict package version dependencies.
If we're only talking about end-user "binaries" you can also package Python protects into exe files or similar format that bundle all the dependencies and are ready to run.
>Wow, so interesting to see the "depth" of anti-python feeling in some quarters. I guess that is the backlash from all the hordes of Python-bros.
I think you are generalizing.
I do not hate on Python the language but this ML projects are a very , very terrible experience. Maybe you can lame the devs of this ML projects, or the ones of the dependencies but the experience is shit. You can follow a step by step instruction that worked 11 day ago and today is broken.
I had similar issues with Python GTK apps, if the app is old enough then you are crewed because that old GTK version is no longer packaged, if the app is very new then you are screwed again because it depends on latest version of some GTK wrapper/helper.
I think what has happened is that because Python is sweet and easy to use for many things, it generated irrational expectations that is perfect for all things. But its just an interpreted language that started as a scripting and gluing oriented language.
Its deployment story is where this gap frequently shows. Desktop apps at best passable, whereas e.g. android apps practically non-existing despite the efforts of projects like kivy.
The problem seems to be getting a project that works on the developer machine packaged and distributed to non developers. Some type of projects seem harder to distribute, this ML dependencies seem to change fast and everything breaks(maybe because the dependencies are not locked correctly).
I don't even write python, really, but I've been interfacing with llama.cpp and whisper.cpp through it recently because it's been most convenient. Before that I was using nodejs libraries that just wrap those two cpp libs.
I guess since these models are meant to be run "client side" or "at the edge" or whatever you want to call it, it helps if they can be neutrally used with just about any wrapper. Using them from Javascript instead of Python is sort of huge for moving ML off the server and into the client.
I haven't really dipped my toes into the space until llama and whisper cpp came along because they dropped the barrier extremely low. The documentation is simple, and excellent. The tools that it's enabled on top like text-generation-webui are next level easy.
Quality of HN comments are getting bad for a few months. This is nothing to do with python ML ecosystem and what you have to realize llmcpp doing is it is inferencing already built models- which is running the models.
Building (training) machine learning, deep learning models are much more complex , order of magnitude complex than just running the models and doing that in C or C++ would take you years which would take just a few month with python.
And complexity of `pip install` is nothing compared to that.
That's why no real ETL+Deep learning, training work is done in c or c++.
The point is, as you pointed out, that you code against the appropriate level of abstraction. You write a ML workflow appropriate language like Python in something like C++/rust, and ML flows in Python. That should really not be that hard to understand.
I am not an expert, but from what I've seen, PyTorch is mostly a thin wrapper over the C++ libtorch. The same is true for DOM as well, but nobody uses DOM directly, whereas everybody uses PyTorch.
The big shift is Jupyter, but that's mainly for exploratory programming. If you already know what you're doing, there's no reason why C++ should be worse than Python for training. It's likely that most ML engineers do not have experience with C++.
I don't have that experience either but from what I've seen, C++ is very powerful, so once you subtract the jupyter, there's not really too much left.
BTW: You cannot use DOM/CSSOM from C++, the only API is JS, so your argument is theoretical.
Yes Python is incredibly annoying to use. Their dependency management is a total mess, and it's incredible how brittle packages if there are even minor point changes in versions anywhere in a stack.
I have to agree. Installing dependencies for some git repos is a total crapshoot. I ended up wasting so much hard drive space with copies of pytorch. Meanwhile llama.cpp is just "make" and takes less time to build than to download one copy of pytorch.
So, the solution is that everyone should write code as self-contained C++ code and not use any software libraries ever. Dependency hell has been solved for all time!
There is a happy medium... somewhere. After following Postgres development for the better part of a decade, I think it's definitely closer to the python side of things... But man they (python) do make it hard to like using that ecosystem.
The flip side is like you said... You will just have to reimplement everything yourself and then you can never worry about dependencies again! Just hope you didn't introduce some obscure security issue in your hashmap implementation .
I personally really don't like much Python, I find it as tedious to write as Go but without the added performance, typesafety and autocomplete benefits that comes with it in exchange.
If I have use a dynamic language, at least make it battery included like Ruby. Sure it's also not performant but I get something back in exchange.
Python sits in a very uncomfortable spot which I don't find a use for. Too verbose for a dynamic language and not performant enough compared to a more static language.
The testing culture is also pretty poor in my opinion, packages rarely have proper tests, (and especially in the ML field)
For me Python's main use cases are being BASIC replacement, and a saner alternative to Perl (I like Perl though) for OS scripting.
For everything else, I rather use compiled languages with JIT/AOT toolchains, and since most "Python libraries" for machine learning are actually C and C++, any language goes, there is nothing special about Python there.
The Python apologists are more annoying than the language.
Its always been obvious that ML’s marriage to python has always been credential ladened proponents in tangentially related fields following group think.
As soon as we got a reason to ignore those PhDs, their gatekept moat evaporated overnight and the community of [co-]dependencies became irrelevant overnight.
As far back as 2015, it’s been common to take neural net inference and get a C++ version of it. That’s what this is.
It didn’t make python obsolete then and it won’t now.
Training (orchestration, architecture definition etc.) and data munging (scraping cleaning analyzing etc.) are much easier with python than C++, and so there is no chance that C++ takes over as the lingua Franca of machine learning until those activities are rare
Even "compiled" JAX or PyTorch can leave some performance even if you hit the common path (It's also "praying" that the compiler actually works if you do anything non-standard).
But memory wise, there is almost no optimization or reuse (and it's sacrificed for performance), which leads to insane memory usage.
And it's not that they are bad - but optimal compilation of a graph is combinatorially explosive problem, impossible without heuristics and guesswork (what to reuse and waste memory vs recompute). A good programmer can do a significantly better job.
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts.
Exploring the landscape ends up with you having 29384232938792834234 different python environments, because that one thing requires specific versions of one set of libraries, while that other thing requires different versions of the same library and there is no middle ground.
It's horribly annoying and I absolutely love python!
I'm currently a Python dev for a living, but I spend a portion of my personal time trying to get better at C/C++, and in my case, it's strictly about the potential for writing faster code. I'm interesting in getting into DSP stuff specifically, so it's 100% necessary if I want to do that, and I would also like to get my head wrapped around OpenCV for similarly creative reasons.
Python opened up the world of code to a lot more people, but there was a cost to that, and the real action as far as actual computer systems go is always gonna be at a much lower level, in much the same way that a lot of people can top up their fluids but most of us pay someone to change the oil.
I could actually totally see oil changes becoming completely robotic in the future, but we would first have to establish open standards for oil pans that all automakers adhered to.
The whole computers designing computers thing, outside of someone cracking cold fusion I don't think we'll ever have the juice for it. In my lifetime, a C-level programmer will never be out of work, but I suspect that the demand for Python programmers is going to slack off, while the supply continues to grow.
Yet another TEDIOUS BATTLE: Python vs. C++/C stack.
This project gained popularity due to the HIGH DEMAND for running large models with 1B+ parameters, like `llama`. Python dominates the interface and training ecosystem, but prior to llama.cpp, non-ML professionals showed little interest in a fast C++ interface library. While existing solutions like tensorflow-serving [1] in C++ were sufficiently fast with GPU support, llama.cpp took the initiative to optimize for CPU and trim unnecessary code, essentially code-golfing and sacrificing some algorithm correctness for improved performance, which isn't favored by "ML research".
NOTE: In my opinion, a true pioneer was DarkNet, which implemented the YOLO model series and significantly outperformed others [2]. Same trick basically like llama.cpp
I believe it's moreso for the (actively pursued) speed optimizations it provides. When inference is already computationally expensive any bit of performance is a big plus
yeah it's sad I guess but half the reason I am recommending this is that it "just works" so much more easily than installing half the Python ecosystem into a conda environment (ironically, just so that Python can then behave as a thin wrapper for calling the underlying native libraries ...)
I've been wondering for awhile now - what was ever the benefit to building these things in Python, other than pytorch and numpy being there to experiment on in hobbyist ways? There's no way that a serious AI is really going to be built in a scripting language, is there? Once you know what you actually want it to do, you're definitely going to rebuild it as close to the metal as you can, right?
Not to mention, to really protect source code and all the sugar around the training systems, it's going to be a good investment to get out of hobby land and manage your own memory and just code them in C/C++.
It strikes me that the hobby AI ethos aligns very well with scripting languages in that they both assume the availability of endless resources to push things a little dirtier and messier and see if anything interesting emerges. Which is great for hobby AI. It's probably not the future, though, unless resource availability outpaces the imagination of people to write more and more bloated scripts to accomplish what's already been proven.
incredibly condescending which always pairs well with ignorance. "Hobby AI" are the people (mathematicians, domain experts etc) that made this all possible so that you can now "just code it in C/C++".
Have you ever tried to iterate developing any serious class of algorithms in C++?
> Once you know what you actually want it to do
When is that exactly? Even the last few months of LLM land development show very clearly how everything is rapidly evolving (and will very likely continue for quite some time).
Numerical linear algebra stabilized decades ago so you do have low-level libraries in C++ (or even fortran) but there is quite some distance between an LLM and linear algebra.
> Have you ever tried to iterate developing any serious class of algorithms in C++?
Yes, and it's not so bad. A lot of ML deployments have been based on C/C++ for inference anyway (with Python driving the training). So that's really nothing new. I.e. most Python research code is not deployable in terms of quality / performance.
In the absence of alternatives its workable. Libraries like eigen or armadillo help a lot. But Python with numpy has been extremely popular for a good reason.
Not quite. Nothing to do directly with python. This was the introduction of 4 and 8 bit quantization to a large number of people.
There wasn’t a python library like that anyone was used to using. Would have always been a C extension anyway.
Starting with the cpu in this situation made sense, strangely. There are python wrappers now. I tried to make one for ya’ll in rust in April, but haha I had a compiler issue I never solved.
It's a great project and an impressive achievement, but I'm also struggling to understand what people use it for that PyTorch wasn't offering. Easy deployment on iOS I guess? I would have thought that's a pretty small use case though.
Given the author hand-rolled his own FFT, I'm also guessing it's not as performant?
Pytorch (+GPU) dependency and python container type diversity are particularly bad. Programmers may not perceive this since they're already managing their python environment keeping all the OS/libs/containers/applications in the alignment required for things to work but it's quite complex. I couldn't do it.
In comparison I could just type git clone https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU acceleration working with my AMD RX 580 8GB. Plus with the llama.cpp CPU mmap stuff I can run multiple LLM IRC bot processes using the same model all sharing the RAM representation for free. Are there even ways to run 2 or 3 bit models in pytorch implementations like llama.cpp can do? It's pretty rad I could run a 65B llama in 27 GB of RAM on my 32GB RAM system (and still get better perplexity than 30B 8 bit).
You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different language and proclaiming that an entire language/ecosystem is crap.
The only real take away is that many projects, independent of language, put way too little effort towards onboarding users.
That's exactly it. Who wants to try pulling in an entire language with wide depenencies and it's ecosystem of envs/containers/etc when a single program will do it? Not people who just want to run inference on pre-made models.
Easy deployment anywhere, not just iOS. I haven't used Python in years so I have no idea what package manager is the best now, completely forgot how to use virtualenv, and it only took a few weeks to completely fuck up my local Python install ("Your version of CUDA doesn't match the one used to blah blah")
Python is a mess. llama.cpp was literally a git clone followed by "cd llama.cpp && make && ./main" - I can recite the commands from memory and I haven't done any C/C++ development in a long time.
For most modern ML projects in python you can just do something like `conda env create -f environment.yaml` then straight to `./main.py`. This handles very complex dependencies in a single command.
The example you gave works because llama.cpp specifically strives to have no dependencies. But this is not an intrinsically useful goal; there's a reason software libraries were invented. I always have fun when I find out that the thing I'm trying to compile needs -std=C++26 and glibc 3.0, and I'm running on a HPC cluster where I can't use a system level package manager, and I don't want to be arsed to dockerize every small thing I want to run.
For scientific and ML uses, conda has basically solved the whole "python packaging is a mess" that people seem to still complain about, at least on the end-user side. Sure, conda is slow as hell but there's a drop in replacement (mamba) that solves that issue.
Conda? Mamba? Or should I use Venv? What are the commands to “activate” an environment? And why do I have to do that anyway, given thats not needed in any other programming language? Which of those systems support nested dependencies properly? Do any of them support the dependency graph containing multiple, mutually incompatible copies of the same package?
Coming from rust (and nodejs before that), the package management situation in python feels like a mess. It’s barely better than C and C++ - both of which are also a disaster. (Make? Autotools? CMake? Use vendored dependencies? System dependencies? Which openblas package should I install from apt? Are any of them recent enough? Kill me.)
Node: npm install. npm start.
Rust: cargo run. Cargo run —-release.
I don’t want to pick from 18 flavours of “virtual environments” that I have to remember how to to “activate”. And I don’t want to deal with transitive dependency conflicts, and I don’t want to be wading through my distro’s packages to figure out how to manually install dependencies.
I just want to run the program. Python and C both make that much more difficult than it needs to be.
Honest question, as I don't follow rust, does the cargo package manager handle non-rust dependencies? I've played around with it enough that I can say it's for sure a joy to use, but what if your rust program has to link against a very specific combination of versions of, say, Nvidia drivers and openssl and blas?
In general solving such environments (where some versions are left floating or only have like >= requirements) is an NP-hard problem. And it requires care and has to draw source code and/or binaries from various sources.
This is the problem that conda/mamba solves.
If you just want to install python packages, use pip+virtualenv. It's officially supported by python. And while pip has traditionally been a mess, there's been a bunch of active development lately, the major version number has gone from like 9.xx to 23.xx in like the past two or three years. They are trying, better late than never, especially for an ecosystem as developed as python's.
So, if you want to compare rust/cargo, and it handles non-rust deps, then the equivalent is conda. Otherwise, it's virtualenv+pip. I don't think there are any other serious options, but I agree that two is not necessarily better than one in this case. Not defending this state of affairs, just pointing out which would be the relevant comparison to rust.
> Honest question, as I don't follow rust, does the cargo package manager handle non-rust dependencies?
You can run arbitrary code at build time, and you can link to external libraries. For the likes of C libraries, you’ll typically bind to the library and either require that it be installed locally already, or bundle its source and compile it yourself (most commonly with the `cc` crate), or support both techniques (probably via a feature flag). The libsqlite3-sys crate is a fairly complex example of offering both, if you happen to want to look at an actual build.rs and Cargo.toml and the associated files.
> [pip’s] major version number has gone from like 9.xx to 23.xx in like the past two or three years.
It skipped from 10.0.1 to 18.0 in mid-2018, and the “major” part has since corresponded to the year it was released in, minus 2000. (Look through the table of contents in https://pip.pypa.io/en/stable/news/ to see all this easily.)
Yeah, no… my current $DAYJOB has a confusing mix of nx, pnpm, npm commands in multiple projects. Python is bad but node is absolutely not a good example.
Eh. Node / npm on its own is generally fine, especially if you use a private package repository for internally shared libraries. The problems show up when compiling javascript for the web as well as nodejs. If you stick to server side javascript using node and npm, it all works pretty well. It’s much nicer than venv / conda. And it handles transitive dependency conflicts and all sorts of wacky problems.
It’s just that almost nobody does that.
What we want instead is to combine typescript, js bundlers, wasm, es modules and node packages all together to make web apps. And that’s more than enough to bring seasoned engineers to tears. Let alone adding in svelte / solidjs compilation on top of all that. I have sweats just thinking about it.
> If you stick to server side javascript using node and npm, it all works pretty well.
Rose colored glasses if you ask me. The difference is it seems you use Node often (daily?) and have rationalized the pain. Same goes for everyone defending Python (which I’m sorta in that camp, full disclosure), they are just used to the worts and have muscle memory for how to work around them just like you seem to be able to do with Node.
> Rose colored glasses if you ask me. The difference is it seems you use Node often (daily?) and have rationalized the pain.
It’s not just that. Node can also handle arbitrary dependency graphs, including transitive dependency conflicts, sibling dependencies and dev dependencies. And node doesn’t need symlink hacks to find locally installed dependencies. Node packages have also been expected to maintain semver compatibility from inception. They’re usually pretty good about it.
I have no dog in this fight, but the fact that you end your "conda solves this complexity" explanation with "and mamba is a replacement for conda" doesn't really sell me on it. Is it just speed as the main reason for mamba? The fact that the "one ring to rule them all" solve for pythons packaging woes even has a replacement sort of defeats the purpose a little.
I understand why people find Python's packaging story painful, I guess is what I'm saying.
conda takes up to several minutes to figure out dependencies just to tell you that one library requires an old version of a dependency and another requires a new one, making the whole setup impossible.
As sibling elaborated, mamba just accelerates the NP-hard step of resolving conflicts using mathy SAT-solver methods. Also it has a much prettier cli/tui. I would be surprised if conda doesn't eventually merge the project, but you can already use it as an optional/experimental backend on the official conda distro. Haven't followed it at all but I suspect the folks who developed it wanted to use it right away so they forked the project, while the people who maintain conda are more conservative since it's critical infrastructure for ${large_number} of orgs.
> But this is not an intrinsically useful goal; there's a reason software libraries were invented.
No need to denigrate those who don't need to import all their code; it's perfectly fine to not rely on third parties when developing software - some of our best works result from this.
Thanks for the response. Interesting to see a lot of other people echoing the same comments - that dependency management in Python is an absolute PITA.
I know exactly what you mean, but I'm probably so inured to it by now that I've just come to accept it. Obviously not everyone feels this way!
Python sits on the C-glue segment of programming languages (where Perl, PHP, Ruby and Node are also notable members). Being a glue language means having APIs to a lot of external toolchains written in not only C/C++ but many other compiled languages, APIs and system resources. Conda, virtualenv, etc. are godsend modules for making it all work, or even better, to freeze things once they all work, without resourcing to Docker, VMs or shell scripts. It's meant for application and DevOps people who need to slap together, ie, ML, Numpy, Elasticsearch, AWS APIs and REST endpoints and Get $hit Done.
It's annoying to see them "glueys" compared to the binary compiled segment where the heavy lifting is done. Python and others exist to latch on and assimilate. Resistance is futile:
Deploying a PyTorch running Python app to anything in a way a user can just run it is a struggle. Even iOS aside that’s not a small use case that’s all local and offline ML potential.
Really makes me wonder about the whole language, did they ever expect the code to have to run elsewhere than the machines the writer controls.
llama cpp is just cpp inference on the llama model. PyTorch is a library to train neural networks. I'm not sure why people are conflating these two totally different projects..
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason?
The desktop/laptop LLMs use case is one where resource efficiency often makes the difference between “I can’t use this at all” rather than the more frequent “I can use it but it maybe runs a little slower”, and llama.cpp offers that. It also has offered new quantization options that the Python-based tooling hasn’t, which compounds the basic resource efficiency point.
(It’s also not an “everyone” thing: plenty of people are using the Python-based toolchains for LLMs, its possible for their to be multiple popular options in a space . Not everything is all or nothing.)
I think Python and R are generally superior (in terms of developer experience) when you have to do end-to-end ML work, including acquiring and munging data, plotting results etc. But even then, the core algorithms are generally implemented in libraries built out of native code (C/C++/Fortran), just wrapped in friendly bindings.
For LLMs, unless you're doing extensive work refactoring the inputs, there are fewer productivity gains to be had around the edges - the main gains are just speeding up training, evaluation and inference, i.e. pure performance.
I find it interesting that it's an example of an ML software that's totally detached from Python ML ecosystem and also popular.
Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts.