I would not recommend FFI + ctypes. Maintaining the bindings is tedious and error-prone. Also, Rust FFI/unsafe can be tricky even for experienced Rust devs.
Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.
Right now FFI + ctypes has one big advantage: it supports true parallelism with GIL-per-subinterpreter in Python 3.12. AFAIK all the higher-level binding libraries/tools (PyO3, pybind11, nanobind, Cython) don't support this yet; and in fact can't really support it without API-breaking fundamental design changes.
But wasn't the whole point to drop down to a lower language, release the GIL and then do parallel processing? Gil per subinterpretee makes things easy python side but I'm not sure how much of it's relevant if you are using something like pyO3
I think you are right. But as a proof of concept is an interesting read. I've work extensively with Rust and PyO3 and Maturin and even for a medium to large project is great. I wrote a bit about that here:
I'd also recommend PyO3 and Maturin. The amount of help these crates give is mind boggling (automatic type marshaling and even github CI jobs for creating cross platform precompiled wheels).
I've created a library using this crate (https://github.com/wiktor-k/pysequoia/) and most of the time I could just focus on the problem domain instead of technical details of the bindings.
There are just a couple of smaller issues (eg. Python to Rust async is not built in) but overall it's really nice.
Agreed. It's also instructive to look at how C extensions are written for the CPython.
I wouldn't necessarily expect a demo; but not mentioning it at all seems like it could lead readers down a path where they choose the "most performant" option presented (FFI + ctypes). The post has a "To go further:" link section where even a quick link to PyO3 would go a long way. So I thought I'd mention the downsides and an alternative on HN.
Absolutely, I was surprised anyone would recommend ctypes for Python/Rust today. I've written a few Python libraries using Rust to do the slow parts, and PyO3/maturin is a great experience.
Even a decade ago I was seeing projects stop using ctypes and instead use things like cffi. Honestly I couldn't tell you exactly why ctypes is so bad but I do know that cffi integrates a little closer with the c compiler and has better header support.
Well it depends. C is the lingua franca of Computer Science. These days I write a lot of code that needs to be accessible to a variety of languages and environments. Python, C#, C++, Matlab, Unity, Unreal, etc.
Write a C API and your code can be used anywhere. Now if you know you only ever need to support Python then sure look into PyO3. But if you have mixed use you can’t be the flexibility of a C API.
Honestly though, I would likely still choose to implement the logic in Rust as a crate/library in a workspace, the (C)Python extensions as a PyO3 dylib/separate crate, and a C FFI dylib as yet another crate.
The benefit is that you only need to ship the CPython extension, you can build it with cargo, and everything's in one place. I've maintained bindings for Python and C# to native libraries, and it's usually a pain for various reasons.
The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.
The Python<'_> type appearing as a parameter means the caller has already acquired the GIL for you. The GIL is held when Python calls into extension modules, and PyO3 doesn't automatically release it (but you can do so yourself).
It's essentially a dummy parameter that serves as proof that the GIL is held. You need to provide it to PyO3 when calling back into the Python runtime, that way the compiler ensures the GIL is still held where required.
PyO3 is great. I have been using it to rewrite some slow python code. My only complaint is that documentation is a bit rough. For someone who understands python, but not rust, there was a pretty big hill to climb to understand type conversions.
There are also things I am still struggling with. For example, there is a rust function I want to repeatedly call from python, that takes in a large unchanging python array. How can I pass the array from python to rust only once, i.e do the type conversion just once, so I don't have to pay that cost repeatedly.
I recommend creating a pyclass over a struct that stores the converted data. Instantiate it once and then repeatedly call a function on it in a pymethods impl with the parameters that differ.
FFI+cffi has the advantage that you leave Python specifics out of your Rust binding, so it's effectively a proper C binding that can be used for other FFI integrations as well.
Small note for the author: I was a bit confused at first by the use of "Nix-based" to mean POSIX-compliant [1] since Nix is the name of a package manager that NixOS is based on. I've only seen "*nix" to refer to Unix-like systems.
[1] Which seems like the right condition here? since I believe Unix domain sockets are required by POSIX.
As others have pointed out: none of these three “generic” methods is appropriate when a more precise one (such as dedicated bindings between a pair of languages) exists.
PyO3 is practical evidence of Rust’s claims around holistic security: a user can write safe Rust and safe Python without having to directly unsafely traverse the C ABI between them, or take a potentially unacceptable performance hit from RPC or IPC.
as long as you don't need to cross compile pyO3 makes calling python from rust in ffi extremely simple and convenient as it's abstracts away so the ffi, ctypes parts
Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.
[1] https://github.com/PyO3/pyo3