This is a practical handbook on how to get started using the Rust programming language for Data Engineering tasks. I'll show you practical examples of how Rust can be used for most of the tasks expected from a data engineer.
This project is a personal hobby project made by Karim Jedda. Data with Rust is no way endorsed nor affiliated with the Rust foundation. Rust & Cargo are registered trademarks of the Rust foundation.
With that out of the way, let me start by saying that most of the content will be openly accessible, however some sections and upcoming features will be reserved to members of my newsletter. This website is a project I started working after getting multiple requests and positive signals to compile resources and guides on using Rust as an alternative to Python for Data Engineering.
As a newsletter member, you'll get notified as soon as code, videos, as well as a paperback and pdf copies of the book become available. Newsletter members will get a discount of 30% on the paid section once they go live.
Alternatively, you can regularly check out the changelog to see whenever something new is added or updated to this guide.
As a short teaser, here's the recording of my EuroPython 2023 talk, titled "Rust for Python Data Engineers" in which I make the case for using Rust for data engineering workloads:
I'm not a Rust expert but I've been writing Python code for at least a decade now (feeling old...). One thing I know is how to get things done, maintainable and simple (mostly). I've worked in a lot of different industries and in different settings. You can find an example of my teaching style, approach to learning and a glimpse of how my brain works on my personal blog.
I'll focus here on giving out practical ways of introducing Rust into your day to day tasks, theoretical concepts will not be discussed in depth however I'll refer to people, guides, videos or websites who can explain it better than me - wherever applicable.
The code and examples in this guide will be practical and very hands-on, the main focus being on getting things done.
As a preview, check out the table of contents on the left panel, we'll be covering a lot of topics by increasing levels of complexity.
As stated above, I wrote a small article on my personal blog that resonated with a lot of people. That small article got roughly 100K views in 2 days. I was very happy about the feedback as it seems that there is at least some interest of exploring what Rust has to offer in the data space.
There are currently a lot of resources online, however, I didn't find any that could introduce data engineers to Rust in a structured and practical way. It isn't an easy task since it'll be extremely difficult to convince Pythonistas to switch and that's not the goal. Rust is a good additional asset to have in your toolbox.
This could have been a paper book but the Rust ecosystem evolves pretty quickly so an online guide that is constantly updated makes a bit more sense.
I also believe a website is way easier to consume when programming and learning the concepts. I'd like to make it as interactive as possible, if you notice an error or think an improvement might make sense, reach out by sending me some feedback
You might be wondering who this guide could be for and it's a fair question. The audience for this book I'll keep in mind are as follows:
- Anyone curious about how Rust can be useful for data problems
- Anyone who heard about Rust but doesn't know where to start
- Python developers that like a hands-on guide starting point for Rust
- Developers searching for a practical and updated guide tailored to data workloads
- Developers looking to add a new skill but like to see some code before investing more time
- Junior to Senior developers, everyone should be able to find something interesting
- Myself: Someone who knows Python very well and got interested in Rust in the context of my job
One thing is for sure though: This guide might not be for Rust experts that certainly know already more than me. Feedback and corrections are always welcome, which is why most of the guide is available for free.
To be completely fair and transparent, learning Rust is a little bit difficult.
This guide won't be an academic lecture of what parts constitute the Rust language but rather a compilation of useful things you can do with Rust, today, to solve a wide variety of data challenges.
By seeing the code and understanding what task it's supposed to solve, I believe that understanding of how Rust works can build up over time.
In other words, some concepts might not click immediately or overnight. It might take some time and that's completely normal.
It doesn't mean that there won't be any explanation or theoretical concepts, it's just that they take second place to making things work.
The bet is that it might make a complicated topic approachable and perhaps fun too.