Data Engineering with Rust

Welcome!

This is a practical handbook on how to get started using the Rust programming language for Data Engineering tasks. I'll show you practical examples of how Rust can be used for most of the tasks expected from a data engineer.

Disclaimer

This project is a personal hobby project made by Karim Jedda. Data with Rust is no way endorsed nor affiliated with the Rust foundation. Rust & Cargo are registered trademarks of the Rust foundation.

With that out of the way, let me start by saying that most of the content will be openly accessible, however some sections and upcoming features will be reserved to members of my newsletter. This website is a project I started working after getting multiple requests and positive signals to compile resources and guides on using Rust as an alternative to Python for Data Engineering.


Subscribtions temporarily closed

Stay tuned for a new 💅 & refreshed look. ✨

Want a sneak peek? 👀 Shoot me an email at → karim.jedda@gmail.com ←


As a newsletter member, you'll get notified as soon as code, videos, as well as a paperback and pdf copies of the book become available. Newsletter members will get a discount of 30% on the paid section once they go live.

Alternatively, you can regularly check out the changelog to see whenever something new is added or updated to this guide.

As a short teaser, here's the recording of my EuroPython 2023 talk, titled "Rust for Python Data Engineers" in which I make the case for using Rust for data engineering workloads:

What you can expect

I'm not a Rust expert but I've been writing Python code for at least a decade now (feeling old...). One thing I know is how to get things done, maintainable and simple (mostly). I've worked in a lot of different industries and in different settings. You can find an example of my teaching style, approach to learning and a glimpse of how my brain works on my personal blog.

I'll focus here on giving out practical ways of introducing Rust into your day to day tasks, theoretical concepts will not be discussed in depth however I'll refer to people, guides, videos or websites who can explain it better than me - wherever applicable.

The code and examples in this guide will be practical and very hands-on, the main focus being on getting things done.

The chapters are going to be published and updated regularly, you can subscribe to the newsletter to be the first to get the new chapters and updates. You can also follow me on LinkedIn or alternatively on Twitter as I'll post updates there, if that's easier and requires less commitment. Here's a link to the privacy policy for this project, just in case.

As a preview, check out the table of contents on the left panel, we'll be covering a lot of topics by increasing levels of complexity.

Motivations

As stated above, I wrote a small article on my personal blog that resonated with a lot of people. That small article got roughly 100K views in 2 days. I was very happy about the feedback as it seems that there is at least some interest of exploring what Rust has to offer in the data space.

There are currently a lot of resources online, however, I didn't find any that could introduce data engineers to Rust in a structured and practical way. It isn't an easy task since it'll be extremely difficult to convince Pythonistas to switch and that's not the goal. Rust is a good additional asset to have in your toolbox.

This could have been a paper book but the Rust ecosystem evolves pretty quickly so an online guide that is constantly updated makes a bit more sense.

I also believe a website is way easier to consume when programming and learning the concepts. I'd like to make it as interactive as possible, if you notice an error or think an improvement might make sense, reach out by sending me some feedback

Who is this course/guide for?

You might be wondering who this guide could be for and it's a fair question. The audience for this book I'll keep in mind are as follows:

  • Anyone curious about how Rust can be useful for data problems
  • Anyone who heard about Rust but doesn't know where to start
  • Python developers that like a hands-on guide starting point for Rust
  • Developers searching for a practical and updated guide tailored to data workloads
  • Developers looking to add a new skill but like to see some code before investing more time
  • Junior to Senior developers, everyone should be able to find something interesting
  • Myself: Someone who knows Python very well and got interested in Rust in the context of my job

One thing is for sure though: This guide might not be for Rust experts that certainly know already more than me. Feedback and corrections are always welcome, which is why most of the guide is available for free.

Having fun

To be completely fair and transparent, learning Rust is a little bit difficult.

This guide won't be an academic lecture of what parts constitute the Rust language but rather a compilation of useful things you can do with Rust, today, to solve a wide variety of data challenges.

By seeing the code and understanding what task it's supposed to solve, I believe that understanding of how Rust works can build up over time.

In other words, some concepts might not click immediately or overnight. It might take some time and that's completely normal.

It doesn't mean that there won't be any explanation or theoretical concepts, it's just that they take second place to making things work.

The bet is that it might make a complicated topic approachable and perhaps fun too.

Let's start.