If you were linked here from Twitch chat, you probably want the FAQ at the end of this post.
This week I started live streaming development of Ruma on my Twitch channel.
Surprisingly, I've never mentioned Ruma on my blog here!
Ruma is a project I started in late 2015 to implement the Matrix communication protocol in Rust.
It's been slow going, for a number of reasons, but in the last few months development has picked up again.
Partly to motivate myself to continue working on it, and partly because I love to teach and help other people learn to program, I decided to start doing live streams of my work on the project.
I did a couple of streams on the topic more than two years ago (and sadly only managed to save the recording of one of them), but I plan to do them more regularly now.
I've already announced that I'm doing this on Twitter, reddit, and in the Matrix chat room for Ruma, but I'm doing it here as well, to have a central place to link to when talking about the streams.
If all you want is to check out the streams (or the recordings that are archived on YouTube after the fact), here are the relevant links:
If you were linked here from the chat on Twitch during a stream, the FAQ below is what you're here for.
These are the most commonly asked questions on the stream, and to avoid repeating myself on-stream too much, this information is provided here to bring you up to speed.
I'll keep this post updated over time if/when the most frequent questions change.
Frequently asked questions
Q: What is Ruma?
A: Ruma is a Matrix homeserver, client, and supporting libraries written in the Rust programming language.
You can dig into the details of Ruma on the Ruma website.
Q: What is Matrix?
A: Matrix is an open specification for an online communication protocol.
It includes all the features you'd expect from a modern chat platform including instant messaging, group chats, audio and video calls, searchable message history, synchronization across all your devices, and end-to-end encryption.
Matrix is federated, so no single company controls the system or your data.
You can use an existing server you trust or run your own, and the servers synchronize messages seamlessly.
To get a better understanding of what Matrix is and why you should care, check out the Introduction to Matrix on the Ruma website.
Q: What is Rust?
A: Rust is a systems programming language from Mozilla built with safety, concurrency, and performance in mind.
Its novel approach to memory safety and its rich type system make it an excellent choice for writing fast, secure, and reliable programs.
Learn more about Rust on the Rust website.
If you're ready to start learning Rust, the book is where everyone starts, and it's a fantastic resource.
Q: What type of software is Rust good for?
Rust was designed largely for use cases where you would otherwise have used C or C++.
This means that it's suitable for high performance, low-level software, including kernel modules or even full operating systems.
Rust also includes a lot of niceties from higher-level programming languages that have become popular since C and C++ were first introduced, so it's also a totally reasonable choice for people coming from languages like Ruby, Python, and JavaScript.
Q: Can I do web development in Rust?
Yes, but the ecosystem of libraries and frameworks is still pretty young.
The best resource for learning about Rust's suitability for web development is Are We Web Yet?.
Notably, asynchronous programming in Rust has been under heavy development, but is about to gain significant traction, as the long-desired async/await feature finally reaches Rust's stable release channel in November 2019.
See Are We Async Yet? for details about this.
More than a year ago, I wrote about my experience programming in Rust and what I felt were its high and low points.
Recently, I was asked if the things I wrote then are still relevant, and if the highs and lows are the same now.
I realized there is enough to talk about since my last review that it was worth writing a new post.
The highs
The high points of using Rust for me are essentially the same as before, so I'd suggest reading that part if you didn't read it then.
To summarize, Rust has been a paradigm shift for me as a programmer.
It greatly raised the bar for me in terms of what I require out of a programming language, and like all the best changes of its kind, left me with the sentiment, "How did I ever go without this?"
I remember a time when I felt that about Ruby (my previous programming paradigm shift), but using Ruby drives me crazy now.
Rust gets attention largely because of its memory safety guarantees, making it a potential replacement for C and C++ for systems that require the highest levels of performance and low-level control.
However, I'm not a low-level programmer, so for me, the thing that Rust brings to the table is correctness.
Others have described this quality as "fearless" (e.g. fearless concurrency), and while a bit jargony, I agree with the sentiment.
What Rust has done for me is allowed me to let go of the immense worry I didn't even realize I had in other languages.
In dynamically typed languages, or even statically typed languages with weak guarantees (e.g. Go), I had to program defensively, always worrying about things like a value being unexpectedly nil.
In Rust, the guarantees I get from reaching the simple goal of "it compiles" gives me more confidence in the correctness of my program than a full-blown test suite ever did in Ruby.
Rust's value is not just that it's statically typed.
There are many languages that are that.
Rust's value is that it brings all the benefits of static typing without sacrificing the expressive, natural feeling of writing code in a language like Ruby.
I get the same feeling of freedom to design and explore that I felt with Ruby, but with essentially none of the dangers.
This is largely because of Rust's fantastic type system.
Algebraic data types—specifically Rust's enums—are something I simply cannot imagine programming without at this point.
Many of the other great parts of Rust are not new ideas, but Rust melds so many great ideas from different languages together cohesively that it feels like you're getting the best of every world.
The only other mainstream language I know of with a comparable type system to Rust is Haskell, but Rust doesn't force me into a purely functional world, and provides a lot of other great benefits Haskell doesn't.
The lows
I'm happy to report that the two specific lows I mentioned last year are now mostly resolved.
The big pain point of doing serialization on stable Rust was resolved in Rust 1.15 when "Macros 1.1" was stabilized.
This allows any crate to do automatic implementations of traits using custom derive annotations.
We now have a 1.0 version of Serde, which has matured into an absolutely fantastic serialization library.
The only minor downside to the custom derive feature is that it is limited to generating implementations based on type declarations and does not offer a full-featured procedural macro system.
The procedural macro system is being revamped, however, and is already partially available in the nightly compiler as "Macros 2.0."
The other specific pain point I mentioned last year, about not having a stable, robust crypto library has improved as ring has matured and become the de facto crate for crypto in Rust.
It's still not 1.0 and still not audited, so we're not quite there yet.
In my work I also frequently need to create and manage X.509 certificates, and there is still nothing in Rust that does that yet.
We don't have a 1.0, audited, pure-Rust TLS implementation yet, but rustls is on its way, and unlike last year, we also have native-tls which greatly improves the TLS story on macOS and Windows.
Interestingly, there are more lows this year than there were last year.
Not only are there more, but despite my perhaps-overzealous love of Rust, I actually feel more negative about the language this year.
My negative feelings are not because anything in Rust is bad.
It's because of the things that are on the horizon that we don't have yet.
Rust the language has been 1.0 since May 2015.
The problem is that 1.0 only means that what's there is stable.
It doesn't mean that it is featureful enough to write anything you might want, realistically.
It doesn't mean that the ecosystem of libraries and supporting tools is featureful and easy enough to use that you will convince non-early adopters to try it out or use it for real work.
This, by itself, is largely the same sentiment as last year: Rust the ecosystem is still just too young and immature for a lot of use cases.
The thing that's worse this year is that I've come much further in the development of my Rust programs, to the point where finishing things is blocked by certain features of Rust not being implemented yet.
While I don't have to worry about things I write today breaking in some future version of Rust, the critical problem is this:
Knowing what features are coming, the APIs I would write when they're stable are significantly different than how I'd write them today.
This knowledge of what is coming, but isn't here yet, completely paralyzes me.
I'm not going to declare any of my libraries 1.0 when I know for sure that I will make breaking changes to them once a feature I wanted but didn't have previously becomes available.
Here is a list of specific features that are either implemented but unstable, or still in the RFC process and not even implemented.
The stabilization of each of these would change the design of at least one piece of code in one of my applications or libraries:
All of these are language features, and say nothing of the huge amount of unstable crates that would need to be 1.0, many of which themselves are blocked on unstable or nonexistent features.
Right now, The most discussed of these features within the Rust community is asynchronous I/O.
For my use cases, this is the current state of things:
- Anything that uses HTTP needs to be asynchronous.
- This requires a 1.0 version of Hyper.
- Hyper, in turn, requires a 1.0 version of the Tokio stack.
- Tokio, in turn, requires a 1.0 version of futures.
- Futures, in turn, require
impl Trait for realistic adoption.
- Futures are likely to be migrated to the standard library.
- Even with all of the above 1.0 and stabilized,
async/await is also needed to make async APIs ergonomic enough to consider stabilizing.
Even this particular chain of dependencies is going to take a while, and this is the stuff on my wishlist that the Rust team has prioritized most highly.
It's likely going to be multiple years before all this stuff is done.
I'd like to make it very clear that the Rust teams and the Rust community are not doing anything wrong.
All of the things I've mentioned here are long since known as desired by the Rust team, and there's a plan to get there.
All of these things are making progress, and a lot of very smart and hard working people are making it happen.
My negative feelings are quite simply because of the paralysis I feel knowing how different Rust will be once we have all these things, and having no recourse but to simply continue waiting, contributing to discussions and generally staying involved.
It wouldn't be so bad if I could continue using other languages in the meantime and consider Rust something I'd consider picking up again in a few years.
The good parts of Rust, even right now, are so good that I have trouble bringing myself to go back to any other language I know.
So I'm trapped in this limbo between a crippled version of the language I want, and this fantastic version of the language I know is coming.
I feel like a pouty little child writing this, but this is honestly how I feel right now.
Should you use Rust?
It's always hard to generalize when the answer is nuanced, but if I had to pick an absolute yes or no for everyone, I'd have to say no.
Rust is most certainly not ready for massive, widespread adoption.
I can't confidently claim that yes, it will work well for whatever you want to do with it, as I could say for Go, Java, C++, etc.
What I can say confidently is that there will be a time when I will unequivocably say, "Yes, you should use Rust."
The only reason I'm not saying yes today is because Rust is still young and its foundational pieces are still being built.
Everything that is in Rust today is awesome, and for many use cases it is already enough.
If you value the correctness of your programs over delivering quickly with minimal investment (i.e. the "always be shipping" mentality), you will already benefit from using Rust today.
That said, if you try to build anything significantly complex in Rust right now, I think you're likely to come across at least one place where you're unable to do something because the language doesn't support it yet.
This may be acceptable for building an application, where you're the only consumer of your API, but for a library it can be a major blocker.
Of course, even this may be less of an issue if you are more willing to stabilize code you know for sure will have breaking changes in the near-to-medium future than I am.
My current, totally unscientific estimate is that roughly two years from now, I will be able to recommend that people choose Rust for their next project, full stop.
After all the recent announcements and hype about these async libraries, I was still a little confused about what each of these crates does and how they relate to each other.
The crates I'm talking about are Futures, MIO, Tokio, and to a lesser extent Hyper and even Iron.
Futures and MIO were especially confusing considering that there are also (or were, at least) several futures-foo crates and tokio-foo crates.
After reading a bit more, I think I understand how they all relate now, so I wanted to share the knowledge (and please correct me if I'm wrong!)
Futures contains primitives for general purpose non-blocking computation, not necessarily specific to IO.
The most important types here are the Future trait, which represents a single non-blocking computation, and Stream, which is like an iterator that yields a sequence of non-blocking computations.
All the related futures-foo crates that were in the repo when it was first announced seem to have been renamed to tokio-foo and moved into the tokio-rs organization on GitHub.
Most of them were just examples of how Futures could be used as the underlying mechanism for a few different purposes.
MIO contains primitives for building cross-platform asynchronous IO systems, generally focused around network IO.
Tokio (as the overarching project) marries Futures and MIO to provide asynchronous IO using the Futures APIs.
Tokio is split up into several crates, which are, roughly in order from lowest level to highest level abstractions: tokio-core, tokio-service, tokio-proto, and the currently vaporware tokio.
tokio-core has the low level guts of asynchronous IO.
tokio-service contains the the Service trait, which similar to the futures crate's Future and Stream traits, is the central abstraction that the project provides for writing composable network programs.
tokio-proto provides additional types that are helpful for implementing a network protocol such as HTTP.
Finally, the crate actually called tokio will provide a higher level API that combines the features of the lower level crates.
This is the crate that most of us will use when we want to write an asynchronous network service.
The other crates exist separately just as a nice separation of concerns and to allow programs with more specific requirements to cherry-pick only the functionality they need.
The tokio crate itself does not exist at the time I'm writing this because the lower level building blocks are still under heavy development and the APIs are not finalized.
The other tokio-foo projects in the tokio-rs GitHub organization are either helpers types for specific use cases or examples of how you would build a network service using Tokio.
For those of us writing HTTP clients and servers, Hyper is the HTTP library we've come to know and love.
Hyper was originally synchronous, but since MIO's initial release has been undergoing some major architectural changes to switch to an asynchronous model.
According to Carl Lerche's Tokio announcement post, Hyper is in the process of moving its async implementation to build on top of Tokio instead of MIO directly.
And last but not least, Iron is a higher level web development framework built on Hyper.
It's one of the more popular frameworks of its kind currently, though development activity has been very quiet for several months now.
It's not clear to me whether or not the primary authors are still working on the project, whether they have run out of time and need help maintaining it, whether it's intentionally abandoned, or whether they're simply waiting for all these lower level components to stabilize before revising Iron's own APIs to use the Futures/MIO/Tokio/Hyper stack.
Whether or not Iron becomes a framework that uses this stack, surely a web development framework using this stack will materialize sooner rather than later!
