Rust is doing this correctly. It’s better to model things as they are rather than requiring a bunch of implicit knowledge.
The simplicity of C is dangerously misleading.
In Rust when you see
let y = x.clone();
You don’t know if it’s a cheap reference count increment or an expensive allocation and memcopy. You can do this instead:
let y = Arc::clone(x);
And enforce it with a clippy lint:
#![deny(clippy::clone_on_ref_ptr)]
Reading large csv files in parallel chunks is tricky because you can’t be sure a line end isn’t in a quoted field. But I found a paper showing how to make a good guess that works most of the time!
I think I’m now at a point where I’m more productive in Rust than TypeScript or PHP because I spend almost no time debugging. An expressive type system and pattern matching is a game changer.
If it compiles it works.
This clickable diagram from Effective Rust showing all the transforms for Option and Rust is an absolute masterpiece:
The only other thing you need to know is when to use as_ref.
The arrival of an Effective book for Rust seems like a coming of age thing. Rust has matured to the point where best practice guidelines are established.
I’m a fan because it reduces the cognitive load of making things go brrr.
Rust has a fear problem.
I semi-seriously implied to someone that they could re-write their Typescript lambdas in Rust and save a bunch of money.
They were terrified that it would be impossible to find people capable of doing it.
I've been implementing a simple binary tree. This has really driven home for me that collections need to implement three different iterators: one for references, one for mutable references and one that consumes the collection - which is the one shown here for my binary tree.
Thoughtful critiques of Rust with proposed solutions:
Between incremental linking, a parallel frontend, macro expansion caching, release mode-macros, and precompiled binaries, I’m positive we could cut Rust compile times by 90% or more.
This PR is a banger:
UTF8 validation is fastest with long strings (it can use SIMD). But string view data is stored sequentially with length before each string. Luckily, lengths less than 128 are valid ascii allowing efficient batching of shorter strings!
I don't find Rust syntax ugly now that I (mostly) understand all of it. It is admittedly somewhat common to end up with deeply nested types:
Result<Option<Box<Generic<Specific>>>>
but that part doesn't seem specific to Rust?
@tem_dev
This obscures the business logic and makes it hard to reason about correctness. I assume you’re doing this because you think it’s more efficient. But that’s usually far less important than clarity.
And it’s not even obvious to me it would be faster.
The top level architecture of grouping in Data Fusion has similarities to the Google MapReduce paper from 2004 that ignited the Big Data era. But the concerns are very different.
Efficient use of multiple SIMD powered cores vs using multiple computers.
@kiyov09
The advantage of starting with C is that it hides much less of the machinery and is a relatively simple language.
The effort required to accomplish things might be a bit terrifying!
“Rewrite Big Data in Rust (RBIR)”
That’s a good summary. I’ve been getting involved in Arrow and Data Fusion and keeping an eye on Iceberg Rust. Once you get started things start to get progressively easier. There is a lot of very interesting work happening.
Published a new post about my personal VISION: Data Freedom.
Access any data, across any service, using any method, in any language.
Contact me if our VISIONs align.
Rather than “you should learn C before learning Rust” how about “I found having some knowledge of C useful when I was learning Rust.”
People understandably dislike being told they SHOULD do something but often appreciate hearing about experiences.
“Despite many protestations that SQL was terrible by the end of the 2010s almost every NoSql DBMS added a SQL interface.”
And they all added transaction support after initially contending that eventual consistency was sufficient.
It took three years to finish, but our follow-up to the 2006 "What Goes Around Comes Around" is finally out! Stonebraker and I examine the last 20 years in databases and discuss why relational databases + SQL will continue to remain on top.
📄PDF:
Another source of cruft that pollutes Rust codebases: the builder pattern. It feels like the Rust ecosystem has Stockholm Syndrome… how could anyone believe that builders are at a sane default for large bags of fields?
Named and optional function parameters are needed.
Thoughtful critiques of Rust with proposed solutions:
Between incremental linking, a parallel frontend, macro expansion caching, release mode-macros, and precompiled binaries, I’m positive we could cut Rust compile times by 90% or more.
@GeorgeMayer
Trust really is the foundation needed to accomplish so many things. I think its importance is often under appreciated.
Whenever I see some sort of dysfunction I often try to start by first considering what trust issues might be involved.
The One Billion Row Challenge looks like a great one: write a Java program for retrieving temperature measurement values from a text file and calculating min, mean, and max temperature per weather station. The file has 1,000,000,000 rows!
Is there any practical reason to learn assembly today unless you’re working on a compiler and need to emit the assembly code?
I mean other than very specialized cases like writing a video codec?
Core databases like Postgres are typically written in C or C++. The tools emerging from the big data era - Lucene, Spark, Presto/Trino… are all on the JVM (Java, Scala).
Now there’s a new generation written in Rust: Arrow, DataFusion, Tantivity… We live in interesting times!
Most new systems languages stop short of full memory safety as supported in Rust. This includes the C successors like Zig and Odin as well as the C++ successor Carbon. It also includes Mojo.
Sean’s Safe C++ goes all the way. It’s a good read.
This week I started documenting Safe C++. I figure this will take me the rest of the month to complete. But it's good to have information written down in one place.
When you read a parquet file using Arrow in Rust the schema is inferred from the file. But sometimes you might need to cast the data type of some columns - read an Int32 as a Timestamp for example. Just landed a PR that lets you provide a schema.
I've been playing around with different ways of implementing binary trees in Rust. It's one of those things that seems to be treated as a black art but that seems unwarranted.
This is a simple one just using Box.
Vortex is an Apache Arrow-compatible toolkit for working with compressed array data. We are using Vortex to develop a next-generation columnar file format for multidimensional arrays.
Very interesting indeed.
Spiral is quietly doing some really cool work. Their Github repo is a lot of fun to poke at.
Glad to see their first blog post on some of their engineering.
@jamonholmgren
I think people are falling into the “it doesn’t always work and sometimes does dumb things so I’m not going to use it all” trap.
Or perhaps the even worse trap: “I can’t imagine how it could work so I’m not going to try it.”
Or the worst trap: “I read that it doesn’t work.”
I like trees rather than linked lists as a learning exercise because trees and graphs come up all the time in real work. I’ve never had a good reason to implement a linked list.
I think the database per customer approach to multi-tenancy is a good one and agree that SQLite becomes a reasonable choice.
We live in interesting times.
"I for one am terribly excited about what conceptual compressions might lie just around the corner, if single-tenancy at scale pans out as a strategy. Let’s go find out!"
Rust can be verbose. The types compose well but you often have nested types like Result<Option<T>,E> and then have to return a result with Ok(Some(thing)). It’s taken me a while to get used to that.
Tonbo is implemented with a Log Structured Merge Tree. The cool thing is that the data - the Sorted String Tables - are stored in parquet format. This provides many benefits:
- structured schema for the values
- fast and flexible OLAP queries
- leverages Arrow eco-system
It's key-value store in Rust day on the internet!
Tonbo provides essential KV-like methods: insert, filter, and range scan integrated into the Arrow eco-system. Very interesting!
I can confirm that as my fluency with Rust improves memory management and the borrow checker has receded into the background. And lifetimes hardly ever come up.
A lot of people feeling the need to show us how smart they are by mentioning that a compiler won’t save you from math, logic and domain errors.
Thank you for your service. There might be someone who doesn’t know this.
I think I’m now at a point where I’m more productive in Rust than TypeScript or PHP because I spend almost no time debugging. An expressive type system and pattern matching is a game changer.
If it compiles it works.
I think of Python as the configuration layer. The heavy lifting needs to be made as efficient as possible because it needs so much data and so many calculations. That’s going to favour Rust and C++ - possibly Mojo in the future.
Python is definitely the language AI/ML. But training, and other data science-y thing. For inference there is inertia that a lot of serving systems have python in them, but that will definitely change to be 99% rust/c++ over time. From the AI consumption perspective, all
The C and Rust folks are fighting (again?). And some are suggesting that everything would be better if we just had one programming language.
And the AI bois are still saying we won’t need any soon enough.
Yesterday a few of us were talking about CUDA kernels in PyTorch. This paper introduces the idea of database kernels running on SSD devices. That seems like a very cool idea.
Slowly working out some general principles on how kernel is used in computer science.
PGVECTOR IS NOW FASTER THAN PINECONE. And 75% cheaper thanks to a new open-source extension – introducing pgvectorscale.
🐘 What is pgvectorscale?
Pgvectorscale is an open-source PostgreSQL extension that builds on pgvector, enabling greater performance and scalability (keep
@DanielW_Kiwi
I just grind it out. Create some data sets that exercise various scenarios. Run the query against the data sets. Check the results match what’s expected. As an automated test.
It’s not glamorous work.
F# seems like an interesting language. I just can’t work up the energy to take on the .net ecosystem. I feel like I can get things done in the JVM world of if I need to and .net just seems similar but different.
DuckDB is very fast performing aggregations on a large csv. But queries on a parquet version of that file are at least an order of magnitude faster (20X typically).
@JosephJacks_
On the other hand it’s possible that forcing people to create models that learn more efficiently with less data will inspire innovation and accelerate progress. Current techniques might just be the lazy ones made possible only by cheap data.
Ownership in Rust is usually discussed alongside borrowing but this post makes it clear that ownership and borrowing are two distinct concepts. It also provides good examples of the difference between affine and linear types. A good Saturday morning read.
Error handling in Rust scales well. You can start by YOLO ing it with the ? operator and then add in more sophisticated error handling as it becomes necessary.
I like using Rust because I can write expressive and idiomatic code that is fast by default and doesn't tend to encounter unexpected performance cliffs.
@exhaze
In Rust I have deeply nested enums and structs that I can flexibly serialize and deserialize in a type safe way and map to other forms with pattern matching.
Typescript unions and switch statements are more limited leading to a lot of “manual” code that is error prone.
Rust 1.78 has just been released!
The new diagnostic attribute unlocks *massive* quality of life improvements for day-to-day Rust usage.
Library authors finally get access to (some of) the machinery that allows Rust's standard library to emit high quality compiler errors.
Rust provides a lot of degrees of freedom for structuring your projects. This can be overwhelming at first because it's not clear which options you should choose in various situations. I think some more prescriptive conventions might be helpful.
The Rust borrow checker doesn't allow the following so the entry implementation in BTreeMap needs to do some contortions. There is definitely a lot I don't know about unsafe Rust code.
I have been spending a lot of time exploring various corners of the standard library. Most recently I discovered DoubleEndedIterator and realized my little BinaryTree implementation should support it!
If you have Xcode tools installed building ARM 64 assembly on an Apple Silicon Mac is straight forward. It can be as simple as
gcc hello.s -o hello
It takes a bit of poking around to get there. Some useful links in the replies.
It’s a bad sign how much OpenAI is talking about “things that might be possible in the future”.
And journalists just uncritically repeating it should be embarrassed.
I was researching using Debezium to replicate Postgres data to Iceberg tables in S3. It’s definitely a thing.
Then I wanted to know if it could handle sharded data. I was not finding good answers until I gave Duck Duck Go a try.
Nailed it:
@GeorgeMayer
I honestly can’t tell if they are serious or it’s just memes all the way down.
All the comments on the post are about the quality of the presentation and nobody discussing the substance.
It's key-value store in Rust day on the internet!
Tonbo provides essential KV-like methods: insert, filter, and range scan integrated into the Arrow eco-system. Very interesting!
I started out writing C and assembly device drivers in DOS. If my code did memory unsafe things there were no seg faults. Instead, it would just scribble over some other memory in the system leading to random and unpredictable behaviour. Good times! I learnt to be careful.
@algo_luca
The definition in the standard library seems good:
More specifically, this is the offset in bytes between successive elements in an array with that item type including alignment padding. Thus, for any type T and length n, [T; n] has a size of n * size_of::<T>().
In Rust, deref on the Deref trait and as_deref on Option and Result are confusingly named.
They return a reference to an underlying thing that can be dereferenced. The naming implies it’s performing the deference. Which confused me because the methods return &T.
@unclebobmartin
If you added a keyword to your political posts I could mute it and filter them out? They just aren’t interesting to me. Seems like a missing feature on this site. But you could just choose a word as a service.
The create method for File has this signature:
pub fn create<P: AsRef<Path>>(path: P) -> Result<File>
This allows it to accept a String, a string slice, a Path or a PathBuf because the AsRef<Path> trait is implemented on all of them.