Eric Fredine
@fredine
Followers
948
Following
2,704
Media
69
Statuses
4,760
Principal engineer writing code for fun and profit. Lapsed photographer. Husband. Father. Aspiring fly fisher.
Vancouver, British Columbia
Joined July 2008
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#Mchoice2024
• 1834169 Tweets
#บัสซิ่งไทยแลนด์FinalEP
• 219712 Tweets
Texas -6.5
• 186576 Tweets
MCHOICE X ML
• 126915 Tweets
#Venue101
• 100171 Tweets
#liSTeningPARTY
• 94238 Tweets
#SixTONESANN
• 84702 Tweets
TOP ELITE ARTIST V
• 43087 Tweets
#ZMZMINZM
• 33589 Tweets
リスアニ
• 27449 Tweets
ROCKSTA SA CALAMBA
• 26718 Tweets
デスゲーム
• 19387 Tweets
紫耀くん
• 16492 Tweets
Hysteria
• 14384 Tweets
Lee Carsley
• 11772 Tweets
Wayne Rooney
• 11115 Tweets
アクキヨ
• 10841 Tweets
カンスト
• 10724 Tweets
ビーフシチュー
• 10558 Tweets
#LEO_HBD_26th
• 10253 Tweets
#HAPPYLEODAY_2024
• 10068 Tweets
Rust is doing this correctly. It’s better to model things as they are rather than requiring a bunch of implicit knowledge.
The simplicity of C is dangerously misleading.
52
109
2K
In Rust when you see
let y = x.clone();
You don’t know if it’s a cheap reference count increment or an expensive allocation and memcopy. You can do this instead:
let y = Arc::clone(x);
And enforce it with a clippy lint:
#![deny(clippy::clone_on_ref_ptr)]
20
28
534
Reading large csv files in parallel chunks is tricky because you can’t be sure a line end isn’t in a quoted field. But I found a paper showing how to make a good guess that works most of the time!
11
42
460
I think I’m now at a point where I’m more productive in Rust than TypeScript or PHP because I spend almost no time debugging. An expressive type system and pattern matching is a game changer.
If it compiles it works.
27
21
409
This clickable diagram from Effective Rust showing all the transforms for Option and Rust is an absolute masterpiece:
The only other thing you need to know is when to use as_ref.
3
23
160
The arrival of an Effective book for Rust seems like a coming of age thing. Rust has matured to the point where best practice guidelines are established.
I’m a fan because it reduces the cognitive load of making things go brrr.
3
19
131
Rust has a fear problem.
I semi-seriously implied to someone that they could re-write their Typescript lambdas in Rust and save a bunch of money.
They were terrified that it would be impossible to find people capable of doing it.
17
5
124
I've been implementing a simple binary tree. This has really driven home for me that collections need to implement three different iterators: one for references, one for mutable references and one that consumes the collection - which is the one shown here for my binary tree.
14
3
125
Thoughtful critiques of Rust with proposed solutions:
Between incremental linking, a parallel frontend, macro expansion caching, release mode-macros, and precompiled binaries, I’m positive we could cut Rust compile times by 90% or more.
7
11
121
Mind you, I think the score is more likely to be a float, so then you need to do something like this
9
0
118
This PR is a banger:
UTF8 validation is fastest with long strings (it can use SIMD). But string view data is stored sequentially with length before each string. Luckily, lengths less than 128 are valid ascii allowing efficient batching of shorter strings!
3
11
96
I don't find Rust syntax ugly now that I (mostly) understand all of it. It is admittedly somewhat common to end up with deeply nested types:
Result<Option<Box<Generic<Specific>>>>
but that part doesn't seem specific to Rust?
31
0
94
The top level architecture of grouping in Data Fusion has similarities to the Google MapReduce paper from 2004 that ignited the Big Data era. But the concerns are very different.
Efficient use of multiple SIMD powered cores vs using multiple computers.
1
8
70
“Rewrite Big Data in Rust (RBIR)”
That’s a good summary. I’ve been getting involved in Arrow and Data Fusion and keeping an eye on Iceberg Rust. Once you get started things start to get progressively easier. There is a lot of very interesting work happening.
Published a new post about my personal VISION: Data Freedom.
Access any data, across any service, using any method, in any language.
Contact me if our VISIONs align.
3
5
45
1
4
64
Rather than “you should learn C before learning Rust” how about “I found having some knowledge of C useful when I was learning Rust.”
People understandably dislike being told they SHOULD do something but often appreciate hearing about experiences.
11
2
54
“Despite many protestations that SQL was terrible by the end of the 2010s almost every NoSql DBMS added a SQL interface.”
And they all added transaction support after initially contending that eventual consistency was sufficient.
It took three years to finish, but our follow-up to the 2006 "What Goes Around Comes Around" is finally out! Stonebraker and I examine the last 20 years in databases and discuss why relational databases + SQL will continue to remain on top.
📄PDF:
26
357
1K
2
10
53
Another source of cruft that pollutes Rust codebases: the builder pattern. It feels like the Rust ecosystem has Stockholm Syndrome… how could anyone believe that builders are at a sane default for large bags of fields?
Named and optional function parameters are needed.
Thoughtful critiques of Rust with proposed solutions:
Between incremental linking, a parallel frontend, macro expansion caching, release mode-macros, and precompiled binaries, I’m positive we could cut Rust compile times by 90% or more.
7
11
121
5
2
50
@GeorgeMayer
Trust really is the foundation needed to accomplish so many things. I think its importance is often under appreciated.
Whenever I see some sort of dysfunction I often try to start by first considering what trust issues might be involved.
1
0
51
@sean_j_roberts
I think so - if you don’t want to do an allocation and return an owned string. But I’m still a bit hazy on this myself.
1
0
51
The One Billion Row Challenge looks like a great one: write a Java program for retrieving temperature measurement values from a text file and calculating min, mean, and max temperature per weather station. The file has 1,000,000,000 rows!
1
4
49
Is there any practical reason to learn assembly today unless you’re working on a compiler and need to emit the assembly code?
I mean other than very specialized cases like writing a video codec?
57
0
44
Core databases like Postgres are typically written in C or C++. The tools emerging from the big data era - Lucene, Spark, Presto/Trino… are all on the JVM (Java, Scala).
Now there’s a new generation written in Rust: Arrow, DataFusion, Tantivity… We live in interesting times!
6
1
42
Most new systems languages stop short of full memory safety as supported in Rust. This includes the C successors like Zig and Odin as well as the C++ successor Carbon. It also includes Mojo.
Sean’s Safe C++ goes all the way. It’s a good read.
This week I started documenting Safe C++. I figure this will take me the rest of the month to complete. But it's good to have information written down in one place.
15
52
322
4
3
40
When you read a parquet file using Arrow in Rust the schema is inferred from the file. But sometimes you might need to cast the data type of some columns - read an Int32 as a Timestamp for example. Just landed a PR that lets you provide a schema.
1
5
39
I've been playing around with different ways of implementing binary trees in Rust. It's one of those things that seems to be treated as a black art but that seems unwarranted.
This is a simple one just using Box.
3
0
36
Vortex is an Apache Arrow-compatible toolkit for working with compressed array data. We are using Vortex to develop a next-generation columnar file format for multidimensional arrays.
Very interesting indeed.
Spiral is quietly doing some really cool work. Their Github repo is a lot of fun to poke at.
Glad to see their first blog post on some of their engineering.
0
2
34
1
4
35
Landed my first PR in the Apache Data Fusion project.
Very well run and well organized project. This was indeed a fine first issue to tackle.
4
2
33
@DrawsMiguel
No. A bag isn’t ordered.
You could do tup but also tuple is fine and everyone knows what it is.
1
0
32
@jamonholmgren
I think people are falling into the “it doesn’t always work and sometimes does dumb things so I’m not going to use it all” trap.
Or perhaps the even worse trap: “I can’t imagine how it could work so I’m not going to try it.”
Or the worst trap: “I read that it doesn’t work.”
3
0
33
I’ve been reading documentation for async Rust and Tokio. I think some of the negative sentiment around async Rust might be over blown.
9
2
33
@ludwigABAP
That’s my perspective. I still hate everything related to python installation and packaging.
3
0
27
Even when you write Rust code that is hilariously inefficient because you don’t know what you’re doing it ends up being more than fast enough.
2
3
28
I’m using Rust as a faster Python,
That’s the explicit goal of Mojo. But it seems like it still has a ways to go until it’s ready for real world use?
13
0
28
I like trees rather than linked lists as a learning exercise because trees and graphs come up all the time in real work. I’ve never had a good reason to implement a linked list.
6
1
26
@AdamRackis
Typescript has a high ceremony to value threshold for me compared to something like Rust or C++.
1
0
26
The “big data” era tools were built in Java. The new exploded database tools are being built in Rust.
2
3
26
I think the database per customer approach to multi-tenancy is a good one and agree that SQLite becomes a reasonable choice.
We live in interesting times.
"I for one am terribly excited about what conceptual compressions might lie just around the corner, if single-tenancy at scale pans out as a strategy. Let’s go find out!"
30
36
380
2
5
24
Rust can be verbose. The types compose well but you often have nested types like Result<Option<T>,E> and then have to return a result with Ok(Some(thing)). It’s taken me a while to get used to that.
4
0
23
Tonbo is implemented with a Log Structured Merge Tree. The cool thing is that the data - the Sorted String Tables - are stored in parquet format. This provides many benefits:
- structured schema for the values
- fast and flexible OLAP queries
- leverages Arrow eco-system
It's key-value store in Rust day on the internet!
Tonbo provides essential KV-like methods: insert, filter, and range scan integrated into the Arrow eco-system. Very interesting!
0
3
14
2
1
24
There seem to be a lot of people identifying as C programmers who don’t understand character encodings?
1
0
23
I can confirm that as my fluency with Rust improves memory management and the borrow checker has receded into the background. And lifetimes hardly ever come up.
2
4
21
@TheEduardoRFS
That is the crux of the problem. It’s the one the proposed Claim trait tries to address directly.
0
0
21
A lot of people feeling the need to show us how smart they are by mentioning that a compiler won’t save you from math, logic and domain errors.
Thank you for your service. There might be someone who doesn’t know this.
I think I’m now at a point where I’m more productive in Rust than TypeScript or PHP because I spend almost no time debugging. An expressive type system and pattern matching is a game changer.
If it compiles it works.
27
21
409
5
0
20
I think of Python as the configuration layer. The heavy lifting needs to be made as efficient as possible because it needs so much data and so many calculations. That’s going to favour Rust and C++ - possibly Mojo in the future.
Python is definitely the language AI/ML. But training, and other data science-y thing. For inference there is inertia that a lot of serving systems have python in them, but that will definitely change to be 99% rust/c++ over time. From the AI consumption perspective, all
1
2
16
2
1
21
The C and Rust folks are fighting (again?). And some are suggesting that everything would be better if we just had one programming language.
And the AI bois are still saying we won’t need any soon enough.
7
2
19
Yesterday a few of us were talking about CUDA kernels in PyTorch. This paper introduces the idea of database kernels running on SSD devices. That seems like a very cool idea.
Slowly working out some general principles on how kernel is used in computer science.
This week's paper in the DistSys reading group looks fantastic.
4
21
192
1
5
18
Improvements in vector search performance are still arriving at furious pace.
Postgres extensions written in Rust btw.
PGVECTOR IS NOW FASTER THAN PINECONE. And 75% cheaper thanks to a new open-source extension – introducing pgvectorscale.
🐘 What is pgvectorscale?
Pgvectorscale is an open-source PostgreSQL extension that builds on pgvector, enabling greater performance and scalability (keep
41
226
1K
1
0
19
@DanielW_Kiwi
I just grind it out. Create some data sets that exercise various scenarios. Run the query against the data sets. Check the results match what’s expected. As an automated test.
It’s not glamorous work.
2
1
17
F# seems like an interesting language. I just can’t work up the energy to take on the .net ecosystem. I feel like I can get things done in the JVM world of if I need to and .net just seems similar but different.
6
1
17
DuckDB is very fast performing aggregations on a large csv. But queries on a parquet version of that file are at least an order of magnitude faster (20X typically).
3
1
18
@JosephJacks_
On the other hand it’s possible that forcing people to create models that learn more efficiently with less data will inspire innovation and accelerate progress. Current techniques might just be the lazy ones made possible only by cheap data.
3
0
17
Ownership in Rust is usually discussed alongside borrowing but this post makes it clear that ownership and borrowing are two distinct concepts. It also provides good examples of the difference between affine and linear types. A good Saturday morning read.
0
1
18
Error handling in Rust scales well. You can start by YOLO ing it with the ? operator and then add in more sophisticated error handling as it becomes necessary.
4
1
17
I like using Rust because I can write expressive and idiomatic code that is fast by default and doesn't tend to encounter unexpected performance cliffs.
3
0
17
@DrawsMiguel
@_Felipe
Unironically why I’ve developed an interest in compilers. I accidentally built one and realized I didn’t know what I was doing.
1
0
16
@RichHHawk
Exactly. You are now responsible for interpreting those bytes correctly. I prefer to let a compiler do that for me.
1
0
16
@greg_johnston
This kind of post is usually from people who aren’t seriously using Rust. The serious ones are just busy getting things done.
1
0
15
@exhaze
In Rust I have deeply nested enums and structs that I can flexibly serialize and deserialize in a type safe way and map to other forms with pattern matching.
Typescript unions and switch statements are more limited leading to a lot of “manual” code that is error prone.
2
0
14
@ludwigABAP
My callused hands (from fly-fishing BTW) are ready to rewrite it all in Rust. I feel the warm glow of job security.
1
0
14
I had an epic day dry fly fishing on the eastern slopes of the Alberta Rockies last week. One of 16 brown trout landed.
5
1
14
This sort of attention to detail and care for quality error messages is a Rust super power.
Rust 1.78 has just been released!
The new diagnostic attribute unlocks *massive* quality of life improvements for day-to-day Rust usage.
Library authors finally get access to (some of) the machinery that allows Rust's standard library to emit high quality compiler errors.
8
70
458
0
1
15
Rust provides a lot of degrees of freedom for structuring your projects. This can be overwhelming at first because it's not clear which options you should choose in various situations. I think some more prescriptive conventions might be helpful.
2
0
14
The Rust borrow checker doesn't allow the following so the entry implementation in BTreeMap needs to do some contortions. There is definitely a lot I don't know about unsafe Rust code.
4
0
15
I have been spending a lot of time exploring various corners of the standard library. Most recently I discovered DoubleEndedIterator and realized my little BinaryTree implementation should support it!
0
0
15
I might have a legitimate reason to sneak a little Rust into production.
5
0
14
@ewanmakepeace
It’s probably the case that the more expressive your type system is the more likely you are to have this experience.
0
0
13
What’s stopping you from writing assembly code on your Mac?
If you have Xcode tools installed building ARM 64 assembly on an Apple Silicon Mac is straight forward. It can be as simple as
gcc hello.s -o hello
It takes a bit of poking around to get there. Some useful links in the replies.
1
0
3
10
0
14
It’s a bad sign how much OpenAI is talking about “things that might be possible in the future”.
And journalists just uncritically repeating it should be embarrassed.
3
0
14
I was researching using Debezium to replicate Postgres data to Iceberg tables in S3. It’s definitely a thing.
Then I wanted to know if it could handle sharded data. I was not finding good answers until I gave Duck Duck Go a try.
Nailed it:
0
1
14
@GeorgeMayer
I honestly can’t tell if they are serious or it’s just memes all the way down.
All the comments on the post are about the quality of the presentation and nobody discussing the substance.
3
0
13
Dynamic programming solution to calculate the length of the longest common subsequence in Rust:
0
1
14
Finding the React compiler discourse tedious but can’t mute compiler because I like reading about compilers.
2
0
14
I started out writing C and assembly device drivers in DOS. If my code did memory unsafe things there were no seg faults. Instead, it would just scribble over some other memory in the system leading to random and unpredictable behaviour. Good times! I learnt to be careful.
3
0
13
@algo_luca
The definition in the standard library seems good:
More specifically, this is the offset in bytes between successive elements in an array with that item type including alignment padding. Thus, for any type T and length n, [T; n] has a size of n * size_of::<T>().
0
0
13
In Rust, deref on the Deref trait and as_deref on Option and Result are confusingly named.
They return a reference to an underlying thing that can be dereferenced. The naming implies it’s performing the deference. Which confused me because the methods return &T.
2
1
13
@unclebobmartin
If you added a keyword to your political posts I could mute it and filter them out? They just aren’t interesting to me. Seems like a missing feature on this site. But you could just choose a word as a service.
2
0
13
The create method for File has this signature:
pub fn create<P: AsRef<Path>>(path: P) -> Result<File>
This allows it to accept a String, a string slice, a Path or a PathBuf because the AsRef<Path> trait is implemented on all of them.
2
2
13