ABC
@Ubunta
Followers
3,950
Following
3,157
Media
280
Statuses
5,210
Data & ML Infrastructure for Healthcare Opinions are पड़ोसी' | DhanvantriAI 📍 🇩🇪Berlin & 🇮🇳Kolkata
Berlin, Germany
Joined August 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#AMAs
• 216623 Tweets
D-10
• 214171 Tweets
RENJUN
• 130932 Tweets
STRAY KIDS BOYBAND LEGACY
• 100162 Tweets
Dodgers
• 99310 Tweets
Halo
• 95455 Tweets
Giants
• 74170 Tweets
Steelers
• 73162 Tweets
Hobi
• 49679 Tweets
49ers
• 44977 Tweets
Cowboys
• 42595 Tweets
斉藤慎二
• 25646 Tweets
書類送検
• 23520 Tweets
斉藤メンバー
• 23291 Tweets
ジャングルポケット
• 23237 Tweets
#ドンキで見つけた
• 15960 Tweets
Mariah
• 15909 Tweets
Bronny
• 14539 Tweets
Profar
• 13694 Tweets
The Next Prince Q8
• 13528 Tweets
Tatis
• 13238 Tweets
#DigitalFasting
• 13217 Tweets
G DE GARIME
• 12029 Tweets
ロケバス
• 11627 Tweets
ダルビッシュ
• 10947 Tweets
Niners
• 10692 Tweets
Pinned Tweet
As a Senior Staff Data Engineer, my top five tasks over the past 2 years include:
1. Simplifying Kubernetes for Data Scientists/Engineers: Developed user-friendly libraries and containers, enabling Data Scientists to utilize Kubernetes effortlessly. Achieved a complete
12
40
335
Lazydocker - A very useful terminal UI based application to manage Docker
This is really a brilliant application for simplifying docker management
22
133
636
DrawDB is an excellent tool for database design and ER modeling. I found it very user-friendly, and it also allows you to upload existing schemas.
📌You can check it out here: ().
I used the generated SQL for PostgreSQL!
3
81
324
Data Engineering and Machine Learning are currently in one of their most exciting phases:
- Single-node data stacks, like
@DataPolars
and Apache Arrow, are now capable of handling 80% of data use cases, even with terabytes of data.
-
@duckdb
is rapidly gaining traction, with
6
25
254
Apache Arrow is on Fire 🔥🔥🔥
🙏 Data Fusion
🔥
@duckdb
⚡Polars Data
To me,
@ApacheArrow
is now the most important component in the data and ML community
7
21
244
Data Engineering offers good pay if you're skilled in several technologies
- Streaming engines: Flink & Kafka
- DWH: Spark , snow, trino, clickhs
- Distributed DB: hbase, cockroachdb, yugabyte
- Infrastructure: elk stack, docker
+ Python & sql
"Ability to explain ☝️ these"
3
37
214
The most effective and modern approach to Data Engineering and Machine Learning involves using
@duckdb
,
@DataPolars
, or Data Fusion with external data formats like Delta, Iceberg, or Parquet.
If you're still using CSV, the first step should be converting it to Parquet!
7
13
180
A very minimal Data Engineering technology stack designed to address the majority of data use cases for a team newly established in close collaboration with Data Science.
1. Apache Spark: This is our primary tool for data processing, ETL, and SQL queries. Apache Spark is highly
6
29
154
Here are some potential shifts we may see in modern data engineering:
- High-performance DataFrame technologies such as
@DataPolars
,
@daft_dataframe
, and DataFusion are set to gradually replace Pandas as the standard for data manipulation and analysis.
-
@duckdb
will continue to
5
18
153
The biggest problem with the current state of Data Engineering is that we are trying to do *almost everything with SQL.
Previously, we were trying everything to Avoid SQL!
17
8
146
Over the past two years, I have conducted over 100 interviews for Data Engineering positions and have noticed the following trends:
1. Over 80% of Data Engineers have not collaborated with Data Science teams, predominantly having backgrounds in analytics.
2. More than 90% of
6
17
139
A very Data Engineering Problem: When working with data that contains rows with very similar content, it's more effective to deduplicate these rows with a certain degree of probability rather than attempting to process all the data.
For this purpose, I utilized Splink, and the
3
10
136
Avoid
@ApacheSpark
when:
- Your dataset is smaller than 50GB; opt for
@duckdb
&
@DataPolars
- Lack of experienced data engineers (infra) and budget for Databricks
- Priority is cost-saving
- Project is short-term
- Need only 2-3 medium nodes; explore vertical scaling.
5
21
134
I demoed
@duckdb
to 10 data scientists last Friday & they used it regularly for the week. Feedback -
* Speed is the most impressive thing about DuckDB.
* It's not exactly a Pandas replacement ( not yet)
* Most of them found out the real power of
@ApacheArrow
1st time
9
17
128
Inefficient Data Engineering Practices:
- Apache Flink pipeline for handling 1kb data/sec
- Apache Airflow for manual task execution
- Switching to dbt without need, just following trends.
- Handling massive CSV files inefficiently.
- 1000s of Dashboards with no users
9
15
122
When a majority of the data/ML community was busy building tools for large-scale systems,
@duckdb
built an extremely fast database &
@ApacheArrow
designed a memory format to optimize a single node.
Both have a massive impact on the overall data/ML ecosystem.
3
17
121
Data Engineering involves:
- Crafting SQL/dbt pipelines for dbs & dashboards.
- Designing Spark jobs & orchestrating via Airflow.
- Launching real-time tasks with Flink, Kafka, & KV.
- Analyzing data using pandas & polars.
- Building data infrastructure.
- Converting NB to prod
2
20
117
As a Data Engineer, working closely with many data scientists has provided me with valuable insights. Here are my 5 key takeaways:
1. Diverse Interests in Data: It's a misconception that all data scientists primarily focus on model building. Many are deeply interested in
3
10
109
The biggest problem of
@duckdb
& entire
@ApacheArrow
stack including Ibis is Awareness.
Most of the Data/ML community are not even aware of these technologies, forget usecases.
7
11
104
The evolution of highly performant single-node data tools like
@duckdb
,
@ApacheArrow
@IbisData
, and Polars, which can process vast amounts of data and exceed the limits of RAM, is a remarkable development for cloud deployment.
With these tools, I can now design highly complex
3
14
102
How will I process over 100GB+ of data on the **cloud:
Choose an instance with ~32GB+ RAM and SSD storage. Install
@ApacheArrow
,
@duckdb
, and/or DataFusion to utilize lazy evaluation and perform complex queries.
Begin with the initial setup and run your data queries.
If the
4
9
95
In Data Engineering,
Key Aspects to Focus On:
1. Dependable Database Systems
2. The choice of data format is critical; while CSV is widely recognized, Parquet offers superior efficiency and optimization.
3. Effective Logging: Even simple print statements can be invaluable.
4.
3
8
93
Data Engineers continue to face the same major challenge as they did in 2023.
1. Clear Communication with Clients: Avoid technical jargon to prevent confusion. Understanding client needs clearly is vital for the success of data projects.
2. Explaining Data Pipelines to
1
20
89
Deploying
@duckdb
on Kubernetes & interact with the duckdb over
@apachesuperset
is impressive.
It's by far the cheapest option (*for me) to explore data quickly for building data science applications.
Native integration with
#apachearrow
made life much easier
5
11
84
While building Data Engineering or Machine Learning pipelines using CSV data, you should first:
🔥Convert the CSV data to parquet and save it using
@DataPolars
.
Using Parquet +
@ApacheArrow
combination with almost any data engine is significantly faster than CSV
7
11
77
The "Tools don't matter" argument falls flat in Data Engineering and ML.
- Working with large datasets on a Spark cluster can quickly shift your focus from business problems to infrastructure challenges.
- Using Pandas for 10GB+ data demands huge RAM, but switching to
@duckdb
3
3
77
You don't need any cloud subscription to start learning Data Engineering.
- Setup Docker with
@Minio
@duckdb
@apachesuperset
- Use Minio to create S3 compatible object store
- Use DuckDB as a database by specifying the S3 endpoint
- Use Apache Superset to create dashboards-
6
10
75
Several startups in Berlin 🇩🇪 are finding data warehouses extremely expensive & pulling out. I am personally aware of 3 companies from India who dropped DWH licenses & moved entirely to object store data.
Speed is good but comes with a cost & not always make 💵 .
15
5
73
In data engineering, I encountered some issues due to overcomplicating things with advanced tools and technologies:
1. Instead of using complex programs like PySpark, a simple pipe could have been enough for the task.
2. Reading data directly from a storage service like Amazon
3
14
72
A few strategies I implemented in my Data Engineering production systems to make it more "cost effective💰" and "less cool"
1. Maximize the use of caching wherever feasible. adding a cache layer between the client and data warehouse, resulting in over 60% cache hits weekly
2.
2
6
71
When a data scientist finds out that a big data warehouse isn't the only way and finds modern data stack:
- Big data warehouses can be costly for testing and exploring data.
- It might take a while for data scientists to switch to
@duckdb
or new data stack, but they get there.
1
7
70
Some interesting Data Engineering benchmarks for production
1. Measure how long different technologies take to write data to blob storage. Not all tech behaves save while writing to the blob.
2. Track the time required to serialize and deserialize data. This is crucial for
0
13
71
Guidelines for Data Engineers to collaborate easily with Data Scientists
- Don't assume Data Scientists will like or be familiar with containers; automate containerization and deployment steps as much as possible.
- Data Engineers may not prefer Notebooks in production, but for
1
5
71
Tech Update: Over a million dashboards were created last month, and the total estimated number of views they received was 100.
2
6
70
I blogged how to set up and explore
@duckdb
using
@apachesuperset
on Kubernetes
This is becoming an important setup for my ML Platform and I am convincing all my data scientists to use it.
3
11
70
I typically default to PostgreSQL for most data use cases, but it’s not always convenient for data science applications. Setting up PostgreSQL involves significant effort in creating specific users, schemas, and loading backups.
That's why
@duckdb
is becoming my preferred
3
5
68
Why aren't
@ApacheArrow
or Protocol Buffers the universal standard for data formats in Data Engineering and Machine Learning applications?
- What factors lead Data Engineers and ML professionals to pay minimal attention to the data format?
- What barriers prevent vendors from
9
6
66
🍻
@clickhouse
is fast but not designed for Single Node
🦆
@duckdb
is the best OLAP local but is not good with object storage
❄️
@SnowflakeDB
& DBC are highly scalable but not cheap
🐘
@PostgreSQL
is the best but sharding is still a pain
12
5
63
I've been using
@SnowflakeDB
Copilot for the past two days and have successfully onboarded my entire Data Science team using Copilot. Snowflake copilot is pretty fast and way faster than OpenAi.
Text-to-SQL is likely to be one of the first challenges successfully tackled by
3
7
63
Apache Airflow remains one of the most widely used orchestration platforms for data engineering, primarily due to its simplicity.
While you might encounter a bit more work and a somewhat sluggish UI, the Airflow operator model is very simple, and creating your own operators is
7
2
61
New Data Engineering is Doing More by using Less
1. Choose
@DataPolars
/
@duckdb
instead of data warehouses.
2. Utilize Jupyter/
@streamlit
for temporary dashboards.
3. Implement code from tools like Copilot/GenAI, avoiding rewrites.
4. Focus on core principles over tools.
1
10
61
A simple Kubernetes open source based MLOps platform
*
@MLflow
for tracking & versioning
*
@DVCorg
for Data Versioning
* Seldon Core for
#MachineLearning
Microservices
*
@bentomlai
for Model Deployment
*
@flyteorg
for k8s native orchestration
1
9
58
In the field of data engineering and machine learning, Apache Parquet and Apache Arrow are becoming the cornerstones
2
5
57
A small but efficient Data Engineering and Machine Learning infrastructure stack for seamless collaboration between engineers and data scientists.
Data Stack:
- Python-based with Apache Arrow, DuckDB, Polars, and Pandas
- Jupyter Notebook endpoint
- Integration with a separately
1
5
59
Challenging Data Engineering Practices That Are Difficult to Avoid
1. Reliance on Spark UDF: Despite its inefficiency, Spark UDFs are widely used for their speed and ability to handle complex tasks, making them a tempting option.
2. Overusing
@streamlit
: Streamlit's ease of
0
1
54
If you effectively utilize
@ApacheIceberg
or
@apachehudi
, you can bypass the need for a managed data warehouse for a medium-sized data engineering team.
Incorporating
@ApacheArrow
as the foundational layer for data further enhances system efficiency, rivaling that of any
2
10
55
Insights gained from Young Data Engineers
- Many young DEs primarily focus on specific platforms like AWS or Azure, often missing the broader context of their field and concentrating solely on tool-specific functionalities.
- They are influenced by popular posts on social media
7
7
53
What a Data Engineer wants
- Setting up
@DataPolars
and using a data stack around
@ApacheArrow
- Converting all code to PySpark and bringing in a cluster for production
- Adding all possible alerts for metrics and resource utilization
What a Data Scientist wants
- Developing a
2
2
52
As a data engineer, implement a centralized logging framework immediately if you're collaborating with data scientists.
It streamlines the process, saving time, and prevents them from navigating the chaotic maze of sourcing logs from separate data tools.
3
8
51
Very Data Engineering decisions 😅:
1. Overlooking Caching: Treating Spark's 'cache' as the sole caching mechanism.
2. Ignoring Query Optimization: Not considering query plans despite using large databases.
3. Minimizing Code: Favoring tools over high level programming,
0
7
52
Peak Data Engineering moments:
-Attempted loading a hefty csv with pandas/spark - no luck.
-Gave
@duckdb
@DataPolars
a shot - still no go.
-Went old-school: read line by line, fixed errors, then rewrote.
-Transformed the polished csv to parquet.
-feeling lucky as parquet exists😅
4
4
51
Over the past weeks, I've delved deeply into the evolving DataFrame ecosystem and have realized one fundamental truth: Apache Arrow is essential and default.
@DataPolars
is impressively fast, but the performance difference between Polars and Pandas 2.0 with Arrow integration
8
3
52
Most of the advancements in Data and Machine Learning are now being led by Rust, with
@ApacheArrow
playing a crucial role in this development.
The data, stored in Arrow format, allows for seamless operation with both Rust and Python because of 0 copy!
3
4
51
Challenges in Data Engineering and Data Science
1. Dockerfile isn't popular among many.
2. GitHub workflow is still an unknown territory
3. While data accessibility is crucial, data security poses challenges
4. Thinks DE/DS is a gold mine 💰
5. No standard Production setup
3
5
50
Avoid data engineering pitfalls:
1. Don't migrate to a large database/dwh when simpler options like Postgres suffice.
2. Avoid costly clusters for non-revenue projects.
3. Never compromise data security for convenience.
4. Find your real customer
5. Setup the metrics
2
4
48
Unspoken Principles of Data Engineering
1. Once established in the production environment, a table is eternal. Whether it’s actively used or not, it’s destined to stay indefinitely.
2. The data platform that seems straightforward to you will invariably be perceived as the most
1
3
47
My fine-tuned GenAI 7B Model for Data Engineering is in prod:
1. Domain-specific txt-to-SQL
2. Pyspark, Airflow, Polars, Pandas & internal libraries code generation, tailored to your codebase
3. Explaining code/stats to the non tech
4. Mining unique codes from domain docs
8
5
47
Reflecting on my experience using a
@streamlit
app in production over the past six months, even with a limited user base of 20 users, I've learned a few interesting lessons:
- From the very beginning, I found it crucial to use an external database for maintaining the state and
3
2
47
Each of Iceberg, Hudi, and Delta has a dedicated audience:
1. Spark users who are Databricks customers lean towards Delta.
2. Snowflake users not keen on Delta(random reasons) often go for Iceberg.
3. Those with a long-term interest in this technology generally prefer Hudi.
8
4
46
Why are new DataFrame technologies not developing APIs similar to Pandas?
API comfort is a thing, and it's challenging to persuade Data Scientists to adopt new methods for DataFrame transformations when they can simply argue, "Pandas has been around forever; why should I change
7
4
46
Today, I had a call with three senior data leaders, and we discussed my preferred data warehouse solutions.
I believe modern data warehouses are incredibly powerful and can address almost all use cases. However, they can also become the most expensive data assets within an
0
6
45
Apache Airflow is a good orchestration platform for DE/ML but it doesn't solve all problems and it's really not easy we just got used to it.
But now it's changing and looking for alternatives are the most obvious & common step.
16
5
43
The field of Data Engineering is currently grappling with a significant challenge related to Data Format.
Generally, there are two primary types of data formats:
1. Human-Readable formats: These include familiar formats such as Excel, CSV, and JSON. They are highly preferred by
3
11
44
If your data engineering system is composed of multiple platforms like Databricks,
@SnowflakeDB
,
@duckdb
, etc., just use SQL 🏋️♂️
Trying to use Python is not gonna make your life easier and everything is neither Pandas nor Pyspark but SQL is indeed possible
3
1
44
If you don't have a very large dataset, try to consider polars dataframe,
@duckdb
&
@ApacheArrow
for a cheaper but faster analytics need.
Local development is extremely fast with the above package and you can avoid connecting to a data warehouse cluster for many use cases!
1
3
43
Data Engineering Anti patterns
*Overkill: A data warehouse for tiny public datasets.
*One-offs: Data pipelines for yearly tasks.
*Misjudged Scale: Treating under 50Gb as big data.
*Underestimation: Overlooking Postgres/MySQL's capacity
*Neglecting CSV->strict schema conversion
2
3
41
Processing data locally with Pandas2 and
@DataPolars
reveals Polars as the superior choice.
When processing approximately 3.5GB of CSV text data row by row, including extracting a set of keywords, using regex, splitting and merging strings, and then writing the results back as
Which is quicker for processing a ~10GB text dataset locally: using
@IbisData
with
@duckdb
or
@DataPolars
?
There are numerous methods for processing text data, and while Pandas remains a convenient option, it's time to consider alternatives. 😇
13
5
40
4
3
43
Data compression plays a crucial role in establishing a robust data infrastructure and is vital for efficient data transmission over networks.
zstd has emerged as a leading compression algorithm, reducing data size by 55-60% when transmitted. Notably, its 'Fastest' mode offers
4
3
43
Now Data engineering and ML stack is composed of
🏋️♂️
@databricks
or Spark
❄️
@SnowflakeDB
or other DWH
🐍
@getdbt
&\| Python pandas, numpy, scikit & other ml libraries
🐻❄️
@DataPolars
for pandas
🦢
@duckdb
for OLAP
⏩1 orchestrator
The missing part is
@IbisData
to bring consistency
1
8
43
How I *finally persuaded data scientists to adopt
@duckdb
&
@DataPolars
🛠️🔍
- Developed a streamlined library mirroring Pandas functions for polars/duckdb
- Implemented an AI interface to recommend exact functions in polars/duckdb docs that is not in lib (still refining)
4
3
41
Created a Data Engineering basic pipeline in 10 mins using
@OpenAI
Vision &
@phidatahq
's sdk.
Smooth image-to-code transformation—futuristic!
- Read CSV data and convert to pqt by polars
- Pyspark Data Processing
- Snowflake data loading
- Orchestrating
All with an image
3
7
42
A significant challenge in transitioning from Pandas to
@DataPolars
is the syntax difference. There's often debate over the numerous changes required.
Then I realized that
@IbisData
functions as a DataFrame programming language, allowing the use of
@duckdb
, Polars, or any other
1
7
42
The B+Tree is an indispensable data structure that underpins many of the databases we rely on daily. While it's often a topic of discussion in interviews, its practical implementation can be quite challenging. This blog post demystified the complexities of the B+Tree.
Real-world
1
7
41
Which is quicker for processing a ~10GB text dataset locally: using
@IbisData
with
@duckdb
or
@DataPolars
?
There are numerous methods for processing text data, and while Pandas remains a convenient option, it's time to consider alternatives. 😇
13
5
40
Engineers are usually more interested in advocating their preferred technologies than understanding the actual business requirements 😵💫
Many startups have ❄️ db for an overall data size of less than 100GB with a revenue of 100k💲
3
1
41
Here are 3 Dashboards that integrate well with existing data and machine learning ecosystems:
1. Apache Superset: Offers excellent integration and is user-friendly.
2. Metabase: Provides an intuitive interface, especially for joining tables.
3. DataEase: A newer option I'm
6
2
40
Another open-source database,
@CockroachDB
, has moved away from the open-source model.
I believe PostgreSQL will continue to be the dominant choice now and in the future!
1
5
39
ADBC is a single API for getting Arrow data in and out of different databases.
JDBC and ODBC drivers are widely popular drivers but these are more database specific.
But ODBC Columnar format representation is not a perfect match with Arrow.
1
6
40
The key issues in Data Engineering:
1. Resource Optimization: Challenges include inefficient code reliant on scalability, premature adoption of new data technologies leading to increased costs and complexity, and a lack of deep optimization or understanding of these systems.
2.
0
2
39
More than half of Data Engineering challenges can be addressed more effectively through smart caching rather than by scaling.
2
3
39
Developing a GPT-powered tool simplifies tasks and could revolutionize the engineering field
- Data Engineering Assistant
- DE bootcamp
- Domain Specific Code generation
Consider an SDK tailored for data engineering that could automate half of the workload.
1
1
39