Data Engineer
@techtonic190427
Followers
3,981
Following
1,698
Media
191
Statuses
675
Data Engineer | Pythonist | Only data can judge on behalf of God; all else is mere opinion. Those who control the data control both you and me.
Joined May 2023
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
فياريال
• 661407 Tweets
Butler
• 411228 Tweets
Saka
• 182160 Tweets
#الهلال_الاهلي
• 162981 Tweets
Macron
• 160883 Tweets
ناصر
• 61756 Tweets
Doom
• 42128 Tweets
Phillies
• 40950 Tweets
Salas
• 38545 Tweets
Missouri
• 30107 Tweets
Gordon
• 27851 Tweets
سالم
• 27773 Tweets
Valverde
• 27570 Tweets
#KadınaVeÇocuğaDokunma
• 27127 Tweets
Texas A&M
• 24330 Tweets
Everton
• 23999 Tweets
Celta
• 20678 Tweets
Mizzou
• 19208 Tweets
ياسر
• 17100 Tweets
جيسوس
• 14751 Tweets
الشوط الاول
• 13831 Tweets
مالكوم
• 12159 Tweets
الشوط الثاني
• 11924 Tweets
While preparing for the Databricks Certified Engineer Exam, I found it challenging to locate practice questions. Therefore, I created practice questions for my fellow professionals.
2
3
30
609 Data Engineer roles were hired this week in the USA. According to Glassdoor, the total annual pay a data engineer earns is $111,998, including base salary and additional pay like bonuses and profit sharing . The average total pay for a senior data engineer, meanwhile, is
5
3
21
Same Concept, Different Cloud Platforms
Big Data and Analytics
- AWS Redshift
- Azure SQL Data Warehouse
- Google BigQuery
Storage Services
- AWS S3
- Azure Blob Storage
- Google Cloud Storage
Database Services
- AWS RDS
- Azure SQL Database
- Google Cloud SQL
-
1
3
19
Databricks SQL is a warehouse engine packed with thousands of optimizations to provide you with the best performance for all your tools, query types and real-world applications.
0
6
20
@PsycheWizard
Certainly! The principles from *The Art of War* by Sun Tzu extend beyond the battlefield and can be applied to various aspects of daily life. Let's explore how you can incorporate these strategies:
1. **Know Yourself and Others**:
- **Self-awareness**: Understand your
0
7
18
Databricks announced Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines. It includes new native, highly scalable connectors for databases including MySQL, Postgres, SQL Server and Oracle and enterprise applications
3
4
16
Believe it or not, ChatGPT and Veed created this video in just a few seconds. My prompt: create a 30-second short video about the data market.
2
1
17
LinkedIn suggested some job openings for me this week. I'm not planning to leave my current company, but these look like solid opportunities for anyone searching for a job. Netflix is willing to pay $720k — that’s great.
1
2
17
My department is relocating. I will miss this view. Beautiful Michigan Lake.
2
1
17
Unlike front-end and back-end developers, data engineering tech interviews focus more on understanding technology rather than pure coding skills. Instead of spending too much time on coding challenges, prioritize building real-world data pipelines. Also, GCP tends to have a
4
6
20
The demand for data engineers in the U.S. job market is strong and continues to grow in 2024. Data engineering is one of the fastest-growing fields, with a projected job growth rate of 21% from 2018 to 2028, and salaries for data engineers have increased by 10% over the last five
1
5
16
Yang Kyoungjong, a Korean from Manchukuo, was conscripted into the Imperial Japanese Army before being captured by the Red Army at the Battle of Khalkhin Gol, Mongolia.
After imprisonment in a labor camp, he was forced to join the Red Army against the Germans, only to be
3
2
14
There are several strategies to optimize joins and minimize shuffling in Spark:
Broadcast Join: If one DataFrame is much smaller than the other, you can broadcast the smaller DataFrame to all nodes. This avoids shuffling the larger DataFrame because the smaller DataFrame is
3
2
13
Black Monday, on October 19, 1987, saw the Dow Jones drop 508 points (22.6%), the largest single-day percentage fall in history. Global losses hit $1.71 trillion, driven by automated trading systems that triggered sell-offs and panic.
2
4
15
Before implementing Delta Lake Time Travel, errors in the ingestion pipeline were costly for the company. Now, such mistakes are no longer a significant concern.
3
2
16
As a data engineer, my mornings start with checking the health of a variety of pipelines. If everything's green, it means I can relax for a bit, grab a coffee, and chat with coworkers. Green days are the best.
6
3
20
Databricks Delta Live Tables is a strong choice due to its data quality management, simplified ETL pipeline development, automated maintenance, and support for both batch and streaming data. It integrates seamlessly with Delta Lake, offering robust monitoring, version control,
2
0
11
Limits exist only in our minds, don’t they? I've got a burst of motivation!
0
2
13
In an analysis of over 40 million tweets, Americans were more likely than Canadians to use words like sh*t, b*tch, hate, and damn, while Canadians favored more argeeable words like thanks, great, good, and sure.
2
4
13
My 8-year-old daughter asked for a princess coloring book. Instead of buying one, I asked ChatGPT for help and printed out the pages. She really enjoyed it!
4
0
13
@SidNahataJain
Languages:
- Python
- PySpark
- Scala
- SQL
Cloud Technologies:
- Databricks
- Azure
- GCP
Additional Skills:
- Data Warehousing
- ETL Processes
- Data Modeling
- Version Control
- Containerization
2
1
12
The Databricks team is encouraging all certified engineers to share photos on social media using specific hashtags. In return, participants will receive a stylish custom jacket and other exciting swag. It's definitely worth it!
#dataAISummit
#LearningHub
0
0
12
As a data engineer, I'm happy as long as everything in the data pipeline is green.
1
2
11
If it's true, I have to respect the CIA's persistence. Personally, I'd give up after 100 attempts—or maybe after just reading the mission briefing.
The CIA unsuccessfully tried to assassinate the late Cuban dictator Fidel Castro over 634 times, including using an exploding cigar.
193
425
4K
4
0
10
1966: 5 megabytes of data required 62,500 punched cards, taking 4 days to load.
2024: 5 megabytes of data is equivalent to a single high-resolution photo or a small PDF document
6
1
10
$125,000 for an entry-level position? That's an incredible opportunity!
2
1
10
Invest only 30 minutes per day for two months to prepare for the exam and get certified. It's the best investment you can make in your lifetime. Source:
1
2
9
It’s shocking how much hatred and anger can reside in such a small brain.
3
1
9
Many of the most successful individuals in the United States are from minority communities. America is the land of opportunity!
0
0
7
Craft a Pandas User Defined Function in Python, and seamlessly execute it within an SQL query. It's a symphony of efficiency and simplicity in data processing.
#PandasUDF
#SQLIntegration
#PythonPower
#SparkSQL
0
2
9
This is one of the modern approaches to dimensional modeling.
0
1
7
I'm unsure if that quote is attributed to Confucius, but it resonates with me!
1
2
8
I noticed that many people are confused between Lakehouse and Delta Lake. I came across an image that perfectly illustrates the concept – proving that one picture is worth a thousand words.
0
1
8
He’s just made unity catalog open 😮 minutes ago. Don’t believe check it here.
1
1
8
After a long day, you deserve to relax with a video like this.
0
0
9
Here's my new weekend hobby! If you recognize the game, I'd love to hear your thoughts.
3
1
7
Pandas is an extremely useful library for analyzing small datasets, but for handling big data, memory becomes a crucial factor. In such cases, PySpark is the ideal tool.
0
2
7
This is my favorite scene from my all-time favorite movie. It makes me appreciate the beauty of love because I am a dad too.
2
0
8
The more vivid and detailed the image, the more memorable it will be.
0
3
5
According to the PYPL Index, Python was the most searched programming language on Google as of January 2024, with a share of 31.23% of all searches.
0
3
7
I am currently working with both Azure Synapse Analytics and Google BigQuery which are scalable data analytics services with key differences.
Azure Synapse Analytics:
Advantages: Seamless integration with Microsoft products, supports cloud and on-premises data, on-demand
0
3
6
An example solution for resolving schema detection issues when reading a CSV file as a delta table in Databricks using CTAS.
0
4
7
Databricks offers a pay-as-you-go approach with no up-front costs. Here are some of the pricing details:
Workflows & Streaming Jobs: Starting at $0.07 per Databricks Unit (DBU).
Delta Live Tables: Starting at $0.20 per DBU.
Databricks SQL: Starting at $0.22 per DBU.
All Purpose
0
1
6
Data engineer job growth summary. After extensive research, interviews, and analysis, Zippia's data science team found that:
* The projected data engineer job growth rate is 21% from 2018-2028.
* About 284,100 new jobs for data engineers are projected over the next decade.
*
1
1
5
More than twelve thousand data and AI professionals have gathered in San Francisco. We are anticipating significant announcements during the keynote presentations tomorrow.
#dataai
0
1
6
My company leverages Databricks Delta Live Tables, and when it operates smoothly, it delivers an exceptional experience. Nevertheless, when issues arise, troubleshooting becomes time-consuming due to the intricate workings beneath the hood.
0
1
5
In today’s data-driven world, the field of Data Engineering has emerged as a crucial pillar for organizations seeking to harness the full potential of their data.
0
1
5
I love this game.
So here's an overview of...
EVERY GAME currently playable or coming soon on
@Ronin_Network
(aka the no.1 blockchain in the world by DAU)
One to share with a friend who's new!
Timestamps and games below ⬇️
34
62
136
0
0
5
Transform your data management with my expert services! I offer a free Azure Databricks data pipeline setup and ongoing monitoring and optimization for just $100/month. Contact me to get started and ensure your data operations are efficient and reliable.
#DataManagement
0
0
6
Data engineering is a fast-growing and exciting field that combines software engineering, data science, and big data. If you want to learn more about data engineering, here are some of the best blogs and websites to follow in 2024.
0
1
5
Data engineering is crucial in a data-driven world, with data projected to hit 175ZB by 2025. Effective pipelines can boost revenue by 10%, and big data investments can increase profits by 8-10%.
0
0
4
The Art of War, attributed to the ancient Chinese military strategist Sun Tzu, offers timeless strategies for success in warfare and beyond. Here are some key principles from this influential work:
◦Know Your Enemy and Yourself:
◦Sun Tzu emphasizes the importance of
1
0
4
I borrowed a joke from a colleague for the stand-up meeting. It goes like this: 'Time flies like an arrow; fruit flies like a banana.
1
0
4
I learned the hard way that starting to build a pipeline without consulting solution architects leads to a challenging and painful migration later on because the lack of initial expert input often results in a design that does not align with best practices or technical standards.
0
0
5
India is know for famous CEOs.
Canada is known for Maple Syrup.
Australia is known for Kangaroos.
Switzerland is known for Chocolate.
🇮🇳 What is India known for?
2K
211
4K
2
0
4
Wow 😮
🚨 Over 1,800 people had shown up for an open interview for 10 vacancies advertised by an engineering company in Jhagadia, Gujarat. (📷-NDTV)
2K
8K
50K
3
2
4
This is incredible.
If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all
567
3K
13K
0
0
4
Unlike data lakes, which often lack governance, and data warehouses, which may not handle unstructured data well, lakehouses provide a unified platform for all types of data while maintaining governance and reliability.
2
1
3
A thousand doctors gathering for a thousand hours to exchange information can't create super doctors. However, thousands of AI doctors, trained by various companies, can integrate their knowledge in seconds. Our future looks bright with such advancements.
2
1
3
I am available daily after 4:30 pm and throughout the weekend to participate in data engineering projects.
0
0
4
@omoalhajaabiola
@norel_12
Data analytics has gained popularity due to its crucial role in decision-making and strategy formulation in today's data-driven world.
0
0
4
Target, one of the largest retail networks in America, has had an intriguing experience demonstrating the power of big data.
A confused father walked into one of their stores, frustrated and baffled. His complaint? His 15-year-old daughter had been receiving mail advertising baby
1
1
3
We are enjoying Delta Lake's fun features like Deletion Vectors, Liquid Clustering, and Auto-Compaction. And yet, Delta 4.0 was just announced less than a week ago. Only UniForm GA makes your life way easier.
0
0
3
Coding languages are like human languages: the more you know, the more you can create! What's your go-to programming language when you want to whip up a quick project and why? Share your dev dialect!
1
0
2
Be careful when using Databricks Auto Loader – its ease and efficiency are so addictive, you might never want to use anything else!
0
2
4
Ask a programmer about their love life, and they might just tell you it's like searching for a bug in a 5000-line code—complicated but worth it when everything finally runs smoothly! 😄❤️💻
#CodingIsFun
0
0
3
I asked ChatGPT, 'The world's data is expected to grow 61 percent to 175 zettabytes by 2025. Can you explain how much data that is in terms understandable to non-tech people?
0
1
3
Heads or Tails: Which side does your code land on when fixing a bug – the elegant fix or the code-that-shall-not-be-named? Share your victory or facepalm moments!
#CodingIsFun
0
0
2
Debugging: the process where you become a detective, a scientist, and a frantic programmer all at once searching for an elusive bug. Who else loves the thrill of the hunt? Share your most epic bug battle stories!
#CodingIsFun
0
0
3
Last year, several thousand data and AI professionals gathered in San Francisco, where many shared that, thanks to ChatGPT, the traditional 8-hour workday is no longer a necessity. I've experienced this too, often working just 5 to 6 hours a day. ChatGPT has truly become a
0
1
3
Can you believe that just a year ago, we questioned whether a machine could be this creative?
#ImageByAI
0
1
3
Many companies prefer hiring experienced professionals. For entry-level or junior engineers without job experience, I recommend engaging in freelancing platforms such as Fiverr and Upwork while searching for full-time roles. This approach offers two major advantages: gaining
0
1
3
With just a few lines of Python code, you can achieve full automation of Google Sheets. Feel free to clone my public GitHub repository.
0
0
3
Labor Day is often considered the unofficial end of summer in the U.S., marked by parades, barbecues, and the last weekend for outdoor pools.
0
0
3
I come across this video several times before. It always makes me laugh.
0
0
3
Daily Tech News: This Smart Mirror Scans Your Face to Evaluate Your Vital Signs,
0
1
1