It's about time for a new and improved
@ApacheAirflow
UI.
Take a sneak peek at our very own
@ryanahamilton
's PR, which includes a refresh of existing colors, icons, typography and top-level navigation.
Coming in Airflow 2.0. You won't want to miss it 🚀
"Astronomer: The Airflow Company." It's got a good ring to it.
Our team could not be more excited for this new era, as we rally around 1 thing: to help organizations adopt
@ApacheAirflow
.
Enormous thank you to everyone who helped us get here 🚀
Today, we are *thrilled* to announce
@marclambertiml
's much-awaited Airflow 2.0 YouTube series 🎉
Marc has prepared 6 engaging (read: addicting) walk-throughs of the biggest features in
@ApacheAirflow
2.0.
Trust us, you won't want to miss these. 🎬👉
Today is the day.
@ApacheAirflow
2.0 is here!! 🎉🎉
This historic open-source release comes with a flurry of significant features that we can't wait for you to try.
Thank you to the hundreds of you who contributed. We couldn't have done it without you.
Introducing KEDA (Kubernetes-based Event Driven Autoscaler) for
@ApacheAirflow
. Our team has been working tirelessly to get this into the hands of our users and it's almost here! Learn more about it 👇
Today, we’re excited to launch the Astronomer Registry: a discovery and distribution hub for
@ApacheAirflow
integrations. Read our announcement post below!
#ApacheAirflow
#DAGs
Airflow 2.0 is almost here.
An HA Scheduler. A new REST API. Smart Sensors. Task Groups. A modern UI... and so much more.
To those contributing - thank you. This is a huge milestone for
@ApacheAirflow
and we could not be more excited.
Stay tuned. 🤓🚀
Looking for some holiday reading? Fan of
@apacheairflow
and
@getdbt
? Do we have something for you...
We teamed up with our friends
@updater
to write a series on building an Airflow + dbt architecture that is just...:chef's kiss:
Part 1 out now 👇
Astronomer is making news! Read this morning’s
@TechCrunch
report on our accelerating growth—including the news of our just-completed, $213 million Series C round and the acquisition of Datakin, the data lineage tool.
#company
The Scheduler we contributed to
@ApacheAirflow
2.0 is faster than ever, highly-available, and ready for scale.
Check out our latest blog post for official performance benchmarks and a full, technical deep-dive into Airflow's critical component. 🚄🔥🤓
➡️
Tasks not scheduling? DAG stuck? Webserver not spinning up?
Ah, DAG troubleshooting woes on
@ApacheAirflow
. We know the feeling. Check out our newest blog post to help you in hard times 😉
📣 If you're an
@ApacheAirflow
expert, would you want a badge to show for it?
Introducing Apache Airflow Core Certification, a course + exam created by our very own
@marclambertiml
that covers concepts, architecture & use cases.🎓
Pre-register today 👉
Astronomer has acquired Datakin, the data lineage tool from founders of the
@OpenLineage
and
@MarquezProject
.
The move accelerates our shared goal of helping organizations build and manage reliable ecosystems that deliver trusted data.
#blog
When you bake the boss pies for Pi Day but he's too busy taking your
@ProductHunt
launch seriously to care.
PS Check out our managed
@ApacheAirflow
offering on Product Hunt today!
Executing SQL is one of the most popular use cases for data pipelines - and we're pumped to have a fresh new guide on how to do so with
@ApacheAirflow
.
Best practices, a run-through of existing operators, and example DAGs all included.👩🏫
Check it out 👇
🥁And now for the moment you've all been waiting for...🥁
Pt 2 of our
@ApacheAirflow
and
@getdbt
series is live, written in collaboration with our friends
@updater
.
Check it out below 👇
The PR has been merged!
Huge nod to
@AshBerlin
+
@kaxil
for working tirelessly to introduce the ability to run multiple schedulers on
@ApacheAirflow
as part of the momentous 2.0 release coming soon.
Stay tuned for performance benchmarks that speak for themselves. 😉🔥🚄
It's done. After almost 4 months of work, I've finally merged to add the ability to run more than one scheduler concurrently to
@ApacheAirflow
2.0, not to mention a serious boost to performance to boot!
@astronomerio
is commited to open source!
Next Generation Astronomer Cloud is here. Built on new infra with our latest flagship features, the release includes:
-
@ApacheAirflow
1.10.5
- KubernetesExecutor
- Log Streaming
- RBAC
- Metrics interface
Check out our blog post for more info ⤵️
Today is the day we give
@marclambertiml
the warmest of welcomes to the
@astronomerio
team!
He's been running the prodigious
@udemy
course on
@ApacheAirflow
and will help us build out a larger collection of educational content for Airflow users everywhere. 🧑🏫
We are excited.
Looking for some good
@ApacheAirflow
content to sink your teeth into this weekend? Tired of manually replicating connections for each one of your Airflow environments? Check out our new guide on pulling secrets from
@hashicorp
Vault in Airflow! ⤵️
In Airflow 2.0+, Airflow Providers are released and versioned independently of the core project.
To prepare for the new world, we worked w/ the team
@expectgreatdata
to build an official Great Expectations Provider for
@ApacheAirflow
.💥😎
Check it out ➡️
Check out the library of
@ApacheAirflow
plugins that our team maintains with ❤️! We're proud to be the Airflow company and can't wait to see what we can do for the future of
#dataengineering
.
Yes! Thanks for your hard work,
@casassaez
. We're pumped about the TaskFlow API and Task Decorator coming in Airflow 2.0.
For those following, this feature builds on XComs to give users an easier way to pass info between tasks.
It also makes for much easier DAG writing. ⌨️😏
Second large contribution (3rd PR) to
@ApacheAirflow
merged:
New decorator that allows you to generate DAGs by wrapping a function with operators. -
Coming soon in Airflow 2.0
Today is a great day. Astronomer v0.9 is here 💥
- RBAC integration with
@ApacheAirflow
- A new "Metrics" tab in the Astronomer UI
- Initial support for the
@kubernetesio
Executor
-
@okta
Auth Integration
- Streamed Logs to the CLI
Full release notes 👉
New episode of the
@ApacheAirflow
Podcast hot off the press!
This week, we linked up with Daniel Imberman, Senior Software Engineer at Bloomberg, and
@schniebot
, Astronomer CTO, to discuss running Airflow on
@kubernetesio
. Hope you enjoy!
🚨🎙️ The latest episode of The Airflow Podcast is here. It's a good one.
Tune in to hear Data Engineers from
@Netlify
talk about adopting
@ApacheAirflow
to manage high-volume application + marketing data. Thanks, Brian and Florian.
Check it out! 👉
🚨Season 2 of The
@ApacheAirflow
Podcast is here!🚨
To kick things off, we linked up with
@chimerasaurus
to talk open source and Airflow at Google. Give it a listen at the link below or in your Podcast app of choice ⤵️ 🚀
Another episode of the Airflow Podcast hot off the press, folks!🎙️
Tune in to hear
@higrys
from
@polidea
talk about creating Airflow Breeze, a tool built to lower
@ApacheAirflow
's barrier-to-entry for anyone interested in contributing to the project.
🎙️The latest episode of the Airflow Podcast is out today.
Tune in to hear our conversation with Ben Wisegarver, Data Scientist
@reddit
, talk about how they use
@ApacheAirflow
to leverage the petabytes of data users generate every single day.
Enjoy! 🙂⤵️
After 4 years of keeping a low profile, Astronomer is making some noise.
Read our CEO Joe Otto's personal take on the journey that got us where we are today, with our acquisition of Datakin and a completed round of Series C funding.
#blog
Improving Airflow’s scheduler is one of our top priorities at Astronomer.
In our most recent blog post,
@ApacheAirflow
committer
@AshBerlin
shows us how he’s been testing and profiling the scheduler in preparation for Airflow 2.0🚀
Check it out here👇
We're proud to be a data bridge between Airflow users and their critical data sources.
Thanks to
@astronomerio
for making us a part of the partner registry!
Excited and humbled to be on
@Accel
's
#Open100
! Love to see these teams each contributing expertise to open-source communities of end-users and developers alike.
To those following
@ApacheAirflow
, stay tuned 🤓
Today is a great day. Astronomer v0.10 is here 💥
- Support for
@ApacheAirflow
v1.10.4
- System Admin Dashboard
- Service Account Improvements
- UI Improvements
- Security Improvements
Full release notes 👉
If you're running a data pipeline and the data source isn't up to date, would you want to be notified?
Check out
@BriskiBob
from our partners at
@RaybeamInc
write about the status plugin his team contributed to
@ApacheAirflow
to solve that very problem 👉
🎙️ The latest episode of the
@ApacheAirflow
podcast covering GDPR, self-service data, and infrastructure automation with the
@typeform
team is here! 🎉🚀
Check it out wherever you get your podcasts ⤵️
Interested in doing
#MachineLearning
with
@ApacheAirflow
? One of our own Airflow core committers
@danimberman
gave a 💯 talk on Bridging Data Science and Data Infrastructure with Airflow at PyCon2020! See below for link ↘️
We've written and helped troubleshoot hundreds of data pipelines on
@ApacheAirflow
(read: DAGs). It can often feel like more of an art than a science. 🧪
To hear our tips & tricks, join us on Monday, Feb 22 for a webinar on DAG-writing best practices. 🤓
Exciting news! The
@ApacheAirflow
Community just released
#Airflow
2.4. Find out how the new data-dependent scheduling logic enables faster data delivery, how you can leverage the new
#Dataset
class, and more!
This is a super insightful overview of
@ApacheAirflow
at
@lyfteng
by
@photoft45
, who helps manage Lyft's 600+ daily DAGs (!).
Check it out for pro tips on backfills, upstream gotchas, and new features he contributed in 1.10.3 👏
We’ve just released Apache Airflow 2.2.5 🎉
📦 PyPI:
📚 Docs:
🛠 Changelog:
🐳 Docker Image: "docker pull apache/airflow:2.2.5"
Thanks to all the contributors who made this possible.
Logging within
@ApacheAirflow
can be mystifying, and we finally have a guide for it.
If you're wondering how to set up custom logging and want to better understand its moving pieces, check this out 👇
Airflow 1.10.14 is out and available now on Astronomer 😎
For those early awaiting for
@ApacheAirflow
2.0, get a head start by upgrading to 1.10.14 and moving to Python 3 if you haven't already.
More to come next week! Stay with us. 😏🥁
In the past few months, other workflow management tools have raised the question, "Why should I be using Airflow over other schedulers?"
Our co-founder and Head of Product
@rywalker
wants to remind us of what makes
@ApacheAirflow
so great...
The Ultimate Guide to Getting Started with
#ApacheAirflow
is here! Get it today to:
- Discover premium pieces written by our Airflow practitioners
- Get the roadmap of go-to guides, docs and videos
- Step up you data orchestration game
Learn more: