We just sent notice we are terminating service for 8chan. There comes a time when enough is enough. But this isn't the end. We need to have a broader conversation about addressing the root causes of hate online.
We just blocked Kiwifarms. The threats on the site escalated enough in the last 48 hours that, in spite of proactively working with law enforcement, it became enough of an imminent emergency we could no longer wait for them to act. Details of our decision:
Not all records you’re happy about breaking:
@Cloudflare
recently mitigated the largest ever reported hyper-volumetric
#DDoS
attack. 3.8 terabits per second (Tbps) and 2.14 billion packets per second (Bpps). Handled automatically any without any customer impact. Details to come.
There’s a lot of buzz right now about a “massive DDoS attack” targeting the US, complete with scary-looking graphs (see Tweet below). While it makes for a good headline in these already dramatic times, it’s not accurate. The reality is far more boring. 1/X
I know this story. It’s very bad. Don’t know if David is most evil person in SV. Lots of competition. But can say: we overlapped a year in law school and even way back then he had a reputation for being a complete asshole. And that was at
@UChicagoLaw
, which is full of assholes.
@DavidSacks
Do you really want the full story of what you did to Parker to be told publicly? Because it's the worst case of an investor maltreating a founder that I've ever heard, and I've heard practically all of them.
One of the smartest decisions we made at
@Cloudflare
was recognizing that the primary purpose of our blog was attracting employees, not attracting customers.
I’m told mine is a contrarian view on the events of the last few days, so here goes…
Contrary to what
@kevinroose
and others have written, Microsoft was not a winner of the events of the last few days around
#OpenAI
. They were in a much better place on Friday morning last week
@paulg
Oh Paul, you know better. I had to borrow money from my mom to pay my taxes when we were starting Cloudflare. But I certainly came from a relatively privileged background, and so did the AirBnB founders. It’s hard to take risks if you don’t have a safety net.
#bereal
Trying to come up with analogy to explain
#xzbackdoor
to my mom. Best I’ve come up with so far:
Imagine a grizzled, tired, old oil pipeline maintainer was approached by a plucky young kid who was new to the small town the oil pipeline passed through. Slowly, over years, the kid
@Austen
We fired ~40 sales people out of over 1,500 in our go to market org. That’s a normal quarter. When we’re doing performance management right, we can often tell within 3 months or less of a sales hire, even during the holidays, whether they’re going to be successful or not. Sadly,
@parastang95
Mistakes happen. The root problem was we didn’t have systems in place to keep them from causing a widespread issue. That’s a problem of leadership that I am more responsible for than the engineer who made the typo.
Details on how we caused an 23 min outage for~50% of
@Cloudflare
's network today. The root cause was a typo in a router configuration on our private backbone. We've applied safeguards to ensure a mistake like this will not cause problems in the future.
Just sent the last
@Cloudflare
employment offer of 2021. I still personally send all the offers out because nothing is more important than hiring. Some numbers: We received more than 200,000 applications. We extended 1,455 offers. And we had a 92% offer acceptance rate.
Nothing we're seeing related to the Facebook services outage suggests it was an attack. Most likely explanation is that the company's Internet routes (BGP) were withdrawn by mistake during maintenance.
#hugops
We’ve made the determination that
#Log4J
is so bad we’re going to try and roll out at least some protection for all
@Cloudflare
customers by default, even free customers who do not have our WAF. Working on how to do that safely now.
Except T-Mobile, which is having a bad day almost certainly entirely of their own team’s making. So, please,
#hugops
. And don’t worry, this is one thing that does not need to get added to the list of craziness that has been 2020. 8/8
Proud of our whole team for creating 1.1.1.1, the Internet’s fastest, privacy-first DNS resolver. It’s
@Cloudflare
’s first consumer product. And, if you’re wondering whose dopey idea it was to launch on April Fools (and Easter), look no further than me.
When I read this story by
@troyhunt
about how
@Cloudflare
and
@Azure
misfired to cause a huge bill I felt terrible. I reached out to
@scottgu
and proposed we split Troy's cost. Scott immediately agreed. Great to support our mutual customers!
It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly. The result has been for around the last 6 hours a series of cascading failures for their users, impacting both their voice and data networks. 2/X
Shhh… 🤫
@Cloudflare
Registrar just quietly rolled out support for the following TLDs:
.app
.boo
.channel
.dad
.day
.dev
.esq
.foo
.how
.mov
.new
.nexus
.page
.phd
.prof
.rsvp
.soy
At
@Cloudflare
, we understand the Russian cyber attack capabilities and stand prepared to defend our clients against any cyber retaliation that results from global sanctions.
The collective hallucination that this is about “taxes” rather than finding somewhere talented junior engineers can afford their own apartment and senior engineering managers can afford a nice house—because sensible housing policies—will be what kills SF.
Earliest evidence we’ve found so far of
#Log4J
exploit is 2021-12-01 04:36:50 UTC. That suggests it was in the wild at least 9 days before publicly disclosed. However, don’t see evidence of mass exploitation until after public disclosure.
So now people are looking around for an explanation and they stumble across sites like the Arbor Networks attack map. It looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services so that’s not surprising. 4/X
We are aware that
@Okta
may have been compromised. There is no evidence that Cloudflare has been compromised. Okta is merely an identity provider for Cloudflare. Thankfully, we have multiple layers of security beyond Okta, and would never consider them to be a standalone option.
@antoniogm
Had a sales guy who lived in San Jose. He’d commute to SF every day. At about Menlo Park he’d turn on his Uber app as a driver. 95% of days picked up someone going to the airport or downtown SF. Always senior tech execs. Filled 80%+ of his pipeline from those rides. Genius.
Okta compromised… again. Here’s how
@Cloudflare
, even though we were (again) targeted, was able to mitigate the attack. And some best security practice suggestions for
@okta
and their customers.
Just sent out the last
@Cloudflare
employment offers of 2022. Received roughly 400,000 applications, up 49% over 2021. Of those, we interviewed 15,805 candidates to ultimately hire 1,418. 37% of the hires were designated fully remote, up from 14% in 2021. (1/2)
Dear
@antoniocostapm
, when your team promises to work with companies in exchange for moving significant jobs to Portugal and shows laws to support those promises; then your bureaucrats refuse to follow those laws and promises — is that ok? I feel lied to. Cc:
@paddycosgrave
Just published an initial post mortem on the incident that impacted many
@Cloudflare
customers use of our dashboard and APIs. Lots of lessons.
#CodeOrange
Wow!
@Cloudflare
's 1.1.1.1 () has now passed handling more than 1 trillion requests per day. Not bad for a project we launched on April Fools Day a little over four years ago.
Lots of reports of Russian censorship of Western media. We are seeing evidence of that. But, generally, consumption of Western media in Russia is up more than 3x in the last month — in spite of censorship.
#truthfindsaway
Seeing a marked increase in cyberattacks this evening. Combined with the deeply disturbing headlines, fear the world just turned up the crazy dial another notch. We’re ready online at
@Cloudflare
. But… worried for the world.
From
@Cloudflare
’s vantage point, we can see a number of things that show there is no massive DDoS attack. First, traffic from WARP to supposedly impacted services is normal and has no increase in errors. 5/X
When
@Cloudflare
started we didn't want to build a DNS service, so we reached out to Dyn & UltraDNS about becoming a customer. Both blew us off because we were "too small." So we built our own. Good lesson about serving all customers, no matter how small.
We are resetting the
@Okta
credentials of any employees who’ve changed their passwords in the last 4 months, out of abundance of caution. We’ve confirmed no compromise. Okta is one layer of security. Given they may have an issue we’re evaluating alternatives for that layer.
That caused a lot of T-Mobile users to complain on Twitter and other forums that they weren’t able to reach popular services. Then services like Down Detector scraps Twitter and report those services as being offline. 3/X
This is the sad story about what happened to my friend and
@Cloudflare
’s brilliant third cofounder Lee Holloway (
@icqheretic
). We’d never have been able to pull off what we did without him on the team. I miss him every day.
Just sent the last
@Cloudflare
offer of 2023. What an incredible year!
* 1,162,526 applicants
* Offer extended to less than 0.1%
* 90%+ offer acceptance rate
Companies are just collections of people; our team is incredible and continues to get better and better! Happy New Year!
Second, there is no spike in traffic to any of the major Internet Exchanges, which you do see during actual DDoS attacks and definitely would during one allegedly this disruptive. 6/X
This is very insightful of
@matt_levine
. And kind of think it’s a skill
@sama
learned from
@elonmusk
, as well as watching what worked bizarrely well with
@ycombinator
companies. Fascinating to watch.
Remember reading when Google's 8.8.8.8 crossed through over 1.2 trillion requests per day back in 2018 and thinking: that's unfathomable. Well, 1.1.1.1 is now handling more than 1.3 trillion requests per day. Still feels unfathomable.
As a precaution, we’ve removed all
@Cloudflare
customer cryptographic material from servers in Ukraine. We continue to serve traffic there for Ukrainian users, for now, via our
#Keyless
technology.
Finally, our team know the network operators at nearly all the other major Internet services and platforms and none of them are reporting anything anomalous. 7/X
In other words: sometimes owning a call option on an asset is better for multiple reasons than owning the asset itself. Last week Microsoft roughly owned a call option on OpenAI. Today, at best, they own some fraction of the asset itself.
We had an issue that impacted some portions of the
@Cloudflare
network. It appears that a router in Atlanta had an error that caused bad routes across our backbone. That resulted in misrouted traffic to PoPs that connect to our backbone. 1/2
Here’s the scary thing that’s likely to happen based on the facts of the day if we don’t pay attention. Microsoft, who competes with
@CrowdStrike
, will argue that they should lock all third-party security vendors out of their OS. “It’s the only way we can be safe,” they’ll
The Juniper Networks news reminded me that when
@Zatlyn
and I were first starting
@Cloudflare
my initial prediction on our “exit” was selling to them for $250M. Michelle disagreed. She turned out to be a much better predictor of the future.
Here's what went wrong on the Internet earlier today causing
@Cloudflare
and several other networks to be unreachable for many users. It's time for providers like
@Verizon
to be held responsible for not filtering BGP routes and implementing RPKI.
The teams at
@verizon
and
@noction
should be incredibly embarrassed at their failings this morning which impacted
@Cloudflare
and other large chunks of the Internet. It’s absurd BGP is so fragile. It’s more absurd Verizon would blindly accept routes without basic filters.
The first five
@Cloudflare
employees —
@zatlyn
,
@icqheretic
,
@IanPye
,
@mtourne
, and I — officially started on January 4, 2010. We were above a nail salon in Palo Alto at 542 Emerson. Our first task was to assemble the BBQ for the deck. It's been quite a decade.
We’ve seen reports of service outages across the Internet. Confirmed
@Cloudflare
’s services all operational. No uptick in attacks. We are seeing local drops in traffic from some upstream providers. Not yet clear if they’re related or not. All indications: not a Cloudflare issue.
Everyone has a bad day. This one really sucked for
@CrowdStrike
. Continue to have faith in them as a partner and the best end point security solution on the market.
#HugOps
Every ~8 years there’s a 10x improvement in how computing is provisioned. Bare metal > VMs > Managed VMs/containers. We’re due for the next step-function improvement and at
@Cloudflare
we’re convinced it’ll be built around a technology called Isolates:
Wow!
@Cloudflare
's 1.1.1.1 just crossed handling 300 billion queries per day. Still well shy of the trillion+ that Google's 8.8.8.8 handles, but growing fast!
The theme of the age of AI:
The demo is extremely impressive and relatively easy.
The product is really, really, really hard.
It is, perhaps, the perfect VC trap.
@_FiveM
@Cloudflare
Completely unacceptable. I’ve ordered the account be restored. Called on team to investigate why an automated system took such draconian action without any warning. It may be you’re doing something that breaks plan limits, but we need to have more nuanced solutions. Apologies.
AWS’s bandwidth charges are egregious. Their wholesale cost in AWS-East is likely less than $200/Gbps/mo. That equates to a 10,000%+ markup.
#nevertrustamazon
My take on this is a bit different than others: it’s really hard to make money as a dev tools company unless you find a way to sell storage, compute, and bandwidth. So clearly they needed to build these hooks.
Tens of millions of websites (4% of the web) uses Polyfill(.)io. Extremely concerning malware has been discovered impacting any site using Polyfill. Cloudflare is stepping in with a secure clone and a service to automatically replace Polyfill on pages.
Proud of the
@Cloudflare
team’s work defeating yet another patent troll. As part of
#ProjectJengo
we took this one all the way to trial and not only prevailed for ourselves but got the troll’s patent cancelled. This is a win not just for us but for the industry as a whole.
The entire problem with Carta is they’re the Facebook of B2B. From day one the whole premise was “if we hoover up private company’s cap table information we can eventually build the world’s biggest secondary market.” That’s the only way to justify their multiple & valuation. 1/2
Years ago, I remember reading that Google's 8.8.8.8 handled 2 trillion queries per day and being blown away. Today, just 3 1/2 years after launch, 1.1.1.1 is a quarter of the way to that same milestone. And >15% of its queries are encrypted!
#progress
@Sativa888
Pretty sure a major DDoS attack would be amazing for us given we’re one of the only companies that could stop it.
Good lesson: when you hear hoof beats, think 🐎 not 🦓.
Incredible that
@Cloudflare
stopped the largest DDoS attack in history and it was just another day at the office. I wasn't even aware of the scale until I read this post.
All threat research groups have cool names. Most of them are full of BS.
@Cloudflare
is planning to launch a threat research group that's no full of BS. But still need a cool name. Any suggestions? If we choose your suggestion I'll make sure you get lots of branded swag.
In 2012 I went to Portland to meet with Intel’s engineering team. Cloudflare was tiny, but didn’t stop me from banging the table saying cores per watt was what we cared most about. They told me we were “doing it wrong.” Glad to see the data (still) back me up (and AMD listened).
Proud of the role
@Cloudflare
played in ensuring the Internet stayed online in 2020. We stand ready for whatever 2021 brings (but, for the record, will be totally cool if it’s less eventful).
Thrilled to partner with NVIDIA to bring AI to the edge!!
@Cloudflare
Workers is the largest, fastest, most used edge computing working. With NVIDIA's hardware running at our edge we open a whole new class of applications for developers.
#DeveloperWeek
Excited to work with
@PalantirTech
to help customers understand what they're spending on the cloud and how they can optimize those costs using
@Cloudflare
Workers.