parithosh | 🐼👉👈🐼
@parithosh_j
Followers
9K
Following
18K
Media
172
Statuses
3K
@ethPandaOps at the Ethereum Foundation ENS: parithosh.eth Website: https://t.co/iujOhlI6I4 Mastodon: @[email protected]
Berlin
Joined June 2009
The goerli fork finalized! :D. After the fix was patched in, the validators came back online and the chain started finalizing again. The MEV circuitbreaker automatically disables and mev-blocks have started flowing through as well. Yay client diversity!.
TLDR on what Historical roots are :D.
294
463
646
The Goerli testnet will merge in the next ~2 days, this will be the last testnet to merge before mainnet. If you are running a Goerli node, please update and follow the numerous guides to ensure we can have a smooth merge. More information here:
155
268
505
Mainnet-Shadow-Fork-4 looks perfect! All clients went through the transition without a hitch. Only erigon missing (unrelated sync issues, we'll know in a bit if its synced to head as expected). We finalized and just keep chugging!.#TestingTheMerge.
112
175
495
We shadow forked mainnet successfully a second time! .#TestingTheMerge.
18
74
475
After 4 days of PoW, merge-devnet-5 hit TTD today. The chain continued finalizing post TTD, as expected. This means the merge was successful. We ran a few zombie miners. Miners ran older versions that don't know about TTD and just continue mining post TTD. No issues were found.
merge-devnet-5 has been launched. The beaconchain genesis happens at 4PM UTC and the PoW chain has already been launched. The TTD has been set to be really high to allow everyone to join pre-merge. Tooling can be found here:
8
88
414
The merge pandas have arrived!.mainnet-shadow-fork-1 hit TTD ~half an hour ago. We've been finalizing and producing blocks! . We quickly noticed some seemingly minor issues with Nethermind and Besu(triage ongoing). Erigon is currently syncing to head, status update later.
Wondering what the #TestingTheMerge team has been up to? . The aim of the Kiln merge testnet was to allow the community to practice running their nodes, deploying contracts, testing infrastructure, etc. We hope it's helping the community get a sense of the post merge world!.
28
103
383
Ropsten beacon chain is live! . We've finalized as expected and the chain is chugging along till the un-announced TTD. Users can join the chain and practice running their nodes! .Initial docs: If you make docs/vids/etc do tag me and #TestingTheMerge
17
76
371
Blobs are now flowing in Sepolia :D. Uneventful testnet forks are the best ones!
Everything looking smooth so far 😄.
13
51
293
Shapella is live and finalized on Sepolia #SepoliaShapella. Next would be Goerli, then Mainnet🦉
9
88
277
The configs for the Kiln testnet are out!. You can find all the information you need here: Help us with #TestingTheMerge !.
11
73
272
Wondering what the #TestingTheMerge team has been up to? . The aim of the Kiln merge testnet was to allow the community to practice running their nodes, deploying contracts, testing infrastructure, etc. We hope it's helping the community get a sense of the post merge world!.
11
66
244
We managed to get a testnet up and running with a total of 12 combinations of execution and consensus clients! Expect some refining and many more testnets till the merge! - wagmi. Huge props to @protolambda and all the client teams for their efforts this week :D #Ethereum
7
39
230
This was the first shadow fork where every client combination survived the transition and managed to stay in sync afterwards. We're making real progress here!
5
29
228
Sepolia merged successfully! . After some config issues, the chain is finalizing. There were no client related issues. 1 EF node had a hiccup during the transition, but it self healed without intervention. Of a possible ~95% participation rate, we are generally seeing ~95%.
I wanted to share some thoughts on the events of the Sepolia merge. I speak for myself, not necessarily the rest of @sigp_io. Needless to say, it was less than ideal and we will be extra vigilant moving forward. I've also seen some understandable concerns from the community:. 1/6.
2
32
182
The new merge testnet Pithos is now live! . Explorer: Instructions to join:
49
98
169
Goerli has successfully upgraded to Shapella! 🦉. We had some validators running older versions of clients, once that was fixed the attestation rate went back up and we are currently finalizing! .#GoerliShapella. Mainnet comes next!.
4
38
183
Blobs are live on Ethereum Mainnet!. The Dencun fork finalized without issues and the network has been running stably since then. We can see healthy blob propagation statistics (expect a deeper analysis over the next weeks). Congrats to all the client teams and contributors!
7
22
180
I made my first post about a potential testing path to figure out gas limit increases: We should be doing this systematically and figure out our constraints, current limits and projected limits before committing to an increase.
8
30
167
As we move closer to shipping Dencun on Ethereum, let's have a look at the latest test results:. Goerli-shadowfork-1 went live recently. Its a 300 node shadowfork (largest public one to date) with almost every client combo as they are on mainnet.
9
40
160
New #TestingTheMerge milestone!. The Kiln testnet is now processing more transactions than mainnet (for the last days)!
7
28
149
Let's bring @EFDevcon to Namma Bengaluru, India ! :D. The proposal for Devcon VII is now live! Check it out and share it if you want to enjoy DevCon with a Dosa and amazing weather!. @devfolio @ETHIndiaco.
14
28
153
Holesky is finalized for Deneb :D. Churn limit looks good so far and blobs are flowing smoothly! . Next stop, Mainnet!.
We have blobs on Holesky!. Another un-eventful testnet fork for the books :)
4
25
151
After the insane chaos spread by the #Kintsugi testnet fuzzer, the testnet finalizes again!
Another day, another bug! All hail the #Kintsugi testnet fuzzer. Thread. So as you might remember from my last thread, the fuzzer created a block which uncovered two separate issue in the besu and nethermind code for verifying payload as well as the optimistic sync code.
2
24
135
Time for devnet updates again! :D. We had a sepolia shadowfork this week, this shadowfork successfully added EIP-4844 support to sepolia with a subset of clients!
11
107
129
We're going to monitor the chain over the next few weeks to make sure no client pairs go out of sync. We'd hope that community members raise alarms if they see dapps/tooling not work as expected! Sync nodes in weird ways to find bugs! Send txs! . Help with #TestingTheMerge.
3
7
130
Top to bottom: Geth, Nethermind,Besu(with lighthouse,teku).Left to right: Prysm,Lighthouse, Teku, Nimbus,Lodestar. Join us on Kintsugi: #TestingTheMerge
10
18
131
Not only is Ropsten the first existing testnet to get merged, its also the first testnet where every client team is running an equal part of the network. That's at least a dozen unique ways of setting up a node!. A hearty congratulations to every client team on a great merge! :D.
2
4
125
Withdrowls are live and finalizing! Congrats to everyone on a successful upgrade! :D.#ShapellaUpgrade
6
13
127
Goerli forked and blobs are now enabled on the testnet! . We're debugging a client issue right now, we should have more info in a bit. Expect to hear more later today and a deeper dive at ACD tomorrow!.
Goerli forked 🥳 @parithosh_j currently monitoring things live:
5
24
122
The Holešovice (Holesky) testnet is planned for launch roughly on Sept 15th(Merge day!). We've had 2 coordination calls and the testnet aims to give client teams a platform to test clients at a scale larger than mainnet and for home stakers to test their setups without risks. 1/n.
9
33
115
The Shanghai+Capella upgrade is going full steam ahead! The first public withdrawals testnet is launching 1st of Feb at 15 UTC. All info here: #TestingTheWithdrurge.
The Zhejiang public testnet is going live tomorrow (1st of Feb 15:00 UTC, 2023). Shanghai+Capella will be triggered 6 days later (at epoch 1350). You will be able to deposit validators, practice BLS change and exit without risk. All links are here:
8
28
110
After the fixes were in place, we quickly saw the participation and proposal rates go back up towards ~99%! About what we had pre-merge!
2
3
102
merge-devnet-5 has been launched. The beaconchain genesis happens at 4PM UTC and the PoW chain has already been launched. The TTD has been set to be really high to allow everyone to join pre-merge. Tooling can be found here:
1
21
99
Best part of the merge being done - deprecating and deleting old configs/infra
5
0
93
The Ropsten testnet hit TTD at ~16:00 UTC (~2h after the planned time, great accuracy for PoW standards!). The chain was at ~99.2% participation rate with healthy proposals and sync committee performance - Pretty much the perfect beacon chain to merge, easy to see what breaks!.
1
2
92
All L2s using Goerli can start testing EIP-4844 now, Looking forward to seeing the process unfold!.
Interested in estimated timelines of all L2s here. EIP-4844 is hitting L1 testnets soon, and more different L2 testruns are needed. What would be a safe timeframe between testnets and mainnet L1 4844?.
2
21
94
wen merge? .
2
2
93
merge-devnet-4 is live! We're still working on getting the tooling links up, but the configs can be found here:
4
29
98
The Ethereum mainnet fork is scheduled for 22:27:35 UTC on Apr. 12, 2023!. Don't forget to update all of your nodes! This includes the EL (geth/nether./erigon/besu. ), CL beacon(lighth./nimbus/prysm/lodestar/teku),CL validator and mev-boost!. Update the host OS while you're at it.
It's happening 🎊 . Shapella is scheduled on mainnet for epoch 194048, scheduled for 22:27:35 UTC on Apr. 12, 2023 📆. Client releases compatible with the upgrade are listed in the announcement below 👇.
2
36
88
Does this mean there were no bugs? Almost. While we have proposals (vital for consensus), some client pairs timeout while building a block and propose with 0 transactions in them. This would affect the throughput of the chain. We saw this in shadow forks too! Fixes incoming!.
2
3
87
Announcing EIP-7665, Adding the EXTSELFDESTRUCT opcode to Ethereum. This would help the fight against state growth and allow us to perform the verkle transition in a matter of minutes. @lightclients and I are hoping that the EIP gets included for Electra.
11
15
89
Time flies quickly in the crypto sphere. A year ago today @protolambda showed me how to setup my first consensus layer testnet, now we've even got merge testnets! If you want to help with merge testnets or any other DevOps tasks, join us at the EF!
9
21
86
Huge shout out to all the client teams who have been working tirelessly! Show the client teams some love :D.
4
8
87
Finally, RIP Ropsten!. Feel free to use it short term, but don't expect long term support. Migrate stuff onto Goerli/Sepolia for the long term!.
My (personal) 2c wrt #Ethereum testnest:. - Fork Ropsten into merge, deprecate (no more forks).- Fork Sepolia into merge, recommend as new testnet.- Fork Goerli into merge, maintain as legacy testnet.- Not fork Rinkeby into merge, deprecate (no more forks).
2
2
77
merge-devnet-2 was launched yesterday and has been running well! We'd love to have people come test it with us!. @vdWijden has created an awesome doc with ideas and links to resources here:
3
16
76
We should be releasing information about the kiln testnet in the next few days.
0
5
81
A hearty congratulations to the client teams and everyone helping!.
4
1
81
We found 2 non-critical bugs that didn't stop finality, both were patched really quickly by client teams. I'm really glad we're looking at optimization bugs rather than critical bugs.
1
4
79
Holesky is happening soon! We crowdsourced requests for running genesis validators and created a genesis state that's 1,460,000 validators: Roughly 2x mainnet! Genesis will happen in less than a month :D.
Holesky public testnet genesis file has been generated, and merged into main branch of Launch date Sept/15, 2023, 14:00 UTC.Chain ID-17000 (zip code of Holešovice). The genesis will have 1,460,000 validators. Our biggest public network to date.
6
18
77
The releases for Goerli, the last testnet merge, are out! . Please make sure your node is up to date and configured properly!. Some config tips and common pitfalls can be found here: #TestingTheMerge.
📢📢📢 Goerli/Prater Merge Announcement 📢 📢📢. Prater will run through the Bellatrix upgrade on August 4th, and merge with Goerli between August 6-12th: if you run a node or validator, this is your last chance to go through the process before mainnet 🚨.
2
19
70
Blobs are coming to a node near you! :D.
Deneb timeline update🎉.Client release: Feb/22.Mainnet: March/13.
1
5
75
Tune in to ACD tomorrow for a full incident report. But I do want to mention that we went from fork -> issue -> triage -> fix -> finality in under 4h. The debugging apparatus has gotten great!.
4
5
72
To anyone wondering how to perform local shadowforks, you can do so with a single line command now!. We made some modifications across the stack to make it easier to pull a snapshot, run the genesis tools and get a local shadowfork in record time (for Holesky).
Local fully featured Ethereum devnets in one line, now with shadow forking and node selectors.🟢. Check out the Ethereum package v1.40.0 here:
6
17
75
Its the shadowfork time of the year again :D.We had a great goerli shadowfork yesterday (goerli-sf-0)!. This forked goerli and enabled the Dencun upgrade. You can find the explorer here: #TestingTheSurge.
10
18
67
We launched a website :D. will now contain blog posts on topics we find interesting, a summary of all the tools we maintain and more(links to services we host is a WIP)!.
We'd like to introduce our new website - your gateway to all things EthPandaOps and a way to keep up to date with projects we’re developing! 🌿.
1
18
69
Client releases are out! This is the final step before the merge goes live on Ethereum. The first part of The Merge happens on the 6th of Sept in the Beaconchain, called Bellatrix. The full transition will happen ~15th Sept! . Update your nodes now!.
This is it: The Merge is coming. UPGRADE 👏YOUR 👏 CLIENTS! . More👇.
3
11
65
Please don't, use sepolia instead! . Tons of resources, faucets here: And here's a landing page with links:
8
18
69
Next shadow fork will be once everyone is back home and rested, #TestingTheMerge won't stop until we merge :D.
1
3
69
This was the first time we attempted a mainnet-shadow-fork, we were expecting to learn a lot from the transition! The next week or so would be spent with sync tests against this fork and trying to trigger more edge cases. We plan on repeating it next week, for advanced users ;).
5
3
70
Thank you for hosting us for a Dencun watch party!.
𝗕𝗹𝗼𝗯𝘀 𝗮𝗿𝗲 𝗹𝗶𝘃𝗲 𝗼𝗻 𝗘𝘁𝗵𝗲𝗿𝗲𝘂𝗺! 🐡🐡🐡. Cheers to the EF team for smoothly delivering the long-awaited upgrade. ️. Time to celebrate with some Dencun Donuts! 😋🍩
5
3
67
Are you interested in setting up a robust monitoring system for your Ethereum node? . Have a look at the guide from @Kiln_finance: Most of it is overkill for home stakers, but extremely important for node operators! Either ways, peek behind the curtain :D.
5
16
63
With the help of the community, we hit TTD and the participation rate dropped by ~13%. We began looking at the missing indexes and knocked on some doors. A large % of the drop was from a config issue on the Nimbus team's nodes, that was easy fixable.
2
1
61
The last missing percentage points are explained by a bug found by the nethermind team shortly before the ropsten merge. Thankfully a simple restart fixed the issue.
Few nodes might have failed on race condition during the transition. Just before Ropsten merge we were reviewing transition on a call and found out that this can potentially happen. Here is the error log. If it happened to you simple restart fixed the issue on our nodes.
2
1
59
We're happy to open source Attacknet! We've been playing with this tool behind the scenes since Nov and have used it for Dencun testing!. The ability to control network latency in certain network layouts will definitely help us on the path to peerDAS.
Attacknet, a tool we collaborated with @trailofbits on is out! It gives us the ability to perform chaos engineering on local devnets, allowing us to simulate edge cases before they happen on mainnet. Find more info here:
2
12
64
The remaining missing participants are from a config issue in the besu team's nodes that affected just the nimbus CL. The EF nimbus-besu node continues to work fine, reinforcing that it isn't a bigger issue.
2
2
60
The node split on this shadow-fork attempts to mimic mainnet(roughly). Here's a grafana dashboard with how we're doing -> Green=Good, Red=Bad.
1
3
62
We had a misconfiguration in the el genesis file and that led to the holesky network improperly launching. Some validators manually fixed the config and were able to start the chain, but not enough for the network to finalize.
Holesky launch update! . Due to a misconfiguration in the ExtraData field (applied to EL configs but not to CL ones), #Holesky didn't initiate properly. A consensus has been reached among the EF DevOps and client teams to relaunch a week from now.
2
14
58
The underlying PoW testnet had multiple forks, so different beacon node detected different terminal block hashes. The chains however agreed on a block hash and moved ahead as expected, the chain is finalizing and we had no downtime with 0 intervention. Great test overall! :).
2
4
57
Genesis for the Kintsugi testnet is in ~15mins, all links can be found here: Information on joining has been updated here:
2
17
58
The kaustinen testnet is now live. again!. Kaustinen is a verkle testnet. This time with proofs in blocks that conform to the new consensus spec: Find tooling links here:
4
15
56
Updates from testing land!.Dencun-devnet-8 is live! Its the next iteration in devnets, aimed to allow clients to test interoperability with the latest spec .
8
11
52
Is anyone still reliant on Goerli for testing? If so, please move to a supported testnet!. Reach out if you need help moving. Sepolia -> Dapps.Holesky -> Large validators.Ephemery -> Short term validator tests.
6
22
56
Participation rate did drop, but we are still well above the minimum required for finality. The missing % needs to be 0 before we're going to ship the merge!
5
7
58
Why can't that day be today? . The entire bee movie script is a blob on devnet-10. Next step, the movie itself!. Thanks for @__flcl for the tooling ;).
blob morning. can't wait till blobs come out, so I can distribute this via the mempool
8
10
55
Introducing the new testing tool on the block: Assertoor. As the name suggests, it asserts behavior on a network. Imagine an end-to-end integration testing tool thats built to support client interop testing.
Our newest blog post introduces Assertoor, a new end-to-end, cross-client integration testing tool that operates at a higher abstraction level than Hive. Check out how it works and how we're using it to do automated testing on full scale ethereum testnets:.
2
10
56
We're seeing a healthy number of blobs submitted to the network. You can expect deeper analysis over the next days. We'll be relying heavily on Xatu for the data:
2
5
54
Client hotfix incoming, seems like they've identified an issue. In the meantime, the network is processing blobs already! We've ingested >1k blobs
Goerli forked and blobs are now enabled on the testnet! . We're debugging a client issue right now, we should have more info in a bit. Expect to hear more later today and a deeper dive at ACD tomorrow!.
2
4
51
Another small difference is that this shadow fork used the develop/unstable branch of every client, so we aren't using merge branches anymore. This is the code that will make its way into releases.
1
3
52
We had the @EthereumOnARM team taking part in the shadow forks too! So we will have blocks produced post merge by an ARM client.
1
0
51
If you ever wondered how we setup test network, what tools exist or what hidden dragons lie in the darkness, come for our workshop at @ETHBerlin :D. Pro tip: We'll make sure to show you a workflow on how you can test your EIPs with a local testnet, takes just one command!.
* @parithosh_j and @BarnabasBusa, both devops engineers at @ethereum, will be running a workshop diving deep into the extensive tooling they use for single-host and multi-host testnets, aiming to showcase preferred options for various use cases.
4
6
43
Reminder that the Goerli network is deprecated and will not be supported for the future. Please make preps to move over your testing to Ephemery, Holesky or Sepolia networks. Ephemery-> small validator testing.Holesky-> long term, large validator testing.Sepolia-> Dapp testing.
Reminder the Goerli testnet is being deprecated!.
1
16
45
Estimating TTD for merge devnets is quite hard since we have no idea what hashrate to expect, its a bit easier if we control all the miners. There seems to be some external miners on Kiln, it'd be great if they could reach out to us so we can work together :) #TestingTheMerge.
1
16
48
The beaconchain genesis will be Friday at 14:00 UTC. The PoW chain has already started and we expect the PoW phase to last ~a week. Deploy your smart contracts now to know how they behave pre and post merge!.
3
6
46
We're gonna continue monitoring the network over the week to look for issues, perform sync tests and stress the nodes.
1
2
46
@evan_van_ness We'll soon have a more public testnet for everyone to play around with withdrawals :D.
2
3
46
@dimsomedim @protolambda The devnets all forked at <256 epochs, meaning the network was too new to even hit one cycle of SLOTS_PER_HISTORICAL_ROOT. So we never triggered this issue in devnets. Shadowforks are more of an EL stress test, underneath they're just fresh CL devnets - so same reason.
3
3
45
We'd like to tear down the Zhejiang testnet on Wednesday (April 19th). Please migrate any infra that still runs there before then. The testnet was meant to test the Shapella upgrade and is now considered deprecated as the upgrade is live on mainnet. Thank you for testing!.
3
16
43
We still need tools built to help us debug issues more efficiently, there's a non-complete list here:
- Get intermediate state roots.- Trace block(by RLP).- A way to pause/flush state in a client(Prevent pruning).- Get Block receipt RLP.- PR in clients to dump dump invalid blocks/non-deserializable stuff received in gossip.- Visualize when merge is successful!!.
3
3
44
Earlier shadowforks required us to sync and run a set of scripts to generate the forked network. We realized this wouldn't scale well with Verkle, as shadowforks would be one of the primary testing tools. We were able to use @KurtosisTech to switch to local and automatic sfs.
Slow and painful in the making, a shadowfork converting the Holesky state to verkle has finally occurred 🎉
3
7
45
New record by the team, looks like we can pull off mainnet size tests that don't cost an arm and a leg too :D.
EF DevOps' largest testnet (in terms of validator count) has launched successfully this morning. The aim was to have a network run at mainnet validator set in order to test BLS key changes. Huge thanks to @beaconcha_in guys to help us get up and running with the explorer.
2
6
44
We reused the mainnet deposit contract with a new fork ID. This means that every mainnet deposit needs to be processed and listed as invalid on the shadow fork. This huge computation triggered some edge cases in some clients, the good news is that the network still chugged along.
1
3
42
Interested in testnets and what tools we use? Check out my talk at #DevConBogota! I'll try to highlight some of the tooling we built during the merge testing efforts, We built most of it for reuse by the community!: 🧵.
6/11. 🎥 Watch the #Devcon Main Stage talk: . 'Quest for the Best Tests' by @parithosh_j, part of the DevOps Crew at the @Ethereum Foundation. 👇.
1
10
40