Genius Yield ISPO Update
As many in the community have noticed, GENS4 staking pool ceased producing blocks from Epoch 315. Our development team, with the help of community Stake Pool Operators (SPOs), have been intensely investigating the issue and were able to pinpoint the root cause on January 30th. After careful consideration, Genius Yield have decided to officially shut down GENS4 by February 18, 2022. As a result, the Genius Team asks delegators to move their stake from GENS4 to GENS1, GENS2, GENS3 or GENSX as soon as possible.
In the spirit of transparency, integrity and accountability, this article describes what happened, how we responded, and what happens next.
TLDR: The Genius Team asks that you move your delegation out of GENS4 and into GENS1 (100% GENS), GENS2 (50% GENS / 50% ADA), GENS3 or GENSX (100% GENS) as soon as possible. The GENS4 pool will stop accumulating GENS on February 13 and will be shut down on February 18th, 2022.
What happened?
During a standard operational procedure, a signing key file was accidentally overwritten. This key permits us to sign pool-related transactions and create pool certificates expected by the Cardano network. Unfortunately, it was not possible to recover this key, which subsequently halted the production of blocks in the GENS4 staking pool. Delegators’ stakes are in no way threatened, and the pool pledge and rewards are intact.
How the issue was identified
This particular situation was very difficult to diagnose because the block producer can run without errors even with a corrupted signing key. The stake pool operates as expected, but can no longer mint blocks. To help us with troubleshooting, we connected with top stake pool operators (SPOs) from the community. After more than 10 days of reading through logs, testing different configurations, and comparing stake pool deployments with fellow SPOs, Andrew Westberg (twitter / github / spo) gave a suggestion that helped us pinpoint the issue.
Our response
At Genius Yield, we strive to build systems that are reliable and resilient to failures. Immediately after discovering this issue, we have made the following changes to drastically improve the safety and security of all critical data associated with our stake pools:
- Updated file permissions to prevent overwriting or deleting of files even as a super user
- Backed up all keys to a password-encrypted physical device
- Created multiple backups of our cold environment disk
- Created a weekly disk snapshot schedule for our block producers
- Started using a VPN to connect to our block producers from our local environments
We have also made the following infrastructure changes to enhance the overall performance of our pools:
- Doubled the CPUs of all block producers and relays
- Updated our relay topologies to connect to at least 15 actively running relays in the network
- Started using the Topology Updater tool written by the Guild Operators
- Started using the CNCLI tool to track our scheduled slots and plan operational procedures accordingly
Furthermore, GENS1, GENS2, GENS3 and GENSX have been operating reliably and on par with the rest of the network.
What happens next?
The Genius Team asks that you move your delegation out of GENS4 and into GENS1 (100% GENS), GENS2 (50% GENS / 50% ADA), GENS3 (100% GENS) or GENSX (100% GENS) as soon as possible. The GENS4 pool will stop accumulating GENS on February 13, 2022 and will be shut down one epoch later on February 18th, 2022.
200K GENS Compensation Program
Genius Yield will roll out a compensation program to encourage GENS4 delegators to move their ADA into GENS1, GENS2, GENS3 or GENSX.
A delegator must meet the following requirements to qualify for the program:
- Must be delegating to GENS4 pool at the time of publication of this announcement (February 3rd, 2022)
- Must switch their delegations to GENS1, GENS2, GENS3 or GENSX no later than February 13th, 2022
- Must maintain their stake until the end of the ISPO: June 15th, 2022.
200,000 GENS will be divided proportionally to current GENS4 delegators that have met all of the criteria described above. These will be added to delegators’ total allocation and airdroped according to the ISPO vesting schedule.
The SPO community
The SPOs of the following pools dedicated many hours to helping us investigate and troubleshoot GENS4:
To show our appreciation, we will reward them 1000 GENS each!
Moving forward
The Genius Yield team has used this opportunity to revamp our stake pool infrastructure by focusing meticulously on security, automation and resilience. We have incorporated industry best practices, and we will continue to strive for greatness and quality across all our systems. Together we will continue to grow stronger.
