Hosting a large number of Rocket Pool minipools & Stader ETHx validators

For institutions and non-technical funders

27 Dec 2023

Blog Hosting a large number of Rocket Pool minipools & Stader ETHx validators

In this report we investigate and compare different methods for hosting hundreds or thousands of Rocket Pool 8ETH minipools and Stader ETHx validators (hereafter collectively referred to as validators) for institutions or non-technical funders. That is to say, any situation other than having a single technical funder who does everything.

The 3 hosting options considered are:

  • Allnodes staking as a service
  • Avado Cloud self staking platform
  • Self-managed servers in the cloud

Parameters, assumptions and initial values

ETH price: $2000
ETH to invest: 15,000
ETH to invest in USD: $30,000,000
Portion of capital assigned to each of Rocket Pool and Stader: 50%
Rocket Pool validators: 705 
Stader validators: 1689 
Total validators: 2394

Attestations per validator per month: 6848.43
Missed attestation cost ETH: 0.000011
Missed attestation cost USD: $0.022

Allnodes uptime: 99.9%
Avado uptime: 99%
Self-managed uptime: 99%

ETH consensus rewards: 4.15%
ETH execution rewards: 1.77%
Rocket Pool governance token rewards: 6.20%
Stader governance token rewards: 22.65%

Rocket Pool governance token ratio (30% minimum): 33%
Stader governance token ratio (10% minimum): 11%

Allnodes Rocket Pool validators fee: $7.5
Allnodes Stader validators fee: $5
Avado server fee: $200 (5 required)
OVH Rise-LE-2 on demand self-managed server: $100 (6 required)
Sysadmin annual salary: $100,000
Sysadmin time spent: 20%/1 day per week
Sysadmin cost yearly: $20,000
Sysadmin cost monthly: $1667

Executive summary

Allnodes

Pros:

  • Reliability
  • Security
  • Simplicity 
  • GUI dashboard

Cons:

  • High cost
  • Custom closed source software

Avado Cloud

Pros:

  • Low cost
  • GUI dashboard 

Cons:

  • New and not well tested
  • Security tradeoffs
  • Need to hire a Sysadmin

Self-managed

Pros:

  • Low cost
  • Flexibility
  • Official node software developed by the Rocket Pool/Stader teams

Cons:

  • Security tradeoffs
  • Complexity
  • Need to hire a Sysadmin
  • No dashboard

Revenue, expenses and ROI summary

All figures are monthly unless stated otherwise.

AllnodesAvado CloudSelf-managed
Revenue$169,599$168,071$168,071
Hosting cost-$13,733-$1,000-$600
Services cost$0-$100-$100
Labor cost$0-$1,667-$1,667
Penalties cost-$356-$3,560-$3,560
Total costs-$14,089-$6,327-$5,927
Profit$155,511$161,744$161,744
Annual ROI6.22%6.47%6.49%

Recommendations

Allnodes provides the most secure, reliable and simple option for running validators. Given the ROI from hosting on Allnodes is not much lower than running self-managed servers we recommend using Allnodes.

We recommend against Avado Cloud which offers no significant benefit over self-managed and is new and risky.

For diversification of risk we recommend a mix of both Rocket Pool and Stader validators. Stader is riskier than Rocket Pool and its current advantage in APR will reduce over time. However Rocket Pool’s much higher governance token collateral requirement of 30% is a large disadvantage compared to Stader’s more reasonable 10%. 

Security

Hacks on funds/nodes/validators

Allnodes

When validators are created through Allnodes there are two factors that should ensure Allnodes/hackers are unable to steal the funds:

  • The collateral is transferred directly from your hardware wallet into the Rocket Pool/Stader smart contract.
  • The withdrawal/reward address of the node is set to your hardware wallet before you transfer the collateral. There is no way to change this without you authorising it from your hardware wallet.

There is an attack vector where a hacker changes the Allnodes code so the transactions you are prompted to sign are not the intended ones. It is therefore important to carefully read and verify each transaction before signing.

An attacker can also cause loss of funds even if they cannot steal the funds. They can do this by making the validator misbehave or launching multiple validators with the same keys (Allnodes and you both keep a copy of the validator’s keys).

Another aspect to consider is that Allnodes is running custom, closed-source software – not the official and open source node software developed by the Rocket Pool/Stader teams. This increases the risk of vulnerabilities.

Avado Cloud

When setting up validators on Avado Cloud, you must send funds to a hot wallet that runs on the server. Supposedly this wallet is encrypted with a password you entered but you are trusting the Avado server:

  1. To securely generate the private keys.
  2. To securely encrypt the private keys.
  3. Not to steal the funds when the wallet is decrypted in the process of setting up validators.

Since the user doesn’t get a chance to verify transactions before they are broadcast, this is putting a lot of trust in the Avado server.

The user can set the validator withdrawal address to their hardware wallet so once the validator is funded hackers are no longer able to steal the funds.

Since Avado Cloud runs custom software there is a risk of bugs/vulnerabilities in this software. The software is open source but there are a small number of contributors and users so it is unlikely the code is extensively reviewed.

There also seems to be a negative impression of Avado amongst Rocket Pool community members with comments such as:

Avado is making some questionable trade-offs re: security, so I would strongly advise against them.

Discord user @[object Object]:

A security advantage of Avado Cloud is the funder could use the GUI to set up the validators without technical help to avoid the risk of the Sysadmins stealing funds from the hot wallet. Once the validators are created the Sysadmins can take responsibility for maintaining the server and at that stage the funds can only be withdrawn to the funder’s hardware wallet.

Self-managed

The official node software from the Rocket Pool/Stader teams currently requires validators to be funded from a hot wallet on the node. The Rocket Pool team is working on enabling an external wallet to fund the node directly but this feature is not expected until around April 2024.

Assuming the funder does not feel comfortable using the command line, the most secure way to create the validators is as follows:

  1. Sysadmin creates the node hot wallet on the server
  2. Funder sends a small amount to the node wallet
  3. Sysadmin registers the node
  4. Sysadmin changes the withdrawal address of the node to funder’s hardware wallet
  5. Funder confirms the node withdrawal address matches his hardware wallet
  6. Funder sends collateral to the node wallet
  7. Sysadmin creates the validators

Once the validators are created, the funds can only be withdrawn to the funder’s hardware wallet. Unfortunately there is a small window between steps 6 and 7 where hackers could steal the funds from the node wallet.

The whole process is complicated and confusing enough that the non-technical funder could be tricked out of his funds by an evil Sysadmin. So only highly trusted staff should be involved in setting up the validators.

The possibility of external hackers stealing the funds can be minimized by ensuring the server is as secure as possible. Assuming the Sysadmin follows security best practices the general risks of a server attack apply equally to all cloud hosting methods.  


The following 3 security risk should be considered, however they are mostly the same no matter which hosting option is used

  • Attack causing loss of funds without theft
  • Hackers or human error on Rocket Pool/Stader
  • $5 wrench attacks

Attack causing loss of funds without theft

For all the hosting options considered, the hosting provider and Sysadmins have access to the validator keys. Therefore it is also possible that attackers could get access to these keys. Having these keys does not allow attackers to steal funds from the validator but they may be able to cause slashing by making the validator misbehave or by running a second node with the same validator keys. This could result in the loss of 100% of the collateral.

Hackers or human error on Rocket Pool/Stader

There is a risk that even if you and Allnodes/Sysadmins do everything right that you could lose your funds due to a bug or vulnerability in the Rocket Pool/Stader protocols or software. One would hope that the more popular a protocol, the less likely this is to happen since there will be more eyes on the source code and more security audits performed. However the more popular a project is, the bigger the incentive for hackers to find and exploit a vulnerability.

Smart contract audits have been performed on the protocols but these aren’t foolproof and it is possible for smart contracts to be updated or for new smart contracts to be introduced that are not audited.

On top of this there are many pieces of software involved in both projects and it is possible for there to be a bug in these applications that could result in loss of funds.

Sticking to popular protocols run by teams with proven records will reduce the risks. In this regard Rocket Pool with around 26,000 active validators would be preferred over Stader which has around 2,000 validators.

$5 wrench attacks

This is when an adversary uses physical threats against you or your loved one to force you to hand over funds. To minimize the risk, in brief:

  • Have your hardware wallet with you as rarely as possible. Keep it somewhere safe and remote, such as a safety deposit box in a bank.
  • Consider multisig options.
  • Set up so as to require hardware wallet signing as infrequently as possible, and not on a set predictable schedule. 
  • Maintain a low profile and anonymity. Don’t tell people you have crypto.
  • Use Trezor’s built-in shamir share backups.
  • Maintain the best possible physical and online security in general.

Reliability

Allnodes

Since Allnodes’ business depends on operating validators reliably you can expect this is a priority of theirs. They offer a 99% SLA on their basic plan and a 99.9% SLA on their advanced plan. If Allnodes fails to reach these guarantees there won’t be any compensation, and the SLAs only refer to server uptime and do not guarantee the validators will function correctly.

Stader is a newer project and the integration with Allnodes has proven buggy. However this will likely only impact creating and destroying validators. Once the validators are up and running the newness of the Stader protocol should not have a large impact on validator reliability. 

Another effect of Allnodes using custom, closed-source software, rather than the official node software developed by the Rocket Pool/Stader teams, is that it increases the risk of software bugs or incompatibilities causing downtime.

Given Allnodes have always exceeded their SLAs and normally achieve greater than 99.9% uptime on the basic plan, we will assume 99.9% validator uptime. This higher uptime is why we assign Allnodes a slighter higher revenue and lower missed attestation penalties than the other hosting options.

Avado Cloud

It is difficult to judge the reliability of Avado Cloud without having used it, but there are a few things going against them:

  • The cloud hosting is a new feature; their website still says “we are currently building and testing this solution before launching it to the general public”.
  • It is partially a self-managed option and you don’t know to what extent you can count on Avado support to resolve issues.
  • They seem quite small so it is likely there will be bugs that you are the first to encounter.
  • Since they are running custom software, if there are any issues you might have trouble getting help from the Rocket Pool/Stader community.
  • There is no SLA listed on their website, however it is likely they will have one once the product is in general release.

We can therefore predict their reliability to be worse than Allnodes and possibly worse than self-managed nodes running the official Rocket Pool/Stader software. We roughly estimate validator uptime of 99%.

Self-managed

Since we don’t have much experience running validators it is hard to estimate the expected reliability of self-managed servers. The hosting providers that you would use have very high availability so you can expect server uptime of over 99.9% (AWS SLA, OVH SLA). We roughly estimate validator uptime of 99%, the same as Avado.

Costs

There are 4 main costs to consider:

  • Penalties: Ethereum/Rocket Pool/Stader validator penalties for offline/misbehaving validators
  • Labor: Deploying, patching, upgrading etc
  • Hosting: Compute, storage and network traffic
  • Other paid services as detailed below

Penalties

There are three types of penalties that validators can incur, shown in the table below. These are paid entirely by the node operator.

TypeWhen it happensAmount
Missed attestationEvery epoch (6.4 minutes) the validator is offline for0.000011 ETH per epoch
Missed sync committeeWhen your validator is offline and selected for a sync committee (only happens every 2 years on average)0.00047 ETH per epoch up to 0.1 ETH
SlashingFor misbehaving validator – unlikely to happen unless the same validator keys are used on two different nodes1-32 ETH

The only penalties that are likely to impact your nodes are missed attestations. Based on the expected uptime mentioned above, the attestation penalties are given in the following table:

Avado/Self-managedAllnodes
Reliability99%99.9%
Missed attestations per validator per month68.486.85
Penalties per month-$3,560-$356

Hosting and Labor

Allnodes

Since Allnodes is a fully-managed node hosting solution there are no labor or services costs. The only costs are hosting fees of $7.5 per Rocket Pool 8ETH minipool per month and $5 per Stader ETHx validator per month.

Avado and self-managed cloud hosting

A single Avado Cloud server costs $200 per month and offers the following specifications:

AMD Ryzen 5 3600 Hexa-Core “Matisse” (Zen2)
64 GB DDR4 RAM
4 TB NVMe SSD
1 Gbits bandwidth

For self-managed servers, considering that you will be running a large number of validators on the nodes, we target the following specs (ref1, ref2, ref3):

CPU6 cores
RAM32 GB
SSD2 TB15,000 IOPS500 Mbps
Network50 Mbps
DataInbound: 2 TB per month
Outbound: 1.5 TB per month

Servers on AWS and OVH that meet these requirements, and their costs when including storage and network are:

  • AWS: t2.2xlarge -$640.98 per server so -$3,845.88 for the required 6 servers.
  • OVH: Rise-LE-2 -$100 per server so -$600 for the required 6 servers.

To decide how many servers you should provision to run the validators you need to consider these factors:

  1. How many validators can a server handle?
  2. Should you run both Rocket Pool and Stader on the same server?
  3. How can you ensure minimal penalties when performing upgrades?
  4. Do you care about smoothing out variance in income due to planned/unplanned downtime?

How many validators can a server handle?
A single node can handle a very large number of validators (ref1,ref2); definitely several hundred and possibly in the thousands. If you look at the number of validators run on Rocket Pool/Stader nodes in practice you see that only a few run more than 400 validators, with 940 being the highest number of validators per node (Rocket Pool validators per node, Stader validators per node).

Number of validatorsStaderRocket Pool
<1001752,731
100-200125
200-400018
400-80002
>80001

Based on this you would be an outlier if you tried to run more than 1,000 validators on a node and it is therefore recommended you (at least initially) spread the validators out.

Should you run both Rocket Pool and Stader on the same server?
The default Rocket Pool/Stader node deployments will not share processes so running both a Rocket Pool node and Stader node on the same server will consume significantly more resources. Unless you wanted to customise the software, which we recommend against, you should only run either Rocket Pool or Stader validators on a server.

How can you ensure minimal penalties when performing upgrades?
To minimize penalties when performing upgrades you should have one server with a smaller number of validators (at least 2) that is used to test updates. Once the process for updating is confirmed on this mini-server you can apply the updates on the other servers one by one. This will result in minimal downtime for the servers running the bulk of the validators and therefore minimal penalties.

Do you care about smoothing out variance in income due to planned/unplanned downtime?
Distributing validators across more servers may have a smoothing impact on the income variance since when there is an outage (planned or otherwise) it may only impact a subset of the servers.

Based on the above, we would recommend 1 server for every 900 validators at most. However this should be tested with lower amounts first and discussed with Avado support. This gives us a total of 5 servers for Avado and 6 for self-managed.

  1. ~2 x Rocket Pool validators 
  2. ~703 x Rocket Pool validators
  3. ~2 x Stader ETHx validators
  4. ~843 x Stader ETHx validators
  5. ~844 x Stader ETHx validators
  6. Testing server for self-managed hosting

Some cost savings could be realized by using lower spec servers for mini-servers and testing server (number 1,3 and 6 in the above list) however significant testing and time in production is required to ascertain the details of this saving.

Services and components

There are a number of technical services and components required to sysadmin Avado and self-managed servers. The choice of which product to use for each service would require investigation and would depend on the cloud server hosting provider, however a couple of popular options are listed as examples for each. 

  • Git repo
    • GitLab
    • GitHub
  • Logging
    • NewRelic
    • CloudWatch
  • Uptime monitoring
    • CloudWatch
    • New Relic
    • Splunk
    • UptimeRobot
  • Alerting and incident response
    • PagerDuty
    • New Relic
    • Splunk
    • OpsGenie
  • Metrics dashboard
    • Grafana
    • New Relic
    • Splunk
  • Secrets management
    • AWS Secrets Manager
    • HashiCorp Vault
    • SOPS
  • Backups
    • Wasabi
    • AWS S3

On average the 7 services each cost around $15 per month. We estimate the total cost to be around $100.

Labor costs

You also need to consider labor costs since Avado Cloud is mostly a self-managed system. Avado servers have a GUI which simplifies setting up the node and automatically updates the node. However, given this software is unlikely to be well tested and isn’t officially supported by the Rocket Pool/Stader developers, it could actually increase the amount of labor required to maintain these nodes compared with using the official node software. We therefore estimate a full time employee would spend 20% of their work time (1 day a week) maintaining the validators for both Avado and self-managed servers.

Profitability

Both Rocket Pool and Stader give separate APRs on the ETH collateral and governance token collateral. The governance tokens are RLP for Rocket Pool and SD for Stader. Assuming a staking ratio of governance token to ETH of x, the overall APR is:

apr = (eth_apr + x gov_apr) / (1 + x)

The smallest permitted staking ratio is 30% for Rocket Pool and 10% for Stader. However it is prudent to stake a bit more than this to avoid price movements resulting in the node becoming under-collateralized which would result in forfeiting governance token rewards.

Rocket PoolStader
RPL/SD collateral33%11%
ETH APR5.92%5.92%
RPL/SD APR6.20%22.65%
Aggregate APR5.99%7.58%

Flexibility

Allnodes

There are two things to consider here:

  1. When new desired features/protocols are developed, will Allnodes be fast to support them?
  2. If Allnodes does not support these new opportunities, how easy would it be to switch to another hosting option?

Given customers will only choose to host with Allnodes if it provides a competitive APY, you can expect that Allnodes will try to support new desired features and protocols. However the speed with which they add support will depend on their resources, the technical complexity and the degree of customer demand. It is safe to assume they will have more technical expertise and resources and so can probably add support for new features/protocols faster than Avado Could, or that you could with self-managed servers.

If new opportunities arise that Allnodes doesn’t intend to support it will be relatively easy to exit the validators from Allnodes. Allnodes has a “Move to other hosting” flow, however we have not tested this and details of how it works are not documented. 

In any case there is no long-term contract and the downside of manually exiting validators from Allnodes and spinning up new validators somewhere else (fees, time in queue, risk of losing funds) apply equally to all hosting options.

Avado Cloud

Avado Cloud’s flexibility characteristics are similar to Allnodes. However Avado Cloud has fewer Engineers and customers so could be expected to take longer to support new features/protocols. Their implementation of such may also be less well tested.

Self-managed

Self-managed provides the most flexibility given your Engineers will control the servers. Provided you are willing to invest the resources to add support for any new features/protocols there should be no blockers. However the reliability of the changes will depend in part on the quality of your Sysadmins.

Other alternatives for hosting Rocket Pool and Stader validators

Allnodes is currently the only managed hosting provider that supports Rocket Pool and Stader validators. Similarly there are no direct competitors to Avado Cloud.

One alternative for Rocket Pool validators, that we didn’t consider in this report because they do not support Stader, is running Dappnode on a self-managed server. This provides a more user-friendly GUI but could make ongoing maintenance more challenging. To quote someone from the Rocket Pool discord:

Q: I’ve been told Dappnode would be easier to use, but this doesn’t seem easy?
A: You’ve been lied to. The existence of a GUI does not imply ease of use. There are a bunch of things which are hard or impossible to do on Dappnode/Avado and quite easy when using the Smartnode CLI.

Hiring a specialist crypto node Sysadmin

Managing Rocket Pool and Stader validators does not require specialized skills that would require a specialist crypto Sysadmin. Given the high level of trust you would need to establish with the Sysadmin we would not recommend hiring someone new for this task. If anything you would want to hire Engineers to develop software/smart contracts to enable deploying validators in a more secure and trustless way. However this would come with significant additional costs.

Standard 32 ETH validators

All of the above also applies to hosting a large number of vanilla 32 ETH validators, but only when comparing these same 3 hosting options. The big difference with hosting ETH validators is that the costs are lower and there are more good options including some that compete directly with Allnodes and compare favorably to it, for example StakeFish.

Small reductions in cost come from ETH validators lower complexity and greater popularity compared to Rocket Pool and Stader validators. This produces more and better documentation, a larger and more active developer community and lower risks. 

When combined with the smaller number of ETH validators hosted for the same capital investment (468 instead of 2394 for this 15k ETH, $30m example) this results in less Sysadmin time needed and lower hosting costs for the Avado and self-managed options.

Gentleman James

Gentleman James: New money trash connoisseur

Gentleman James
New money trash connoisseur


Crypto Financial Planning

Take a step towards financial independence with iYields combined crypto and fiat wealth and budget dashboards.