Q: What is the methodology behind the Data Accuracy Benchmarking Tool?

A: In our tests, we query the JSON RPC method eth_blockNumber a total of ~1,072,000 times to get the latest block number from each provider over a period of 24 hours. This is one simple example for benchmarking data inaccuracy.

Q: How do we measure inaccurate data?

A: Let’s assume that we have the following sequence of block numbers after polling eth_blockNumber a total of 13 times.

1, 1, 1, 1, 1, 2, 1, 1, 3, 2, 1, 1, 3

Given this sequence, we find a total of 5 errors at block indices 6, 7, 9, 10, & 11 (the red numbers). What this means is that at blocks 6, 7, 9, 10 & 11, the returned blocks did not follow the expected sequence of ascending values, evidencing some underlying data accuracy issue.

NOTE: Once we record that the node provider is on block number 2, we assume that any subsequent values of 1 are incorrect and equivalent to the provider time-traveling and providing a current block value that is 1 block late. Similarly, once we record that the node provider is on block number 3, any instances of blocks 1 or 2 showing up are now misaligned. To simplify this benchmarking test, we explicitly measure data accuracy from individual providers themselves and do not cross-compare block data between different providers.

Q: Why do we use eth_blockNumber as the call in the test?

In thinking about benchmarking accuracy, we started from first principles and tested one of the most basic Ethereum RPC calls: eth_blockNumber. eth_blockNumber returns the Ethereum network’s latest block number, and in a decentralized network, the block with the highest block number contains the most recent transactions. Because of this, in normal network conditions, calling eth_blockNumber from a single node/node provider should only return block numbers that ascend in value. Any inconsistency in ascending data is indicative of a reversion or, when it happens more frequently, of potential provider data inaccuracy.