By: Brett McLain, Director of Engineering – Crypto, Fiat, Staking If you’re interested in cryptocurrencies, payments, or staking and want to help build the financial system of the future, the funding engineering team @ Kraken is hiring! When Kraken launched almost a decade ago, only three cryptocurrencies were supported: BTC, LTC, and XRP.  Today, Kraken supports 82 assets on 33 blockchains, and staking services for 8 cryptocurrencies.  To facilitate the millions of deposits, withdrawals, and staking transactions a year at Kraken, the crypto engineering team operates hundreds of services to ensure the smooth flow of funds in and out of the exchange. The blockchain software that underpins these services is updated frequently; for some of the more active blockchains, hard and soft forks can be monthly in nature while others like Ethereum are twice annual events. In general there are at least a few software updates to our blockchain infrastructure every week. The challenge of supporting and updating such a large number of different services, while at the same time building new ones can be daunting. In the last 12 months our team has added support for: - 60 new cryptocurrencies: 39 x ERC20 tokens Polkadot (at mainnet launch) Kusama Filecoin (at mainnet launch) Flow (at mainnet launch) Kava Energy Web Token (at mainnet launch) USDT (TRC20) 10 x Parachain Crowdloans Solana 1 x SPL token (Serum) Mina - 39 x ERC20 tokens - Polkadot (at mainnet launch) - Kusama - Filecoin (at mainnet launch) - Flow (at mainnet launch) - Kava - Energy Web Token (at mainnet launch) - USDT (TRC20) - 10 x Parachain Crowdloans - Solana - 1 x SPL token (Serum) - Mina - 8 new staking assets: Polkadot (at mainnet launch) Kusama Ethereum 2.0  (at mainnet launch) Flow (at mainnet launch) Cardano Cosmos Kava Solana - Polkadot (at mainnet launch) - Kusama - Ethereum 2.0  (at mainnet launch) - Flow (at mainnet launch) - Cardano - Cosmos - Kava - Solana These achievements were accomplished alongside the maintenance of our existing integrations. The engineers of the crypto team are responsible for not only the gateway software written in house, but also for the maintenance and deployment of our blockchain infrastructure that our gateways rely on. The cadence of blockchain development on these projects can be a blistering one, with breaking changes and novel new features coming frequently and sometimes with little warning. So, how does Kraken manage to release dozens of new products every year while keeping up with the fast pace of blockchain development? End to End (E2E) Tests!Why we value E2E tests and avoid mocks Since the early days at Kraken, the emphasis has been that E2E tests are the most valuable kind of tests an engineer can build. Unit tests have their place, but many developers inexperienced with complex integrations tend to write unit tests for every piece of code they build in the belief that they’re improving the overall quality of the software they’re developing.  This path, while full of good intentions, can often lead to a lot of pain down the road. Over reliance on unit tests tends to cement your architecture; it’s like pouring a layer of epoxy on top of your entire code base. You’re tightly coupling the code to its tests, making the code more rigid, inflexible, and resistant to being refactored. If you need to make a change, you’re likely going to need to make significant changes to the tests, and in some cases, throw them out entirely. Refactoring code is a key capability for an engineering team to have in their toolkit and anything that adds friction to the ease of refactoring should be carefully evaluated before being introduced. When refactoring code, a well designed E2E test often doesn’t require many changes, and provides flexibility in adjusting the internal guts of an application while ensuring that it continues to operate as expected. Does this mean you shouldn’t write unit tests? Not at all! There are many scenarios where unit tests are the perfect solution, however we’ve found that for complex integrations, E2E tests work better. In general, unit tests are most effective when written for code that meets the following criteria: - Algorithmically complex with many edge cases. - Tightly scoped with well defined requirements. - Completes a single unit of work. - Stateless. These small, tightly scoped pieces of complex code are often the building blocks of a larger application and even if a refactor were to occur, these functions would be unlikely to change. In our world this would be things like address derivation, address validation, transaction signing, etc. The key takeaway here is that as a small engineering team, there’s no way we could ever maintain the volume of services we currently support, and build new products without end to end tests. Unit tests should be considered table stakes, but in isolation, they would not be sufficient for us to keep up in this evolving space. Instead, we’ve chosen to invest heavily in robust sets of integration and E2E tests that validate that our services will operate successfully in their most common modes of operation.Challenges of E2E tests Although E2E tests can be powerful, they’re not a panacea. When integrating with third party services these kinds of tests often lose a great deal of their value since certain endpoints or interfaces need to be mocked in order to fully test the flow of a specific function or call. Mocks are only as good as your understanding of the service you’re mocking, and as a result, they can be error prone when updates are frequent and large in nature. Maintaining your own code as well as your mocks is a violation of the DRY principle (don’t repeat yourself), a term coined by David Thomas and Andrew Hunt in “The Pragmatic Programmer.”  In their book they state that “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” Creating a mocked version of any service means that there are now two potentially divergent copies of said service: your mocked version and the actual version. Errors in translating the mocked dependency’s behavior is now another concern to account for.Regtests to the rescue Thankfully for us, most blockchains support the ability to run temporary private nets that can be spun up as part of our continuous integration (CI) / continuous deployment (CD) process. The most popular example of this is Bitcoin’s regression test (regtest) mode. When you start bitcoind with the `-regtest` option, it creates a new local blockchain environment that you have complete control over. The key feature of regtest mode is that you can mine an arbitrary number of blocks at will, allowing your E2E tests to complete round trips for deposits & withdrawals of all types and variations, simulating hundreds of scenarios within seconds. Edge cases and other unique scenarios can easily be simulated in regtest mode, like multisig transactions, re-orgs, replace by fee (RBF), child pays for parent (CPFP), and more! Not only do these tests ensure that our code contains no errors, it also validates the end state of the blockchain and our ledgers to ensure that everything is operating as expected. As part of the process to add support for a new cryptocurrency on Kraken, the funding team builds out a regtest framework for all new listings. This code is the foundation of our maintenance regime: any time a new version is released, it’s simply a matter of updating the blockchain node version and re-running our CI pipeline to ensure that there’s no breaking changes. Careful reading of the release notes and collaboration with the community is still very much necessary, but these tests give us confidence in releasing new versions that we otherwise wouldn’t have.Creative Solutions Unfortunately for us, not all blockchains are as battle tested as Bitcoin. New blockchains often introduce novel concepts, and in order to offer our clients access to the most exciting new technologies, Kraken prefers to launch support for new blockchains as close to the start of the mainnet as possible. To safely support a new asset on or close to the launch date, Kraken sometimes needs to develop complex test harnesses to gain confidence in the integration and to ensure that client funds are not at risk. A perfect illustration of this is when Kraken launched support for Ethereum 2.0 only 3 days after the mainnet went live on December 1st, 2020. Although thousands of individuals and companies around the world helped test Ethereum 2.0 on multiple testnets like Medalla and Spadina, we still decided to take the concept of regtests to a whole other level with this integration. We knew early on that Ethereum 2.0 would be a significant development, and that belief has proven true as millions of ETH has so far been staked on the beacon chain, including more than 800,000 ETH that has been staked by Kraken clients. Below you can see a diagram of the set of services that our continuous integration (CI) pipeline spins up and tears down every single time a developer commits code to one of our ETH2 code repositories. At a high level, the test flow is: - Start ETH1 primary and alternate nodes (they each take turns mining for consensus) with a genesis that contains an initial amount of ETH for testing. - Start ETH2 beacon chain node as a private chain using a special minimal config mode where only 16 validators are necessary to activate genesis. - Deploy ETH2 smart contract to the ETH1 blockchain. - Deposit ETH into the ETH2 deposit contract where the funds are burned and validators are created on the ETH2 external validator node. These are validators that are just operating the ETH2 network and are treated as if they’re external to any Kraken validators. - Start ETH1 & ETH2 block explorers. - Start Database. - Start Gateway and Signatories. - Insert client requests to stake ETH -> ETH2. - Gateway picks up client requests and sends ETH to the deposit contract on the ETH1 blockchain and creates a corresponding number of validators on the ETH2 internal validator node. Validators are segregated into internal and external validator sets so that we can test what happens when our validators go down (to test slashing, penalties, lost rewards), and to see what happens when the rest of the network goes down or offline but our validators remain up. - Monitor until the validators are active on the ETH2 chain, begin tracking rewards, payouts, test slashing and penalties, lost reward detection, and pay rewards to clients. - Run our separate financial reconciliation process on all transactions to ensure everything in all of our ledgers matches correctly. The above is only a high level summary of what’s going on within our test framework; there are a number of other tests, checks, and validations that occur. If a developer needs to debug something or look at the state of either network, they can consult the block explorers to see what exactly has occurred at a glance. We don’t usually include block explorers in our CI pipeline, but given the complexity of the integration, it was helpful during the development phase to visualize what was happening on-chain. You might think that this adds an enormous delay to our CI pipeline, but that’s not the case thankfully. Currently, the full CI pipeline for our Ethereum 2.0 repo takes only 14 minutes to run. This includes auditing/building all dependencies, starting all the services, deploying various smart contracts to the blockchain, mining blocks, creating validators, and then running through all the 100+ test scenarios.Final Thoughts Developing comprehensive E2E tests for every single blockchain integration at Kraken consumes a significant amount of engineering resources. It’s a price we gladly pay, as our foremost concern is the safety of our clients’ funds and ensuring that they have a quality experience on our platform. Could our team release more products if we spent less time on tests when building new integrations? Without question. However, doing so would go against the ethos and values of not only the engineering team, but the company as a whole. These tests ensure that we can safely update to new versions of blockchain software, increase confidence during hard/soft forks, and reduce developer stress when deploying changes. Why are Kraken engineers some of the most respected in the industry? This message from Steve Hunt, Kraken’s VP of Engineering, outlines our values and dedication to helping other blockchain engineers. Share this: - Twitter - Facebook - Like this:Like Loading...