Automation of our Testnet Deployment
Spawning a Substrate Testnet in a Kubernetes Cluster within Minutes
Over the past couple of months, we have been investing heavily in setting up cutting-edge infrastructure to support the development efforts behind Basilisk. We believe that having the future-proof infra is key to the growth of any project for two reasons.
In the first place, quick builds enable rapid development. With our shiny new infrastructure, we are now able to build and setup a testnet environment (Relay Chain + Parachain) in less than 10min. In the second place, it allows us to quickly onboard new engineers (e.g. front-end) who will benefit from a production-like environment, thereby paving the road towards a larger degree of community dev involvement.
This article provides an overview of the journey which we undertook when setting up our testnet infrastructure using Kubernetes (K8s). We hope that the insights which we gathered will be useful to other teams who are looking to setup their testnet infra.
Overview of technologies used
Kubernetes - we run it in the cloud (AWS Fargate), mainly for convenience reasons. However, you can adapt the findings in this article to also spin up your own K8s cluster.
Terraform - because we like having our infra as code.
GitHub Actions - for CI/CD.
For all relevant yaml manifests and config files used in this post, please refer to the Basilisk docs.
Containerize everything!
In terms of flexibility and scalability, one can hardly do better than containers. So we naturally chose to dockerize our nodes. This quickly led to a discussion on the orchestration of these containers. We talked about Kubernetes vs Docker Swarm, and (quite quickly) decided to go with Kubernetes for various reasons. It is well documented and adopted in the wider tech world, and also the technology of choice for our team. Furthermore, the distributed nature of Kubernetes offers a more extensible scaling, as far as production would be concerned. It is also easier and more intuitive to manage Services and Ingress Controllers in a Kubernetes Cluster, especially when running it in a Cloud Environment. To give an example, being able to use ALB Controllers to expose our Nodes and make them accessible through a Route53 URL was a life changing experience!
Focus on building cool stuff, not on administering servers
Once Kubernetes was declared the Orchestration tool of choice, we needed to decide between two execution modes: Either going with our own EC2 (services) instances, or using a Managed Service to run our images and orchestrate the pods.
The two differ in a sense that, with EC2, you have less flexibility as far as controlling the resources is concerned; if you have no idea about the number of pods you need to be running in the future, you most likely will have to overestimate the capacity of your instances (CPU / RAM, as well as pods count) which may result in waste of capacity and higher bills. Another reason is that these EC2 instances need to be administrated, which requires time and resources.
For these reasons, we came to the conclusion that using Fargate is be a better solution for dealing with our deployments and being able to scale them (either up or down) correctly. With Fargate, you don't need to worry about instances or servers, all you have to do (in a nutshell) is write your Kubernetes Manifests, apply these, and AWS will take care of the rest; i.e. provisioning the servers, planning the pods, etc.
Challenges on the way
Apart from the choice that we had to make between EC2 and Fargate instances, we had an issue that wasn't that easy to deal with: namely, the Volumes. During our deployment, we found out that we had to pass a configuration to our Basilisk Command, which could not be stored in a config-map, since the configuration was more than 4MB in size, whereas config-maps can only store up to 1MB. Now the problem is that while this is something pretty straight forward to do in Kubernetes (create a Volume, put a file or folder inside and use it from other pods) when using EC2 instances, it is not that simple with Fargate. Fargate only started supporting Volumes in August 2020, meaning that the feature is still somewhat immature. So if you are heavily reliant on Volumes in your Kubernetes Deployment, please take this into account. Eventually, we were able to solve this issue using AWS EFS following this documentation.).
Other technologies used
Finally, we should say a couple of words about the other technologies which played a role when setting up the automated deployment of our testnet.
Our Infra as Code (we actually like to call it Infra as Configuration!) is based on Terraform. For instance, we use it to automate the creation of our ACM Certificates and our Route53 Entries to expose the Nodes after creation à la volée.
Our CI/CD is based on Github Actions. This was the CI/CD tool from the beginning, and we have not found any limitations up to the present date, at least as far as our use cases are concerned.
It is worthwhile mentioning that we also automated our build infrastructure using Terraform and Github Actions. Every push to a branch spawns a dedicated server to accelerate the build (the built-in GH nodes can be very limiting for a Substrate project), and to ameliorate the collaboration between our contributors.
Key takeaways
Here is a summary of the 3 key takeaways from our testnet infra journey:
Think scalability along with simplicity. Do not always make the easiest choice (choosing the simplest one is fine and always a good idea, though).
Take into account the maturity of a technology first, then its features. You don't want to struggle with a matter that no one can help you with.
Always choose to focus on what makes your product great, rather than reinventing the wheel (if you have an alternative of using a Cloud Managed Service and you can afford it, please do so. No need to administer your own servers, unless really necessary).
Epilogue
We hope that the insights we shared in this article will inspire you on your own DevOps journey in Substrate.
If you are building your own parachain, we would like to invite you to come join our Testnet party! There's room for plenty, and XCMP fun is guaranteed.
Please join our 📞 Hydration Discord 📞 and get in touch with us. Hydration a growing community of passionate builders and researchers who hang out together and do cool stuff.