How we improved productivity with an internal CLI?

At Qonto, we want all engineering teams to be 100% autonomous because we believe that if a team can work independently, they’ll be able to ship new features, improvements & bug fixes faster. The Platform Reliability team aims to ensure systems are resilient and scalable for developers & customers.

The resilience of our production system parallels how we run the Qonto platform. Robust automation is much more powerful than manual operations that require advance training. As a result, we do our best to set up automatic processes across other teams, like Security, Data, or Backend to save ourselves time and effort.

Here’s a concrete example of how we set up a quick automation: one of our teams needed on-demand, temporarily readable replicas of a PostgreSQL database to run expensive queries. One way of going about it would have been to ask the SRE team (through Slack or a ticketing system) to manually create a replica every time they need a replica and use the same process to delete it once it was used. This process is hard to scale, so instead we built a script allowing developers to create and manage these replicas autonomously.

One issue we faced with this approach was having scripts stored in multiple places. Some scripts were stored in a dedicated Git repository, some lived inside project repositories, and some weren’t shared at all or through unconventional channels. In short, they were hard to find. Having to track down a script — that could be in one of many places — every time you want to run a command is not efficient. Not to mention it runs the risk of creating double work: people were sometimes completely rewriting a script that existed elsewhere because they didn’t know about it or couldn’t find it.

Another common issue with standalone scripts is that they are not robust enough, or poorly tested. They often don’t implement software development best practices and are not included in a continuous integration platform. This makes them hard to maintain in the long term, subject to bugs, or mastered by only a few people.

This is why we built qontoctl at Qonto. Qontoctl is a simple CLI meant to house all of our scripts at the same quality standard as all our code bases, covering essential aspects such as security and user authentication.

Language choice

When we launched the project, we had to decide which programming language to use to build it. Infrastructure teams often use the shell language (or a derivative like Bash) for their scripts. We chose not to follow this path.

The main issue with shell is that it lacks many useful features that are available in full-fledged programming languages:

• Strong typing: shell variables and functions are untyped. It’s easy to inadvertently introduce typing bugs which would otherwise be caught in the compilation phase with another language. Manipulating basic data structures like lists or hash maps is also incredibly tricky. You also can’t define custom types in shell. This becomes an issue when, for instance, dealing with external APIs and trying to manipulate request and response payloads.
• Error handling: Instructions like set -eto exit on errors or using || to handle errors on commands is possible but not efficient compared to what modern languages allow.
• Code abstractions and design patterns: A lot of development best practices can’t be implemented in shell, which doesn’t have notions of interfaces or protocols, for example.
• Dependency management: Modern programming languages have a vast ecosystem of libraries, and dependencies management is integrated into the build system. It’s also easy to create and share new libraries if needed.

All in all, using a full-fledged programming language brings a lot of benefits without meaningful drawbacks. We can work on the CLI as we would work on regular programs writing tests, using linters, and integrating them into a continuous integration pipeline.

We chose the Go programming language to build qontoctl for several reasons:

• The Golang ecosystem is huge, especially in the cloud-native landscape. We also use the powerful Cobra library to manage CLI subcommands and flags. Cobra also allows us to generate CLI documentation in markdown.
• Our Backend Developers use Go heavily at Qonto, so we have a huge pool of potential contributors who already know the language and can reuse existing internal libraries.
• The build output is a static binary, which can be installed easily without any requirements.

We had some scripts written in other programming languages (Bash and Python, for example). Picking Golang and deciding to centralize scripts into a single CLI meant that these scripts had to be rewritten. We’re still in the process of bringing these scripts over to Golang, because we’re advancing step by step, depending on how critical the script is. We also had to train people in Golang and found pair programming to be an excellent technique to do that.

Auto-update

At the outset of qontoctl, users had to install and reinstall the program manually on their computers every time we released a new version. Realizing that a tool is not up to date when using it is frustrating, especially during an emergency like following a runbook during an incident in the middle of the night.

We solved this issue by deploying the qontoctl binary on all Qonto computers, thanks to JamF, our device management system. Each new release is now automatically deployed across our computer network. No one is running an outdated version of the tool anymore.

Use cases

Every team at Qonto is specialized in different topics. We found that having a common CLI is an excellent way to break down knowledge silos and offer this expertise to others. The commands are intuitive, well-documented (with examples) and tested, and users can trust the CLI and use it right away. It’s also important to make sure that people know about the tool and the philosophy behind it. Developing a great tool is great but ineffective if few people know about it or use it. We gave several presentations on our CLI and communicated it widely within the SRE teams and during our Global Tech meetings. We encourage teams to develop new scripts in qontoctl to make sure that our tool is consistent and available company-wide.

The SRE team is, for example, providing features to developers. While our team is rather small, comparatively our Developer teams are growing a lot, so we can’t provide them with tools that require a lot of training. This way, people don’t need to know command implementation directions — they only need to know how to use the CLI interface.

Managing local configuration files

A common issue we had involved local configuration files, for example Kubernetes or AWS-related files. These files should have specific values per team and environment, and often people assigned these files invalid configurations.

Since qontoctl is installed on all Qonto computers, we used it to be able to quickly check (and repair) configuration files. For example, we developed a qontoctl config troubleshoot command which scans all required configuration files for Qontoers and reports invalid configurations. We also created a qontoctl config k8s command which prints the list of commands to execute to configure local kubeconfig files, according to the user team setup.

Terminating or canceling PostgreSQL queries

We extensively use PostgreSQL at Qonto. It can sometimes be helpful to be able to cancel or terminate a query running on a database, for example, due to a bug on a query that makes it too long to execute. Doing this procedure manually is painful: users need specific permissions on PostgreSQL, find the proper instance to log into, and then cancel or terminate the query using SQL commands (for instance, pg_cancel_backend). Doing this during an incident was slow and prone to human errors.

To counter this, our team built an API capable of canceling and terminating queries. Fine-grained permissions can be configured by user and/ or team, and the API helps us audit performed actions if needed.

We then build new qontoctl commands (qontoctl rds cancel-query and qontoctl rds terminate-query), which use this API and allow us to resolve potential incidents in seconds.

Migrating development environments between Kubernetes clusters

We have multiple Kubernetes clusters where we can deploy full-featured development environments in less than 3 minutes.

Consequently, we wanted to be able to move given environments between clusters. Instead of doing it manually, which is time-consuming and complicated, we created a qontoctl feature-branch move command that does exactly that.

Conclusion

We use many managed services at Qonto, especially from the AWS catalog. Using these services significantly simplifies infrastructure management and gives us time to invest in improving the quality of our internal tools, not only for us but for the whole Tech department. Building a CLI was part of this effort, and qontoctl is slowly becoming the unique entry point to perform actions on our environments.

Today, Qontoctl provides more than 30 commands ready to be used by tech teams. Several teams already contributed to the project. For example, our Security team merged several tools they provide to developers into qontoctl in order to simplify their distribution and usage. New projects providing on-demand capabilities to teams now always start with a qontoctl integration. The goal of the Qonto Platform team was to create a tool to which every team can contribute autonomously, which is the case today.

This project is also an important reminder that teams working on infrastructure topics should use the same tools and practices as Backend Developers when they need to produce code. Having quality tools is the key to making scaling them and increasing the productivity of the people using them.

About Qonto

Qonto is a business finance solution designed for SMEs and freelancers founded in 2016 by Steve Anavi and Alexandre Prot. Since our launch in, Qonto has made business financing easier for more than 250,000 companies.

Business owners save time thanks to Qonto’s streamlined account set-up, an intuitive day-to-day user experience with unlimited transaction history, accounting exports, and a practical expense management feature.

Entrepreneurs stay in control while being able to give their teams more autonomy via real-time notifications and a user-rights management system. They benefit from improved cash-flow visibility by means of smart dashboards, transaction auto-tagging, and cash-flow monitoring tools. They also enjoy stellar customer support at a fair and transparent price.