Building a software-defined engineering platform

In the next few blogs, we will dive into applying platform engineering practices to create the foundational parts of an engineering platform for an imaginary company. These foundational components are crucial for any effective platform.

You’ll see how:

• Platform domains relate to the pipelines we build
• We can extend the Kubernetes control plane for more value
• And what a self-serve user experience should look like

Let’s assume this will be a brand-new product. While this is a common approach for learning exercises, there are plenty of good reasons for an enterprise to consider doing the same when building an engineering platform. The most pragmatic reason is described best by Gall’s Law:

By now, nearly every organization has been using many, if not all, of the various technologies that will go into an engineering platform. They have teams scattered everywhere using Kubernetes, creating infrastructure using Terraform, using Git, and “doing DevOps.” 

These implementations are often difficult to adopt because they’re either too focused on the needs and preferences of a small group of users, or they were developed without user input and are instead optimized for the needs of the team providing the technology.

In other cases, while the implementation is meant for general use, it is managed across many different traditional IT silos where API access and self-serve experiences were not originally considered and by teams without the resources or experience to evolve. Changes need a lot of planning, and long delays are normal. In all these scenarios, user experience is rarely among the top priorities. 

Organizations trying to create a developer platform while sticking to a traditional IT or their original DevOps model spend a lot of time and money, only to realize they are not achieving their goals and have to start over with a unified team in a greenfield setting.

The engineering platform product domains show the basic dependencies and almost exactly the order in which we’ll set up the product infrastructure pipelines. This diagram shows the pipelines we will build in each domain to create the foundation of our engineering platform.

But first, let’s talk about the prerequisites.

The figure above shows resources we will need to begin building our engineering platform product foundation.

Tools included in the platform

Besides the cloud accounts where the platform infrastructure resides, you will notice that these tools are also tools the platform product includes for use by platform customers. As platform engineers, we will use these tools ourselves to deliver the platform. 

Not every potential developer tool is needed initially, so which tools are needed to bootstrap an engineering platform effectively? 

• Source code version control 
• Secrets store 
• Infrastructure state store 
• Pipeline orchestrator 

Starting from scratch, we have a bootstrap challenge. How can we deploy the first configuration in a software-defined manner using source control, managed secrets, state store, and a pipeline if we must first deploy these tools before using them? 

Right away, you can see the accelerating impact of using SAAS tools as the solution to the bootstrap challenge. The reference code examples in the Effective Platform Engineering GitHub organization will demonstrate several highly effective tools. 

• GitHub 
• 1Password 
• Terraform
• Cloud CircleCI 

Using these tools is not required to apply the principles in this book, though it will allow you to get the most from the sample exercise solutions.

In an Enterprise setting, each tool would typically be integrated using an SSO solution. We will talk more about how that fits into the product experience in the Effective Platform Engineering book section on Customer Identity Provider (you can download the entire book here.) 

But for now, let’s go ahead and set up access to the initial tools as an individual user. The respective tool’s product documentation provides detailed instructions for performing the following steps. 

If you do not already have one, create a free personal account on github.com. Then, create a free-tier GitHub Organization for our imaginary VitalSigns company, in which we and all the VitalSigns platform developers will be members. Create a GitHub team called platform-team to represent our product delivery team for the VitalSigns exercises. 

Later, we will use GitHub Teams and team membership to manage access permissions. Add yourself to this team. Create a personal access token (PAT) and be sure to upload your personal SSH keys, and enable support for signed commits.

Create a free-tier Terraform Cloud organization. Go to the settings area (from the left-hand menu) within the organization, create a team, and add yourself to the team. From the team settings page, also generate a team API token. Create a free-tier CircleCI organization and link it with our GitHub organization. Generate an access token. 

Lastly, 1Password offers individual plans for less than $3 per month. Create a dedicated 1Password vault for these exercises then go to the Developer options and generate a service account credential with read and write permissions. 

Or, if you are using some other secrets management tool, also have the access credentials for pipeline usage available. Store all of the above access tokens in this vault. 

Most of the SAAS tools used in the example exercise solutions offer a free tier adequate to cover the exercises in the book or very affordable personal options, and alternative tools will often be discussed. 

Occasionally, we will use alternative tools to demonstrate the differences among effective choices within the example exercise solutions. 

For any tool where you need an access token to use in automation, such as in our pipelines, if the tool doesn’t allow you to create something like a team or organizational level token, where anyone on your team can manage it, then you are left with needing to create a personal access token. This introduces a problem. 

What if you leave the company or take on a new role with a different team? 

Either of these events can cause your personal access token to be revoked, and any automation that depends on the token will break. The two most effective ways of dealing with this situation are: 

Service Accounts. Sometimes called machine users, these are identities created within the appropriate system in the same way as human users, except that no single person has control over the identity. It becomes a team resource with the username and password stored in the Team secrets store for management. Often, systems provide an actual feature designed to support this type of User. We will use an example in AWS later in a later excerpt blog. Just for these exercises, creating and using a personal token when needed is fine. 

OpenID Connect Tokens. Many tools and most cloud resources, such as AWS, also provide a means of establishing direct trust between other tools or cloud resources. This approach requires more behind-the-scenes automation to create a self-service experience for Platform users, but it is an option. 

Next time we’ll cover: Developer tools selection criteria