Terraform and Multi-Tier Architecture

This article was written by Ironclad’s renowned team of software engineers. 

In general, every organization prefers to have multiple environments, some run production workloads, and a few more run development+test environments. In infrastructure management, one of the major asks from all stakeholders is to have a similar setup across all environments such as production, staging, development, etc, from the perspective of the network, application runtimes and monitoring, etc, while allowing environments to differ in capacity such as size or number of machines. Having a similar environment setup would help engineering teams to detect any infrastructure or application runtime issues before it goes to production.

In all major cloud providers (GCP, AWS, Azure, etc.), it’s easy to provision a network, virtual machine, or a cloud service like Kubernetes, SQL, etc., with a few clicks on their UI portal or through CLI/API. Manually provisioning multiple environments with a consistent configuration is not a viable option in the long run. But if the entire environment could be released as a versioned piece of code and can be provisioned by a simple execution, then it would alleviate the manual work and consistency issues.

Now, the infrastructure team would like to create multiple environments that all look alike and are provisioned by a piece of code. Various technologies exist today that can be used to provision environments via code like terraform, chef, puppet, etc. After various technical evaluations of these technologies, Terraform is proven out to be a good choice to provision an environment as it has support for all major public cloud providers with great community support and allows us to define infrastructure as a code. 

Defining each environment as a terraform code doesn’t solve the problem of creating look-a-like environments. For example, the infrastructure team can simply create a git repository for each environment and add all terraform code for provisioning and configuration in there, intending to keep all these environment-specific repositories in sync.

This may work initially, but code in these repositories gets drifted over time, and they may very well look completely different. One solution to solve this problem is to create a templating system using terraform whereby filling up a template with the required information would simply spit out an environment. Various design patterns have been implemented to create an efficient templating system and one such example would be the Template method design pattern in java using inheritance. 

Let’s examine some well-known design patterns/concepts implemented in software engineering and check if some of them can be used to design an efficient templating system in Terraform.

• Looking at a typical multi-tier architecture of a web application(Model-View-Adapterpattern), the model is responsible for data/state, the view represents the presentation, andan adapter to mediate the interactions between these two. In this design, the view is oblivious to the model and vice versa to have a clear separation of concerns, and the adapter layer can be extended further to more layers to create data flows.

multi-tier architecture of a web application(Model-View-Adapterpattern)

• Another popular principle of software development is composition. In this approach, a large complex object/task/function can be constructed from a combination of smaller objects/tasks/functions. These smaller tasks can be completely encapsulated and do a small scope of work. This would give a lot of flexibility to change implementations without breaking the system. In the below UML representation, class A is composed of classes X, Y & Z

UML representation

• One more popular principle in functional programming is pure functions. In pure functions, the output of the function is solely dependent on the inputs and any mutation of non-local variables isn’t allowed. This would allow us to write a clear composable code that is idempotent, testable, cacheable, and highly parallelizable.

input and output of pure functions

If we can incorporate some of the above principles into the terraform design system, we can create a highly scalable and cross-functional design irrespective of the cloud infrastructure type.

Considerations

• Infrastructure as a code
• Platform agnostic
• Work across all cloud providers as well as traditional data centers
• All the code is versioned and managed inside a version control system like git
• Deployed through an automated system
• Should follow all standards of traditional Software development life cycle (SDLC)
• Adhere to SOLID principles

Plan

Creating an isolated environment would typically consist of provisioning of the components below (not an exhaustive list, just to name a few)

• Organization
• Account/Project
• Virtual private cloud (VPC)
• Network components such as NAT gateways, routers, peering connections, etc…
• Security components such as security groups/firewalls, IAM roles/members, Service accounts, etc
• Application runtime components such as databases, Kubernetes clusters, compute nodes, Redis cluster, queues, object storage buckets, ETL pipelines, etc…
• Monitoring components for both infrastructure and application such as logs, metrics, alerts, traces, etc
• Deployment system to deploy applications

Creating some of the above services would require the availability of others but as long as dependencies are satisfied, each service can be provisioned explicitly and independently via API/CLI calls in the cloud. For example, the creation of a database would require the availability of a network but once it is available then we can create many databases with their specific configuration via a terraform resource declaration/API/CLI. 

Having this above notion, we can split the provisioning of an infrastructure into multiple tiers/layers.

A diagram to visualize this tiered structure

tiered structure visualization

Tier1

This layer comprises a terraform module for each service with all dependencies configured as input parameters and is responsible for provisioning the service and its state without mutating any global state or input parameters. This would give us smaller composable functions that are only responsible for managing that one service alone. This function encapsulates all internal provisioning details of the service. These pure functions/TF modules would resemble the model in a model-view-adapter pattern. This layer can be just called “service_definition”. Every service definition can be versioned and tested independently.

Tier2

After having these individual pure functions for each of the services, this tier would serve as a controller layer where all functions can be composed together. Dependencies among services/functions can be defined here to ensure the order of the execution, input parameters, validations, and appropriately passing them around across functions. Terraform modules that correspond to this tier would resemble a template that produces a set of components (ex: network, organization, application runtimes, etc…) with the given inputs. Here a component can be defined as a combination of multiple services. This layer can be called a “component_defnition” (ex: org_definition, network_definition, application_definition, etc …). The splitting of an environment into multiple components is dependent on how small the terraform state file should be. Having many services inside a single component would lead to a huge state file and longer terraform runtimes. Every component definition can be versioned and tested independently.

Tier3

This is the top layer that corresponds to “view”, which represents the actual environment. Here the entire environment-specific configuration is either supplied or created. This layer would compose versioned components together to create a user-defined environment.

Each tier has a well-defined scope and can be used as a stand-alone. We can create more tiers by just composing the previous tier’s terraform modules into a single module. This is a very powerful scheme where pure functions serve as a base and all subsequent layers use the composition principle to define an environment. This would allow us to run all terraform provisioning with high concurrency and less erroneous possibilities.

Versioning

All service and component modules are versioned following the semantic pattern as detailed on the multi tier-ed diagram. A versioned component definition can pick and choose all service definition versions that form it and an environment that comprises multiple components can choose their versions as well. This would give the flexibility of releasing new changes sequentially one environment at a time and better validation.

Over the last couple of months, we have successfully tested the above design in our GCP Environment, stay tuned for the implementation blog post. In addition, Ironclad also has an AWS Environment where we will be implementing the same design & share it with you all.

If this sounds exciting to you and you’d like to help the Platform Engineering team to deliver even a better experience with automation and internal tooling, Ironclad is always growing our platform engineering team. Please click here for more information.

Nari Mulakala, Himay Desai, Nate Schlitt, Magnus Thorne,John Fiedler

Outside Ironclad

Salesforce: Raymond Sun, an Architect from Salesforce, helped collaborate on the overall design. Special thanks to Raymond!