Infrastructure as Code (IaC)
Developing Modular Infrastructure for Environment Consistency
Master the creation of composable modules in Terraform and CloudFormation to reduce duplication and enforce standard resource configurations.
In this article
The Evolution from Manual Triage to Composable Infrastructure
In the early days of cloud computing, infrastructure was often managed through a series of manual clicks in a web console or by running brittle shell scripts. While this approach worked for small projects, it quickly became a liability as organizations scaled their digital presence to hundreds of services and environments. Manual changes are difficult to track, impossible to audit effectively, and almost always lead to configuration drift where your actual resources no longer match your documentation.
Infrastructure as Code solved the problem of repeatability by allowing us to define our data centers in the same way we define our software. However, many teams fall into the trap of creating monolithic files that contain every single resource for an entire application. These large files become difficult to navigate and even more dangerous to modify because a small change in one area can have unintended side effects across the entire stack.
Composability is the architectural principle that allows us to break these large systems down into small, self-contained units called modules. Instead of one giant template, you build a library of reusable building blocks such as networking modules, database modules, and application cluster modules. This approach mirrors modern software engineering practices like microservices and object-oriented programming.
By focusing on composable modules, you can establish clear boundaries of responsibility within your engineering team. A security team can maintain a hardened virtual private cloud module while application developers consume it as a black box. This separation of concerns reduces cognitive load and allows teams to move faster with higher confidence in their deployments.
Understanding the Why of Modularity
The primary goal of modularity is to reduce duplication and enforce organizational standards across different environments. If every team in your company is responsible for writing their own firewall rules from scratch, you will inevitably end up with inconsistent security postures. Modules allow you to bake your security policies, tagging requirements, and logging configurations directly into the template.
Another critical benefit is the reduction of the blast radius when things go wrong. In a monolithic configuration, a simple error in a load balancer resource might cause the entire stack to fail its update or even roll back critical database changes. When infrastructure is partitioned into modules, failures are often isolated to a specific component, making recovery much faster and less stressful.
Architecting High-Quality Modules in Terraform
Terraform is one of the most popular tools for managing infrastructure because of its expressive syntax and powerful module system. A module in Terraform is simply a directory containing one or more configuration files that work together to perform a specific task. To make these modules truly composable, you must treat them as an application programming interface with strictly defined inputs and outputs.
Good module design starts with variable definitions that use clear types and descriptive help text. You should avoid hardcoding any values that might change between environments, such as region names, instance sizes, or environment tags. Instead, expose these as variables so that the consumer of the module can tailor the behavior to their specific needs without modifying the internal code.
1# variables.tf
2variable "db_instance_class" {
3 description = "The size of the database instance"
4 type = string
5 default = "db.t3.micro"
6}
7
8variable "allocated_storage" {
9 description = "Storage size in gigabytes"
10 type = number
11 default = 20
12}
13
14# main.tf
15resource "aws_db_instance" "application_db" {
16 instance_class = var.db_instance_class
17 allocated_storage = var.allocated_storage
18 engine = "postgres"
19 engine_version = "15.4"
20
21 # Enforce encryption by default in the module
22 storage_encrypted = true
23
24 # Ensure proper cleanup during deletion
25 skip_final_snapshot = false
26 final_snapshot_identifier = "final-db-snap-${var.db_instance_class}"
27}
28
29# outputs.tf
30output "db_endpoint" {
31 description = "The connection string for the database"
32 value = aws_db_instance.application_db.endpoint
33}In the example above, the module encapsulates the complexity of setting up an encrypted database while providing a simple interface to the user. Notice how the storage_encrypted attribute is set to true within the module code rather than being a variable. This is a common pattern for enforcing organizational compliance where certain security features are non-negotiable.
Outputs are just as important as inputs because they allow you to pass information from one module to another. For instance, a networking module might output the unique identifier of a virtual private cloud, which is then passed as an input to a database module. This chaining of modules creates a dependency graph that Terraform uses to determine the correct order of resource creation.
The Pattern of Dependency Injection
One common pitfall is hardcoding resource lookups inside a module, which makes the module less portable and harder to test. Instead, you should practice dependency injection by passing resource identifiers or configuration objects into the module from the outside. If a module needs to place an instance inside a specific subnet, the subnet identifier should be an input variable rather than something the module tries to find on its own.
This pattern makes your code significantly easier to refactor because the module doesn't care about the implementation details of the network it sits in. It simply expects a valid identifier. This level of abstraction allows you to swap out the underlying network infrastructure entirely without ever having to touch the application-level module code.
Mastering Abstraction in AWS CloudFormation
While Terraform uses a directory-based module system, AWS CloudFormation utilizes a concept called Nested Stacks to achieve modularity. A nested stack is a resource within a parent template that points to another template file stored in an S3 bucket. This allows you to build a hierarchy of templates where a master root stack orchestrates the deployment of several child stacks.
The primary advantage of nested stacks in CloudFormation is the ability to bypass the individual template size limits. Large, complex environments can easily exceed the maximum character count for a single file, but by nesting stacks, you can spread the configuration across dozens of independent files. This also makes the CloudFormation console much easier to read as it displays the parent-child relationships clearly.
1AWSTemplateFormatVersion: '2010-09-09'
2Description: Parent stack for a multi-tier web application
3
4Resources:
5 NetworkStack:
6 Type: AWS::CloudFormation::Stack
7 Properties:
8 TemplateURL: https://s3.amazonaws.com/my-templates/network.yaml
9 Parameters:
10 VpcCidr: 10.0.0.0/16
11
12 DatabaseStack:
13 Type: AWS::CloudFormation::Stack
14 Properties:
15 TemplateURL: https://s3.amazonaws.com/my-templates/database.yaml
16 Parameters:
17 # Injecting the VPC ID from the network stack output
18 VpcId: !GetAtt NetworkStack.Outputs.VpcId
19 DbSubnetIds: !GetAtt NetworkStack.Outputs.PrivateSubnetsPassing data between these stacks is handled through parameters and outputs. When the network stack finishes creating the virtual network, it publishes the network identifier as an output. The parent stack then captures that value and feeds it into the database stack as a parameter, ensuring the database is placed correctly within the newly created network.
CloudFormation also offers a newer feature called Modules, which allow you to register a collection of resources in the CloudFormation Registry. Once registered, these modules can be used in your templates just like native AWS resource types. This is particularly useful for building small, reusable snippets like a standardized security group or an encrypted S3 bucket that needs to be used across hundreds of templates.
Nested Stacks versus Cross-Stack References
Developers often struggle with the choice between using nested stacks and using cross-stack references via exports. Use cross-stack references when you have a shared resource that has its own lifecycle, such as a core networking layer used by multiple different application teams. This allows the network team to update their stack independently of the applications that consume it.
In contrast, use nested stacks when the resources are part of a single application and should be created or deleted together. Nested stacks create a strong bond between the components, meaning you can manage the entire application lifecycle from a single entry point. This provides a much tighter feedback loop during development and simplifies the process of tearing down staging environments.
Strategic Trade-offs and Best Practices
While modularity provides many benefits, it is possible to over-engineer your infrastructure by creating too many small, granular modules. Every module you create introduces a new layer of abstraction that your team must learn and maintain. If a module only contains a single resource with no added logic or standardization, it might be adding more complexity than it is worth.
A good rule of thumb is the rule of three: if you find yourself writing the same resource configuration in three different places, it is time to abstract it into a module. Until that point, keeping the configuration inline can actually make the code more readable by keeping related resources close together. Always prioritize clarity over cleverness when designing your infrastructure architecture.
- Maintain clear versioning for your modules to prevent breaking changes from affecting production environments.
- Keep modules focused on a single responsibility like compute, storage, or networking.
- Avoid deep nesting of modules beyond two or three levels to keep the dependency graph manageable.
- Include a README file for every module explaining the required inputs and the expected outputs.
- Use semantic versioning for your module releases so consumers know when to expect breaking changes.
Testing your modules is another critical aspect of building composable infrastructure. You should treat your infrastructure code like application code by implementing linting, static analysis, and automated testing. Tools like TFLint for Terraform can catch common configuration errors before you ever try to deploy them to a real cloud provider.
For more advanced validation, you can use integration testing frameworks that actually spin up the resources in a sandbox account and run assertions against them. While these tests are slower and more expensive, they provide the highest level of confidence that your module will perform correctly in a production environment. This is especially important for modules that handle critical data or security configurations.
Managing the Lifecycle of Changes
Infrastructure modules are living documents that will evolve as your cloud provider releases new features or your security requirements change. When you need to make a breaking change to a module, you should use a versioning strategy that allows existing users to continue using the old version while new projects adopt the new one. This is typically done by tagging your Git repository with semantic version numbers.
By pinning your infrastructure to specific module versions, you ensure that a change made by one engineer doesn't accidentally trigger a destructive update in an unrelated project. This stability is the foundation of a reliable deployment pipeline. It allows you to test changes in isolation and roll them out across your fleet with controlled, predictable steps.
The Blast Radius and Safety First
The most dangerous moment for any infrastructure is not during the initial build, but during an update of a shared component that has hidden dependencies across the stack.
Always visualize your dependency graph before applying major changes to core modules. In Terraform, you can use the graph command to generate a visual representation of how your modules are connected. In CloudFormation, reviewing the change set before executing an update can save you from catastrophic downtime by highlighting which resources will be replaced rather than updated.