The project relies on EC2 instances, Auto Scaling, CodeDeploy, and CI/CD pipelines to ensure scalability, availability, and continuous code updates.
Specifically, the project includes:
- Amazon EC2 (Elastic Compute Cloud): the AWS service for creating and managing virtual servers in the cloud.
- Auto Scaling: a mechanism that automatically launches and/or terminates EC2 instances based on the workload.
- AWS CodeDeploy: an AWS service that automates software deployment across multiple servers.
- CI/CD pipeline: an automated process for testing, validating, and releasing code quickly and reliably: CI (Continuous Integration) refers to developers continuously integrating their code, while CD (Continuous Delivery/Deployment) covers automated deployment to test or production environments.
During releases of new application versions, errors occasionally occurred that interrupted the automated software deployment process.
The context
Applications are released using the CodeDeploy blue/green deployment type. This mechanism creates a new version of the Auto Scaling group with EC2 instances running the new software release. At the end of the deployment, the new instances gradually replace the previous ones, which run the old version. The configuration files CodeDeploy needs are versioned in a Git repository, from which the CI/CD pipeline retrieves them during the release process.
The problem
During releases of some applications, CodeDeploy occasionally failed the deployment and sent the corresponding error email:
These failures called for an in-depth analysis to identify their causes and implement an effective solution.
The objectives
The main objective was to pinpoint the cause of the deployment failures and implement a solution that would make the process more robust, preventing future errors tied to non-essential dependencies.
The solution: EC2 Auto Scaling and deployment with CodeDeploy
Troubleshooting proceeded in several phases.
1. CodeDeploy analysis
We identified the exact phase in which the failure occurred: the initial one, before CodeDeploy even ran.
2. Auto Scaling
To create the EC2 instances, Auto Scaling uses a launch template that defines parameters such as the AMI, instance type, network, security groups, user data, and so on.
3. CodeDeploy agent
The CodeDeploy agent is downloaded and installed in the final stage of the user data script, which suggested that something might have gone wrong there.
4. User data analysis
Checking the user data logs revealed problems with the custom EPEL (Extra Packages for Enterprise Linux) repositories, which had been installed for some optional dependencies (weak dependencies) and for custom fonts the application needs.
Specifically, the user data script was trying to access the EPEL repositories.
However, the repositories were temporarily unreachable, causing checksum mismatch errors and preventing the installation process from completing, so the CodeDeploy agent was never installed. This interruption prevented the deployment from starting, leaving the instance unable to receive application traffic.
Fix applied
Once we had clear evidence of the problem, we changed how the custom fonts are installed and configured.
- Removed the EPEL repositories from the user data script, since they were no longer needed.
- Moved the custom fonts and their configuration into the Git repository, so they are available and installed through the CodeDeploy stages instead of in the user data.
The results
Here are the results after the change was applied.
- More stable deployments, a few minutes faster.
- No more errors related to package downloads.
- Shorter overall release times.
- No new failures recorded in subsequent deployments.
Final recommendations
This case taught us an important lesson, from which we draw the recommendations below.
Writing user data and CodeDeploy configurations carefully is essential.
Setting up the various AWS services in an orderly, clear, and functional way avoids redundant installations or configurations and optimizes the infrastructure. This approach not only keeps the environment lean but also helps prevent malfunctions caused by unnecessary components.
A structured, analytical approach to problems is essential.
Progressively narrowing down the possible causes until we found the source of the problem proved decisive in resolving the failures. This shows the value of a technology partner with solid expertise.
Environment standardization
Having separate, well-defined environments (development, test, and production) is essential to software quality and reliability. It is just as essential, however, that these environments be aligned as closely as possible in configurations, versions, dependencies, and infrastructure.
Only then can problems be caught early, during development or acceptance testing, greatly reducing the risk of unexpected errors in production. Consistent environments make troubleshooting more effective and allow changes to be validated in realistic conditions, minimizing outages and costs.
The importance of monitoring
Setting up targeted metrics and alarms makes it possible to detect and resolve issues promptly.
It is this combination of technical expertise and a methodical approach that makes Miriade a reliable, strategic partner, able to handle even complex challenges effectively, always focused on stable, scalable solutions tailored to each customer's specific goals.
