The CI/CD (Continuous Integration/Continuous Deployment) pipeline is a major ingredient in the DevOps recipe. As a DevSecOps practitioner, you must consider the security implications for this pipeline. In this article, we’ll look at the main things to consider when it comes to DevSecOps and CI/CD.
The type of CI/CD pipeline you choose, whether managed, open source, or a bespoke solution you build in-house, will affect the availability of certain out-of-the-box security features or the need particular attention to their implementation.
Let’s dive in
Secrets management for your CI/CD pipeline
Your CI/CD pipeline holds the keys to the kingdom: it can provision infrastructure and deploy workloads to your system. From a security perspective, the CI/CD pipeline should be the only way to perform these actions. To manage your infrastructure, the CI/CD pipeline needs credentials to access cloud service APIs, databases, service accounts, etc., and those credentials must be secure.
Managed or hosted CI/CD pipelines provide a secure way to store these secrets. If you build your CI/CD solution, you are responsible for ensuring that secrets are stored securely. CI/CD secrets should be encrypted at rest and only decrypted in memory when the CI/CD pipeline needs to use them.
You should tightly lock down access to your CI/CD pipeline configuration. If every engineer can access these secrets, the potential for leaks is enormous. Avoid the temptation to let engineers debug and fix issues using CI/CD credentials.
Some secrets (for example, access tokens) need to be refreshed periodically. CI/CD pipelines often use static secrets, which have much longer lifetimes and therefore do not require regular refresh, to avoid the complexity of token refresh.
Inject secrets into workloads
Cloud workloads themselves also use secrets and credentials to access other resources and services that their functionality depends on. These secrets can be provided in several ways. If you deploy your system as packages using VM images or containers, you can embed the secrets directly into the image, making them available in a file when the payload runs. job.
Another approach is to encrypt secrets and store them in source control. Next, inject the decryption key into the workload, which can then retrieve, decrypt, and use the secrets.
Kubernetes allows secrets that are managed outof the workload image, but exposed as environment variableor one to file. One of the advantages of secrets as files is that rotating secrets does not require redeploying the workload.
Infrastructure as Code: A Security Perspective
Infrastructure as code is not just an operational best practice; it is also a good security practice.
software systems = infrastructure + workloads
When ad hoc changes are made to infrastructure configurations, this drift can introduce security risks. When resources are provisioned without auditing or governance, it becomes difficult to maintain proper security measures across all resources.
Manage your infrastructure like you manage your code. Use declarative configurations (like those in Terraform, AWS CloudFormationWhere CRD Kubernetes). Review and audit every change.
Bring your own security tools
CI/CD pipelines are flexible. Generally speaking, they allow you to perform a sequence of steps and manage artifacts. The steps themselves are up to you. As a security engineer, you should take advantage of the security tools that already exist in your environment (especially in the cloud). For example, both GitHub and GitLab scan your commits for the presence of secrets or credentials. Some managed CI/CD solutions incorporate API scans or application security scans. However, you may also prefer to add tools and checks into the mix.
You can also add static code analysis (like SonarQube) to ensure that the code adheres to conventions and best practices. As another example, you can integrate vulnerability scanning (like trivialWhere nibble) to your CI/CD pipeline, checking container images or third-party dependencies for security vulnerabilities.
Comprehensive detection and response
Application observability, monitoring, and alerting are core concerns of DevOps Day 2. Although your CI/CD pipeline is not directly involved in these activities, you should use your CI/CD pipeline to deploy the tools for security you use for these purposes. From a CI/CD pipeline perspective, these are just additional workloads to deploy and configure.
Your CI/CD pipeline should include early detection of security issues that trigger with every change affecting workloads or infrastructure. After the changes are deployed, you should run periodic checks and respond to events that occur after the deployment.
In case of defective CI/CD, break the glass
The CI/CD pipeline is an essential part of your system. If your CI/CD is broken or compromised, your application may continue to run, but you lose the ability to safely make changes. Large-scale applications require constant updates and modifications. In the event of a security breach, you need to be able to safely shut down and isolate parts of your application.
To do this, your CI/CD pipeline must be highly available and securely deployed. Whenever you need to update, restore or redeploy your application, you depend on your CI/CD pipeline.
What should you do if your CI/CD pipeline is broken? Prepare in advance for such a case, determining how your team and system will continue to operate (at reduced capacity most likely) until you can repair your CI/CD pipeline. For complex systems you should have runbooks. Test how you will operate when the CI/CD is down or compromised.
The CI/CD pipeline is at the heart of the DevOps process. When adding security to the mix, you should be aware of the implications, paying particular attention to the security of the CI/CD pipeline itself, and the secrets and artifacts it consumes and produces. The security features that work best for you will be those that match the type of CI/CD pipeline you have chosen. Once you know the key areas to consider when it comes to DevSecOps and CI/CD, you can make that bespoke selection with confidence.
Dive into CI/CD pipelines on the Cisco Developer Blogthen explore DevNet CI/CD Sandbox.