LAS VEGAS — When it comes to the software supply chain, organizations shouldn’t just worry about external threats like poisoned open source dependencies.
During a Black Hat 2022 session on Wednesday, cybersecurity vendor NCC Group presented findings from studying continuous integration and continuous delivery software pipelines across numerous organizations over the past five years. Companies have embraced CI/CD pipelines in recent years to increase the speed and efficiency of software development.
As supply chain security has become a concern for companies amid growing reports of poisoned software libraries and compromised repositories, NCC Group researchers argued that the pipelines themselves pose a significant risk. for organizations.
During the Black Hat presentation, Iain Smart, Principal Security Consultant at NCC Group, and Viktor Gazdag, Security Management Consultant at NCC Group, explained how they were able to compromise virtually every CI/CD pipeline they have tested over the past five years, with very few exceptions. They described how gaining unauthorized access to pipelines, which are often not properly segmented and hardened, could allow threat actors to turn them into “remote code execution as a service” threats. ” that deliver signed malware instead of enterprise applications.
In an interview with SearchSecurity, Smart said CI/CD pipelines are a dangerous attack surface that hasn’t been realized in a long time. But that might not last, as threat actors continue to search for weak spots in enterprise environments, especially around the software supply chain.
“I think a lot of it comes down to a lack of threat modeling, which is a formal way of saying you just don’t think about what you’re deploying,” he said.
CI/CD pipeline attacks
Smart and Gazdag detailed several customer engagement where they were tasked with compromising a CI/CD pipeline and then breaking out of the pipeline to gain access to a company’s core infrastructure.
Smart described an engagement where the client granted NCC Group researchers a set of developer credentials, which lacked admin privileges, to see if they could get out of the pipeline. Although they only had access to a single Bitbucket repository, they were also able to modify the dependencies necessary for the pipeline to build an application.
Smart’s team added their own malicious addiction to the mix, and when the pipeline connected to it, NCC Group gained access to a Jenkins runner. While the runner had a limited environment, Smart said NCC Group discovered a private SSH key hiding there for unknown reasons. With the key, researchers were able to access all Jenkins control nodes in the organization and obtain all secrets (which NCC Group described as non-public data such as credentials and API keys) on all the client’s Jenkins projects.
“In this case, it turned out that we had full access to deploy whatever we wanted into the customer’s production workload,” including malware, Smart said.
In another red team engagement, NCC Group compromised a client who Smart said was a “relatively difficult target.” The researchers were unable to access the CI/CD pipeline through their usual methods, but they managed to phish a developer and used their credentials to gain access to a wide range of repositories.
Even though the customer had hardened their development environment and restricted permissions and access, the red team was able to modify some of the pipelines functions and recover secrets that had not been filtered out. Eventually, the researchers found a single pipeline that had domain admin access, which gave the red team significant control.
“That shouldn’t be said in 2022 — don’t do everything as a domain admin, please,” Smart said. “It’s just not a good idea.”
Gazdag and Smart discussed other customer engagements where the research team was able to break out of a compromised pipeline not only in the full product environment, but also in customer cloud and on-premises infrastructure. In one case involving a particularly well-defended client with extremely limited developer access, Smart was able to obtain production API keys and modify the company’s AWS environment.
Smart said the client initially denied such an attack was possible and was so confident that it allowed him to execute the attack. “Two redacted screenshots later, we were on an incident response call,” he said.
He also said that code signing software in production is a good practice, but it is not a magic bullet to prevent malicious activity. In fact, he said, it could be used by threat actors to create an RCE-as-a-service.
“A lot of people think, ‘Well, I’m signing at the very end of my pipeline, so I’m safe,’ and they’re pretty much right,” he said. “But if an attacker compromised your pipeline before this step, then congratulations – you just programmatically signed my malware, and now everyone is going to trust it more.”
Lessons learned
Gazdag pointed out that none of the CI/CD pipeline engagements that NCC Group has worked on involved exotic exploits or new techniques. “None of these issues are new, and we haven’t used any zero days,” he told the audience.
In an interview with SearchSecurity, Gazdag said there were a few customer engagements where the research team couldn’t get into the pipelines. “When we couldn’t compromise a pipeline, it was because the company had the security controls in place that we recommended or they had already deployed,” he said.
These controls include relatively simple steps such as segmenting production environments, applying the principle of least privilege, using role-based access control, and configuring alerts and logging. suitable for pipelines. Gazdag and Smart also encouraged organizations to limit pipeline access to external dependencies that have been validated.
Smart told SearchSecurity that many of the customers NCC Group worked with did not fully understand the scale of the attack surface of their CI/CD pipelines and the extent of the damage that could be caused if compromised. .
“These pipelines are, by their very nature, privileged. The more automation you add, the more privileges you need to empower your automation,” he said. “If you have a box that can automatically do things for you, and someone compromises the box, then that attacker has all the permissions the box had.”