Why the shift left dream has become a nightmare for security and developers

Written by Ivan Milenkovic, Vice President Risk Technology EMEA, Qualys

For the better part of the last decade,we have engaged in a comfortable fiction around security and development. If we could only “shift left” and get developers to take a modicum more responsibility for security alongside their coding, testing and infrastructure deployment, the digital world would become a safer, faster and cheaper place. Instead, the fundamental conflict between speed and security has got worse.

Why did this fail? Developers are under crushing pressure. The classic triangle of project management – Fast, Good, Cheap; pick two – has been smashed to pieces.

Businesses demand fast, good, cheap and secure. When push comes to shove, “fast” always wins. At the same time, we pushed too much cognitive load onto developers who were already drowning.

When they choose to use public container images to speed up development, they are trying to meet their goals, but they are also open to potential risk. So how can we understand what the real problem is, and then work to solve that?

Business demands beat security recommendations

There is a pervasive narrative in the security industry that developers are lazy or careless. This is absolutely not true. Developers are not lazy; they are overloaded, pragmatic professionals reacting to the incentives placed before them. If their bonus depends on shipping features by Friday and the security scan takes four hours to run and blocks the build, they will find a way around the scan.

Businesses demand results faster and faster, which has created an environment where security protocols are seen as a barrier to productivity rather than an integral part of engineering. When security tools are noisy, slow, and disconnected from the workflow, they are a barrier.

However, the result of this is that organisations have lost control of what is actually running in their environments. We have pipelines that deploy code automatically, infrastructure that scales up and down without human intervention, and AI agents that can now write and execute their own scripts.

Into this high-speed, automated chaos, we treat public registries like curated libraries, assuming that because an image is on Docker Hub, it must be safe. But pulling a container from a public registry like Docker Hub is a trust decision.

The likes of Docker, Amazon, Google and Microsoft all operate public container registries, so there is a natural assumption that they are safe.

This trust is misplaced. By the time that container image makes it to the deployment pipeline, it is already a trusted artifact, baked into the application.

The 34,000 Image Reality Check

Qualys Threat Research Unit (TRU) recently conducted an exhaustive analysis of over 34,000 container images pulled from public repositories to see what is really going on beneath the manifest.

Of that total, around 2,500 images – approximately 7.3 percent of the sample – were malicious. Of the malicious images, 70 percent contained cryptomining software.

On top of this, 42 percent of images contained more than five secrets that could be used to get access to other resources or accounts. This includes valuable items like AWS access keys, GitHub API tokens, and database credentials baked directly into the image layers.

In our analysis, the biggest issues around malicious containers are still very simple. Typosquatting is one of the most common methods that attackers use to get their malicious containers downloaded. The standard advice to “check the spelling” is essential, yes, but it is also a low-energy response to a high-stakes problem.

Telling a developer to “be more careful” is not a security strategy. While public registries are handy for speed, we should not be letting developers pull from public registries at all.

In a mature environment, every external image should be proxied through an internal artifact repository that acts as a quarantine zone. Yet that need for speed is not going to go away. Instead, we have to work on how to help developers move faster while keeping security in place.

This does mean more work for the infrastructure team, but that work should enable developers to move ahead faster and with less risk.

Shift down

The logic is that it is cheaper to fix a bug during design or coding than in production. Therefore, moving security earlier in the Software Development Life Cycle (SDLC) should reduce risks later. While this makes sense in theory, it asks developers to scan their own code, check their own dependencies, and manage their own infrastructure.

In reality, we just shifted the pain onward. It asks developers to manage vulnerabilities, configuration hardening, secret detection, compliance auditing, and so on. At the same time, those developers are measured primarily on feature velocity.

“Shift left” was supposed to make security collaborative. Instead, it simply moved the problem into every developer’s IDE. To fix this problem, we have to make security within infrastructure the default, rather than by design.

This involves real collaboration between developers and security – developers have to understand what they want to achieve and what will be required of what they build, while security will have to work around those requirements so they can be delivered securely. Both teams are responsible, but they both have to work at the speed that the business needs.

In practice, we can create a “golden path” for developers. If they use the standard templates, the pre-approved base images, and the official CI pipelines, security is free. If they want to go “off-road” and build something custom, then they have to do the additional work of security reviews and manual configurations.

This is also something that should be flagged back to the business from the start, so security and development present a united front around what the cost is.

Taking this approach incentivises secure deployment by making it the path of least resistance. It moves the responsibility down the stack to the infrastructure layer, managed by a specialised Platform Engineering team. And if something different is needed, that work can be done collaboratively to ensure it is right first time, rather than leading to more issues that need to be remediated.

For example, instead of asking a developer to please enable versioning on a specific S3 bucket, the platform team writes a policy using Terraform modules, Crossplane compositions, or Open Policy Agent that simply doesn’t allow a bucket to exist without versioning. The developer literally cannot make the mistake.

The platform corrects it automatically or rejects the request. Similarly, developers shouldn’t have to remember container scanning in their workflows, the CI pipeline should do it automatically. The admission controller should reject non-compliant images before they ever hit a cluster. The developer doesn’t need to know how the scan works, only that if they try to deploy a critical vulnerability, the door will be locked.

“Shift down” also means automating the fix. For instance if a vulnerability is found in a base image, the platform should automatically generate a Pull Request to upgrade it. If a runtime security tool detects a container behaving badly (e.g., spawning a shell for persistence), it shouldn’t just send an alert. It should kill the pod and isolate the node autonomously.

Rather than sticking with existing ways of running across security and development, we have to react to what is happening. This can mean we fundamentally change how we operate across teams.

If we continue with the “shift left” mentality of piling cognitive load onto developers, we will fail. We will burn them out, and they will bypass our controls simply so they can get what needs to be done for the business.

Instead, security has to be proactive around how to implement and support the right platforms for the business, so they can be made secure automatically.

Sponsored and written by Qualys.