What Is Container Security? 8 Best Practices You Must Know

What Is Container Security?

The purpose of container security is to protect containerized environments, which are much more complex than traditional workloads. Production environments deploy massive amounts of containers. Security experts and administrators need to secure more components in a containerized environment than they would in traditional deployments.

Container security involves the implementation and maintenance of security controls that protect containers and the underlying infrastructure. Integrating security into the development pipeline can help ensure that all components are secured from the initial development phase and until the end of their lifecycle.

This is part of an extensive series of guides about Kubernetes.

In this article:

  • Container Security Challenges
  • Container Security Best Practices
    1. Securing Images
    2. Securing Registries
    3. Securing Deployment
    4. Securing Container Runtime
    5. Securing Kubernetes
    6. Using Thin, Short-Lived Containers to Reduce Your Attack Surface
    7. Using Container Security Tools
    8. Monitoring Container Activity
  • 4 Common Container Security Mistakes to Avoid
  • A Quick Guide to Secure Image Development
    • Secure Code Running in Containers
    • Use Minimal Base Images
    • Use Trusted Images
    • Be Aware of Container Image Layers
  • Improving Container Security with Calico

Container Security Challenges

Containers offer many advantages, but also pose certain security challenges that can be difficult to overcome. Perhaps the most noticeable security challenge is the larger attack surface containers create, as opposed to traditional workloads, because of the large number of containers based on many different underlying images, each of which can have vulnerabilities.

Another key issue is the underlying kernel architecture shared by containers. Securing the host is not enough to ensure protection. You also need to maintain secure configurations to limit container permissions and ensure proper isolation between containers.

Because containerized environments are highly dynamic, containerized workloads also pose visibility challenges. This is because traditional monitoring tools may not be able to see which containers are running, what they are running, or scrutinize their network behavior. It is critical to improve visibility as much as possible to ensure timely remediation and prevent breaches.

Container Security Best Practices

1. Securing Images

Container images are used to create containers. A misconfiguration or malicious activity in container images can introduce vulnerabilities into containers deployed in production. To ensure the health of your containerized workloads and applications, you need to secure container images. Here are several ways to do this:

  • Include your application in a container image – A container image holds a subset of the operating system along with the application designed to run in the container. Each library and tool you pull into the image poses a potential threat. To mitigate these threats, you need to include the application within the container image. This should be a statically compiled binary that contains all required dependencies.
  • Include as little as possible – Remove all components the application does not need. For example, remove the “sed” and “awk” binaries, which are present by default on any UNIX system. This can help you reduce the attack surface.
  • Use trustworthy images – If you are not creating the image from scratch, you should choose images that are trustworthy. Public image repositories, such as Docker Hub, can be used by anyone and may contain malware or misconfigurations.

Related content: Read our guide to container security scanning

2. Securing Registries

Container images are usually stored in private or public registries. It is critical to secure these registries to ensure all team members and collaborators are using images that are as free of vulnerabilities as possible. Here are several ways to secure container registries:

  • Use access control – If you have your own private registry, you need to establish access controls that define exactly who can access and publish images and who cannot perform these actions. Access control is a basic security measure that can prevent unauthorized parties from modifying, publishing, or deleting your images.
  • Sign your images – Signatures help track images to the people who signed them. This makes it difficult to substitute the signed image for a compromised one. The Docker Content Trust mechanism provides information about signing images. You can also use Notary, an open-source tool that helps you sign and verify images.
  • Scan your images – Vulnerability scanners are designed to identify known vulnerabilities. These tools can help you find critical vulnerabilities and detect critical threats. You can use scanners on a continuous basis to ensure that your registries do not contain critical vulnerabilities.

3. Securing Deployment

Here are several methods that can help you secure your deployments:

  • Secure the target environment – You can do this by hardening the underlying host operating system. You can also establish firewall and VPC rules or create special accounts that limit access.
  • Use an orchestration platform – These systems usually provide secure API endpoints as well as role-based access control (RBAC), which can help minimize the risk of unauthorized access.
  • Use immutable deployments – This involves creating an instance image during your build steps. Your deployment can then use this image to create new instances. To update the application, you need to create new images, spin up new instances, and then destroy the old ones.

4. Securing Container Runtime

Here are several best practices that can help you secure runtime:

  • Create separate virtual networks for your containers – This introduces a level of isolation that can reduce the attack surface.
  • Apply the principle of least privilege – Allow connectivity only between containers that truly need it.
  • Expose only the ports that serve the application – Do not expose any other ports, except for SSH. Apply this principle to containers as well as the underlying machines.
  • Use TLS to secure communication between services – This process encrypts traffic and ensures only authorized endpoints are allowed.
  • Use the Docker Image policy plugin – This plugin is designed to prevent any process from pulling images that were not previously allow-listed.

Learn more in our detailed guide to Docker security

5. Securing Kubernetes

Here are several best practices to help you secure Kubernetes:

  • Enable TLS everywhere – You should enable TLS for all supported components to defend against traffic sniffing and authenticate identities at both ends of each connection.
  • Use a service mesh architecture – Service meshes are networks of persistent encrypted connections between high-performance sidecar proxies. They provide traffic monitoring, management, and policy enforcement without affecting microservices.
  • Use OPA – Open Policy Agent (OPA) enforces custom policies on a Kubernetes object without reconfiguring or recompiling the Kubernetes API server.
  • Apply network policies – The default Kubernetes networking permits all traffic between pods, but you can restrict this with a network policy.
  • Implement private networks – Deploy each Kubernetes worker and master node on a private subnet to secure the connections to corporate networks, make nodes unreachable from the public Internet, and minimize your overall attack surface.
  • Keep the etcd cluster separate – Use a firewall to protect the etcd cluster, which stores state and secret information and requires special protection compared to other Kubernetes components.
  • Ensure the regular rotation of encryption keys – Regularly rotating encryption keys and certificates helps minimize the blast radius of an attack that compromises keys.
  • Use static analysis for YAML – Statically analyze where pod security policies deny access to API servers. This should be part of the development workflow because it helps identify the organization’s risk tolerance and compliance requirements.
  • Manage secrets – Integrate clusters using a secret management system to ensure application pods automatically receive all secrets and passwords needed at runtime (based on the app roles associated with each pod).
  • Check the code – Scan the code and use static analysis to ensure automation security. You must scan the source code for all application code in Kubernetes to identify vulnerabilities and hard-coded errors.
  • Use RBAC policies based on the principle of least privilege – Role-based access control (RBAC) helps manage access policies at a granular level to protect resources. A centralized authentication and authorization system like SSO throughout the organization makes onboarding and offboarding easier.
  • Use third-party authorization for API servers – You can manage and secure Kubernetes API access by creating detailed auth policies and applying advanced, full-featured verification technologies.

Learn more in our detailed guide to Kubernetes security

6. Using Thin, Short-Lived Containers to Reduce Your Attack Surface

Containers are designed to be ephemeral and lightweight. They are not supposed to be used like servers. You should not constantly add files to the container or update only every couple of weeks or months. This approach can weaken your security posture because, essentially, you are creating a larger attack surface that is not maintained on a regular basis.

You should strive to minimize the number of components in each container and keep all containers as thin as possible. This approach can help reduce the attack surface. Additionally, when identifying vulnerabilities in standard images, you should quickly resolve the issue and then deploy a new, clean container.

7. Using Container Security Tools

Container orchestration platforms, like Kubernetes, provide native security capabilities. However, these controls are not enough to ensure the security and health of containerized applications. A particular challenge is ensuring that none of the third-party software components included in the workload contain critical vulnerabilities.

Containers can be vulnerable to rogue processes, which may be able to bypass isolation and gain unauthorized access to other containers and container images. If a container image includes a vulnerability, it may be deployed in applications. Additionally, misconfigured permissions might multiply these issues.

You can mitigate security risks by using container security tools, which can help you manage access, test security, and protect your infrastructure. You can employ tools that help you gain visibility into activity across your containerized applications. Testing features can help you develop security policies, simulate attacks from common threat sources, and discover zero-day vulnerabilities.

Related content: Read our guide to container security tools

8. Monitoring Container Activity

Containerized workloads are highly dynamic. One container image often has multiple running instances. Additionally, new images and new versions are deployed at a high-paced speed. Issues can, therefore, quickly propagate across multiple containers and applications. This is why it is highly critical to identify these issues quickly and remediate each issue at the source.

Containerized workloads require a granular level of monitoring, to provide visibility for IT and security teams into elements running inside the environment.

Monitoring tools enable seamless container security and operations. They help identify anomalous behaviors and respond to events in a timely and appropriate manner. For example, the timely identification of a faulty image can ensure that admins quickly fix it and rebuild all relevant containers using the new image.

To maintain security, you should implement security practices and tools that can help you achieve observability for the following components:

  • Master nodes (when using a container orchestration platform like Kubernetes)
  • Container engines
  • Workloads running in containers
  • Containerized middleware and networking

Related content: Read our guide to Docker container monitoring

4 Common Container Security Mistakes to Avoid

Here are several common container security pitfalls to avoid:

  • Forgetting basic security hygiene – Containers are generally considered a new technology, which requires the use of new security methods. However, certain security fundamentals still apply. For example, you need to keep all systems patched and up-to-date, including operating systems and container runtimes.
  • Failure to harden and configure tools and environments – Container orchestration platforms offer a set of unique security capabilities. However, to ensure security, you need to properly configure them for each environment. You should never run security configurations using a platform’s default settings. For example, you should grant containers only the privileges needed to run. This can significantly minimize risks associated with privilege escalation attacks.
  • Failure to log, monitor, and test – When you first run containers in production, you might lose visibility into the health of your application and environments. If this occurs and you do not catch the issue in time, you might run into critical risks. This is especially important for highly distributed systems spanning multiple clouds and on-premise infrastructure. You need to make sure you have properly configured monitoring, logging, and testing. This can help minimize the amount of unknown vulnerabilities as well as reduce other blind spots.
  • Failure to secure all phases of the CI/CD pipeline – Do not ignore other components of your software development pipeline. You can avoid this issue by implementing a “shift left” philosophy, which means you implement security early in the development cycle. This often requires consistently applying relevant tools and policies across the pipeline and making changes as needed.

A Quick Guide to Secure Image Development

While it is critical to scan images, that alone is not enough. You should aim to shift image security left, avoid using unsafe images in the first place, and take care not to add vulnerable components to new images you create. Here are a few steps you can use to improve the security of your container images early in the development process.

Secure Code Running in Containers

Containers are intended to run software applications. Typically, these applications will consist of a combination of proprietary code and open source components. It is important to realize that a critical part of securing images is ensuring the code packaged within them is secure.

A variety of automated tools are available that can help you scan your code for vulnerabilities:

  • Software Composition Analysis (SCA) can help discover vulnerabilities in open source components
  • Static Application Security Testing (SAST) can scan your proprietary code for security flaws, bugs, and code quality issues
  • Dynamic Application Security Testing (DAST) can help you test the application at runtime to discover exploitable vulnerabilities

It is important to have these or similar tools as a mandatory step in your CI/CD pipeline, to ensure that all code you add to a container image is known to be safe.

Use Minimal Base Images

Almost all container images are derived from a base image (this is the FROM line in the Dockerfile). There are a huge number of public images to choose from when selecting your base image. Compare all the images that meet your needs and select the minimal image—the one with the smallest footprint and fewest features, components, and dependencies.

Selecting a minimal image will reduce your attack surface from the outset. It will also make your containers more lightweight and improve resource utilization. Remember that a base image can run in hundreds or even thousands of containers and the overhead adds up.

Use Trusted Images

Never use container images from unknown publishers in public repositories. There are a few sources of trusted images where there is some assurance that the image does not contain vulnerabilities and has not been tampered with by attackers. For example:

  • Docker Hub Official Images are curated by Docker experts and tested for functionality and security.
  • The Docker Verified Publishers badge means that an image is high-quality, and is directly maintained by a company affiliated with Docker. For example, a MySQL image that is directly maintained by Oracle.
  • You can use Notary or similar tools to verify that an image was signed by a trusted entity and has not been changed.

Be Aware of Container Image Layers

In a Dockerfile, you start from a base image and then add additional components needed for your containers to function. You do this using RUN, COPY, and ADD commands. Technically, each of these commands adds another layer to the container image, and each layer creates a new attack surface.

It is important to be aware of the layers you are adding to your containers, and reduce security risks using these guidelines:

  • Ensure each layer adds exactly the tools you need for the relevant stage of the development lifecycle and no more.
  • Ensure tools or components added in these middle layers use the latest version and do not themselves have security vulnerabilities.
  • Most of the tools will only be needed in development and testing stages—remove them in production. The best way to do this is to use a multi-stage build.

Improving Container Security with Calico

Calico Cloud offers active build, deploy and runtime security for containers and Kubernetes.

  • Build – With Calico’s Image Assurance, DevOps teams can secure the build pipeline with an integrated Image Scanner, an admission controller to automatically block vulnerable images from being deployed, and a runtime view of high-risk workloads in the application to prioritize remediation activities.
  • Deploy – Detect configuration violations and misconfigurations in the Kubernetes environment to prevent breaches. Calico can generate CIS benchmark reports to conform to security best practices.
  • Runtime – Protect container workloads from network-based and container-based threats with zero-trust workload access controls, malware protection, and identity-based microsegmentation. Calico uses machine learning and global threat feeds to detect anomalies and zero-day threats in containerized applications.

Visit our container security solutions page to learn more about Calico Cloud’s container security features.

Next Steps

  • Try one of our 30-minute, hands-on tutorials for container security
  • Read about container security for cloud workload protection
  • Calico Cloud: Plug-and-Play Active Container Security
  • Try Calico Cloud for free

See Our Additional Guides on Key Kubernetes Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of Kubernetes.

Kubernetes Security

  • AKS Security: The Basics and 5 Critical Best Practices
  • Kubernetes Vulnerability Scanning: Why, How, and 5 Free Scanners
  • Kubernetes Security Policy Design: 10 Critical Best Practices

Kubernetes Networking

  • Container Networking: What You Should Know
  • Kubernetes CNI Explained

Kubernetes Troubleshooting

Authored by Komodor

  • How to Fix CrashLoopBackOff Kubernetes Error
  • How to Fix ErrImagePull and ImagePullBackoff
  • How to Fix Kubernetes ‘Node Not Ready’ Error