Kubernetes Security landscape

Kubernetes!?!?

What on earth is this and how do you even pronounce it? … Well I can’t help you with that, but I can help you to understand a bit about its security posture, and how you can leverage some open source tools to reduce the attack surface!

I first looked into Kubernetes security during an event called KringleCon. Thanks to Jay Beale of InGuardian, he talked and provided some amazing examples on K8 attacks. I have also read endless blogs and interviews from Ian Coldwater. I would like to thank both of them for being amazing advocate on K8 security, and other who wrote many articles and blogs that I have done my research on!

What it is

You might have heard of Docker, or Docker Swarm. K8 is the cool kid in town that’s coming to take over. You may see some companies deploying Docker containers inside a K8 clusters too.

Kubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerized applications. Kubernetes makes it easy to run distributed systems, and that necessitates opening communication protocols between machines that weren’t previously there. By opening connections between machines, it definitely open up more ways for it to be attacked.

Kubernetes is insecure by design (that is if you run everything in default configuration)… not because the creators didn’t care about security but mainly because Kubernetes have been designed to adapt to their quick adoption rate. It is highly customized, and it allow the users to take control over how their clusters to be configured.

Early back in the ABAC (Attribute-based access control) or policy based, there has been a lot that has been done to improve security. We now have Role Based Access Control. A few main points to know:

Kubernetes by default is pretty insecure, so don’t deploy a default cluster with no customized configurations!
Defaults can vary widely across cloud providers and installers  
User installations can use different combinations of add-ons, plugins and integrated services, and many of those have insecure defaults too
Attack and defense can vary with individual configurations.

Past incidents:

Weightwatcher

First example is Weight Watcher - what’d they do? They ran a no password Kubernetes instance, aka they didn’t set a password for their admin console. And essentially, if anyone did a port scan, or know where to look and go there, they can see all the configurations and have access to the cluster.(Seems like it was done based on default configurations.)

The results of this is that about 46 S3 buckets that contains info such as sensitive logs, passwords, private keys…was exposed.

Tesla

This is a more public case that many people mention and use as an example.

Within 1 Kubernetes pod, the attacker made a query to AWS, get the exposed credentials, had access to their AWS environment, and configured a mining software to make some money.

The attackers hid the malware behind an IP address hosted by security firm Cloudflare. They also configured the mining software to use a non-standard port to reach the Internet and to connect to an unlisted or semi-public endpoint rather than well-known mining pools. The attackers also likely ratcheted down the amount of CPU resources used to mine the digital coin. The measures helped to make the illicit mining harder to detect and lower the chances of it being shut down.

Other companies that had the same thing done to them include Gemalto which is a company to manufacture SIM card, and an insurance company named Aviva.

Digital Ocean

For this one, with a specially crafted network request, any user can establish a connection through the Kubernetes API server to a backend server. Once established, an attacker can send arbitrary requests over the network connection directly to that backend. Adding insult to injury, these requests are authenticated with the Kubernetes API server’s Transport Layer Security (TLS) credentials.

In default configurations, both authenticated and unauthenticated users are allowed to perform API discovery calls and could exploit this vulnerability to escalate their privileges. For example, attackers could list all pods on a node, could run arbitrary commands on those pods and could obtain the output of those commands.

An attacker can leverage this, query for some type of cloud metadata. A lot of times, it’ll return back with some type of API keys for access, or tokens. Think Capital One and how they got exploited. But in this case, DigitalOcean has a concept of Personal Access Tokens to enable you to use their API to configure your account and compute requirements. The value in the digital ocean secret is just this. An access token to your account and knowledge of this is sufficient to control just about any element of it, any other Droplets you have running, or firing up new ones, adding SSH keys to droplets, reading data in Spaces, etc.

Attack surfaces

There’s two main attack surfaces in Kubernetes. There’s the Kubernetes as a binary/application, and then there’s Kubernetes as an API. The results of either can be :

Container compromise - through application misconfigurations
Unauthorized connections between different pods - compromised containers attempting to connect to other pods within the same network.
Data exfiltration from a pod - either through a reverse shell, netcat…

Example:

Exec a command/ shell in a container via the API server (kubectl)
- A pod have one or more containers but very often it have 1 container
Once you’ve landed in a pod, you can ask the API server that lives on the master to accept your command to do an equivalent of a docker exec - and basically run as a shell
Launch a container onto the cluster via the API Server
- Hey - I know im in a container, but can you start another container? Maybe have a container that will have more privilege than this one im in!
Abuse or set up a “volume mount” to steal/modify data or the host itself
Ask a kubelet to execute a command / shell in an existing container
- The thing running on the node, talking to the k8 master, to execute a command or shell inside a container
Interact with docker daemon on the host
Interact with the internal or external networks

Attacker Workflow

What does the attacker do? They have a workflow -> discovery which include scanning, enumeration is trying take an inventory of everything that you have, using that information, they figure out how they can get in.

Then after they get in, post exploitation include either privilege escalation which is trying to go from a local user to local admin, or lateral movement which is trying to get onto other hosts on your network. Find interesting data, find hashes, crack hashes, get passwords, rinse and repeat.

Why do people do this? Motivations can be anything ranging from money, fun, for the challenge, or just because it’s open. ;)

How to protect your cluster?

Discovery:

Don’t leave ports open if you don’t have to
Make sure all management ports are visible externally have and require authentication
Rotate keys regularly, make sure to give admin only when needed
Limit SSH access to kubernetes nodes

Enumeration:

Again with the close ports you don’t need

Initial Compromise:

Keep credentials encrypted and limit access to them.
Handle secrets safely.
Write your own applications as securely as possible. Treat other people’s code with caution!
Run static code analysis on your applications and containers to check for vulnerabilities.
You can use open source tools for this such as Clair by CoreOS.
If you find vulnerabilities, patch or mitigate them.

Prevent Privilege escalation:

Pod Security Policies are good for this.
Don’t run your containers as root!
You can set allowPrivilegeEscalation: false

Preventing Lateral Movement:

Admission Controllers can be useful for this.
Segment your networks and limit communication to only where necessary

Container Breakout control:

Network Policies
Admission Controllers
Role-Based Access Control
Pod Security Policies
Resource Quotas
Logging and Monitoring

At the end of the day, we can’t protect everything. So threat modeling is a big thing.

What are you trying to protect? Who are you trying to protect it from? What type of data are stored on there? How many people do you have to dedicate to securing this? How much budget do you have? What’s the risks of X or Y being exposed?

What to focus on

Securing the cluster: API server, Kubelet, run etcd safely, k8 dashboard, validate configurations
Authentication: identity
Authorization: RBAC (role based access control)
Secure Container Images: Scan, patch, CI/CD, Image storage/trust/supply chain, minimize images to reduce attack surface
Running containers: no root, admission control, set security boundaries, and policies
Secret management: least priv, secret encryption, passing secrets to containerized code, secret rotation/revocation
Other: monitoring / alerting / auditing
Host / Network security; Static Analysis of YAML

Current K8 State:

Ok - so that was a lot of information on the attack surface and what not. So what does the current state of Kubernetes look like? A group (Trail of Bits) recently performed a 4 months long audit of the project and listed out all of their findings. The findings are available on Github. They focused on networking, cryptography, authentication, authorization, secret management, and multi tenacy with 8 different selected components such as the kube apiserver, etcd, kube scheduler/controller, kubelet, kube proxy…all sort of things. I won’t go too much into this, there is a report out there that detail a list of this very very well. So I just want to summarize and show this as a resource. If anyone is interested, I will send the slide out and you can feel free to read it.

The main points I want to make are that the results they find is that:

Policies may not be applied and lead to a false sense of security. And that there are a tons of information leakage that are still happening. For example:
Insecure TLS is in use by default, and a lot of times, credentials are exposed in environment variables and command line arguments.
Name of secrets are leaked in logs,
No certificate revocation
Seccomp is not enabled by default.

Some interesting ones:

Kubernetes doesn’t facilitate cert revocation
HTTPS connections are not authenticated
Adding creds to containers by default is unsafe
Directory traversal of host logs using kube-apiserver and kubelet
Cleartext secret in logs
Hard coded credentials path
Legacy tokens don’t expire
CoreDNS Leaking information across namespaces

How to reduce your attack surface

The CIS (Center for Internet Security) rolled out this benchmark document that basically show you the best way to set up your kubernetes cluster. It’s about 200 pages long, so it’d be a little rough to run it over by hand.

KubeBench Here comes Kubebench, by Aquasecurity, it will automates the CIS Benchmark for Kubernetes, making it easy for operators to check whether each node in their Kubernetes cluster is configured according to security best practices.
KubeHunter Kubehunter is a tool that show you how your cluster look like from an attacker’s point of view. It will hunt for security weakness, and show you the results.

Run kube-hunter on any machine (including your laptop), select Remote scanning and give the IP address or domain name of your Kubernetes cluster.

This will give you an attackers-eye-view of your Kubernetes setup. You can run kube-hunter directly on a machine in the cluster, and select the option to probe all the local network interfaces. You can also run kube-hunter in a pod within the cluster. This gives an indication of how exposed your cluster would be in the event that one of your application pods is compromised (through a software vulnerability, for example).

Sources / Resources:

Published 7 Nov 2019

Security engineer. Blue teamer. What about you?TV0 on Twitter