Whether you're just getting started on your Kubernetes journey or you're ready to start managing workloads, security is critical at every step. In this blog, you'll learn how to keep security in mind when it comes to clusters, images and deployment in GCP.
See more articles
Platform Engineer at CTS
Using services like Google Kubernetes Engine (GKE), you can greatly improve your ability to get started with Kubernetes in just a few clicks, and in a few minutes you could be looking at those famous words - “Hello world!” - in your favourite browser. However, you will inevitably still be left with some burning questions - some of which this series of articles will aim to answer for you!
The first focus of this series will be security. If you start your Kubernetes journey with security at the forefront of your mind, you will thank yourself in a few months. Security is not easy to implement after-the-fact! We’ll start by asking a simple question:
Is my cluster as secure as possible?
It is important to know how to use services like GKE to ensure that your cluster is secure and follows security best practices. There are lots of ways people can take advantage of Kubernetes clusters and deployments which haven’t been set up with security in mind.
Luckily, GCP makes these security practices easy to implement.
The easiest way to make your cluster safe is to remove their external (public) IP addresses leaving them with only an internal (private) IP address making them “private nodes”. This protects the nodes and, by association, your workloads running on them from the public internet.
Using GKE, you can add a managed NAT service to allow your nodes to have internet access in a much more manageable and secure way which allows them to do things like install external images.
On top of this, GCP also allows your Kubernetes cluster to still communicate with other services on GCP without any requirement for access to the public internet. Using Private Google Access, GCP allows communication to most of Google’s APIs from a private network.
Using a non-default service account
When you create a cluster using GKE, a service account called the “default compute” service account is used. It is recommended that user-managed service accounts are created and default service accounts aren’t used because:
- The default service accounts’ behaviour is unpredictable in terms of when it is created and how they show up in your project.
- They usually have a very large scope of permissions. This means that they have the ability to do a lot within GCP which goes against the principle of least privilege that Google recommends.
Restricting access to the master
A private cluster has 2 master endpoints - private and public - which can be used to communicate with the master. By default, nodes in a cluster use the private endpoint for all communications and the public endpoint is used for external connections to the master API (i.e. a developer using kubectl).
Realistically, although technically feasible, you can’t fully restrict your public master API as you will more than likely need to access the resources on the cluster from a local machine for debugging or troubleshooting. If this is the case, you should aim to restrict the public master endpoint as much as possible.
For example, if you will only be accessing the resources from a computer at your office, restrict access to the IP range of the office network.
Similarly, if you work from home, consider a VPN or a static address which you can restrict your master access to.
Security doesn’t stop at the cluster! Now that we have a more secure cluster, we need to think about our deployments.
(W)Get out of here!
Consider the tools that attackers could use if they gain access to your container to download and execute malicious code - “curl” and “wget”, for example.
If your application can run without the need for these tools, you can uninstall them in your Dockerfile or better yet, start small and add only the tools you need.
Think about where your base image comes from. Ensure it is from a reputable source or that you have seen the code which defines this image to be certain that it doesn’t have any vulnerabilities or install anything strange.
If this is all okay, think about the distribution that the source image is based on. For example, many tutorials will tell you to use an Alpine image. This distribution of Linux has been designed with security in mind and is very small, coming in at around 5MB.
If you want to take this a step further and don’t need to have any shell access to the running containers, consider using a “distroless” base image (like the one Google provides). “Distroless” means it houses runtime dependencies and your application without a lot of unnecessary files, services and tools.
Even better than this, if your application is a Go binary, for example, build your image “FROM scratch” which is as minimal as they get! Docker describes this image as “an explicitly empty image”.
On top of the above measures, scanning images for vulnerabilities is worthwhile. Google Container Registry’s Vulnerability Scanning feature only requires an API to be enabled on your project and it will automatically scan any image that gets pushed to GCR for vulnerabilities. It will add “notes” to the image describing the vulnerability, its severity, and how to fix it (if a fix is available).
You can even integrate this into your pipeline to ensure images have been given the “OK” before being deployed to your cluster on GKE.
Now we have a cluster which is secure and all of our images are written with security in mind! Next, we need to deploy them while maintaining our focus on security.
Role-Based Access Control is the method by which access to Kubernetes resources is granted to users and service accounts within Kubernetes.
In order to use this effectively, service accounts should be used on a per-application basis where access to other resources is required. The service account should only have access to the resources it needs with the minimum required access level. This form of access is referred to as the principle of least privilege.
A network policy is a way to define the communication from pods to other pods and endpoints. Labels are used to select pods and rules are defined for what traffic is allowed to and from the selected resources. Using these policies you can ensure pods can only communicate with other pods and services when these connections are required. These should be used to minimise the blast radius of an attack.
A security context can be defined for pods and/or containers and is used to set the privilege level and security access of the container(s). The following are some of the important settings which can be defined in a Pod security context (these values apply to all containers in the pod) along with the container level security context:
- runAsGroup - the GID to run the entrypoint of the container process. Can also be set in the container security context.
- runAsNonRoot - Enforces that the container must run as a non-root user. This setting will ensure that the container will not start if the Kubelet deems that it runs as root. Can also be set in the container security context.
- runAsUser - The UID to run the entrypoint of the container process.
On top of this, the following can be set in the security context of a specific container:
- privileged - If this boolean is set to true, the container runs in privileged mode. Processes in privileged containers are essentially equivalent to root on the host.
- readOnlyRootFilesystem - whether his container has read-only access to the root filesystem.
Capabilities can be defined in a container Security Context. This allows you to add or remove capabilities to/from running containers. A Capability defines a set of superuser privileges as a unit and can be turned on and off. This allows you to run containers as non-root users while giving them only the capabilities they require to run.
As a simple example, if a webserver has been written to bind to a port in the range of 1-1023, it can be run as non-root and still bind to these ports by using the “net_bind_service” capability.
To summarise, we have gone over securing your Kubernetes deployment from start to finish. I outlined the best practices when securing your cluster with Google Kubernetes Engine on GCP.
Then I discussed how to write Docker images and deploy them in Kubernetes with security at the forefront of your mind. Using simple additions like RBAC, Network Policies and Security Contexts alongside your deployment can greatly secure your architecture.
Starting with security gives you a good place to start. In part 2 of this series, I will be discussing how to make the most of GCP to provide a CICD platform for your Kubernetes architecture.
Kubernetes Part 3: Secrets
In the two previous posts in this series, I have discussed security in GKE - starting your...
Kubernetes Part 2: Continuous Deployment and GitOps
The first post in this series discussed securing your cluster and the workloads running inside of...