How to validate Kubernetes YAML files
Introduction
In the previous article, we have learned how to create Kubernetes YAML files. Now, let’s see how to ensure that the files we have created are not only valid YAML but more importantly adhere to best practices of Kubernetes development.
This article is intended as a guide for validating Kubernetes manifest files. If you are a developer, software architect, security specialist or simply interested in learning more about the quality assurance process around creating and managing Kubernetes manifests, this article is for you.
The Shift Left philosophy
The shift left movement or as I like to call it “extend left” started as an idea of empowering developers to be responsible for and own the end to end process of the software lifecycle.
Done right, this means creating tools and supporting organizational structure to set development teams for success by allowing them to take full ownership and control over the software they create and maintain. Done wrong means most of the time “dumping” all the responsibility on developers and hoping for the best.
In the spirit of extending left, the process of authoring and managing Kubernetes artefacts should also be the responsibility of development teams.
Kubernetes already has a powerful and sophisticated process of validating resources in the form of the following tools:
- admission controllers
- policy enforcement via tools like Open Policy Agent, Kyverno or Kubewarden
- pod security policies (deprecated in 1.21)
Those tools are great, but they belong to the cluster operator layer and are more suited as a platform rather than a tool that fits neatly in a standard developer workflow.
What we are going to look at next are various tools and methods that shift the starting point of a validation process to a development workflow and thus greatly improve the quality of the whole system.
The holy grail of shifting validation left is to make illegal states irrepresentable. This is difficult to do with a non-statically typed language like YAML, but with the right tools, it can be done.
Quality in a service or product is not what you put into it. It is what the client or customer gets out of it.
— Peter Drucker
What you will learn
After reading this article, you will learn:
- learn about the impact that shifting validation left has on the end product
- what are the different ways to validate Kubernetes YAML files
- what are the benefits of automating the validation process in a CI/CD pipeline
- learn about tools specializing in validating different kinds of YAML files
Prerequisites
If you would like to follow along and get your hands dirty with YAML, you will need:
- VS Code with remote development extension
- docker desktop or any of the alternatives (need to be able to run containers)
- clone this repository
Please note that the repository uses a devcontainer with all the tools and configuration needed to run the examples build in. Because of this, the image is large, so please modify it accordingly before running the container.
What do we need to validate?
Let’s look a bit closer at the validation process. We can divide the validation process into 3 distinct categories:
- Structural validation of YAML syntax. This is the easiest step to validate if the file is a valid YAML. This is always a good starting point, if our file has YAML errors, we want to know as soon as possible or even better, prevent the error altogether.
- Semantic validation of K8s schema. The next step is to validate whether the file is a correct Kubernetes YAML file. This validation is done automatically by Kubernetes, but this is a bit too late in the process. Remember, we want to shift left. An interesting tool in this space is kubeval.
- Pragmatic validation of the resource. This last category is where the validation process looks at the file from different contexts. We want to check the file and configuration for security vulnerabilities, performance issues, adherence to best practices, versioning schemes and many more. The vehicle best suited for performing such validation is policies. In this space, I would like to focus on two tools, trivy and datree. Both bringing a unique perspective and functionality.
Security checks with trivy
Trivy can scan YAML files for security vulnerabilities, misconfiguration and more. Each check has a web page explaining the reason behind it as well as its severity. The only issue here is that the checks are “static”, we cannot influence directly if any of them should be skipped (this is partially possible with setting up severity flag)
Policy enforcement with datree
Datree takes a different approach, where policies can be switched on and off by the developer.
You can see that datree first checked whether the file is a valid YAML and then validated the content against the Kubernetes schema. This satisfies our Structural -> Semantic -> Pragmatic validation flow!
Each check is associated with a configurable policy set. Policies can be switched on and off to meet personalized validation scheme needs.
This brings us a step closer to shifting the validation process left and empowering developers to take control over the whole process.
Automate with CI/CD pipelines
The end goal of every repetitive development process is to automate it. Adding validation rules from both datree and trivy to our CI pipeline is easy. They provide rich integration with all major git providers like GitHub, GitLab as well as Helm installation plugins etc.
Closing thoughts
Kubernetes story for developers still leaves a lot to be desired. I’m happy to see tools created to address some of the pains of developing on Kubernetes. Tools like datree especially focus on and enable the shift left movement.
Developers are fully in control of their software lifecycle, tools and organizational structures geared towards empowering and supporting development teams in their mission of creating amazing software. This is the future that will benefit all of us.
