Kubernetes has become one of the most popular container orchestrators for deploying applications at scale. And building the cluster securely has always been a matter of concern. Many tools can scan your cluster and list its security threats. But implementing them can be challenging as one has to write and manage custom webhooks for each of them.
So, policy as a code has been introduced to help Kuberentes admins and make their life easier. Policy means a set of rules that must be met to impose security and govern IT operations easily. Policy as a code is an approach for policy management using high-level languages such as YAML or Rego.
There are different policy engines, such as Kyverno and OPA, to impose policies as code to secure the Kubernetes cluster.
In this blog, we will learn more about Kyverno, its policies, its use in the Kubernetes cluster and how to set up a Kyverno project locally. Along with this, we’ll look at how to work with the Kyverno command line interface (CLI), write test cases for validating and mutating policies, and test them with the Kyverno CLI, which has also been my Linux Foundation mentorship work.
Let’s first understand what the admission controller in Kubernetes is as Kyverno uses it.
The Linux Foundation is the world’s leading home for collaboration on open source software, hardware, standards, and data. Linux Foundation projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, ONAP, Hyperledger Foundation, PyTorch, RISC-V, and more.
The latest from Linux Foundation Training
Admission Controller in Kubernetes
An
admission controller in Kubernetes is a piece of code that intercepts the incoming object request to the Kuberentes API server before persisting it into the ETCD. But this is done only after the request is authenticated and authorized. These controllers validate or mutate the incoming request based on the validating and mutating admission webhooks, generally known as
dynamic admission controllers.
The following diagram illustrates its working inside the Kubernetes cluster.
👁 Image
Kyverno inside the cluster works in place of these webhooks, making the work easier through policies.
What Is Kyverno?
Kyverno, a Greek word for “govern,”
is a policy engine natively designed for Kubernetes. Currently, it is an
incubating project under the Cloud Native Computing Foundation (CNCF). Kyverno policies are managed as Kubernetes resources and are written in YAML. So, no new language is required to write Kyverno policies.
Features of Kyverno
Various features of Kyverno make it unique from other policy engines, such as:
- Policies are managed as Kubernetes resources
- Policies are manageable with kubectl, git and Kustomize tools
- Validate, mutate, generate or even remove any resources
- Helps in container image signing and verification for software supply chain security
- Match resources using label selectors and wildcards
- Enable background scans on existing Kubernetes resources to ensure best practices
- Block resources and report policy violations
- Generate policy reports
- Test policies and validate resources with Kyverno CLI in CI/CD pipeline before using them in a cluster
How Kyverno Works
Kyverno is used as a
dynamic admission controller inside the Kuberentes cluster. It receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests.
Kyverno policies can match resources with the help of resource kind, name, label selectors, user, and much more. Policy enforcement is recorded through Kubernetes events, and it also reports violations of existing resources through background scans.
Policy Types and Their Behaviors
Two types of policies are permitted with Kyverno:
- ClusterPolicy: The policy scope is cluster-wide.
- Policy: The policy scope is limited to the namespace in which it is applied.
Different Behaviors of Policies
The different behaviors found in
Kyverno policies for resources are:
- Validate: This uses overlay-style syntax with pattern-matching support and conditional (if-then-else) processing. If the Kubernetes resource matches the pattern specified, then the resource is allowed to be created, otherwise, the creation will be blocked.
- Mutate: This uses RFC 6902 JSONPatch format, which is used to modify the matching resources.
- Generate: This is used when there is a need to create supporting resources based on a new resource.
- Verify Images: With the help of Sigstore’s sub-project Cosign, container image signatures are checked.
Kyverno in the Kubernetes Cluster
Kyverno Installation
-
- To work with Kyverno policies in the Kubernetes cluster, first, install Kyverno custom resource definitions (CRDs) either via Helm or kubectl.
| helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno –create-namespace –set replicaCount=1 |
- Verify the pods in the Kyverno namespace are running.
| kubectl get pods -n kyverno |
For now, we will see Kyverno policies with validate and mutate behavior on Kubernetes resources.
Validating Policy in Kyverno
- To ensure pods are allowed to mount `hostPath` volumes in `readOnly` mode. This validating policy checks all the containers of pods for any 1hostPath1 volumes and ensures that all of them are only in `readOnly` mode.
Policy reference:
https://kyverno.io/policies/other/ensure_readonly_hostpath/ensure_readonly_hostpath/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ensure-readonly-hostpath
annotations:
policies.kyverno.io/title: Ensure Read Only hostPath
policies.kyverno.io/category: Other
policies.kyverno.io/severity: medium
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kyverno-version: 1.6.2
kyverno.io/kubernetes-version: "1.23"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Pods which are allowed to mount hostPath volumes in read/write mode pose a security risk
even if confined to a "safe" file system on the host and may escape those confines (see
https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts). The only true way
to ensure safety is to enforce that all Pods mounting hostPath volumes do so in read only
mode. This policy checks all containers for any hostPath volumes and ensures they are
explicitly mounted in readOnly mode.
spec:
background: false
validationFailureAction: Enforce
rules:
- name: ensure-hostpaths-readonly
match:
any:
- resources:
kinds:
- Pod
preconditions:
all:
- key: "{{ request.operation || 'BACKGROUND' }}"
operator: AnyIn
value:
- CREATE
- UPDATE
validate:
message: All hostPath volumes must be mounted as readOnly.
foreach:
- list: "request.object.spec.volumes[?hostPath][]"
deny:
conditions:
any:
- key: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}'][] | length(@) }}"
operator: NotEquals
value: "{{ request.object.spec.[containers, initContainers, ephemeralContainers][].volumeMounts[?name == '{{element.name}}' && readOnly] [] | length(@) }}"
When the policy attribute `validationFailureAction` is set to “Audit”
(default) mode, it will report policy violations by resources but will not block them from being created. On the other hand, “Enforce” mode will block them as well.
| kubectl apply -f ensure-readonly-hostpath.yaml
kubectl get clusterpolicies |
👁 Image
- Create new pod resources with `hostPath` volume to test policy.
apiVersion: v1
kind: Pod
metadata:
name: nginx-without-readonly
spec:
containers:
- image: nginx:latest
name: nginx
volumeMounts:
- mountPath: /data
name: nginx-volume
volumes:
- name: nginx-volume
hostPath:
path: /data
---
apiVersion: v1
kind: Pod
metadata:
name: nginx-with-readonly
spec:
containers:
- image: nginx:latest
name: nginx
volumeMounts:
- mountPath: /data
name: nginx-volume
readOnly: true
volumes:
- name: nginx-volume
hostPath:
path: /data
On creating the above pod resources, one which has `hostPath` mounted as `readOnly` will be created and the other one will be blocked from being created.
👁 Image
Mutating Policy in Kyverno
- It is always a good practice to work with fully resolved container images that have digest in them rather than the tag. This mutate policy changes the pod container images and sets them as fully resolved.
# Policy reference:
https://kyverno.io/policies/other/resolve_image_to_digest/resolve-image-to-digest/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: resolve-image-to-digest
annotations:
policies.kyverno.io/title: Resolve Image to Digest
policies.kyverno.io/category: Other
policies.kyverno.io/severity: medium
kyverno.io/kyverno-version: 1.6.0
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.23"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Image tags are mutable and the change of an image can result in the same tag.
This policy resolves the image digest of each image in a container and replaces
the image with the fully resolved reference which includes the digest rather than tag.
spec:
background: false
rules:
- name: resolve-to-digest
match:
any:
- resources:
kinds:
- Pod
preconditions:
all:
- key: "{{request.operation || 'BACKGROUND'}}"
operator: NotEquals
value: DELETE
mutate:
foreach:
- list: "request.object.spec.containers"
context:
- name: resolvedRef
imageRegistry:
reference: "{{ element.image }}"
jmesPath: "resolvedImage"
patchStrategicMerge:
spec:
containers:
- name: "{{ element.name }}"
image: "{{ resolvedRef }}"
kubectl apply -f resolve-image-to-digest.yaml
kubectl get clusterpolicies
👁 Image
- Create new pod resource to test the policy
apiVersion: v1
kind: Pod
metadata:
name: nginx-app
spec:
containers:
- image: nginx:latest
name: nginx
👁 ImageFrom the above output, it can be clearly seen that the
NGINX image has been fully resolved along with the image digest.
Setting Up Kyverno Locally
To set up the
Kyverno project locally, you must have
Golang installed on your machine. Ensure that the `GOPATH` environment variable is set up correctly.
- Now clone the Kyverno repository locally on your system under a subdirectory of your `$GOPATH` location, as shown below.
| git clone https://github.com/kyverno/kyverno.git $GOPATH/src/github.com/kyverno/kyverno |
- Similarly, you can clone the policies repository by following the above path.
You can also check out the
Kyverno project wiki for further guidance You can also create a local cluster with
Kind/
Minikube and build a local Kyverno image and use them in your local cluster. Find out more from these
training videos.
- Install the Kyverno CLI. One can find more information about different versions of it on the release page.
| wget https://github.com/kyverno/kyverno/releases/download/v1.9.2/kyverno-cli_v1.9.2_linux_x86_64.tar.gz
tar -xvzf kyverno-cli_v1.9.2_linux_x86_64.tar.gz
mv kyverno /usr/local/bin |
- Verify the Kyverno CLI installation by checking its version.
👁 Image
What Is Kyverno CLI?
The
Kyverno CLI is designed to apply and test policies outside the cluster. This is mainly used to validate and test the policy behavior on resources before adding them to the cluster. This CLI can be used with
kubectl as a plugin, CI/CD pipelines or as a standalone.
Different Kyverno CLI Commands
Different subcommands can be used with the Kyverno CLI, such as:
Kyverno Apply Command
The `kyverno apply` command is used to try out the policies with given resources. For validating policies, it gives out the result as pass/fail/skip depending on the resource given against the policy. In the case of mutate policies, it generates the mutated resource as an output.
The syntax of the command is as follows:
| kyverno apply /path/to/policy.yaml –resource /path/to/resource.yaml -f /path/to/values.yaml |
The `-f
` flag is used to pass variables that are needed to test the policy against the resources. Find out more about it
here.
Kyverno Test Command
The `kyverno test` command is used to test the resources against policies for desired results. For this, a separate (standard) `kyverno-test.yaml` file is created to mention the tests.
The syntax of the command is as follows:
| kyverno test /path/to/testyamlfile |
The `-f` flag is used to set a custom file name, which includes test cases. By default, it will search for a file called `kyverno-test.yaml`. Find out more about it
here.
Now that we learned about the Kyverno CLI, we can discuss how to write the test cases for resources to be tested against the policies.
Writing Test Cases for Kyverno Policies and Testing Them with Kyverno CLI
To write test cases one by one, we will use the same
validating and
mutating policies we used above.
Tests for Validating Policy
- This validating policy will work on pod resources that have a precondition and rule for matching `hostPath` volume. This precondition is used to have more control over rules by defining variables that will be defined in the `values.yaml` file.
It’s vital to have negative and positive cases to ensure when the policy is being passed and when it is being failed.
- Below is the `values.yaml` and`kyverno-test.yaml` file for the validating policy.
# values.yaml
policies:
- name: ensure-readonly-hostpath
resources:
- name: good-pod-01
values:
request.operation: UPDATE
# kyverno-test.yaml
name: ensure-readonly-hostpath
policies:
- ensure-readonly-hostpath.yaml
resources:
- good-pod-01.yaml
- bad-pod-01.yaml
variables: values.yaml
results:
- policy: ensure-readonly-hostpath
rule: ensure-hostpaths-readonly
resource: bad-pod-01
kind: Pod
result: fail
- policy: ensure-readonly-hostpath
rule: ensure-hostpaths-readonly
resource: good-pod-01
kind: Pod
result: pass
- Let’s check the policy with these test cases on Kyverno locally with the Kyverno CLI command.
| kyverno apply ensure-readonly-hostpath.yaml -r good-pod-01.yaml -f values.yaml |
👁 Image
👁 Image
Tests for Mutating Policy
- This mutating policy will work on a pod resource with a mutate rule to change image tag to digest and a precondition with it. The `kyverno apply` command will generate a mutate (`patchedResource.yaml`) file for the resource on which the policy is applied.
- Below is the `values.yaml` , `kyverno-test.yaml` and `patchedResource.yaml` file for the mutating policy.
# values.yaml
policies:
- name: resolve-image-to-digest
rules:
- name: resolve-to-digest
values:
resolvedRef: "busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47"
resources:
- name: busybox
values:
request.operation: UPDATE
# kyverno-test.yaml
name: resolve-image-to-digest
policies:
- resolve-image-to-digest.yaml
resources:
- pod.yaml
variables: values.yaml
results:
- policy: resolve-image-to-digest
rule: resolve-to-digest
resource: busybox
patchedResource: patchedResource.yaml
kind: Pod
result: pass
# patchedResource.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
- Let’s check the policy with these test cases on Kyverno locally with the Kyverno CLI command.
| kyverno apply resolve-image-to-digest.yaml -r pod.yaml -f values.yaml |
👁 Image
👁 ImageSo, this is how to write test cases for Kyverno policies and test them locally.
The Linux Foundation is the world’s leading home for collaboration on open source software, hardware, standards, and data. Linux Foundation projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, ONAP, Hyperledger Foundation, PyTorch, RISC-V, and more.
The latest from Linux Foundation Training