The blueprint
A mock lab here has 16 tasks in 2 hours, split across the domains in the same proportions as the official exam guide. The official pass mark is 67%. FixOps scores practice against a target of 67%.
-
01
Cluster Setup
-
02
Cluster Hardening
-
03
System Hardening
-
04
Minimize Microservice Vulnerabilities
-
05
Supply Chain Security
-
06
Monitoring, Logging and Runtime Security
The free sample lab
Two tasks from the bank, in a simulated cluster you drive with kubectl and helm. You are graded on the state you leave behind, not on the commands you type.
- Close a namespace and open DNS. Deny all traffic in a namespace by default and allow only DNS lookups out.
- Enforce the restricted standard on a namespace. Label a namespace for Pod Security Admission and make its workload comply.
Revision notes: Cluster Setup
The notes for one domain, free to read here and in the app. FixOps Pro has them for all 6 domains.
Network policies that restrict traffic, the CIS benchmark with kube-bench, TLS on an Ingress, keeping pods away from node metadata and endpoints, and verifying binaries before they are used.
Network policies
- A NetworkPolicy selects pods with podSelector in its own namespace; an empty podSelector selects every pod of that namespace.
- A pod that no policy selects accepts and sends everything; once a policy selects it for a direction, only what some policy allows in that direction gets through.
- A default deny is a policy with an empty podSelector, the direction in policyTypes and no rules for it.
- Policies are additive: there is no deny rule and no order, and the result is the union of everything that is allowed.
- In a from or to list, a namespaceSelector and a podSelector in the same item must both match; as two items either may match.
- ipBlock allows a CIDR and can leave parts of it out with except, which is how the metadata address 169.254.169.254 is kept away from pods.
- Egress rules have to allow DNS, UDP and TCP port 53, or pods can no longer resolve the names the other rules are about.
- NetworkPolicy is enforced by the network plugin: with a plugin that does not implement it the objects are stored and nothing is filtered.
- A policy is tested from a pod (kubectl exec POD -- wget -qO- -T 2 SERVICE): a dropped packet is a timeout, and a name that cannot be looked up means the pod's egress to DNS is closed.
The CIS benchmark and kube-bench
- The CIS Kubernetes Benchmark is a list of numbered recommendations for the control plane, etcd, the worker nodes and policies; kube-bench checks a node against it.
- kube-bench run --targets master,etcd,node limits a run to those parts, and --check 1.2.15 runs single recommendations.
- PASS and FAIL are for checks kube-bench can decide; WARN marks the manual ones, and the remediation of everything that did not pass is printed below the list.
- The control plane components of a kubeadm cluster are static pods: their flags are in /etc/kubernetes/manifests, and the kubelet restarts a component when its file changes.
- After a change to the API server's manifest kubectl is refused until the new container is ready; when it stays refused, sudo crictl ps -a and sudo crictl logs show the flag or the file the server did not accept.
- No API server container at all means the kubelet could not read the manifest as a pod: journalctl -u kubelet names the line.
- A file a flag points to has to be mounted into the static pod with a hostPath volume, or the API server does not find it.
- The kubelet's settings are in /var/lib/kubelet/config.yaml and take effect after systemctl restart kubelet.
- Manifests and kubeconfig files should be mode 600 and owned by root:root, and the etcd data directory 700 and owned by etcd.
Ingress with TLS
- An Ingress terminates TLS with a tls entry that names the hosts and a Secret of type kubernetes.io/tls in the same namespace.
- kubectl create secret tls NAME --cert=FILE --key=FILE builds that Secret from a certificate and its key.
- The certificate has to cover the host name of the rule, or clients get a certificate warning although the Ingress works.
Node metadata, endpoints and binaries
- The cloud metadata service answers on 169.254.169.254 and can hand out the node's credentials, so pods are kept from it with an egress policy.
- The kubelet API on port 10250 must require authentication (anonymous disabled) and authorization (mode Webhook), and the read-only port 10255 should be off.
- sha512sum FILE prints a checksum to compare with the one on the release page, and sha512sum -c SUMS checks every file listed in SUMS.
- A binary whose checksum differs must not be used, whatever its version output says.
Easy to mix up
- An empty podSelector selects all pods; a missing from or to in a rule allows all sources or destinations; an empty ingress list allows nothing.
- policyTypes says which directions a policy speaks about: listing Egress with no egress rules denies all egress, while leaving Egress out says nothing about it.
- kube-bench reports on the configuration of a node and changes nothing: a FAIL stays until you edit the file it names.
- Anonymous access exists on the API server and on the kubelet: one is a flag in a manifest, the other a field in the kubelet's configuration file.