Secrets
A secret leaked: what now
The order matters. Deleting the commit does not un-leak anything.
1. Revoke or rotate the credential immediately. Assume it is already copied.
2. Deploy the new credential and confirm services still work.
3. Audit usage of the old credential during the exposure window (cloud audit logs, access logs).
4. Remove it from the repository history and ask the host to purge caches if needed.
5. Add a pre-commit and CI secret scanner so it cannot happen the same way again.
- Rotate first
- Bots scrape public repositories for keys within minutes.
- Exposure window
- From the first push to the moment of revocation.
gitleaks git <repo>
Scan a repository's history for committed secrets such as API keys, tokens and private keys.
gitleaks git --redact -v .
git- Scan commits in git history (gitleaks 8.19+; older versions use detect).
dir PATH- Scan files on disk, ignoring git.
--redact- Hide secret values in the report.
.gitleaksignore- Fingerprints of accepted false positives.
git filter-repo --replace-text <file>
Rewrite history to remove a secret or file everywhere. Everyone must re-clone afterwards.
git filter-repo --invert-paths --path config/prod.env
--invert-paths --path P- Remove a file from every commit.
--replace-text FILE- Replace matching strings in every commit.
Gotcha Rewriting history is cleanup, not remediation. The credential is compromised until it is rotated.
openssl rand -base64 32
Generate a strong random secret for a password, token or signing key.
q8Vf0mB9mZ2yJ8cF0w2Qm2mO4aY3o6Hc2xF3m1nKc5E=
-base64 N- N random bytes, base64-encoded.
-hex N- N random bytes as hex.
vault kv get <path>
Read a secret from HashiCorp Vault's KV engine.
vault kv get -field=password secret/shop/db
-field=NAME- Print one value without formatting.
-format=json- Machine-readable output.
-version=N- Read an older version (KV v2).
vault kv put <path> key=value
Write or rotate a secret. KV v2 keeps previous versions for rollback.
vault kv put secret/shop/db password=@new-password.txt
key=@file- Read the value from a file, keeping it out of shell history.
kv patch- Update some keys without replacing the others.
kv rollback -version=N- Restore an earlier version.
vault token lookup
Inspect your current token: policies, TTL and whether it is renewable.
token renew- Extend a renewable token.
token revoke -self- Revoke your own token when done.
sops -e -i <file>
Encrypt the values in a YAML or JSON file with KMS, age or PGP, so it can live in Git safely.
sops -e -i secrets.prod.yaml
-e / -d- Encrypt or decrypt.
-i- In place.
.sops.yaml- Rules for which keys encrypt which paths.
sops FILE- Open decrypted in your editor, re-encrypt on save.
aws secretsmanager get-secret-value --secret-id <id>
Fetch a secret from AWS Secrets Manager.
aws secretsmanager get-secret-value --secret-id prod/shop/db --query SecretString --output text
--query SecretString- Only the secret payload.
rotate-secret- Trigger the configured rotation Lambda.
Where secrets should live
Ways to hand a secret to a process, from worst to best.
Hard-coded in source or an image layer: never.
Command-line arguments: visible in ps and shell history.
Environment variables: common, but leak into crash dumps, debug pages and child processes.
Mounted files with mode 0400 or 0440: better; easy to rotate.
Fetched at runtime from a secret manager with short-lived credentials: best.
- Least privilege
- Each service reads only its own secrets.
- Short-lived
- Dynamic credentials expire before they can be abused.
Scan & sign
trivy image <image>
Scan an image for known vulnerabilities in OS packages and language dependencies.
trivy image --severity HIGH,CRITICAL --ignore-unfixed myapp:latest
--severity- Only these severities.
--ignore-unfixed- Hide findings without an available fix.
--exit-code 1- Fail CI when findings remain.
-q -f json- A report for jq and for scripts, without the log lines.
Gotcha Fix by rebuilding on a patched base image first. Most CVEs come from the base, not your code.
trivy fs <dir>
Scan a source tree: dependency lockfiles, secrets and misconfigurations.
trivy fs --scanners secret .
--scanners- Choose vuln, secret, misconfig and license.
--exit-code 1- Fail the pipeline when something is found.
trivy config DIR- Misconfiguration scan of Terraform, Kubernetes YAML and Dockerfiles.
Gotcha A secret that was committed once stays in the history after the file is cleaned: rotate it, do not just delete it.
syft <image> -o <format>
Generate a software bill of materials (SBOM) listing every package in an image.
syft myapp:latest -o spdx-json > sbom.spdx.json
-o spdx-json / cyclonedx-json- Standard SBOM formats.
dir:PATH- Scan a directory instead of an image.
grype <image|sbom>
Match an image or an SBOM against vulnerability databases.
grype sbom:./sbom.spdx.json --fail-on high
sbom:FILE- Scan an existing SBOM without pulling the image.
--fail-on SEV- Non-zero exit at or above a severity.
--only-fixed- Only findings with a fix available.
cosign verify <image>
Verify an image signature before deploying it, so only images your pipeline built can run.
cosign verify --certificate-identity-regexp 'https://github.com/acme/.*' --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/acme/api:1.4.2
sign IMAGE- Sign, keyless with OIDC or with --key.
--certificate-identity- Which workflow identity must have signed it.
--certificate-oidc-issuer- Which OIDC provider issued that identity.
semgrep scan --config <rules>
Static analysis with pattern rules: injection, unsafe deserialization and hard-coded secrets.
semgrep scan --config p/owasp-top-ten --error
--config- A registry ruleset or a local rules file.
--error- Exit non-zero when findings exist.
--baseline-commit- Only report findings new since that commit.
Harden a host
ss -tlnp
Audit what is exposed. Anything listening on 0.0.0.0 that should not be public is attack surface.
0.0.0.0 / [::]- Reachable on every interface.
127.0.0.1- Local only.
find / -xdev -perm -4000 -type f
List setuid binaries; an unexpected one is a privilege-escalation path.
-perm -4000- setuid bit set.
-perm -2000- setgid bit set.
find <dir> -xdev -type f -perm -0002
Find world-writable files, which any local user or compromised service could modify.
find /etc /opt -xdev -type f -perm -0002
-perm -0002- Others have write permission.
unattended-upgrade --dry-run
Confirm automatic security updates are configured and working on Debian and Ubuntu.
sudo unattended-upgrade --dry-run --debug
--dry-run- Show what would be installed.
/var/log/unattended-upgrades/- What actually ran.
fail2ban-client status sshd
See which addresses are banned for repeated failed logins.
sudo fail2ban-client status sshd
set sshd unbanip IP- Unban an address, for example a colleague's.
Cluster & runtime
kubectl auth can-i <verb> <resource> --as=<subject>
Ask the API server what a user or a service account may do, instead of reading every Role and binding.
kubectl auth can-i get secret/smtp-credentials -n billing --as=system:serviceaccount:billing:invoice-reader
--as=system:serviceaccount:NS:NAME- A service account is asked for by this full name.
--list- Everything the subject may do in the namespace.
secret/NAME- One object: rules with resourceNames answer differently per name.
Gotcha RBAC only adds. A narrow Role changes nothing while another binding still grants more: find it with kubectl get clusterrolebindings,rolebindings -A -o wide.
kubectl <command> --as=<subject>
Run any command as another user or service account and read what the API server answers them.
kubectl get pods -n billing --as=system:serviceaccount:billing:invoice-reader
--as-group=GROUP- Adds a group to the impersonated user; repeat it for several.
Forbidden- The refusal names the verb, the resource, the API group and the namespace a rule would have to cover.
Gotcha Logs and exec are permissions of their own: pods/log needs get and pods/exec needs create, on top of get for pods.
kubectl label namespace <ns> pod-security.kubernetes.io/enforce=<level>
Switch on Pod Security Admission for a namespace: privileged, baseline or restricted.
kubectl label namespace staging pod-security.kubernetes.io/enforce=baseline pod-security.kubernetes.io/warn=restricted
enforce- Pods that violate the level are refused.
warn- They are admitted, and kubectl prints a warning.
audit- They are admitted, and the audit event is annotated.
--dry-run=server- Shows which running pods the new level would refuse.
Gotcha Labels are checked when a pod is created. Pods that already run stay, and a Deployment only notices at its next rollout: look at the events of its ReplicaSet.
kube-bench run --targets <target>
Check a node against the CIS Kubernetes Benchmark: the flags of the control plane, the kubelet configuration and the modes of their files.
sudo kube-bench run --targets master --check 1.2.16
--targets master,etcd,node- Which part of the node to check.
--check 1.2.16,4.2.1- Only these checks.
--noremediations- Leave out the long advice.
Gotcha Each FAIL comes with the file and the flag to change. On a kubeadm node the control plane restarts by itself when its manifest changes; the kubelet needs systemctl restart kubelet.
crictl ps -a --name <container>
Ask the container runtime of a node what it runs. It answers when the API server does not, which makes it the way to find out why the API server is down.
sudo crictl ps -a --name kube-apiserver
-a- Exited containers too: a crashing one is only in this list.
crictl logs ID- What that container wrote; the first characters of the id are enough.
crictl pods- The pod sandboxes of the node.
Gotcha After a change to /etc/kubernetes/manifests/kube-apiserver.yaml, kubectl is refused until the new container is ready. If it stays refused, the last line of crictl logs names the flag or the file; no container at all means the kubelet could not read the manifest: journalctl -u kubelet.
etcdctl get /registry/secrets/<ns>/<name> | hexdump -C
Look at a Secret as it lies in etcd, to see whether the API server encrypts it at rest.
sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/default/db | hexdump -C | head -n 3
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 64 62 0a 6b 38 73 |s/default/db.k8s|
00000020 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 31 3a 6b |:enc:aescbc:v1:k|
--prefix --keys-only- List the keys below a path instead of one value.
--print-value-only- The value without its key.
kubectl get secrets -A -o json | kubectl replace -f -- Write every Secret again, so that the server encrypts what was stored before.
Gotcha k8s:enc:aescbc:v1:key1: in front means encrypted with that provider and key; k8s followed by the name of the type and readable values means plain. Switching encryption on changes nothing that is already stored, and the API server reads its configuration only when it starts.
apparmor_parser <profile file>
Load an AppArmor profile into the kernel, then check that it is there. A pod can only use a profile that is loaded on its node.
sudo apparmor_parser -q /etc/apparmor.d/k8s-deny-write && sudo aa-status | grep k8s-deny-write
-r- Replace a profile that is already loaded.
-R- Unload it.
aa-status- The loaded profiles, by mode.
Gotcha The name a pod uses is the one inside the file (profile NAME { ... }), not the file name. A pod naming a profile that is not loaded stays in CreateContainerError.
falco -V <rules file>
Ask Falco whether it would load a rules file, and where the first mistake is.
sudo falco -V /etc/falco/falco_rules.yaml -V /etc/falco/falco_rules.local.yaml
-V a -V b- Files in loading order: a later file uses the macros of an earlier one.
-L- Name and description of every rule.
-o json_output=true -l 'RULE'- One rule as Falco holds it, overrides applied: pipe it to jq.
--dry-run- Load the configuration and all rules, then exit.
Gotcha An output that starts with % has to be quoted, or the file is not YAML. A running Falco loads a valid change by itself and refuses a broken one; a restart with a broken file leaves it down, and the journal does not say why.