Skip to main content

Audit & Evidence

Purpose: For security officers, shows how to locate audit logs and generate compliance evidence for regulatory reviews.

Task Summary​

Compliance reviews require evidence that security controls are active and enforced. openCenter produces audit evidence from four sources: Kubernetes API server audit logs, Kyverno policy reports, RBAC configuration, and secrets rotation records. This guide shows where each type of evidence lives and how to collect it.

Prerequisites​

  • kubectl access to the cluster with cluster-admin privileges
  • SSH access to control plane nodes (for API server audit logs)
  • opencenter CLI installed (for key rotation records)

API Server Audit Logs​

The Kubernetes API server logs all requests to the audit log. This includes resource creation, modification, deletion, and authentication events.

Location on control plane nodes:

/var/log/kubernetes/audit/audit.log

Collect Audit Logs​

# SSH to a control plane node
ssh user@control-plane-1

# View recent audit events
tail -100 /var/log/kubernetes/audit/audit.log | jq .

# Filter for specific resource types
grep '"resource":"secrets"' /var/log/kubernetes/audit/audit.log | tail -20 | jq .

# Filter for specific users
grep '"username":"admin@example.com"' /var/log/kubernetes/audit/audit.log | jq .

If Loki is configured, audit logs are also available via Grafana. Query with LogQL:

{job="kubernetes-audit"} | json | resource="secrets"

Kyverno Policy Reports​

Kyverno generates PolicyReport and ClusterPolicyReport resources that record every policy evaluation — both passes and violations.

Collect Policy Reports​

# List all cluster-wide policy reports
kubectl get clusterpolicyreport

# List namespace-scoped reports
kubectl get policyreport -A

# Export a detailed report for a namespace
kubectl get policyreport -n <namespace> -o yaml > evidence/policyreport-<namespace>.yaml

# Count violations by policy
kubectl get policyreport -A -o json | \
  jq '[.items[].results[] | select(.result=="fail")] | group_by(.policy) | map({policy: .[0].policy, count: length})'

Policy reports are retained in the cluster as long as the evaluated resource exists. For long-term retention, export reports periodically or forward them to Loki.

RBAC Audit​

Document the current RBAC configuration to show who has access to what.

# List all ClusterRoleBindings
kubectl get clusterrolebindings -o wide > evidence/clusterrolebindings.txt

# List all RoleBindings across namespaces
kubectl get rolebindings -A -o wide > evidence/rolebindings.txt

# Show RBACDefinitions (openCenter RBAC Manager)
kubectl get rbacdefinitions -o yaml > evidence/rbacdefinitions.yaml

# Check effective permissions for a user
kubectl auth can-i --list --as=admin@example.com

Secrets Rotation Records​

Key rotation events are tracked by the openCenter CLI. Use check-keys to produce a point-in-time snapshot:

opencenter cluster check-keys <cluster-name> > evidence/key-rotation-status.txt

This output includes:

  • Current Age key creation date and age in days
  • Current SSH key creation date and age in days
  • Whether keys are within policy (90-day Age, 180-day SSH)

Git history provides a rotation audit trail. Each key rotation produces commits that update .sops.yaml and re-encrypt secrets:

git log --oneline --all -- '.sops.yaml' 'secrets/' > evidence/rotation-history.txt

Verification​

After collecting evidence, verify completeness:

Evidence TypeSourceExpected Content
API audit logsControl plane nodes or LokiRequest/response records for all API calls
Policy reportskubectl get policyreport -APass/fail results for all 17 ClusterPolicies
RBAC snapshotkubectl get clusterrolebindingsAll role bindings with subjects and roles
Key rotationopencenter cluster check-keysKey ages within policy thresholds

Further Reading​