Author: Steven Crickman
Edited by: Vahe Aslanyan
Every platform team eventually hits the same wall. Self-service infrastructure and CI/CD pipelines let developers move fast, but speed without guardrails produces public S3 buckets, unencrypted databases, untagged resources nobody can bill back to a team, and containers running as root in production. The traditional fix — a human security reviewer manually inspecting every pull request or Terraform plan — doesn't scale, and it reintroduces exactly the kind of ticket-queue bottleneck that self-service platforms exist to eliminate.
Policy-as-code is the alternative: expressing compliance, security, and operational rules as machine-evaluated code that runs automatically in the same pipelines developers already use, rather than as a checklist a human works through by hand. Done well, it's invisible on the happy path — a compliant change sails through in milliseconds — and only surfaces friction when something actually violates a rule, with a clear, specific error message explaining what and why.
This handbook covers two of the dominant tools in this space — Open Policy Agent (OPA), a general-purpose policy engine with its own query language, Rego; and Kyverno, a Kubernetes-native policy engine that uses plain YAML — along with the patterns, pitfalls, and organizational practices that determine whether policy-as-code becomes an accelerant or a bottleneck.
Manual security and compliance review has three structural problems: it's slow (a human has to be available and context-switch into your change), it's inconsistent (different reviewers catch different things, and fatigue sets in on the fiftieth Terraform plan of the day), and it doesn't scale with the number of changes flowing through a platform. Policy-as-code addresses all three by moving the check to the moment of creation — in CI, at admission time, or during Terraform plan — where it's fast, consistent, and infinitely repeatable at zero marginal review cost.
The single most important design decision in a policy-as-code program is where checks run. The further left (earlier in the pipeline) a check runs, the cheaper the failure is to fix — a policy violation caught in a developer's local pre-commit hook costs seconds to fix; the same violation caught by a runtime admission controller after a deploy has already been attempted costs a failed deployment and a context switch; the same violation caught by a nightly compliance scan after the resource has been running in production for a week costs an incident and a retroactive fix. This handbook returns to this principle throughout: policy-as-code is most valuable not because it blocks bad things, but because it blocks them at the cheapest possible point in the pipeline.
Policy-as-code isn't a standalone practice — it's a layer that sits inside CI/CD pipelines, GitOps reconciliation loops, and Kubernetes admission control, validating the artifacts (Terraform plans, Kubernetes manifests, container images) that flow through them. It complements, rather than replaces, the self-service infrastructure patterns (golden paths, scaffolded templates) that reduce how often a bad default gets requested in the first place — policy-as-code is the safety net underneath, catching the exceptions and edge cases a golden path's defaults don't cover.
OPA is a general-purpose policy engine, not tied to any single system. It ingests structured data (JSON-like input) and evaluates it against policies written in Rego, a declarative query language purpose-built for expressing rules over nested data structures. Because OPA's input format is generic, the same engine can evaluate Terraform plans, Kubernetes manifests, API request payloads, or CI/CD pipeline configuration — anywhere you can serialize the thing you want to check into JSON.
OPA typically runs in one of three modes:
opa eval, or via the conftest wrapper), evaluating static files like Terraform plan JSON or Kubernetes YAML in a CI pipeline.kubectl apply requests before they reach etcd.Rego policies are organized into packages and express rules as logical statements over input data. A minimal policy that denies public S3 buckets:
package terraform.security
import future.keywords.in
deny[msg] {
resource := input.resource_changes[_]
resource.type == "aws_s3_bucket_acl"
resource.change.after.acl == "public-read"
msg := sprintf("S3 bucket %v must not have public-read ACL", [resource.address])
}
Reading this rule: for every element in input.resource_changes (a Terraform plan's JSON representation of proposed changes), if the resource type is an S3 bucket ACL and the proposed ACL is public-read, produce a deny message referencing the specific resource address. Rego's evaluation model is declarative — you're not writing an imperative "if/then," you're defining a set of conditions under which deny contains a message, and any non-empty deny set fails the policy check.
A common real-world policy is requiring cost-allocation tags on every taggable resource:
package terraform.tagging
import future.keywords.in
required_tags := {"owner", "environment", "cost-center"}
taggable_types := {
"aws_instance",
"aws_s3_bucket",
"aws_db_instance",
"aws_lambda_function",
}
deny[msg] {
resource := input.resource_changes[_]
resource.type in taggable_types
resource.change.actions[_] in {"create", "update"}
provided_tags := object.keys(object.get(resource.change.after, "tags", {}))
missing := required_tags - {tag | tag := provided_tags[_]}
count(missing) > 0
msg := sprintf(
"%v is missing required tags: %v",
[resource.address, concat(", ", missing)],
)
}
This policy computes the set difference between required tags and provided tags, and only fires when that difference is non-empty — meaning a resource with all required tags present produces no denial, regardless of what other tags it happens to have. Notice the policy is permissive by default on anything not explicitly listed in taggable_types — an important design choice covered further in Section 6.
Untested policy code is exactly as risky as untested application code — a subtly wrong Rego rule can either fail open (letting through what it should block) or fail closed (blocking legitimate changes), and both failure modes erode trust in the platform. OPA ships a native test framework:
package terraform.tagging_test
import data.terraform.tagging.deny
test_denies_missing_tags {
count(deny) > 0 with input as {
"resource_changes": [{
"address": "aws_instance.web",
"type": "aws_instance",
"change": {
"actions": ["create"],
"after": {"tags": {"owner": "team-a"}},
},
}],
}
}
test_allows_complete_tags {
count(deny) == 0 with input as {
"resource_changes": [{
"address": "aws_instance.web",
"type": "aws_instance",
"change": {
"actions": ["create"],
"after": {
"tags": {
"owner": "team-a",
"environment": "production",
"cost-center": "eng-42",
},
},
},
}],
}
}
$ opa test policies/ -v
data.terraform.tagging_test.test_denies_missing_tags: PASS (1.2ms)
data.terraform.tagging_test.test_allows_complete_tags: PASS (0.9ms)
PASS: 2/2
Treat this test suite exactly like application test code: it lives in version control next to the policies, runs in CI on every policy change, and is a required check before a policy update can merge — since a broken policy has blast radius across every team whose changes flow through it.
Conftest is a thin wrapper around OPA purpose-built for testing structured configuration files (Terraform plan JSON, Kubernetes manifests, Dockerfiles) against Rego policies:
# Generate a plan and convert it to JSON
terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json
# Evaluate it against the policy directory
conftest test tfplan.json --policy policies/
FAIL - tfplan.json - terraform.tagging - aws_instance.web is missing required tags: cost-center
2 tests, 1 passed, 0 warnings, 1 failure
This is the pattern from the earlier OpenTofu/Terraform and IDP handbooks in this series — a CI job that fails the build with a specific, actionable message rather than a human catching the same issue two days later in a manual review.
For enforcement at the cluster level — not just in CI, but blocking non-compliant manifests from ever being applied — Gatekeeper wraps OPA as a Kubernetes admission webhook, using ConstraintTemplate and Constraint custom resources:
# constrainttemplate.yaml — defines the reusable policy logic
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg}] {
required := input.parameters.labels
provided := input.review.object.metadata.labels
missing := required[_]
not provided[missing]
msg := sprintf("missing required label: %v", [missing])
}
# constraint.yaml — applies the template with specific parameters, scoped to a namespace
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: require-owner-label
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production"]
parameters:
labels: ["owner", "team"]
Once applied, any attempt to create a Pod in the production namespace without an owner and team label is rejected at admission time, before the object is ever persisted:
$ kubectl apply -f pod.yaml
Error from server ([denied by require-owner-label] missing required label: owner): error when creating "pod.yaml"
This split — ConstraintTemplate as reusable logic, Constraint as scoped parameters — lets a platform team define a policy once and apply it with different parameters or scope (different namespaces, different label requirements) without duplicating Rego code.
Kyverno was built specifically for Kubernetes, and its core design decision is that policies are expressed as native Kubernetes resources in YAML — no separate query language to learn. For teams whose policy needs are entirely Kubernetes-scoped (as opposed to also covering Terraform plans or arbitrary API payloads), this lower barrier to entry is Kyverno's main advantage over OPA/Gatekeeper: a platform engineer who already knows Kubernetes YAML can read, write, and review Kyverno policies without learning Rego.
Kyverno policies support three core actions natively: validate (allow or deny based on a rule), mutate (rewrite a resource before it's persisted), and generate (create additional resources in response to another resource being created) — the latter two have no direct OPA/Gatekeeper equivalent without additional tooling, and are a meaningful part of Kyverno's appeal.
The equivalent of the Gatekeeper label-requirement example, expressed in Kyverno:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-labels
spec:
validationFailureAction: Enforce
background: true
rules:
- name: check-for-labels
match:
any:
- resources:
kinds: ["Pod"]
namespaces: ["production"]
validate:
message: "The labels 'owner' and 'team' are required."
pattern:
metadata:
labels:
owner: "?*"
team: "?*"
The pattern block reads almost like an example of a compliant resource, with ?* meaning "any non-empty value" — this pattern-matching style is deliberately closer to describing "what good looks like" than writing imperative validation logic, which is a large part of why Kyverno policies tend to be quicker to read and write for straightforward rules.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-privileged-containers
spec:
validationFailureAction: Enforce
rules:
- name: privileged-containers
match:
any:
- resources:
kinds: ["Pod"]
validate:
message: "Privileged containers are not allowed."
pattern:
spec:
containers:
- =(securityContext):
=(privileged): "false"
The =() syntax marks a field as conditional — the rule only evaluates privileged if securityContext is present at all, avoiding false positives against Pods that don't set a securityContext block explicitly but also aren't requesting privileged mode.
Where Gatekeeper can only allow or deny, Kyverno can also silently correct non-compliant-but-fixable resources before they're persisted — useful for defaults developers shouldn't need to remember to set themselves:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-default-resources
spec:
rules:
- name: add-resource-limits
match:
any:
- resources:
kinds: ["Pod"]
mutate:
patchStrategicMerge:
spec:
containers:
- (name): "*"
resources:
limits:
+(memory): "512Mi"
+(cpu): "500m"
The +() prefix means "add this field only if it's not already set" — a Pod spec that already declares its own memory limit is left untouched, while one that omits it gets a sane platform default injected automatically, with no developer action required and no build failure to interrupt them.
Kyverno's generate rules can create a resource automatically whenever a triggering resource is created — a common pattern is auto-creating a default NetworkPolicy whenever a new namespace is provisioned:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: generate-default-networkpolicy
spec:
rules:
- name: default-deny-networkpolicy
match:
any:
- resources:
kinds: ["Namespace"]
generate:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
name: default-deny-all
namespace: "{{request.object.metadata.name}}"
synchronize: true
data:
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
Every new namespace gets a default-deny network policy the moment it's created, without a platform engineer or developer needing to remember to add one — the secure default is structurally guaranteed rather than dependent on someone following a checklist.
Kyverno ships its own CLI test framework, following a similar philosophy to OPA's:
# test.yaml
name: require-labels-test
policies:
- require-labels.yaml
resources:
- compliant-pod.yaml
- noncompliant-pod.yaml
results:
- policy: require-labels
rule: check-for-labels
resource: compliant-pod
kind: Pod
result: pass
- policy: require-labels
rule: check-for-labels
resource: noncompliant-pod
kind: Pod
result: fail
$ kyverno test .
Executing require-labels-test...
Checking policy require-labels...
✓ pass: compliant-pod
✓ fail: noncompliant-pod
Test Summary: 2 tests passed, 0 failed
Neither tool is strictly superior — they optimize for different scopes and teams:
Choose OPA/Gatekeeper when:
Choose Kyverno when:
A common pattern in practice: use OPA/Conftest in CI/CD pipelines to validate Terraform plans and other non-Kubernetes artifacts, and use Kyverno as the Kubernetes admission controller for cluster-native policy — leaning on each tool's comparative strength rather than picking one exclusively for the whole stack. Running both is not unusual and doesn't imply either tool was the "wrong" choice.
DimensionOPA / GatekeeperKyvernoPolicy languageRego (general-purpose query language)Native Kubernetes YAMLScopeAny JSON-serializable input — Terraform, Kubernetes, custom APIsKubernetes-native onlyValidateYesYesMutate (auto-fix)Limited, via separate toolingNative, first-classGenerate companion resourcesNot nativeNative, first-classLearning curve for Kubernetes-only teamsSteeper — requires learning RegoLower — YAML patterns familiar to most K8s usersCross-system reuse (same policy engine for IaC and runtime)StrongWeakNative CLI test frameworkYes (opa test)Yes (kyverno test)Typical adoption contextPlatform teams already invested in OPA elsewhere, or needing Terraform + Kubernetes coverage from one engineKubernetes-first teams wanting the fastest path to writing their first policy
Teams sometimes start with one tool and later add or switch to the other as their needs grow — typically starting with Kyverno for its lower barrier to entry on cluster-native rules, then adding OPA/Conftest once Terraform or non-Kubernetes policy needs emerge, rather than trying to force Rego to solve a problem Kyverno's YAML already solves more simply. Moving in the other direction — starting with OPA/Gatekeeper and later adopting Kyverno for its mutate/generate capabilities — is equally common among teams that found themselves reaching for a third-party mutating webhook to fill a gap Gatekeeper alone doesn't cover. Running both concurrently, scoped to what each does best, is the steady-state outcome for most platform teams past a certain size, and isn't a sign of an unfinished decision.
This section is the difference between a policy-as-code program developers tolerate and one they actively resist and route around. Getting the mechanics of OPA or Kyverno right is necessary but not sufficient — how the guardrails are designed and rolled out determines their actual, lasting effectiveness.
Never ship a new policy directly in enforcing mode. Both tools support a non-blocking mode — OPA/Conftest can be run with a --output format that reports without failing the build, and Kyverno has a native validationFailureAction: Audit setting:
spec:
validationFailureAction: Audit # logs violations without blocking
Run every new policy in audit/warn mode for a defined burn-in period — commonly 1-2 weeks — and review what it would have blocked before flipping it to enforcing. This surfaces false positives and legitimate-but-unanticipated use cases before they become a blocked deploy at 5pm on a Friday, and it gives the platform team real data on a policy's actual impact rather than a guess.
A denial message like "denied by policy" sends a developer straight to the platform team's Slack channel. A denial message like the ones in Sections 3.2–3.4 and 4.2 — naming the specific resource, the specific missing requirement, and implicitly the fix — resolves the majority of violations without a human ever getting involved. Treat writing a good msg/message field with the same care as writing a good error message in application code; it is, functionally, exactly that.
Every policy has an implicit answer to "what happens to inputs this policy doesn't recognize?" The tagging policy in Section 3.3 only evaluates resources in an explicit taggable_types set — anything not in that set passes silently. This is usually the right default for early-stage policy rollout (avoiding false positives on resource types nobody has thought about yet), but it means coverage gaps are silent by default. Track policy coverage explicitly — what fraction of actual resource types in your environment are covered by at least one policy — rather than assuming "no violations" means "fully compliant."
Golden paths and policies are opinionated defaults, not absolute mandates — there are always legitimate exceptions (a load test that genuinely needs a larger instance than policy allows; a legacy system that can't yet meet a new tagging standard). An escape hatch that requires zero accountability (a magic annotation anyone can add) undermines the policy entirely; one that requires so much process it's effectively unusable pushes teams to quietly work around the platform instead. A workable middle ground: an explicit, auditable exception mechanism.
# Kyverno exclusion via annotation, requiring an approved exception ticket reference
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-privileged-containers
spec:
rules:
- name: privileged-containers
match:
any:
- resources:
kinds: ["Pod"]
selector:
matchExpressions:
- key: policy.company.io/exception-ticket
operator: DoesNotExist
validate:
message: "Privileged containers are not allowed without an approved exception (see policy.company.io/exception-ticket)."
pattern:
spec:
containers:
- =(securityContext):
=(privileged): "false"
Requiring the annotation to reference a real, reviewed exception ticket means every deviation from the guardrail is discoverable and auditable later, rather than being an invisible, undocumented workaround.
Policies deserve the same rigor as the systems they govern: version control, a required review process, automated tests (Sections 3.4 and 4.6), and a staged rollout across environments rather than a single global flip. A policy change that breaks CI for every team simultaneously is itself an incident, and should be treated with the same care as a bad application deploy — including a fast, well-understood rollback path.
# A minimal CI pipeline for the policy repository itself
opa test policies/ -v # unit tests for Rego logic
conftest verify --policy policies/ # validates policy syntax
kyverno test kyverno-policies/ # unit tests for Kyverno rules
Track the same category of leading indicators a platform team tracks for golden path adoption:
As a policy-as-code program matures, the number of individual rules tends to grow past what any one person can hold in their head, especially across multiple teams contributing policies independently. Organize policies into clearly scoped categories (security, cost, tagging, reliability) with clear ownership per category, and periodically audit for redundant or contradictory rules — two policies that both technically pass but disagree about what "correctly tagged" means are a maintenance trap waiting to surface as a confusing, hard-to-debug denial.
For a platform team introducing policy-as-code for the first time, a reasonable sequencing:
The sections above describe policy-as-code mechanics individually; this section follows one policy — mandatory cost-allocation tagging — through its full lifecycle, from first draft to steady-state enforcement, to show how the pieces connect in practice.
Maria, a platform engineer, is asked to stop the recurring problem of untagged AWS resources showing up in the monthly cost report with no owning team. She starts from the Rego policy in Section 3.3, adds the test cases from Section 3.4, and runs them locally before anything touches a real pipeline:
$ opa test policies/ -v
data.terraform.tagging_test.test_denies_missing_tags: PASS (1.1ms)
data.terraform.tagging_test.test_allows_complete_tags: PASS (0.8ms)
PASS: 2/2
She wires the policy into the shared Terraform CI pipeline, but in audit mode — implemented here as a Conftest run whose exit code is intentionally ignored so it reports without blocking:
# .github/workflows/terraform-ci.yaml (excerpt)
- name: Policy check (audit mode)
run: |
terraform show -json tfplan > tfplan.json
conftest test tfplan.json --policy policies/ --output json > policy-report.json || true
cat policy-report.json
Over the following two weeks, the audit-mode job runs on every Terraform plan across the org without blocking anything. Maria reviews the accumulated policy-report.json output and finds two categories of real violation:
{
"filename": "tfplan.json",
"namespace": "terraform.tagging",
"failures": [
{"msg": "aws_lambda_function.image_resizer is missing required tags: cost-center"},
{"msg": "aws_instance.legacy_batch_job is missing required tags: owner, cost-center"}
]
}
The first is a simple oversight — the team just needs to add the tag. The second, legacy_batch_job, turns out to be a system nobody has clearly owned since a reorg eighteen months earlier — exactly the kind of gap the policy was designed to surface, and exactly the kind of thing that would have gone undetected indefinitely without this check.
Before flipping the policy to blocking mode, Maria adds the exception mechanism from Section 6.4, adapted for Terraform via a variable-driven override rather than a Kubernetes annotation:
package terraform.tagging
exempt_resources := {r | r := input.resource_changes[_]; r.address == input.exceptions[_]}
deny[msg] {
resource := input.resource_changes[_]
not resource in exempt_resources
resource.type in taggable_types
resource.change.actions[_] in {"create", "update"}
provided_tags := object.keys(object.get(resource.change.after, "tags", {}))
missing := required_tags - {tag | tag := provided_tags[_]}
count(missing) > 0
msg := sprintf("%v is missing required tags: %v", [resource.address, concat(", ", missing)])
}
# Exceptions are supplied explicitly and reviewed in the pull request that adds them
conftest test tfplan.json --policy policies/ --data exceptions.json
// exceptions.json — reviewed and merged like any other change, so it's auditable
{
"exceptions": ["aws_instance.legacy_batch_job"]
}
With the exception path in place and reviewed, she flips validationFailureAction-equivalent behavior to blocking for the Growth team first — the team that originally reported the cost-attribution problem and is motivated to see it enforced — by making the CI step's exit code authoritative:
- name: Policy check (enforcing)
run: |
terraform show -json tfplan > tfplan.json
conftest test tfplan.json --policy policies/ --data exceptions.json
# no `|| true` here — a failure now fails the build
After two weeks of clean enforcement for the pilot team with no unexpected blocks, Maria rolls the same enforcing configuration out org-wide. She adds a lightweight dashboard tracking the metrics from Section 6.6:
{
"policy": "terraform.tagging",
"month": "2026-08",
"violations_caught_in_ci": 34,
"exceptions_granted": 2,
"mean_time_to_fix_minutes": 6,
"false_positive_reports": 0
}
A mean time-to-fix of six minutes — the time it takes a developer to read the denial message, add the missing tag, and re-run the plan — is the concrete evidence that the guardrail is doing its job without becoming a bottleneck. Two exceptions, both reviewed and merged deliberately, is a healthy number: enough to show the escape hatch works, not so many that it suggests the policy itself is miscalibrated.
What's notable across this rollout is the same pattern described abstractly in Section 6: audit before enforce, a pilot team before org-wide rollout, a reviewed exception path established before it was needed, and concrete metrics collected from day one rather than relying on the absence of complaints as a proxy for success.
Policy-as-code succeeds when it's judged the way any other piece of developer-facing infrastructure should be judged: by how invisible it is on the happy path, and how clear and actionable it is on the unhappy one. OPA and Kyverno solve overlapping but distinct problems — OPA's generality suits cross-system enforcement from Terraform plans to arbitrary API decisions, while Kyverno's native Kubernetes YAML lowers the barrier for teams whose policy needs stay inside the cluster — and many mature platforms end up running both, each where its strengths matter most.
The technology, though, is the easier half. The harder half is organizational discipline: rolling out new policies in audit mode before enforcing them, writing denial messages that resolve issues without a human in the loop, giving legitimate exceptions an auditable path instead of an invisible one, and treating policies as versioned, tested code rather than a pile of YAML that accretes indefinitely. Done this way, guardrails stop being something teams route around and become something they mostly never notice — which is exactly the point.
ConstraintTemplate and Constraint resources.ClusterPolicy/Policy resources.kubectl apply.
%402x.svg){kind=icon}