Author: Steven Crickman
Editor: Vahe Aslanyan
As organizations scale past a handful of engineering teams, the cognitive load placed on individual developers grows faster than headcount. Provisioning a database, standing up a new service, configuring CI/CD, wiring up observability, and navigating security review all require context that no single engineer can hold for long. A developer who joined two years ago and shipped five services already knows the unwritten rules; a developer who joined two months ago is left guessing, asking around in Slack, or copy-pasting from whichever existing service looks closest to what they need — inheriting its mistakes along with its structure.
The traditional organizational response to this is a dedicated platform or DevOps team that handles requests via tickets: "please provision a database," "please set up a new CI pipeline," "please grant access to this bucket." This doesn't scale either — it just moves the bottleneck from "every developer needs infra knowledge" to "every infra request needs a human in the loop." As the org grows, the queue grows, and the platform team becomes the thing slowing everyone down, which is precisely the opposite of what platform teams are meant to do.
An Internal Developer Platform (IDP) is the architectural and organizational answer to this problem. It is a curated, self-service layer that sits between raw infrastructure (Kubernetes, cloud accounts, CI systems) and the application developers who need to consume it — without requiring those developers to become infrastructure experts themselves.
A good IDP does three things simultaneously:
This handbook walks through building such a platform using Backstage as the developer portal layer, golden path templates as the paving mechanism, and a self-service infrastructure layer underneath, supported by policy-as-code guardrails and a product-oriented operating model for the platform team itself.
Before touching tooling, it's worth being precise about the layers involved, because conflating them is one of the most common causes of failed platform initiatives — teams often start by buying or building a portal UI, discover six months later that there's no underlying self-service automation behind it, and end up with a pretty catalog that still requires tickets for anything useful.
A platform team's job is not to build infrastructure — that's typically already there, owned by SRE or cloud infra — but to build the glue that makes infrastructure consumable without a ticket queue. Treating the IDP as a software product, with developers as customers, is the single most important mental shift platform teams need to make, and it's a theme this handbook returns to repeatedly because it explains most of the difference between IDP efforts that stick and ones that quietly die after the initial launch enthusiasm fades.
It's also worth naming what an IDP is not. It is not a replacement for a service mesh, not a replacement for your CI system, and not a single tool you can buy off the shelf and be done with. It's an integration layer — its value comes from how well it stitches together things that already exist, plus a handful of opinionated defaults that didn't exist before.
Backstage, originally built at Spotify and donated to the CNCF, is an open-source framework for building developer portals. It is not a finished product — it's a framework with a plugin architecture, a software catalog, and a templating engine (Scaffolder) that you assemble into your own portal. This distinction matters because organizations sometimes adopt Backstage expecting an out-of-the-box experience comparable to a SaaS product, then are surprised by the engineering investment required to make it genuinely useful. Budget for a small, dedicated team (often 2-4 engineers to start) who treat Backstage as a product they own, not a tool they install once.
Core components:
catalog-info.yaml) files that live alongside the code they describe.Backstage ships as a Node.js application you deploy yourself — typically on Kubernetes, fronted by an ingress and backed by PostgreSQL. Key early decisions, each of which is expensive to reverse later:
catalog-info.yaml combined with a discovery processor that scans your source control org), or synced from an external system of record (e.g., a CMDB or existing service registry). Auto-discovery scales better than manual registration but requires more upfront tooling work.The catalog models the organization as a graph of entities: Component, System, API, Resource, Group, User, Domain. Each Component (typically a service) declares ownership, its system membership, dependencies, and links to docs, dashboards, and runbooks.
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: payments-api
description: Handles payment authorization and settlement
annotations:
github.com/project-slug: org/payments-api
backstage.io/techdocs-ref: dir:.
pagerduty.com/service-id: PXXXXX
tags:
- payments
- tier-1
spec:
type: service
lifecycle: production
owner: team-payments
system: payments
providesApis:
- payments-api-v2
dependsOn:
- resource:payments-db
- component:fraud-detection
The catalog's value compounds over time: once every service declares its owner, you get instant answers to questions that previously required Slack archaeology — "who owns this," "what depends on this," "is this service deprecated," "what's the blast radius if this goes down." This is the difference between a catalog that's a nice-to-have directory and one that's load-bearing infrastructure for incident response and architectural decision-making.
Common failure mode: treating catalog population as a one-time migration project rather than an ongoing process enforced by CI. A typical pattern is a CI check that fails the build if catalog-info.yaml is missing, malformed, or references a non-existent owner team. Without this kind of enforcement, catalog entropy sets in within months as new services get created without registration and existing entries drift out of date as ownership changes.
Beyond individual components, Backstage's System and Domain entities let you model architecture at higher levels of abstraction — a System groups related components (e.g., "payments" containing payments-api, payments-worker, and payments-db), and a Domain groups related systems under a business capability (e.g., "commerce"). This hierarchical modeling becomes valuable once an org has more than a few dozen services, because it lets new engineers orient themselves at the right altitude — "what systems exist in commerce" — before drilling into individual component details.
Backstage's plugin architecture is what lets the portal become a genuine single pane of glass rather than just a catalog browser. Common plugin categories:
Building a custom plugin is a reasonable investment once you have two or three concrete, recurring developer pain points that existing community plugins don't solve. Resist the urge to build plugins speculatively — a common pattern in struggling platform teams is investing heavily in a plugin nobody asked for while the basics (a populated catalog, one working golden path) remain unfinished.
As Backstage adoption grows, access control becomes a real concern: not every developer should be able to trigger every scaffolder template, and not every catalog entity should be editable by anyone who happens to find it. Backstage's permission framework allows fine-grained policies — for example, restricting "provision a production database" templates to senior engineers or requiring an approval step, while keeping "create a new service" open to everyone.
A reasonable default model:
owner field.Getting this right early avoids a common later-stage problem where the platform team becomes a de facto approval bottleneck for every template, recreating the exact ticket queue the IDP was meant to eliminate.
A golden path is the happy path for a common task — "create a new microservice," "add a Kafka topic," "spin up a preview environment" — implemented so that following it requires no infrastructure knowledge, and deviating from it is possible but explicitly unsupported (meaning: still allowed, but the platform team won't optimize for it, and the developer takes on more operational responsibility by going around the path).
The key word is opinionated. A golden path is not a menu of options; it's a small number of well-tested defaults. Offering twelve database choices in a template defeats the purpose — offer one or two, with a clearly signposted escape hatch for genuine exceptions. The temptation to make templates maximally flexible, to accommodate every possible team's preference, is one of the most reliable ways to turn a golden path into a configuration maze that takes longer to use correctly than the manual process it was meant to replace.
Backstage's Scaffolder executes templates defined in YAML, composed of parameterized input forms and a sequence of actions: fetch a skeleton repo, template variables into it, create a source control repo, register it in the catalog, and trigger an initial CI run.
A minimal "new service" template:
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
name: new-go-service
title: New Go Microservice
description: Creates a production-ready Go service with CI, observability, and catalog registration pre-wired.
spec:
parameters:
- title: Service details
required: [name, owner, system]
properties:
name:
type: string
description: Service name (lowercase, hyphenated)
owner:
type: string
ui:field: OwnerPicker
system:
type: string
ui:field: EntityPicker
ui:options:
catalogFilter:
kind: System
steps:
- id: fetch
name: Fetch skeleton
action: fetch:template
input:
url: ./skeleton
values:
name: ${{ parameters.name }}
owner: ${{ parameters.owner }}
- id: publish
name: Create repository
action: publish:github
input:
repoUrl: github.com?repo=${{ parameters.name }}
defaultBranch: main
- id: register
name: Register in catalog
action: catalog:register
input:
repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
catalogInfoPath: /catalog-info.yaml
output:
links:
- title: Repository
url: ${{ steps.publish.output.remoteUrl }}
- title: Catalog entry
icon: catalog
entityRef: ${{ steps.register.output.entityRef }}
The skeleton repo this references is where the real value lives, and it should already include: a working Dockerfile, a complete CI pipeline configuration (lint, test, build, scan, deploy stages), baseline observability instrumentation (structured logging, a metrics endpoint, trace propagation already wired into the HTTP client), a catalog-info.yaml pre-filled with the parameterized values, and security defaults (non-root container user, dependency scanning enabled, secrets sourced from a vault rather than environment files). The golden path's value isn't the repository creation step itself — it's everything bundled inside the skeleton that the developer didn't have to research, decide on, or configure manually.
Once the new-service path exists and has real adoption, the same scaffolding pattern extends naturally to other recurring, high-friction tasks:
Golden paths are also a governance mechanism, and this is worth stating explicitly because it reframes how security and compliance teams should think about engaging with the platform. By making the paved path the easiest path — easier than going around the platform — you achieve compliance through convenience rather than mandate. Encoding requirements like mandatory resource tagging, encryption-at-rest, or approved base images directly into templates means policy is enforced structurally, at the moment of creation, rather than discovered later through an audit and remediated after the fact at much higher cost.
This doesn't eliminate the need for guardrail enforcement elsewhere (see Section 6 on policy-as-code), since developers can and sometimes will go around the golden path for legitimate reasons. But it shifts the large majority of compliance work left, into the default path, where it costs the developer nothing extra and costs the security team far less ongoing audit effort.
Golden paths are not "build once, done" artifacts. As the org's standards evolve — a new base image, an updated CI tool, a new required security scan — templates need to be versioned and existing services need a path to adopt the update. A practical approach is treating the skeleton repo itself as a dependency: services generated from a template retain a reference to the template version they were scaffolded from, and a periodic (often quarterly) "golden path upgrade" pass — sometimes automated as a bot-generated pull request — proposes the delta to each consuming repo. Without this mechanism, golden paths calcify: the template stays frozen at whatever standards existed when it was first written, while best practices for everything it encodes continue to move forward elsewhere.
Golden paths for code are necessary but insufficient — developers also need infrastructure: databases, queues, storage buckets, caches, secrets. The traditional model routes these requests through tickets to an infrastructure team, with turnaround measured in days. Self-service infrastructure replaces the ticket with an API or form, backed by automation that still respects organizational policy, with turnaround measured in minutes.
Whatever the front-end mechanism, the underlying provisioning logic should be expressed declaratively, typically via one of two complementary approaches:
Database custom resource) that compile down to the underlying cloud resources via a controller. This fits especially naturally into a Kubernetes-centric platform, since the same kubectl apply / GitOps mental model used for application deployment extends directly to infrastructure provisioning.The principle in both cases is identical: developers shouldn't write raw Terraform or click around a cloud console. They request a higher-level abstraction — "give me a Postgres database, small tier, with backups" — and the platform translates that request into compliant, lower-level resources on their behalf, with the complexity of the underlying cloud APIs entirely hidden.
Backstage Scaffolder templates become the front door to this layer. A template for "new database" collects a small number of parameters (engine, size tier, environment) and then takes one of two architectural paths:
The GitOps pattern is generally preferable for production infrastructure: every change is a reviewable, revertible artifact in version control, and the "who changed what, when, and why" question is answered for free by git history and commit messages, which matters enormously during incident postmortems. The direct-API pattern earns its keep for lower-risk, high-frequency operations — e.g., short-lived preview environments — where the overhead of a pull-request cycle outweighs the benefit.
Self-service only works if "no human bottleneck" doesn't mean "no checks." Policy engines — Open Policy Agent (OPA, often via its Conftest tooling), Kyverno for Kubernetes-native resources, or HashiCorp Sentinel for Terraform — validate generated manifests automatically before they're applied: enforcing tagging standards, blocking public S3 buckets, capping instance sizes by environment, requiring encryption at rest, restricting which regions resources can be created in.
These checks run automatically in the CI/CD pipeline as a required step before apply, so the human reviewer (if a review step exists at all) is reviewing business intent — "does this team genuinely need a large instance" — rather than manually checking for compliance details a machine can verify far more reliably. A representative OPA policy might look like:
package terraform.policies
deny[msg] {
resource := input.resource_changes[_]
resource.type == "aws_s3_bucket"
not resource.change.after.tags["owner"]
msg := sprintf("S3 bucket %v is missing required 'owner' tag", [resource.address])
}
Self-service infrastructure without cost visibility tends to produce sprawl, since the natural friction that previously made someone think twice before requesting a large resource (waiting for a ticket, justifying the request to a human) disappears. Two practices mitigate this:
No template library will cover every infrastructure need, and a mature self-service platform needs an explicit process for the long tail of requests that don't fit existing golden paths — a "request a new resource type" path that routes to the platform team, distinct from "I have a question," with a service-level expectation attached. Treating these as product feedback (candidates for the next golden path to build) rather than one-off tickets to dispatch and forget keeps the template library growing in the direction of actual demand.
The most common reason IDP initiatives stall isn't technical — it's that the platform team builds what they assume developers need rather than what developers actually struggle with. Apply standard product practices:
Useful metrics fall into two categories.
Outcome metrics (the org-level effects you ultimately care about):
Adoption and health metrics (leading indicators that predict the outcome metrics above):
Avoid vanity metrics like raw Backstage login counts — they correlate poorly with actual value delivered and can be inflated trivially (e.g., by making the portal the company homepage) without reflecting genuine usefulness.
Technical excellence alone doesn't guarantee adoption; how a platform is introduced matters nearly as much as what it contains.
For organizations starting from scratch, a reasonable sequencing is below, with rough timeline and staffing figures attached. These numbers assume a mid-sized organization (roughly 150-400 engineers, a few dozen services) and a dedicated platform team of 2-4 engineers; adjust up or down for scale, but the relative proportions — most of the early budget going to one excellent golden path rather than breadth — tend to hold regardless of company size.
catalog-info.yaml in every repo, paired with a focused, time-boxed effort to backfill existing services rather than letting backfill drag on indefinitely. Roughly 3-6 weeks, scaling with service count; for 30-50 services this is usually a few days of scripting to auto-generate a baseline catalog-info.yaml from existing repo metadata, plus a couple of weeks of chasing down ownership for the stragglers.Put together, a realistic time-to-first-value — Backstage live, catalog populated, one golden path in daily use — is roughly 3-4 months for a team of this size, with self-service infrastructure following another 6-10 weeks behind that. Organizations frequently underestimate step 3 specifically, treating the Scaffolder template as the hard part when it's usually the skeleton repo's production-readiness that consumes the bulk of the time. This sequencing front-loads trust-building — a working catalog, one genuinely excellent golden path — before expanding scope, which tends to produce durable, organic adoption rather than a portal that launches with fanfare and is quietly abandoned within a quarter once the novelty wears off.
notifications-api Service, Start to FinishThe sections above describe the platform's layers individually; this section follows one realistic service through all of them as a single connected story, with the actual artifacts produced at each step, so the pieces can be seen fitting together rather than just described in the abstract.
Priya, an engineer on the Growth team, needs a new service to handle push and email notifications. She opens Backstage and runs the new-go-service template from Section 4.2, submitting these values through the form:
# Values submitted to the Scaffolder via the template's input form
name: notifications-api
owner: team-growth
system: growth
The Scaffolder executes the template's steps against these values, and within about ninety seconds she has a new repository, org/notifications-api, generated from the skeleton. The generated catalog-info.yaml at the repo root looks like this:
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: notifications-api
description: Handles push and email notification delivery
annotations:
github.com/project-slug: org/notifications-api
backstage.io/techdocs-ref: dir:.
tags:
- growth
spec:
type: service
lifecycle: production
owner: team-growth
system: growth
dependsOn:
- resource:notifications-db
The skeleton also generates a working CI pipeline. The .github/workflows/ci.yaml produced looks like this:
name: CI
on:
pull_request:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
- name: Lint
run: golangci-lint run ./...
- name: Test
run: go test ./... -coverprofile=coverage.out
- name: Build
run: go build -o bin/notifications-api ./cmd/server
- name: Scan image
run: trivy image --exit-code 1 --severity CRITICAL,HIGH notifications-api:${{ github.sha }}
- name: Push image
if: github.ref == 'refs/heads/main'
run: |
docker build -t registry.internal/notifications-api:${{ github.sha }} .
docker push registry.internal/notifications-api:${{ github.sha }}
And a minimal, already-instrumented HTTP entrypoint, cmd/server/main.go:
package main
import (
"log/slog"
"net/http"
"os"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
mux := http.NewServeMux()
mux.Handle("/metrics", promhttp.Handler())
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
logger.Info("starting notifications-api", "port", 8080)
if err := http.ListenAndServe(":8080", mux); err != nil {
logger.Error("server exited", "error", err)
os.Exit(1)
}
}
Priya adds her actual notification-sending handlers to this scaffold that afternoon, rather than spending the day assembling logging, metrics, Dockerfiles, and CI from scratch.
The service needs a Postgres database for notification preferences and delivery history. Priya runs the "new database" golden path from Section 5.3, submitting:
# Values submitted to the "new database" Scaffolder template
service: notifications-api
engine: postgres
size: small
environment: staging
This template step opens a pull request against the platform team's GitOps repo, adding the following Crossplane claim:
# gitops-repo/claims/notifications-db.yaml
apiVersion: platform.org/v1alpha1
kind: Database
metadata:
name: notifications-db
labels:
owner: team-growth
service: notifications-api
spec:
engine: postgres
size: small
environment: staging
This claim is validated against the platform team's existing Database composition, which fixes the settings the org always requires regardless of what the requester specified:
# gitops-repo/compositions/database-composition.yaml (excerpt, pre-existing)
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: postgres-standard
spec:
compositeTypeRef:
apiVersion: platform.org/v1alpha1
kind: Database
resources:
- name: rds-instance
base:
apiVersion: rds.aws.upbound.io/v1beta1
kind: Instance
spec:
forProvider:
engine: postgres
storageEncrypted: true
backupRetentionPeriod: 7
deletionProtection: true
tags:
managed-by: crossplane
The pull request triggers the OPA policy check from Section 5.4 automatically as a CI job:
$ conftest test claims/notifications-db.yaml --policy policies/
PASS - claims/notifications-db.yaml - terraform.policies - owner tag present
PASS - claims/notifications-db.yaml - terraform.policies - encryption enabled
PASS - claims/notifications-db.yaml - terraform.policies - environment is valid
3 tests, 3 passed, 0 warnings, 0 failures
Because the check passes, a platform engineer merges the PR after a thirty-second glance — the real verification already happened in CI, not in their head. ArgoCD detects the merge and reconciles the actual RDS instance within a few minutes; Priya receives a Slack notification from the platform's bot once the claim reaches Ready: True.
Database credentials are delivered to the running service via the platform's standard secrets-injection convention, so no manual credential handling is required. Priya's Kubernetes deployment manifest, generated as part of the skeleton, already references it correctly:
# k8s/deployment.yaml (excerpt, from the skeleton)
apiVersion: apps/v1
kind: Deployment
metadata:
name: notifications-api
spec:
replicas: 2
template:
spec:
containers:
- name: notifications-api
image: registry.internal/notifications-api:latest
envFrom:
- secretRef:
name: notifications-db-credentials # auto-populated by the platform's secrets operator
ports:
- containerPort: 8080
Her first merged pull request runs the CI pipeline shown in 8.1: tests pass, Trivy reports no critical or high vulnerabilities, and the pipeline promotes the build to staging automatically. The notifications-api catalog page now shows a green CI badge, a link to the running staging environment, and — via the cost-visibility plugin — an estimated $340/month in infrastructure spend, computed from the size: small RDS instance and the two-replica deployment, tagged automatically with owner: team-growth for the Growth team's cost dashboard.
In production, notifications-api's catalog page surfaces its DORA metrics, pulled from the CI/CD and incident systems via the DORA plugin:
{
"component": "notifications-api",
"deployment_frequency": "1.1 / day",
"lead_time_for_changes": "1h 42m",
"change_failure_rate": "3.2%",
"time_to_restore_service": "24m"
}
These numbers are consistent with the org-wide averages for services built on the same golden path — a useful signal to the platform team that the path is performing as intended rather than producing a one-off success.
The platform team updates the new-go-service skeleton's base image, per the template versioning practice from Section 4.5. An automated bot, tracking which repos were scaffolded from which template version, opens a pull request against every consuming repo, including notifications-api. The generated diff is small and mechanical:
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM golang:1.22-bookworm AS build
+FROM golang:1.22-bookworm@sha256:9f8e1a2b... AS build
WORKDIR /src
COPY . .
RUN go build -o /out/notifications-api ./cmd/server
-FROM gcr.io/distroless/base-debian11:latest
+FROM gcr.io/distroless/base-debian12:nonroot
COPY --from=build /out/notifications-api /notifications-api
+USER nonroot:nonroot
ENTRYPOINT ["/notifications-api"]
Priya's team reviews the five-line diff and merges it the same day. No ticket was filed, no manual audit was needed to discover that notifications-api was running an outdated base image, and no cross-team meeting had to happen to coordinate the same rollout across the dozens of other services generated from this template — the bot opened the equivalent PR against each of them independently.
What's notable across this story is not any single artifact in isolation, but the absence of friction connecting them: Priya never filed a ticket, never waited on another team, and never had to hand-write a Dockerfile, CI pipeline, Crossplane claim, or Kubernetes manifest from scratch — while the platform team retained full visibility into cost, security posture, and compliance the entire time, verified by machine-checked policy rather than manual review.
Even well-built platforms run into predictable friction points. A few patterns and responses worth anticipating:
An Internal Developer Platform succeeds or fails less on tooling choice and more on discipline: treating the platform as a product, keeping golden paths genuinely opinionated rather than sprawling into configuration menus, and ensuring self-service infrastructure remains both fast and policy-compliant by construction rather than by review. Backstage provides a strong, extensible foundation for the experience layer, but the platform's real value is created in the layers underneath and around it — the quality of the skeleton repos, the soundness of the Terraform and Crossplane modules, the policy-as-code guardrails that make speed and safety compatible, and the organization's willingness to keep iterating based on what developers actually need rather than what seemed elegant at design time.
Done well, an IDP becomes close to invisible: developers stop thinking about infrastructure because the paved path handles it, and the platform team's success is measured by how little anyone has to think about them at all.
catalog-info.yaml validity
%402x.svg){kind=icon}