Author: Bevan To
Editor: Vahe Aslanyan
Every engineering team eventually faces the same painful moment: something ships, something breaks, and someone asks "wait, was this actually ready?" This handbook exists so that question has a clear, shared answer before it ever comes up in an incident retrospective.
"Production-ready" is not a feeling. It is not the absence of known bugs. It is not "it worked in staging." It is a concrete, verifiable state that a system reaches when a defined set of criteria have been met, reviewed, and signed off on by the right people.
The goal of this document is to make that state explicit and to give every engineer, tech lead, PM, and stakeholder a shared vocabulary and a shared standard so that going from demo to production is a deliberate, confident act rather than a stressful guess.
This handbook applies to:
It does not apply to:
A system is production-ready when it can be trusted to:
This definition intentionally does not say "never fail." Systems fail. Production-ready means the failure is survivable, detectable, and recoverable.
Demos are built to show the controlled path. Production has to handle everything else.
Here is what the gap typically looks like:
DimensionDemo / StagingProductionTrafficControlled, predictableSpiky, unpredictableDataSynthetic or anonymizedReal, sensitive, irreversibleUsersInternal teamCustomers with expectationsErrorsTolerable, fixable instantlyVisible, potentially costlyRecoveryManual restart is fineNeeds documented runbookMonitoringOptionalRequired
Closing this gap is what production readiness is about.
"It works on my machine, so it will work during production." Local environments hide network latency, DNS resolution differences, IAM permission boundaries, and secrets management complexity. If it hasn't run in an environment that mirrors production, it hasn't been tested.
"Monitoring can be added later." Monitoring is not a nice-to-have. It is how you find out something is wrong before your users do. Adding it after launch means you are going in blind.
"Staging is close enough." Staging is useful but it is not production. Real production readiness requires that the system has been validated against production-like load, production-like data shapes, and production IAM policies.
"We can roll back if something goes wrong." Only if the rollback has been tested. A rollback plan that exists only in someone's head is not a plan.
Reliability is the foundation. A system that cannot be counted on to do what it promises consistently and under real conditions is not ready for production regardless of how well everything else is set up.
Every external call, every database query, every third-party API request can fail. Production-ready code handles these failures explicitly:
Calls that can hang indefinitely will hang indefinitely. Every outbound call must have:
When a non-critical dependency is unavailable, the system should degrade gracefully rather than fail completely. This means identifying which features can be disabled or stubbed, and ensuring the critical path remains functional.
If you cannot see what a system is doing, you cannot know when it is broken. Observability means three things: logs, metrics, and traces.
ERROR for actionable failures, WARN for unexpected but recoverable states, INFO for significant lifecycle events, DEBUG for development onlyAt minimum, every service exposes the four golden signals:
All new services and significant changes undergo a security review before launch. The review covers:
Before launch, the service must be load tested at a level that represents expected peak traffic, plus a safety margin. The test should:
Results must be documented. If the service cannot meet its SLO under expected load, it is not ready.
Every production service should have defined Service Level Objectives:
These numbers should be agreed upon with stakeholders before launch, not derived after the fact from whatever the system happened to achieve.
Every change to a production service must go through a CI/CD pipeline that:
Manual deployments, such as SSH-ing into a box and running code directly, are not acceptable for production systems.
Before launching, the team must have a documented and tested rollback plan:
If the rollback involves a database migration, it must be verified that the previous version of the code is compatible with both the migrated and un-migrated schema.
For high-risk changes, feature flags allow the new behavior to be enabled for a subset of users or traffic before full rollout. This is particularly valuable for:
Documentation is not a formality. It is what allows the system to be operated, debugged, and improved by people who were not in the room when it was built.
Every service repository has a README that covers:
The runbook is the on-call engineer's guide to the service. It covers:
Significant technical decisions, why this database, why this queue, why this authentication approach, should be documented as Architecture Decision Records (ADRs) so future engineers understand not just what was built, but why.
Testing is one of the strongest indicators that a service is ready for production. While no amount of testing can prove that software is completely free of defects, a comprehensive testing strategy significantly reduces the likelihood of critical failures reaching customers. Production-ready systems are built with the expectation that code will evolve over time, and tests provide confidence that new changes do not unintentionally break existing functionality.
Testing should occur throughout the software development lifecycle rather than being treated as the final step before deployment. Different types of testing address different categories of risk, and together they provide multiple layers of protection against regressions, configuration issues, and unexpected behavior.
Unit tests validate individual functions, methods, or classes in isolation. They are designed to verify that a specific piece of business logic behaves correctly under a variety of conditions without relying on external systems such as databases, APIs, or file systems.
Effective unit tests should be:
A production-ready service should include unit tests for all critical business logic, validation rules, calculations, and utility functions. Unit tests should cover both expected inputs and edge cases, including invalid data, empty values, and boundary conditions.
While code coverage metrics can provide useful insight, they should not be treated as the primary goal. A high coverage percentage does not guarantee meaningful tests. Instead, engineering teams should prioritize testing the areas of the application where failures would have the greatest impact on users or the business.
Most production failures occur not because an individual function is incorrect, but because multiple components fail to work together as expected. Integration testing verifies that these components interact correctly in realistic environments.
Integration tests commonly validate interactions between:
Whenever practical, integration tests should execute against environments that closely resemble production. Mocking external dependencies can be valuable during development, but production readiness also requires validating behavior against real implementations whenever feasible.
These tests should verify not only successful interactions but also failure scenarios, such as unavailable services, malformed responses, network timeouts, and permission errors.
End-to-end (E2E) testing validates complete user workflows from beginning to end. Unlike unit or integration tests, E2E tests simulate how actual users interact with the application.
Examples of workflows that are well suited for end-to-end testing include:
These tests provide confidence that all system components work together correctly, including the frontend, backend, databases, authentication providers, and external integrations.
Because end-to-end tests typically require more infrastructure and execution time than other forms of testing, they should focus on the application's most critical user journeys rather than attempting to exhaustively test every possible scenario.
Every production incident should improve the quality of the system going forward. When a defect is discovered, the first step after identifying the root cause should be adding a test that reproduces the issue.
This practice ensures that once a bug has been fixed, it cannot silently reappear in future releases without being detected by the automated test suite.
Regression testing provides long-term confidence as applications grow in complexity. Over time, the regression suite becomes a valuable record of previously encountered failures and protects against repeating the same mistakes.
Meaningful tests require meaningful data. Test environments should contain representative datasets that accurately reflect the structure, relationships, and scale of production data while ensuring that no sensitive customer information is exposed.
Test datasets should include:
When production data must be used for testing, all personally identifiable information (PII) and confidential business data should be anonymized or removed in accordance with organizational security policies.
Automated testing should be integrated directly into the continuous integration and continuous deployment (CI/CD) pipeline. Every code change should be validated before it can be merged or deployed.
A typical production deployment pipeline includes:
A build should automatically fail if any required validation step does not pass. Production deployments should never rely solely on manual verification.
Testing should extend beyond application correctness. Production systems must also satisfy operational requirements that cannot be verified through traditional functional testing alone.
Teams should evaluate:
These tests help identify issues that may only appear after prolonged operation or under heavy load, allowing teams to address them before customers are affected.
A production-ready system must be capable of supporting not only today's workload but also tomorrow's growth. Scalability is the ability of a system to maintain acceptable performance and reliability as the number of users, requests, and data volume increases. While performance focuses on how well a system operates under its current workload, scalability addresses how effectively that system adapts when demand changes.
Scalability should be considered during system design rather than after performance problems begin to appear. Retrofitting scalability into an existing application is often significantly more expensive than designing for growth from the outset.
Production-ready systems are built with the expectation that usage patterns will evolve over time and that infrastructure should be capable of expanding without requiring major architectural changes.
There are two primary approaches to increasing system capacity.
Vertical scaling involves increasing the resources available to an existing server, such as adding additional CPU cores, memory, or storage. This approach is generally simple to implement and can provide immediate performance improvements for smaller workloads. However, vertical scaling has practical limits, as hardware resources cannot be increased indefinitely.
Horizontal scaling involves adding additional application instances or servers that share the workload. Incoming requests are distributed across multiple instances using a load balancer, allowing the system to handle significantly higher traffic while improving fault tolerance.
Whenever practical, production services should be designed to support horizontal scaling. This approach provides greater flexibility, simplifies maintenance, and reduces the impact of individual server failures.
Stateless applications are significantly easier to scale than stateful ones.
A stateless service does not rely on locally stored session data or application state that exists only within a single server instance. Instead, user sessions and shared application data are stored in external systems such as databases, distributed caches, or dedicated session stores.
Because each request contains all of the information required to process it, any application instance can handle the request. This allows infrastructure platforms to automatically add or remove instances in response to changing demand without disrupting active users.
When application state must be maintained, it should be stored in systems specifically designed for persistence and availability rather than in application memory.
Repeatedly performing expensive computations or database queries can become a significant performance bottleneck as traffic grows. Caching reduces unnecessary work by temporarily storing frequently accessed data so that future requests can be served more efficiently.
Common caching strategies include:
Caches should always include clearly defined expiration policies to prevent stale or inconsistent data from being served indefinitely.
Not every operation must be completed while a user waits for a response. Long-running or resource-intensive tasks should be processed asynchronously whenever possible.
Examples include:
Asynchronous processing typically relies on message queues or job scheduling systems that allow work to be processed independently of user requests.
Production-ready queue systems should include:
Separating background work from user-facing requests improves responsiveness while increasing overall system resilience.
As applications grow, the database frequently becomes one of the primary performance constraints. Database scalability should be considered early to prevent bottlenecks from affecting application performance.
Engineering teams should evaluate:
Database optimization should be driven by measured performance data rather than assumptions. Query profiling and monitoring tools should be used regularly to identify slow or resource-intensive operations.
Scalable systems are supported by thoughtful capacity planning rather than reactive infrastructure expansion.
Before launching a production service, engineering teams should estimate:
These estimates should include reasonable growth projections so that infrastructure can accommodate increased demand without requiring emergency upgrades shortly after launch.
Capacity planning should be reviewed periodically as application usage evolves and new features are introduced.
Designing for scalability is not sufficient unless those assumptions are validated through testing.
Scalability testing should evaluate how the application behaves under increasing levels of traffic and identify the point at which performance begins to degrade.
Typical scenarios include:
Results should be documented and compared against the service's defined Service Level Objectives (SLOs). If performance deteriorates significantly before expected capacity is reached, the underlying bottlenecks should be identified and resolved before production deployment.
Production-ready services should be designed with long-term growth in mind. While not every application requires massive scale on day one, adopting scalable architectural patterns early reduces operational risk and minimizes future redesign efforts.
Engineering teams should verify that:
A scalable system enables an organization to support increasing demand without sacrificing reliability, maintainability, or user experience. By designing for growth before it becomes necessary, engineering teams can deploy services with confidence that they will continue to perform effectively as adoption increases.
This checklist is the operational summary of everything in Section 3. It must be completed and signed off before any new service or major change goes to production.
RoleResponsibilitySign-off required?EngineerCompletes the checklist, writes the runbook, owns the deploymentYesTech LeadReviews the checklist, validates the architecture and rollback planYesSecurityReviews and signs off on the security sectionYesPMConfirms stakeholder alignment, coordinates the go/no-go callYesOn-callBriefed on the launch, aware of the runbook and rollback planNo (but must be informed)
For any significant launch, a go/no-go call is held with the engineer, tech lead, PM, and any other relevant stakeholders. The agenda is:
A go/no-go call is not a blame session. It is a structured way to make sure everyone who needs to know something knows it before the deployment happens.
Launching a service is not the end of the engineering process. Every deployment, whether successful or problematic, provides valuable information that can improve future releases. Production-ready organizations treat operational experience as a learning opportunity and continuously refine their systems and processes based on what they observe.
When an incident occurs, the objective is not to determine who made a mistake but to understand why the system allowed the mistake to have an impact. Production incidents are rarely caused by a single action. More often, they result from a combination of technical decisions, process gaps, environmental conditions, and unforeseen interactions between systems.
All significant production incidents should be followed by a blameless postmortem. The purpose of a postmortem is to identify contributing factors, document lessons learned, and implement improvements that reduce the likelihood of similar incidents occurring in the future.
A blameless culture encourages honest discussion and accurate reporting. Engineers should feel comfortable sharing what happened without fear of punishment, allowing the team to focus on improving systems rather than assigning fault.
A typical postmortem should answer the following questions:
Identifying the immediate failure is only part of the investigation. Teams should continue asking "why" until they understand the underlying process or technical weakness that allowed the failure to occur.
For example:
The root cause is not simply the incorrect configuration. It is the absence of an automated, repeatable deployment process.
By addressing the underlying cause rather than only the symptom, future incidents become less likely.
Every postmortem should conclude with specific, measurable action items that improve the system.
Examples include:
Each action item should have:
Action items should be reviewed during sprint planning or engineering planning meetings until completed.
Lessons learned from production incidents should benefit the entire engineering organization, not only the team responsible for the affected service.
Organizations should maintain a searchable repository of completed postmortems so engineers can learn from previous incidents when designing new systems. Regular engineering meetings or technical presentations can also be used to share significant operational lessons across teams.
Continuous improvement is one of the defining characteristics of mature engineering organizations. Every production incident provides an opportunity to strengthen systems, improve processes, and reduce operational risk for future deployments
Not every deployment should be released to all users simultaneously. Large-scale deployments introduce risk because any undiscovered issue immediately affects the entire user base. Production-ready organizations reduce this risk by releasing software gradually, validating system behavior at each stage before expanding the rollout.
Gradual deployment strategies allow engineers to observe real production traffic while limiting the impact of unexpected issues.
A canary deployment releases a new version of a service to a small percentage of users while the majority continue using the existing version.
During the canary phase, engineers monitor key operational metrics such as:
If the new version performs as expected, the rollout gradually expands until all users are running the updated service. If problems are detected, the deployment can be halted or rolled back before the issue affects the broader customer base.
Canary deployments are particularly valuable for high-risk changes, infrastructure upgrades, and major feature releases.
A blue-green deployment maintains two identical production environments.
Once the Green environment has been validated, production traffic is switched from Blue to Green. If issues arise, traffic can quickly be redirected back to the previous environment with minimal downtime.
Blue-green deployments reduce deployment risk by allowing complete validation before users begin interacting with the new version.
In a rolling deployment, application instances are updated incrementally rather than all at once.
For example, in a cluster of ten application servers:
Rolling deployments minimize downtime while reducing the likelihood of widespread service disruption.
Feature flags separate deployment from feature release. Code can be safely deployed to production while the associated functionality remains disabled until it is ready to be enabled.
Feature flags provide several advantages:
Feature flags should be reviewed periodically and removed once they are no longer needed. Long-lived feature flags increase code complexity and make systems more difficult to maintain.
Every deployment should be accompanied by active monitoring.
Engineering teams should observe:
Deployment monitoring should continue throughout the rollout and for an appropriate observation period after completion. Teams should avoid scheduling additional production changes until they have confidence that the deployment is stable.
A deployment is not considered successful simply because the deployment process completed without errors. Success is confirmed only after the application demonstrates stable, reliable behavior under real production traffic. Gradual rollout strategies provide engineering teams with the confidence to deploy frequently while minimizing the operational risk associated with change.
Sometimes a checklist item cannot be completed before launch. A dependency is not ready, a load testing environment is not available, or a business deadline creates genuine urgency. In these cases:
Exceptions are not shortcuts. They are risk acknowledgments with a documented mitigation plan.
If there is disagreement about whether a service is production-ready, the escalation path is:
Transitioning a system from a successful demonstration to a production-ready service requires significantly more than implementing features. While a demo proves that an idea works under controlled conditions, production software must continue to operate reliably in unpredictable environments, support real users, recover gracefully from failures, and remain maintainable throughout its lifecycle.
Throughout this handbook, we have explored the key principles that define production readiness. Reliability ensures that systems behave consistently even when dependencies fail or unexpected conditions arise. Observability provides engineers with the visibility needed to detect, diagnose, and resolve issues before they significantly impact users. Security protects customer data, infrastructure, and business operations through careful design and ongoing risk management. Performance and scalability ensure that applications continue to meet service expectations as usage grows, while well-defined deployment processes reduce the risk associated with releasing new software. Comprehensive testing validates functionality before changes reach production, and thorough documentation enables future engineers to understand, operate, and improve the system long after its original developers have moved on.
Equally important are the operational practices that surround the software itself. Production readiness is not achieved by engineering alone; it requires collaboration between developers, technical leads, security teams, product managers, and operations personnel. Launch planning, production readiness reviews, gradual rollout strategies, monitoring, post-launch validation, and continuous improvement all contribute to reducing operational risk and increasing confidence in every deployment.
The production readiness checklist presented in this handbook should be viewed as a practical decision-making tool rather than a simple list of tasks. Completing each item demonstrates that the engineering team has considered not only whether the software functions correctly, but also whether it can be safely deployed, monitored, maintained, and supported in a production environment. While there may occasionally be valid reasons to request exceptions, such decisions should always be documented, reviewed, and accompanied by a clear mitigation plan.
It is also important to recognize that production readiness is not a one-time milestone. As systems evolve, new features are introduced, traffic patterns change, and infrastructure grows more complex, production readiness must be continually reassessed. Regular reviews of monitoring, testing, documentation, security practices, deployment procedures, and operational processes help ensure that services remain resilient as they mature.
Ultimately, production-ready software is characterized not by the absence of failure, but by the ability to anticipate, detect, respond to, and recover from failure in a controlled and predictable manner. Organizations that invest in these engineering practices build systems that are more reliable, easier to maintain, and better equipped to meet the expectations of both customers and engineering teams.
By consistently applying the principles outlined in this handbook, engineering teams can move beyond delivering software that simply works in a demonstration and instead deliver software that can be trusted to perform reliably in production. Production readiness is an ongoing commitment to quality, operational excellence, and continuous improvement, ensuring that every deployment strengthens both the product and the organization that supports it.
%402x.svg){kind=icon}