Author: Jim Amuto
Editor: Vahe Aslanyan
The Model Context Protocol, usually shortened to MCP, gives AI assistants a standard way to connect to tools, services, files, databases, and business systems. It lets a client discover available tools, inspect their schemas, and ask the server to execute those tools through a consistent protocol. That standardization matters because it reduces the amount of custom integration work required for every assistant, application, or agent framework.
However, the thing that makes MCP useful also makes it sensitive. An MCP server is not just an API wrapper. It is a trust boundary between an AI system and real resources. It decides what the assistant can see, what it can change, and what evidence exists after an action has been taken. A weak MCP server can leak private data, perform actions in the wrong account, overwrite important records, or allow prompt-injected content to influence tool usage in unsafe ways.
A demo MCP server can be built quickly. A production MCP server requires deeper engineering judgment. It needs authentication, authorization, tenant isolation, validation, rate limits, logging, deployment discipline, monitoring, backups, and recovery procedures. These are not optional extras. They are what turn a working tool into an operational system.
This handbook is intentionally general. The examples may mention documents, notes, workspaces, and object storage, but the same principles apply to MCP servers that connect to CRMs, ticketing systems, databases, code repositories, billing systems, cloud infrastructure, analytics tools, or internal business workflows.
The main idea is simple:
MCP standardizes the connection. Production engineering makes that connection safe.
The first mistake teams make is treating “production-ready” as one universal state. It is not. Production readiness depends on the intended users, data sensitivity, deployment environment, and failure impact.
A local MCP server for one trusted user has a smaller production boundary. It may run over stdio, access a local folder, and rely partly on the operating system user’s permissions. It still needs input validation, safe path handling, output limits, and predictable errors, but it may not need multi-tenant isolation, OAuth, or a public reverse proxy.
A remote MCP server is different. Once a server accepts network traffic, the threat model changes. The server must authenticate every caller, reject unauthorized access, isolate tenant data, handle abuse, and produce durable logs. If the same deployment serves multiple customers, then tenant isolation becomes one of the most important requirements in the entire system.
A useful way to define scope is to ask what the server is allowed to damage. If a tool can only read public documentation, the risk is low. If a tool can read private contracts, create support tickets, deploy code, send email, or update billing records, the server needs stronger controls.
Production scopeTypical usersMain risksRequired controlsLocal single-userOne trusted userPath traversal, accidental overwrite, context bloatPath safety, write limits, local logging, testsInternal teamEmployees or contractorsUnauthorized internal access, mistakes, abuseSSO/OIDC, role checks, audit logs, rate limitsCustomer-facing SaaSMany users/tenantsCross-tenant leaks, data corruption, compliance issuesTenant isolation, strong auth, durable audit, monitoring, backupsHigh-risk enterpriseRegulated or critical systemsLegal, financial, infrastructure, or privacy impactApproval workflows, SIEM logs, policy controls, incident response
The scope should be written down before implementation. If the MCP server is intended only for a local trusted workflow, say so. If it is intended for remote multi-tenant production, design for that from the start.
Local and remote MCP servers solve different problems. Local servers are easier to build and often safer for personal workflows because they are not exposed to the internet. Remote servers are easier to centralize, monitor, update, and share across users, but they require a stronger security and operations model.
DimensionLocal stdio serverRemote HTTP serverLaunch modelStarted by client as local processRuns as network serviceIdentityOften OS user or local configExplicit token/session identityData boundaryLocal machine or configured pathTenant, workspace, account, or organizationDeploymentUser machineServer, container, cloud, or KubernetesSecurity focusFilesystem safety, local permissionsAuth, tenant isolation, abuse controlsObservabilityLocal stderr/log filesCentral logs, metrics, tracing, alertsBest forPersonal tools, desktop workflowsTeams, SaaS, enterprise integrations
A strong pattern is to support both where appropriate. The tool logic can be shared, while each transport gets its own security wrapper. Local stdio can remain convenient for development and personal use. Remote HTTP can enforce bearer tokens, rate limits, workspace checks, and centralized audit logging.
The key is not to accidentally treat a local design as remote-safe. A hardcoded local path or single global API token might work for one user, but it is not a production multi-user model.
Transport is not just a technical detail. It affects authentication, deployment, client compatibility, scalability, and failure behavior.
Stdio is excellent for local development. The MCP client starts the server, communicates over standard input and output, and shuts it down when done. There is no public endpoint, no TLS configuration, and no network routing. But stdio is not a good fit when many users or remote clients need to share one server.
Streamable HTTP or another remote MCP transport is better for production services. It allows a central server to be deployed behind HTTPS, integrated with identity providers, monitored, scaled, and updated without asking every user to install local code.
TransportStrengthsWeaknessesRecommended usestdioSimple, local, no public networkNot centralized, hard to share, local install requiredPersonal tools, local developmentHTTP / Streamable HTTPCentralized, deployable, auth-friendlyRequires full web security modelTeam and SaaS productionSSE / legacy remote patternsUseful for older clientsCompatibility and lifecycle complexityOnly when client requires it
A production codebase should keep the MCP server registration separate from the transport. In practice, this means having one function that creates the MCP server and registers tools, then separate entry points for stdio and HTTP. This avoids duplicating tool logic and makes it easier to apply different controls for different modes.
A flexible production architecture looks like this:
MCP Client
-> HTTPS Reverse Proxy
-> MCP HTTP Server
-> Authentication Middleware
-> Tool Router
-> Authorization Layer
-> Domain Service
-> PostgreSQL Metadata
-> Object Storage / External APIs
-> Audit Log
Each layer has a clear responsibility. The reverse proxy terminates TLS and handles public network concerns. The authentication middleware verifies identity. The tool router maps MCP tool names to handlers. The authorization layer checks whether the user can perform the action. The domain service implements business logic. PostgreSQL stores metadata, memberships, and audit events. Object storage or external APIs hold the actual content or perform the external work.
This architecture is flexible because the domain service can change. A document MCP, billing MCP, DevOps MCP, and CRM MCP may all have different business logic, but they still need identity, authorization, validation, logging, and deployment controls.
Authentication answers the question: who is calling this MCP server? In remote production, every request must map to a stable identity. Without that identity, the server cannot enforce permissions, write meaningful audit logs, or investigate incidents.
OIDC and OAuth2 are common choices because they are standardized and supported by identity providers such as Keycloak, Okta, Auth0, Microsoft Entra, Google, and Cognito. A production MCP server should verify tokens cryptographically. It is not enough to decode a JWT and trust its contents. The server must validate the signature, issuer, expiry, not-before time, and audience.
A typical authentication flow looks like this:
Client obtains access token from identity provider
Client calls MCP endpoint with Authorization: Bearer <token>
MCP server fetches identity provider JWKS
MCP server verifies token signature and claims
MCP server creates auth context
Tool handlers receive auth context
A useful auth context contains the stable subject, username or email, roles, groups, and possibly tenant or organization claims:
type AuthContext = {
userId: string;
username?: string;
email?: string;
roles: string[];
tenantId?: string;
};
If the identity provider supports lightweight access tokens, some claims may be omitted from the access token. In that case, the server may need to call the UserInfo endpoint or introspection endpoint. This should be handled deliberately, not as an afterthought.
Auth approachProsConsBest fitOIDC/OAuth2Standard, SSO-ready, role/group supportMore setup requiredMost remote production systemsAPI keysSimple for developers and servicesHarder user identity, rotation issuesInternal service integrationsmTLSStrong machine identityOperationally heavierhigh-security internal systemsSession cookiesFamiliar for web appsLess universal for MCP clientsBrowser-based MCP clients
Authorization answers the question: what is this caller allowed to do? A production MCP server usually needs both tool-level and resource-level authorization.
Tool-level authorization checks whether the user can call a type of tool. For example, a user may have permission to search documents but not create drafts. Resource-level authorization checks whether the user can access the specific workspace, account, document, project, or tenant named in the request.
ToolTool permissionResource checklist_documentsdocuments:readUser is member of workspacesearch_documentsdocuments:searchUser is member of workspaceread_documentdocuments:readUser can access specific documentcreate_draftdocuments:create_draftUser is editor or ownerdelete_documentdocuments:deleteUser is owner and deletion is enabled
The resource check is what prevents cross-tenant leaks. It is not enough for a user to have documents:read globally. They must have documents:read for the workspace or tenant being accessed.
A strong authorization pattern is:
verify token -> extract user -> check tool permission -> check workspace membership -> perform scoped query
Authorization should be boring and explicit. Avoid generic tools that accept arbitrary operations. Avoid hidden global access. Avoid relying on the model to “only ask for things the user should see.” The server must enforce the rule.
Tenant isolation is one of the most important production MCP concerns. A cross-tenant data leak is often a severe incident because it exposes one customer’s data to another. Even if the leak is accidental, the impact can be legal, contractual, and reputational.
The production problem is usually subtle. A developer may correctly check authentication, but forget to include workspace_id in one search query. Or a cache key may use only the document path, so a result from one tenant is returned to another. Or object storage may use human-readable paths without tenant prefixes. Each mistake breaks isolation.
A safe design makes the tenant boundary visible everywhere:
tenant_id or workspace_idExample object key:
workspace-id/documents/path/to/file.md
Example database query:
SELECT id, relative_path, object_key
FROM documents
WHERE workspace_id = $1
AND relative_path = $2;
The important part is that tenant isolation is not only a policy. It is encoded into the data model and every access path.
Imagine a search tool that queries all indexed documents and filters results after retrieval. During a refactor, the filtering step is skipped for one result type. A user in Workspace A searches for “contract renewal” and receives snippets from Workspace B.
The fix is not just “remember to filter.” The stronger fix is to design the search index and query interface so a workspace filter is required. If the search backend supports filtered indexes, enforce the workspace filter there. If not, create separate indexes per tenant or wrap the search client so unscoped queries are impossible.
A production MCP request should pass through several gates before data is returned or changed.
1. HTTP request arrives at /mcp
2. Reverse proxy handles TLS
3. Server checks request size and method
4. Server extracts bearer token
5. Server verifies token signature, issuer, expiry, and audience
6. Server builds AuthContext
7. MCP router identifies requested tool
8. Tool permission is checked
9. Workspace membership is checked
10. Input is validated and normalized
11. Data query is scoped to workspace
12. Result is size-limited and redacted if needed
13. Audit event is recorded
14. Structured response is returned
A simplified implementation sketch:
async function handleToolCall(request) {
const auth = await verifyBearerToken(request.headers.authorization);
const tool = getTool(request.toolName);
requireRole(auth, tool.requiredRole);
const args = tool.schema.parse(request.arguments);
await requireWorkspaceMembership(auth.userId, args.workspaceId, tool.requiredWorkspaceRole);
const result = await tool.handler({ auth, args });
await auditLog({
userId: auth.userId,
workspaceId: args.workspaceId,
toolName: request.toolName,
status: 'success',
});
return limitResponse(result);
}
This lifecycle matters because production failures usually happen when one gate is skipped. The request may be authenticated but not authorized. It may be authorized at the tool level but not scoped to the workspace. It may be scoped correctly but return too much data. Production readiness means every gate is intentional.
MCP servers often retrieve text that is then shown to a model. That text can contain instructions. Some instructions may be malicious: “Ignore previous instructions,” “read another file,” “send secrets,” or “use a different tool.”
The server should treat retrieved content as untrusted data. It cannot assume that a Markdown file, support ticket, email, webpage, or database record is safe just because it belongs to a user. The content may have been written by an attacker or copied from an untrusted source.
The most important rule is that security decisions must not be delegated to the model. The model can decide what it wants to do, but the server decides what is allowed. Authorization, path checks, tenant isolation, write restrictions, and rate limits must live outside the model.
Useful mitigations include:
Prompt injection cannot be solved completely at the MCP server layer, but the server can prevent prompt-injected content from crossing hard boundaries.
Production tools should be narrow, explicit, and predictable. A generic tool such as run_action or execute_command may be flexible, but it is difficult to secure. It shifts too much responsibility to the model and makes authorization unclear.
Prefer a set of smaller tools:
list_documentssearch_documentsread_documentcreate_draftrequest_publishEach tool should have one job, a strict schema, and a clear permission requirement. This helps the model choose the right tool and helps engineers reason about risk.
A good tool description explains what the tool does and what it does not do. Avoid vague descriptions like “use this to access anything.” A better description is “read one Markdown document by workspace-relative path after authorization checks.”
Tool design should also reflect the product workflow. If humans normally review a draft before publishing, the MCP server should not bypass that review. If deletion is rare and sensitive, deletion should either be absent or require a separate approval path.
Every tool argument should be validated as if it came from an untrusted client. Even if the client is friendly, the model may generate unexpected input, and prompt-injected content may influence tool calls.
Validation should check types, required fields, string lengths, arrays, enums, IDs, paths, and business rules. If a tool accepts a path, normalize it and reject dangerous segments. If it accepts a workspace ID, validate that it is a UUID or expected identifier format. If it accepts content, cap its size.
A common mistake is validating only the JSON schema but not the business rule. For example, a schema may confirm that workspaceId is a string, but only a membership check can confirm the user belongs to that workspace. Both are required.
Search tools are useful because they let the model discover relevant resources before reading them. They are also risky because search can accidentally expose data from a broad scope.
A production search tool should require a workspace or tenant boundary. It should limit query length, result count, snippet length, and execution time. It should return enough information for the model to choose a result, but not dump entire documents into context.
Search should usually be followed by read. The search tool returns metadata and snippets. The read tool returns one selected item after another authorization check. This two-step pattern reduces data exposure and context bloat.
Search result example:
{
"relativePath": "Contracts/acme-renewal.md",
"title": "Acme Renewal Notes",
"snippet": "Renewal discussion focused on support terms...",
"score": 0.82
}
If search uses a shared index, tenant filters must be mandatory. If the backend makes mandatory filters hard, consider separate indexes per tenant or a wrapper that rejects unscoped queries.
Read tools can easily overwhelm a model context or expose more data than necessary. A production read tool should have maximum output size, truncation metadata, and clear source information.
A response should tell the client whether the content was truncated:
{
"relativePath": "Policies/security.md",
"content": "...",
"truncated": true,
"maxBytes": 100000
}
The truncation limit should be based on practical model context size, data sensitivity, and performance. For large documents, consider chunked reading or section-based retrieval.
Read tools should also avoid returning hidden metadata, system files, credentials, binary content, or unsupported file types unless explicitly designed to do so.
Write tools create the highest risk. A read-only server can leak information. A write-capable server can corrupt data, trigger workflows, send messages, or create financial and operational consequences.
A safe maturity model is:
LevelWrite capabilityRiskTypical control0No writesLowestRead-only tools1Create draftsLow-mediumDraft folder or draft state2Update draftsMediumVersion history3Request approvalMediumHuman review4Publish approved changesHigherApproval token or workflow5Direct modificationHighRestricted roles and audit6Destructive actionsHighestStrong confirmation and recovery
Most production MCP servers should start with draft-only writes. Drafts are useful because the assistant can help create work without bypassing human judgment. Direct overwrite, publish, or delete should be introduced only when the product has strong approval, audit, and recovery mechanisms.
Suppose an MCP tool sends an email. The client calls send_email, the server sends the email, but the network connection drops before the response returns. The client retries. Without idempotency, the email is sent twice.
The fix is to design the write operation around an idempotency key or a draft-send workflow. The tool can create a draft and return its ID. A separate approved send action can execute once. If direct send is necessary, the server stores an idempotency key and returns the previous result on retry.
Retries are normal in distributed systems. AI clients, HTTP libraries, proxies, and users can all repeat a request. Production write tools should be safe under retries.
An idempotency key lets the server recognize repeated requests:
workspace_id + user_id + idempotency_key -> previous result
This is especially important for tools that create tickets, send messages, start jobs, charge accounts, publish content, or trigger automation. If a tool cannot be made idempotent, document that clearly and consider requiring confirmation.
Production MCP storage often separates metadata from content. PostgreSQL is a good fit for structured metadata, permissions, memberships, and audit logs. Object storage such as S3, R2, or MinIO is a good fit for larger document bodies or files.
A flexible model:
workspaces stores workspace recordsworkspace_memberships stores who can access each workspacedocuments stores path, object key, size, content type, and timestampsaudit_events stores tool activityThis design supports tenant isolation because every document belongs to a workspace, and every object key is scoped by workspace. It also supports backup and restore because metadata and content are stored in systems designed for production durability.
Caching improves performance but can break authorization if done carelessly. Cache keys must include the same boundary used for authorization.
Unsafe cache key:
document:pricing.md
Safer cache key:
tenant:{tenantId}:workspace:{workspaceId}:document:pricing.md
Permission changes are another concern. If a user loses access but cached data remains available, the cache has become an authorization bypass. Use short TTLs, permission versioning, or cache invalidation when membership changes.
A user has access to a workspace and reads a sensitive document. The result is cached by document path. Later, the user is removed from the workspace. Because the cache key does not include membership state, the same user can still retrieve the cached result.
The fix is to include workspace and permission version in the cache key, or to invalidate relevant cache entries when memberships change. For highly sensitive content, avoid caching full bodies.
Production errors should be structured and predictable. The client should be able to distinguish bad input, missing authentication, forbidden access, not found, conflict, rate limit, and internal failure.
Example:
{
"ok": false,
"error": {
"code": "workspace_forbidden",
"status": "forbidden",
"message": "User is not authorized for this workspace."
}
}
Do not expose stack traces or secrets in tool responses. Detailed errors can go to internal logs with redaction. Public errors should be useful but safe.
Rate limiting protects reliability and cost. MCP clients can call tools repeatedly, and malicious users can intentionally trigger expensive operations. A search tool that scans thousands of documents or an external API tool that makes paid requests can become expensive quickly.
Start with per-user limits, per-IP limits, request body limits, concurrent request limits, tool timeouts, and response size caps. In-memory rate limiting is acceptable for a single-node baseline, but distributed deployments need shared limits through Redis, a database, an API gateway, or platform controls.
The engineering goal is not only to block attackers. It is also to protect the system from accidental loops, retries, or poorly chosen model behavior.
Logs help operate the service. Audit trails explain what happened. Both are necessary, but they are not the same.
Operational logs may include startup events, errors, latency, and dependency failures. Audit logs should record tool calls: who called what, in which workspace, against which resource, and whether it succeeded.
Do log:
Do not log access tokens, refresh tokens, client secrets, full document contents, or sensitive prompts. If the server cannot answer “who did what and when,” incident response becomes guesswork.
Some MCP operations are too slow for a single tool call. Indexing, syncing, importing, exporting, and long-running external workflows should usually become background jobs.
A job-based design returns a job ID and provides status tools:
start_syncget_sync_statuscancel_syncJobs should include workspace scope, user identity, status, progress, retry count, timestamps, and final result. They should have timeouts and dead-letter handling. A failed job should be visible to operators, not silently lost.
Monitoring should cover the MCP server and its dependencies. A healthy HTTP process is not enough if the database is down or object storage is unavailable.
Monitor:
Alert only on conditions that require action. Too many noisy alerts train operators to ignore them.
Backups are not real until restore has been tested. A production MCP server may depend on identity data, metadata, audit logs, and object storage. All of these need backup coverage.
Back up PostgreSQL, object storage, identity provider databases, and configuration required for restore. Store backups outside the primary machine or cluster. Generate checksums where practical.
A restore test should verify that the service can authenticate users, load metadata, retrieve stored content, and serve MCP tool calls from restored data. This should be practiced before an emergency.
Prepare runbooks before incidents happen. Common incidents include leaked secrets, unauthorized access, database outage, object storage outage, bad deployment, failed backup, and high error rates.
A basic response flow:
MCP servers can perform meaningful actions, so audit logs are central to incident response. You need to know which tool was called, by whom, and against what resource.
Deployment should match risk and team maturity.
ModelProsConsBest fitDocker ComposeSimple, self-hosted, understandableManual scaling and opsInternal tools, small teamsManaged containersEasier ops, scalableCloud-specific setupSaaS and production teamsKubernetesFlexible, enterprise-readyMore complexityLarger organizationsServerlessLow ops, scales automaticallyRuntime and transport constraintsLightweight APIs
A self-hosted production stack often includes a reverse proxy, MCP app, PostgreSQL, object storage, identity provider, backup scripts, and monitoring. Keep internal services private and expose only HTTPS.
Production configuration should be explicit and validated. The server should fail fast if required production values are missing.
Never commit real .env files. Commit examples with placeholders. Store real secrets in Docker secrets, Kubernetes secrets, cloud secret managers, or protected environment variables.
Secrets include database passwords, OIDC client secrets, API tokens, storage credentials, and admin passwords. Rotate development defaults before production.
CI should install from the lockfile, build, test, audit dependencies, and validate deployment configuration. A Docker build should run before release.
Deployment should have versioned artifacts and rollback. If a database migration is required, the rollout plan should explain order, compatibility, and rollback limits.
After deployment, run a smoke test that authenticates, calls the MCP endpoint, performs a safe tool call, and verifies expected data changes.
Test unsafe cases, not only happy paths.
Important tests include:
End-to-end tests should create a workspace, authenticate a user, create a resource, list it, read it, search it, verify metadata, and verify object storage.
A production-ready MCP server is one that can be safely used by its intended users, within its intended scope, under expected failure conditions, with enough evidence and recovery capability to operate responsibly.
The strongest MCP servers are not necessarily the ones with the most tools. They are the ones with the clearest boundaries. They verify identity, enforce permissions, isolate tenants, validate inputs, limit outputs, make writes safe, redact logs, survive failures, and provide enough audit evidence to understand what happened.
MCP gives AI systems a standard way to use tools. Production engineering decides which tools are safe, who can use them, what they can touch, and how the system recovers when something goes wrong.
%402x.svg){kind=icon}