Author: Jim Amuto
Editor: Vahe Aslanyan
Draft handbook for review
Audience: engineers, AI product builders, technical founders, and teams moving LLM features from prototype to production.
Core promise: reduce LLM spend without blindly destroying product quality.
This draft is based on current public material from provider documentation and engineering references, including OpenAI prompt caching documentation, vLLM documentation and the original vLLM PagedAttention article, BentoML's LLM inference and quantization material, and recent production-focused writing on semantic caching, model routing, prompt compression, and LLM cache invalidation.
Key source themes used in this handbook:
This handbook avoids treating any single technique as magic. The practical thesis is that major LLM savings come from layering multiple controls: measurement, prompt trimming, caching, routing, output limits, agent budgets, evals, quantization, and only then local inference.
Most LLM applications start cheap. A small team builds a prototype, sends a few prompts during development, maybe demos it to leadership, and the monthly bill looks harmless. The app feels magical. The cost seems manageable. The team ships.
Then real users arrive.
The prompts get longer. The product team asks for better answers, so engineers add more instructions. The support team wants personalization, so the system includes account history. The legal team wants disclaimers. The growth team wants tone control. The data team wants retrieval-augmented generation. The agent now calls tools, retries failed actions, summarizes long conversations, and stores memory. Every layer seems reasonable in isolation, but together they turn one user action into thousands or tens of thousands of tokens.
The surprise is not that LLMs cost money. The surprise is how quickly product complexity turns into token spend.
A simple prototype might look like this:
User question -> LLM answer
A production system often looks more like this:
User question
-> safety classification
-> query rewrite
-> embedding lookup
-> vector search
-> keyword search
-> reranking
-> context assembly
-> main LLM answer
-> structured output repair
-> moderation check
-> follow-up suggestion generation
-> conversation summary update
Each step may be defensible. Each step may improve quality. But each step also has cost, latency, and failure modes.
This is why teams need an LLM cost strategy. Not because cost is more important than quality, but because uncontrolled cost eventually limits product quality. If every request is expensive, teams become afraid to add features, evaluate outputs, run experiments, or serve lower-value users. Cost control is not just finance hygiene. It is product scalability.
The best cost optimization does not start with “replace the model.” It starts with understanding the workload.
You want to answer questions like:
Without this information, optimization is guesswork. With it, cost reduction becomes an engineering process.
LLM cost usually comes from several places.
Input tokens are everything you send to the model:
Input tokens are easy to underestimate because many of them are invisible to the user. A user may type one short sentence, but the application may send a huge prompt behind the scenes.
For example:
User: Can I return this item?
The actual model input might include:
The user sees a small question. The model sees a large document.
Output tokens are what the model generates. They often cost more per token than input tokens, depending on provider and model.
Many teams optimize input context and forget output length. But verbose answers, long JSON outputs, repeated disclaimers, and unnecessary reasoning text can become a major cost center.
Output control is one of the simplest levers:
max_tokensIf a task only needs a label, do not ask the model for a paragraph.
Model choice is the most obvious cost lever. Premium frontier models are powerful, but many production tasks do not need them.
Common tasks that often work with cheaper models:
Common tasks that may need stronger models:
The goal is not to always use the cheapest model. The goal is to use the cheapest model that reliably meets the quality requirement for that task.
RAG systems add cost in places teams forget:
RAG can reduce hallucination and improve freshness, but badly tuned RAG can also explode input tokens. The system retrieves too many chunks, includes irrelevant context, and forces the model to read a pile of text to answer a simple question.
Cost-effective RAG is not “retrieve more.” It is “retrieve enough relevant context.”
Agents are powerful because they can plan, call tools, observe results, and continue. They are expensive for the same reason.
A single user request can become:
LLM planning call
Tool call
LLM observation call
Tool call
LLM observation call
Tool call fails
Retry
LLM recovery call
Final answer call
Memory update call
If the agent has no budget, no stopping rule, and no loop detection, cost can become unpredictable. This is especially dangerous when a tool fails repeatedly or returns confusing output.
For production agents, cost control is part of reliability engineering.
Retries are hidden spend. A flaky provider, a schema validation failure, a timeout, or a tool error may cause the application to call the model again. If the retry uses the same huge context, cost doubles.
Structured output repair is another hidden cost. If a model returns invalid JSON, the system may send the invalid output back to the model and ask it to fix it. That may be necessary, but it should be measured.
Evaluation also costs money. LLM-as-judge systems, regression tests, trace analysis, and synthetic test generation all call models. These are valuable, but they should be budgeted.
A mature team tracks production inference cost and evaluation cost separately.
You cannot cut an LLM bill intelligently if you do not know what created it.
At minimum, log these fields for every LLM call:
request_id
user_id or account_id
feature_name
task_type
model
provider
input_tokens
output_tokens
total_tokens
estimated_cost
latency_ms
cache_hit
retry_count
tool_call_count
success_or_failure
If you use RAG, also log:
retrieved_chunk_count
retrieved_token_count
reranker_used
embedding_model
vector_index
source_documents
If you use agents, log:
agent_turn_count
tool_names_called
failed_tool_calls
loop_stopped_by_budget
final_status
The first useful dashboard is simple:
MetricQuestion It AnswersCost by featureWhich product area is expensive?Cost by accountWhich customers drive spend?Cost by modelAre premium models overused?Avg input tokensAre prompts/context too large?Avg output tokensAre answers too verbose?Cache hit rateAre repeated requests being reused?Retry rateAre failures doubling cost?Agent turns per requestAre agents looping?Cost per successful taskAre failed requests wasting spend?
Measurement often reveals embarrassingly simple waste:
The first 20–40% savings often come from removing obvious waste, not from sophisticated infrastructure.
Token reduction is the most universal optimization. It works whether you use hosted APIs, open-source models, local inference, RAG, or agents.
System prompts tend to grow over time. Every incident adds another instruction. Every stakeholder adds another requirement. Eventually the prompt becomes a policy document.
Review your system prompt like production code.
Ask:
Bad pattern:
Every request receives the same giant universal prompt.
Better pattern:
Common base prompt + task-specific compact instructions.
Do not send instructions for summarization, classification, code generation, and customer support if the current task is only classification.
RAG systems often over-retrieve because teams are afraid of missing context. But sending too much context can hurt both cost and quality. The model may pay attention to irrelevant chunks, mix sources, or produce hedged answers.
Improve retrieval by:
A practical rule: measure answer quality as you reduce the number of chunks. Many systems can drop from 10–20 chunks to 3–6 strong chunks with little quality loss, especially when reranking is used.
Chat history can become one of the largest costs in assistant products. Sending the full transcript on every turn is simple, but expensive.
Alternatives:
Do not treat conversation memory as a raw transcript problem. Treat it as a state management problem.
The model usually does not need every word the user said yesterday. It needs the current task, relevant preferences, unresolved decisions, and recent context.
Many products let the model talk too much.
For high-volume tasks, set strong defaults:
Output token limits can improve both cost and UX. Users often prefer concise answers.
Not every language-shaped task needs an LLM.
Use code for:
One common anti-pattern is asking an LLM to do work that a database query, rule engine, or string function can do perfectly.
LLMs are useful for ambiguity. They are wasteful for deterministic operations.
Caching is the technique most people know first. If the same work repeats, reuse the result instead of paying again.
But LLM caching has more nuance than normal web caching because the answer may depend on prompt wording, model version, retrieved documents, user permissions, tool outputs, and time.
Exact-match caching stores a response for an identical request.
Cache key example:
hash(model + system_prompt_version + user_prompt + parameters)
This works well for:
Exact caching is safe when inputs are truly identical and the answer does not depend on fresh state.
Some providers offer prompt caching, where repeated prompt prefixes can be processed more cheaply and quickly. OpenAI documents prompt caching as automatic for supported models, especially when requests share long common prefixes. This is useful when many requests begin with the same system prompt, examples, or stable context.
This matters because many apps send the same prefix repeatedly:
System instructions
Few-shot examples
Tool schemas
Long policy text
User-specific context
Current user question
If the stable prefix is structured consistently, provider-side caching can reduce input-token cost and latency for that prefix.
Practical implications:
Prompt caching does not eliminate all cost. It reduces cost for repeated prefixes. You still pay for uncached parts and outputs.
Semantic caching tries to reuse answers for similar questions, not just identical prompts.
Example:
Question A: How do I reset my password?
Question B: I forgot my password, what should I do?
These are different strings but likely the same intent. A semantic cache embeds the new query, searches prior cached queries, and reuses a response if similarity is high enough.
This can create major savings for support bots, documentation assistants, and internal knowledge tools. But it is riskier than exact caching.
The hard questions are:
Semantic caching is powerful, but unsafe without boundaries.
Cache invalidation is the core production challenge.
An LLM cache may need invalidation when:
A good cache key includes versions:
model_version
prompt_version
retrieval_corpus_version
tool_version
user_or_tenant_scope
response_schema_version
For RAG answers, consider storing source document IDs and versions with each cached answer. If a source document changes, invalidate cached answers derived from it.
Avoid or be very careful caching:
A stale cached answer can be worse than an expensive fresh answer.
Many teams cache final LLM answers but forget tool calls. Tool results often repeat:
Caching tool results reduces latency and can reduce the amount of context sent back to the model.
Model routing is one of the highest-leverage cost strategies. The idea is simple: classify the task, then choose the cheapest model that can handle it.
A single-model architecture is easy:
All requests -> best model
A routed architecture is more efficient:
Simple classification -> small cheap model
Extraction -> cheap structured-output model
Normal chat -> mid-tier model
Complex reasoning -> premium model
High-risk answer -> premium model + review
Different tasks have different difficulty.
TaskOften Suitable ModelBoolean classificationsmall/cheap model or rulesSentiment taggingsmall modelEntity extractionsmall or mid modelShort summarizationcheap/mid modelFAQ answer from known contextcheap/mid modelComplex synthesispremium modelCoding/debuggingstrong modelAgent planningstrong modelHigh-stakes advicestrong model + safeguards
Routing starts with naming your task types. If every request is just “chat,” routing is hard. If you separate classification, extraction, summarization, answering, planning, and verification, routing becomes straightforward.
Some systems use a cheap model first, then escalate when needed.
Pattern:
Cheap model attempts answer
-> confidence check
-> if low confidence, call stronger model
The confidence check can use:
Be careful with self-reported confidence. Models can be overconfident. Use external signals when possible.
Not all requests have equal business value.
A free-tier user asking a casual question may not justify the same cost as an enterprise admin asking for production incident help. This does not mean free users get bad answers. It means product economics matter.
Possible routing dimensions:
Model routing can reduce cost, but it can also silently reduce quality.
Common mistakes:
A router must be evaluated like any other model component.
Measure:
The best routers are boring. They send easy work to cheap models, hard work to strong models, and uncertain work through a fallback path.
Agents are like interns with a corporate credit card. They can be very useful, but you need spending limits.
An agent that can reason, call tools, retry, and continue has an open-ended cost surface. If the agent gets confused, it may loop. If a tool returns bad data, it may keep trying. If the prompt encourages thoroughness, it may over-plan.
Every agent run should have limits:
max_turns
max_tool_calls
max_runtime_seconds
max_input_tokens
max_output_tokens
max_cost_usd
max_retries_per_tool
When the budget is reached, the agent should fail gracefully:
I could not complete this automatically because the search tool failed twice. Here is what I found and what I recommend next.
A graceful partial answer is better than a runaway loop.
Agents often call the same tool repeatedly with similar arguments. Cache tool outputs inside the agent run and across runs when safe.
Example:
Agent searches docs for “refund policy”
Agent reasons
Agent searches docs for “refund policy” again
The second call should usually be a cache hit.
If a tool call fails with the same error twice, do not let the agent blindly try five more times.
Add loop detection:
Stop and summarize.
An agent may not need a premium model for every step.
Possible split:
This is model routing inside the agent.
Some models and prompts produce long reasoning traces. Depending on provider and configuration, this can increase tokens, latency, and risk. For production, ask for concise final outputs unless intermediate reasoning is needed for audit or debugging.
Keep traces in internal logs when needed, not in user-facing verbose text.
Quantization reduces the numeric precision used to store and compute model weights. In simple terms, it compresses the model so it uses less memory and may run faster or fit on cheaper hardware.
A rough intuition:
Quantization matters because GPU memory is one of the main constraints in LLM serving. If a model uses less memory, you may be able to:
Quantization introduces approximation. The model may become slightly worse, especially on tasks requiring precision, reasoning, math, code, or nuanced instruction following.
The quality impact depends on:
A 4-bit model may be excellent for classification and poor for complex reasoning. An 8-bit model may be nearly indistinguishable for some tasks and noticeably worse for others.
Good candidates:
Riskier candidates:
Never switch to a quantized model based only on benchmark claims.
Build an eval set from real production examples:
Compare:
Quantization is a cost lever, not a religion. Use it where it preserves enough quality.
vLLM comes up in cost conversations because it is one of the most important open-source tools for serving LLMs efficiently.
vLLM is not a model. It is an inference engine/server.
Think of the stack like this:
Model: Llama, Mistral, Qwen, DeepSeek, etc.
Serving engine: vLLM
Hardware: GPUs
API layer: OpenAI-compatible endpoint or custom service
Application: your product
If OpenAI or Anthropic is like renting intelligence through an API, vLLM is part of the machinery you might use when you run an open model yourself.
LLM serving is difficult because generation is memory-intensive and request lengths vary. Each request needs a KV cache, which stores attention-related information from previous tokens so the model does not recompute everything from scratch.
The KV cache grows as generation continues. In a busy server, many users have different prompt lengths and output lengths. Managing that memory efficiently is hard.
Naive serving wastes GPU memory. Wasted GPU memory means fewer concurrent requests. Fewer concurrent requests means lower throughput. Lower throughput means higher cost per token.
vLLM became famous for PagedAttention, a memory-management approach for KV cache inspired by operating-system paging. Instead of reserving large contiguous memory blocks inefficiently, vLLM manages cache memory in blocks/pages, reducing fragmentation and improving utilization.
The practical result: more requests can be served on the same GPU before running out of memory.
LLM requests do not arrive neatly in fixed batches. Users arrive at random times. Some prompts are short. Some are long. Some outputs finish quickly. Some keep generating.
Continuous batching lets the serving engine dynamically batch active requests together as they arrive and finish. This improves GPU utilization.
Better utilization matters because GPU cost is mostly fixed while the machine is running. If your GPU is idle, you are paying for unused capacity. If vLLM helps the GPU process more tokens per second, the cost per token goes down.
vLLM can expose OpenAI-compatible APIs. That matters because many applications are already written around OpenAI-style chat completions.
This can reduce migration friction:
Before:
App -> OpenAI API
After:
App -> vLLM OpenAI-compatible server -> self-hosted model
It is not always a drop-in replacement. Models differ, outputs differ, tool calling support differs, and quality must be evaluated. But API compatibility makes experimentation much easier.
vLLM also supports performance features such as prefix caching and optimized scheduling. Prefix caching is useful when requests share common prompt prefixes, similar in spirit to provider prompt caching: repeated prefixes can be reused instead of recomputed.
This matters for applications with stable system prompts, few-shot examples, long tool schemas, or shared context.
vLLM can help cut costs when:
In that world, the economics can shift from:
Pay provider per token
to:
Pay for GPU capacity and maximize tokens per second
If utilization is high, self-hosting can be cheaper.
Self-hosting is not automatically cheaper.
It may be worse when:
A hosted API includes more than inference. It includes model hosting, scaling, reliability, security programs, monitoring, updates, and operational expertise. When you self-host, you inherit much of that burden.
So the honest advice is:
Do not start with vLLM. Graduate to vLLM when measurement shows inference volume is high enough and model quality is acceptable.
vLLM is special because it makes local/open-model serving much more practical. It is not special because it magically makes all LLM usage cheap.
Batching is another underused lever. Some providers offer cheaper batch APIs for non-real-time jobs. Self-hosted systems also benefit from batching because GPUs are more efficient when processing multiple requests together.
Good batch candidates:
Bad batch candidates:
A common cost mistake is treating every job as interactive. If the user does not need the answer now, batch it.
You can design product UX around this:
Latency requirements drive cost. Relax latency when product experience allows it.
RAG can either reduce or increase cost. It reduces cost when it lets a smaller model answer with relevant context. It increases cost when it dumps huge irrelevant context into the prompt.
Bad chunking creates bad retrieval. If chunks are too small, the model lacks context. If chunks are too large, you waste tokens. If chunks split concepts awkwardly, retrieval returns fragments.
Good chunks often follow document structure:
The goal is not a magic token size. The goal is chunks that represent coherent units of meaning.
Vector search is good for semantic similarity. Keyword search is good for exact terms, IDs, error messages, product names, and legal/policy language. Hybrid search combines both.
Hybrid retrieval can reduce cost by improving relevance. Better retrieval means fewer chunks need to be sent to the model.
A reranker scores candidate chunks and selects the best ones. Reranking adds cost, but it can reduce final prompt size and improve quality.
The tradeoff:
Pay small reranking cost -> send fewer/better chunks -> reduce main LLM cost and improve answer
This is often worth it when the final model is expensive.
Instead of sending raw retrieved chunks, compress them:
Be careful: compression can remove nuance. Evaluate it.
Not every question needs RAG. If the user asks a general question or asks about the current conversation, retrieval may be unnecessary.
Add a retrieval gate:
Does this request need external knowledge?
If yes, retrieve.
If no, skip retrieval.
Skipping unnecessary retrieval saves embedding, search, reranking, and input tokens.
A realistic 10x reduction usually comes from stacking multiple improvements.
Here is a practical sequence.
Add logging for tokens, model, cost, latency, retries, cache hits, and feature names.
Deliverable:
Expected savings: none immediately, but it reveals the path.
Actions:
Expected savings: 20–50% in many messy systems.
Actions:
Expected savings: highly workload-dependent. Repetitive support/docs workloads can see very large savings; bespoke creative workloads may see less.
Actions:
Expected savings: often one of the largest levers, especially if everything currently uses a premium model.
Actions:
Expected savings: large for agentic systems; smaller for simple chat.
Actions:
Expected savings: significant for suitable high-volume tasks.
Actions:
Expected savings: potentially large at scale, but not guaranteed.
Suppose a system costs $10,000/month.
A possible path:
OptimizationRemaining CostStarting point$10,000Prompt/context reduction saves 30%$7,000Caching saves 30% of remaining$4,900Model routing saves 40% of remaining$2,940Agent/retry controls save 20% of remaining$2,352Smaller models/self-hosting for selected workloads saves 50% of remaining$1,176
That is not one magic trick. It is layered engineering.
TechniqueSavings PotentialLatency ImpactQuality RiskComplexityBest ForPrompt trimmingMediumBetterLowLowEveryoneOutput limitsMediumBetterLow/MediumLowChat, generationExact cachingHigh when repeatedBetterLow/MediumMediumFAQs, evals, docsSemantic cachingHighBetterMedium/HighMedium/HighSupport, docsTool result cachingMedium/HighBetterMediumMediumAgents, RAGProvider prompt cachingMedium/HighBetterLowLow/MediumStable prompt prefixesModel routingHighMixedMediumMediumMulti-task appsAgent budgetsHigh for agentsBetterLow/MediumMediumTool-using agentsRAG rerankingMediumWorse or mixedLow/MediumMediumKnowledge assistantsQuantizationMedium/HighBetter or mixedMediumHighSelf-hosted modelsvLLM self-hostingHigh at scaleMixedMediumHighHigh-volume workloadsBatch processingMedium/HighSlower by designLowMediumOffline jobs
Teams jump to smaller models or self-hosting before knowing where spend comes from. This can waste weeks while obvious prompt bloat remains untouched.
Caching can leak stale, wrong, or unauthorized information. Cache keys and invalidation rules matter.
More context is not always better. Irrelevant context costs money and can confuse the model.
A frontier model doing sentiment tagging at scale is usually a routing failure.
Verbose outputs can be expensive. Concise defaults help.
Agents need limits. Otherwise failures become invoices.
vLLM is powerful, but GPU infrastructure is not free. Low utilization kills the economics.
A cheaper model that reduces conversion, increases support tickets, or creates factual errors may be more expensive in business terms.
A production cost-aware architecture might look like this:
User Request
|
v
Request Logger / Cost Context
|
v
Task Classifier
|
+--> Deterministic handler when possible
|
+--> Cache lookup
| |
| +--> Cache hit -> return
| +--> Cache miss
|
v
Retrieval Gate
|
+--> No retrieval
+--> RAG pipeline: search -> rerank -> compress
|
v
Model Router
|
+--> cheap model
+--> mid model
+--> premium model
+--> self-hosted model via vLLM
|
v
Output Validator
|
+--> success -> cache if safe -> return
+--> failure -> repair/escalate within budget
This architecture makes cost a first-class concern without making cost the only concern.
The important design principle is that every expensive step has a gate:
Cost control is mostly about asking these questions automatically.
Here is the plain-language answer:
vLLM matters because serving LLMs efficiently is hard, and vLLM solves important serving problems. It manages the KV cache efficiently with PagedAttention, batches requests continuously, supports high-throughput inference, and can expose an OpenAI-compatible API. That makes it one of the practical tools teams use when they want to run open-source models themselves.
But vLLM is not the first step in reducing cost. It is a later-stage option.
Use this phrasing:
vLLM is special because it helps you get more tokens per second out of your GPUs. It reduces memory waste during generation and improves batching, which lowers cost per token when you have enough volume. But it only saves money if self-hosting makes sense. If traffic is low or model quality is not good enough, hosted APIs may still be cheaper overall.
That answer is balanced and credible.
If you want to cut your LLM bill, work through this checklist in order:
Cutting an LLM bill by 10x is possible, but not usually because of one trick. Caching helps. Model routing helps. Prompt trimming helps. Agent budgets help. Quantization helps. vLLM can help at scale. The real win comes from combining them in the right order.
The most important mindset shift is this: LLM inference is production infrastructure. It needs observability, routing, caching, budgets, evals, and capacity planning. A prototype can send everything to the best model and hope. A production system cannot.
Start with measurement. Remove waste. Cache safe repeated work. Route by difficulty. Control agent loops. Evaluate cheaper models. Only then consider local inference with vLLM.
The goal is not to make the system cheap at any cost. The goal is to spend intelligence where it matters.
%402x.svg){kind=icon}