Why RAG cost is not the embedding bill
Teams often plan RAG budgets around embedding ingestion, then get surprised by the generation bill. Embeddings are a one-time cost per document at a low per-token price. Generation is the recurring cost: every query pulls in retrieved chunks that inflate the input prompt, and you pay the model's per-token price for all of them on every request.
How to reduce RAG cost
- Lower top-k with better re-ranking. A tight top-3 with a re-ranker often beats a naive top-10.
- Compress chunks. Summarize or strip boilerplate before injection.
- Cache retrieval. Repeated queries should skip both vector DB and embedding cost.
- Pick a long-context, low-price model. Gemini Flash and Claude Haiku are strong on price-per-quality with large context.
Frequently asked questions
What are the cost components of a RAG system?+
Four buckets: embedding ingestion (one-time per document), embedding requeries (per user query), vector database storage/queries, and the LLM generation call that consumes retrieved context. In production, LLM generation typically dominates because retrieved chunks inflate input tokens on every request.
Why does retrieved context blow up my token bill?+
Every retrieved chunk is prepended to the prompt. A top-k of 8 chunks at 500 tokens each adds 4,000 input tokens to every query — often 5–10× the user's actual question. That input cost is paid per request, forever.
How can I reduce RAG cost without hurting answer quality?+
Lower top-k with better re-ranking, compress or summarize chunks before injection, cache retrieval for repeated queries, and route to a cheaper long-context model (Gemini Flash, Claude Haiku) for the generation step.
How much does embedding cost compared to generation?+
Embeddings are cheap — typically $0.02–$0.13 per million tokens — so ingestion of even large corpora is a small line item. The recurring cost is generation: every query pays for retrieved-context input tokens plus generated output.
Is the RAG cost calculator free?+
Yes. The calculator and baseline estimate are free. Upgrade to unlock the full optimization report — cheapest-model swap, chunk-size scenarios and monthly savings projection.