TokenAtlasHighest-impact levers
1. Route to cheaper models per task (40–80% savings). 2. Enable prompt caching (50–90% off cached input). 3. Use Batch APIs for async work (50% off). 4. Compress system prompts.
Medium-impact levers
5. Cap output length. 6. Truncate context aggressively for RAG. 7. Switch to fine-tuned smaller models for repeated tasks. 8. Use embeddings for retrieval instead of stuffing context.
Operational levers
9. Set per-user token caps. 10. Log everything and audit weekly. 11. Negotiate enterprise commits above $50K/mo. 12. Re-run TokenAtlas scenarios monthly — prices change.
Worked example: routing vs no routing
100K monthly calls, 3K input / 400 output tokens each. All on GPT-4o ($2.50/$10): input $750, output $400, total $1,150. If 80% of those calls are simple classification and route to GPT-4o mini ($0.15/$0.60): mini share (80K calls) costs about $76 total, flagship share (20K calls) costs about $230. Combined ≈ $306 — a 73% reduction from routing alone, before touching caching or batching.
Mechanics behind batching and caching
Batch APIs process requests asynchronously within a fixed window (commonly 24 hours) in exchange for a flat discount, typically 50%, because the provider can schedule the compute off-peak. Caching works differently: it stores the model's processed representation of a repeated prompt prefix so subsequent calls skip re-processing that portion, cutting the input-token cost for the cached segment specifically, not the whole call.
How to sanity-check whether a lever actually saved money
Before changing anything, record token volume and blended $/1M for a stable week. After the change, hold volume roughly constant and recompute the blended rate the same way. A real saving shows up as a lower $/1M blended rate, not just a lower total bill — a lower bill with lower volume tells you nothing about whether the lever worked.
What this list does not cover
These levers assume you already know your token volumes; this post does not explain how to instrument usage tracking or pull live spend data from a provider account — that's a separate engineering task. It also doesn't cover contract negotiation mechanics beyond noting that commits exist above certain volume thresholds.
Frequently asked questions
- What's the single biggest lever?
- Model routing. Most teams over-use frontier models for tasks a small model would handle.
- Is caching hard to implement?
- OpenAI is implicit; Anthropic needs explicit breakpoints. Both are <1 day of engineering.
- How do I prove savings?
- Run a 7-day A/B and compare token spend per outcome.