Cost, latency, and prompt caching

API pricing is per token, and input and output tokens are usually priced differently — output tokens cost more, since generating each one requires a full forward pass through the model while input tokens are processed in parallel. A longer system prompt or more retrieved context directly raises cost on every single call.

Model choice is a real tradeoff: a larger model is more capable but slower and more expensive per token; a smaller model is cheaper and faster but may need better prompting to hit the same quality bar. Prompt caching lets a provider reuse a previously-processed prefix (like a long, unchanging system prompt) across calls, cutting both cost and latency for the cached portion.