Batch API
Asynchronous API mode (OpenAI, Anthropic) processing jobs at ~50% reduced cost with 24-hour SLA. Ideal for enrichment pipelines and evaluation runs.
Reference
TokenOps Glossary
The canonical TokenOps glossary — 40 terms used across TokenOps practices, dashboards, and vendor contracts. Every concept is defined here once; the guide, playbooks, and calculators all point back to this page.
B
Blended cost per 1M tokens
Total spend ÷ total tokens consumed (in millions). Tracks the efficiency of your model mix over time — the single number that captures whether routing, caching, and prompt work are landing. Target: 20–30% YoY reduction.
C
Cache Hit Rate
Fraction of API calls where a cached prompt prefix was reused instead of reprocessed. Target >80% for workloads with stable system prompts.
Chargeback
Internal billing practice that charges teams for their actual LLM token consumption. Creates cost accountability at the team level.
Chunking
Splitting source documents into smaller segments for embedding and RAG retrieval. Optimal range: 200–512 tokens per chunk.
Context Trimming
Reducing the volume of retrieved or accumulated context passed to the model — by relevance filtering, summarization, or sliding windows. Typical achievable input token reduction: 30–60% when the workload has run without discipline for months.
Context Window
Maximum tokens (input + output) a model processes in one call. Ranges from 128K on older frontier models to 200K (Claude Opus 4.5, Sonnet 5), 400K (GPT-5), 1M (Gemini 3 Pro), and up to 10M on long-context specialists (Llama 4 Scout).
Cost per Outcome
Token cost divided by a meaningful business unit (resolved ticket, converted lead, accurate classification). The ultimate TokenOps KPI.
E
Embeddings Token Cost
Cost of text-to-vector conversion for search indexes. Typically 100× cheaper than generation but accumulates at ingestion scale.
F
FinOps for AI
Application of FinOps disciplines (visibility → optimisation → governance) to AI/LLM workloads. TokenOps is the token-specific sub-discipline.
H
Hard Limit
Budget threshold that actively blocks API calls once reached. Protects against runaway spend; requires fallback logic in application code.
I
Inference Cost
The compute cost of running an LLM API call — distinct from training cost. Priced per token by all major providers. In hosted contexts it is the only cost that shows up on your invoice.
Input Token
Tokens in the request sent to the model: system prompt + history + context + user query. Priced per 1M tokens.
L
LLM Gateway
Centralised proxy between application code and LLM APIs. Enforces tagging, routing, rate limits, caching, and cost attribution.
M
Model Routing
Directing calls to the cheapest model meeting quality and latency thresholds for the task type. Most impactful single optimisation.
Model Tier
Classification by cost and capability: Frontier (GPT-5, Claude Opus 4.5, Gemini 3 Pro), Mid (Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash), Cheap/Nano (GPT-5 Nano, DeepSeek V3.2, Gemini 2.0 Flash Lite).
O
Output Constraints
Prompt techniques or API parameters that limit or structure the model's output — JSON schema, function calls, bullet-count limits. Reduces output tokens 20–40% and cuts parsing failures at the same time.
Output Token
Tokens in the model's generated response. Priced 3–10× higher than input tokens across most providers.
P
Prompt Caching
Re-using a previously computed KV-cache for a stable prefix. OpenAI: automatic at 1,024-token boundaries (50% discount). Anthropic: explicit cache control (90% read discount).
Prompt Compression
Removing redundancy, fluff words, and verbose phrasing from prompts without changing task intent. Typical saving: 15–30% of input tokens. See the Caveman method in /caveman for a specific recipe.
Prompt Versioning
Tracking system prompt changes with version control, enabling rollback when quality or cost metrics degrade and audit trails for compliance. Non-negotiable once prompts drive real revenue.
R
RAG (Retrieval-Augmented Generation)
Architecture injecting retrieved document chunks into the prompt. Primary TokenOps risk: context bloat from over-retrieval.
Reranking
Secondary scoring of retrieved chunks before injection. Allows 3–5 high-quality chunks instead of 10–20 lower-quality ones.
Retry Rate
Fraction of LLM calls retried due to errors or rejected output. >5% indicates prompt instability and inflates cost proportionally.
S
Semantic Caching
Storing LLM responses indexed by query embedding, and serving cached responses for semantically similar subsequent queries. Distinct from prompt caching, which discounts repeated prefixes provider-side. Can reduce token consumption 40–80% on repetitive workloads, at the cost of correctness risk on near-miss queries.
Showback
Reporting token costs to teams without billing them. A governance stepping stone before full chargeback.
Sliding Window
Conversation-history management technique that keeps only the last N turns and discards or summarizes older ones, preventing input token cost from growing linearly with conversation length. The default agent history strategy in 2026.
Soft Limit
Budget threshold triggering an alert without blocking calls. Typically set at 80% of monthly budget.
Structured Output
Constraining model response to JSON schema or function call format. Reduces output verbosity and parsing errors.
System Prompt
Persistent instructions prepended to every call. A 3K-token system prompt at 1M calls/month = 3B tokens = $15,000 at $5/1M. Prime caching target.
T
Token
Atomic billing unit for LLM APIs. ~0.75 English words or ~4 characters. Exact count depends on the model's tokenizer.
Token Budget
An explicit monthly limit on token consumption for a service or feature, enforced at the API gateway. Standard pattern: soft alert at 80%, hard limit at 100%, explicit approval required to increase.
Token Drift
Gradual unnoticed growth in token consumption over time — caused by accumulating history, growing data payloads, or verbose prompts.
Token Velocity
Tokens consumed per day or per hour. The key leading indicator for cost forecasting. Flat or declining velocity alongside feature growth signals successful optimization; rising velocity without feature growth is a drift alarm.
Token Yield
Fraction of generated tokens delivering value. Low yield (retries, ignored output) inflates cost without business benefit. Target: >80%.
Tokenizer
Algorithm converting text to token IDs. Different tokenizers produce different counts. Always benchmark with the target model's tokenizer.
TokenOps
The operational discipline of governing, allocating, optimising, and forecasting LLM token spend. Analogous to FinOps for cloud compute.
TPM (Tokens Per Minute)
Provider-enforced rate limit. Exceeding TPM causes HTTP 429 throttling. Relevant for burst workloads and batch job sizing.
U
Unit Economics
Token cost as a ratio to business metrics: cost per 1K users, cost per $1 ARR, cost per transaction. Makes LLM spend comparable to other COGS.
W
Waste Tokens
Input tokens not contributing to answer quality: stale history, irrelevant context, repeated boilerplate, excessive formatting instructions.