How to Reduce LLM Token Usage: 9 Techniques That Work (2027)

AI Application Development
Guides
Token efficiency
LLM APIs
Prompt compression

Written by: Bhargav Tibadiya
Published: November 12, 2026
Reading time: 13 min read

Illustrative document fragments pass through a relevance filter into a shorter task-ready payload, preserving the facts and constraints. — Conceptual workflow. Count full accepted tasks; this illustration does not claim a measured saving.

The most reliable way to reduce LLM token usage is to stop sending material the task does not need, stop asking for output nobody uses, and stop repeating requests whose valid answers you already have. A new serialization format can help, but it should come after measuring the whole workflow.

There is an important distinction hidden inside many token-saving guides: fewer tokens, fewer model calls and cheaper tokens are different outcomes. Prompt caching can reduce the bill without shrinking the context window. Batch processing can discount the same request. Moving to a cheaper model can leave token volume unchanged or even increase it.

This guide covers nine techniques and tells you which problem each solves. It complements the earlier LLM cost optimization guide, which was money-first. Here, the focus is the mechanics of token volume, preserving meaning and measuring what changed.

For choosing a model before changing the workflow, see the November coding-model comparison. The 2027 label is a planning horizon, not a claim that I know next year's API behavior. Documentation and tools were checked in November 2026. Recheck model support and current rates before adopting a provider-specific feature.

Start with the right denominator

A smaller prompt is not automatically a more efficient task. If a compressed request needs two retries and a longer explanation, the full workflow may use more tokens than the original. Compare accepted results under the same task definition, not just the first input string.

Record ordinary input, cached input, cache writes, visible output, reasoning tokens when exposed, tool calls and retries. Do not add nested usage categories twice. Some providers report cached tokens as a subset of input, while others expose separate fields. Read the provider's usage contract rather than treating every counter as another independent charge.

Keep a small quality check beside the count. For extraction, check required fields and exact values. For summaries, check names, amounts, deadlines and exceptions. For code, use acceptance tests and diff review. A lower token count that changes a date or removes a condition is not a successful optimization.

A useful comparison is total tokens per accepted result, plus bill and latency separately. That does not eliminate judgment, but it prevents a price discount from being mislabeled as token compression. Keep the baseline data, prompt and model identifier so the result can be reproduced later.

1. Trim context before adding a compression model

Remove duplicated instructions, irrelevant document sections, repeated boilerplate and old tool output that is no longer needed. This is ordinary context hygiene, not an excuse to delete qualifications, safety rules or the user's actual constraints.

For a document question, retrieve the passages that answer it instead of sending every available file. Keep citations and identifiers that let the answer be checked. Deduplicate overlapping retrieval chunks before the request, and cap the number of results by relevance rather than always filling the model's window.

For long-running tool workflows, loading every tool definition upfront can also waste context. Anthropic distinguishes tool search, programmatic tool calling, prompt caching and context editing because they target different sources of pressure^[1]. On supported models, tool search loads relevant definitions on demand rather than sending the entire catalog every turn^[2].

There is a tradeoff. An extra discovery step can add latency, and an overly aggressive search can hide a needed tool. Measure the whole run, including lookups. Keep critical instructions and the tools required for safe stopping available. A compact but poorly equipped agent is not efficient.

2. Control output by designing a shorter answer

Specify the smallest useful output contract. A classification task might need a label and one short reason, not an essay. An extraction task may need a fixed set of fields rather than a prose introduction and a second copy of the same information.

Use a sensible output limit as a guardrail, but do not depend on truncation to make an answer concise. A hard cap can cut off valid JSON, omit a required field or stop before an exception. Prefer a clear requested shape and then check whether the result is complete.

Where a provider exposes reasoning effort, treat that as another measured setting, not a universal free saving. Lower effort can change accuracy and retries. Anthropic's optimization guidance separates token hygiene, model selection and reasoning-effort tuning for this reason^[3].

If nobody consumes a long explanation, do not ask for one by default. If a reviewer needs evidence, retain the evidence instead of shaving it away to make the token count look good. The useful boundary is unnecessary output, not necessary accountability.

3. Compact structured data, with TOON as one option

JSON repeats keys and punctuation. Minifying it removes presentation whitespace without changing the parsed structure, so compact JSON is a useful baseline before introducing another format. Do not compare a compact alternative only against heavily indented JSON and call the full difference a format breakthrough.

TOON, Token-Oriented Object Notation, is designed to represent JSON-style data compactly for prompts. Uniform arrays can put field names in a header rather than repeating them for every row. The project's own benchmarks separate flat, tabular data from mixed and deeply nested structures^[4].

That distinction matters. Regular records often compress well; nested objects, long free-text fields and irregular schemas can behave differently. A shorter representation can also require extra format instructions or produce parsing errors in model output. Count those instructions, outputs and repairs when evaluating the workflow.

CSV is another useful candidate for a genuinely flat table, but CSV alone does not preserve every JSON type or nested-object meaning. A model may need an external schema. Keep null, empty string, numeric identifier and missing field distinct where the application depends on them.

A small reproducible count, not a model benchmark

I generated forty synthetic records with id, name, team and active fields. Using gpt-tokenizer version 4.1.0 with its explicit o200k_base encoding, the payloads measured 1,289 tokens for pretty JSON, 685 for compact JSON, 371 for TOON and 286 for a CSV table.

TOON is about 45.8% smaller than compact JSON in this sample, not a universal 60% saving. Relative to pretty JSON the percentage is larger, which is exactly why the baseline must be stated. CSV is smaller still, but it omits the outer users object and carries no independent typing schema. These strings are not all equivalent self-describing interchange contracts.

This count uses one tokenizer, no trailing newline and payload text only. It excludes prompt framing, format instructions, output and retries. No model was called, and no answer-accuracy or roundtrip test was performed. Different tokenizers and data shapes can change the result.

Payload token counts for four formats on forty synthetic uniform records — Original local token count using gpt-tokenizer 4.1.0 and o200k_base. Payload only; no model calls. CSV omits the outer object and type schema, so this is not an equivalent-contract ranking.

Use this as a method to repeat on your own data, not a result to paste into a savings forecast. The sample strings and counts are downloadable. An independent community benchmark also finds that results change with data shape, including cases where nested data reverses the advantage; inspect its methodology before reusing a percentage^[5].

4. Test prompt compression against the details you cannot lose

Prompt compression goes beyond changing punctuation. It selects, removes or transforms pieces of the prompt so fewer tokens reach the target model. LLMLingua provides open-source approaches and examples for this class of work^[6].

Start with a task-specific test set. Protect exact names, numerical values, units, dates, identifiers and words that reverse meaning, such as not or unless. Keep policy conditions and exception clauses intact. A compressed passage that sounds similar can still change the answer.

Measure compressor latency and compute as well as target-model tokens. A second model that saves a little input may not improve end-to-end latency or cost. For stable long documents reused many times, offline preparation may be sensible; for a short one-off request, the extra step may be wasteful.

Do not apply a published compression ratio to your own corpus as a promised saving. The repository's experiments depend on tasks, models and settings. I did not reproduce LLMLingua's model-accuracy benchmarks here. The practical recommendation is a controlled trial, not an untested maximum compression setting.

5. Keep long conversations small without losing the current state

Repeated history grows quickly. A useful compact state should preserve the current goal, user decisions, unresolved questions, exact resource identifiers, deadlines and evidence needed for the next action. Old exploratory chatter and bulky intermediate results are often less valuable.

OpenAI's compaction documentation describes server-side and standalone routes for managing long conversations on supported models^[7]. Anthropic's context-management guidance also distinguishes removing old tool results from other ways of reducing context pressure^[1]. Support and mechanics are provider-specific; do not assume one API's settings transfer to another.

A summary is a lossy representation. It can harden a tentative statement into a decision, drop a refusal or lose the difference between a draft and approval. Preserve those distinctions explicitly and retain a way to recover the original when a consequential action depends on it.

Compaction can also change a cached prefix and reduce cache reuse. That creates a real tradeoff: a shorter new context may save token volume while losing some cheap cached reads^[8]. Compare both counters instead of demanding that one technique improve every metric.

6. Reuse valid answers with semantic caching

An application-level answer cache can avoid a model call entirely. Exact-match caching is easiest to reason about: the same normalized request under the same authorized context returns the same stored result. Semantic caching broadens reuse to similar requests, which increases the chance of a wrong match.

Treat this as application design, not a provider guarantee. The cache key should include the dimensions that can change the answer: relevant source version, user or tenant, permission scope, language, policy version and time sensitivity. Similar wording is not enough when two people have different access or ask about different accounts.

Use short lifetimes or no reuse for live prices, availability, balances, changing policies and other freshness-sensitive answers. Invalidate when source data changes. Do not reuse one user's private response for another user just because the questions have similar embeddings.

Measure false reuse as a quality failure. A high hit rate can look efficient while silently returning stale or unauthorized information. When freshness or permission cannot be established, a cache miss is the correct result. The avoided model tokens count only for a valid reused answer.

7. Cache repeated prefixes, but call it a price optimization

Provider prompt caching reuses processing for a stable prefix. It generally does not remove that prefix from context or turn an oversized request into one that fits. Anthropic's own tool-context guidance states this distinction directly^[1].

OpenAI's current documentation separates model generations, cache breakpoints, lifetimes and write/read rates. It warns that a shared prefix is not automatically a reusable cached prefix under every configuration^[8]. Anthropic also documents model-specific minimum lengths, cache lifetimes and write/read billing^[9].

Keep stable material early and variable content later, then inspect actual cache-read fields. Include write charges, expiry and misses in the calculation. Do not pad an otherwise short prompt merely to reach a cache threshold without measuring whether the added tokens are worthwhile.

This is one of the nine techniques because it belongs in an efficiency plan, but the result should be labeled accurately: lower processing cost for repeated input, not necessarily fewer input tokens. If the objective is fitting a context window, use trimming, retrieval or compaction instead.

8. Use Batch when waiting is acceptable

Batch processing moves suitable requests into an asynchronous service tier. OpenAI documents lower cost, separate limits and a 24-hour completion window^[10]. That can suit backfills, offline classifications and evaluations. It does not suit every interactive answer.

Submitting the same prompt through Batch does not inherently make it shorter. Token volume may remain the same while the bill changes. Keep that improvement on the price line of your report rather than claiming the input has been compressed.

Make the pipeline resilient to failed, expired or partial results. Use stable item identifiers so an output can be matched to its input. Retry only the failed items, not the whole file by default. Unnecessary replay can erase the discount and increase total tokens.

Also check what can be stacked with Batch on the exact model and provider. Do not multiply several headline discounts together and present the product as a guaranteed bill. The earlier cost guide covers money-first arithmetic; this article keeps the token-volume distinction visible.

9. Route and right-size models around accepted results

Model routing and right-sizing are one policy problem: what is the least expensive configuration that meets this task's acceptance standard? A smaller model or lower effort can be useful for a bounded task, but a cheap first attempt plus repeated escalations may use more tokens overall.

Route with explicit criteria. A short classification with a validated label set may go to a small model. A difficult multi-file code change may justify a stronger one. A request containing uncertain facts or missing authority may need a question, not merely a bigger model.

Measure escalation frequency, repeated context, reviewer time and final quality. If a first-stage model creates a long draft that the second model ignores, the routing step has added work instead of removing it. Sharing a compact verified intermediate result can help, but only if the second stage can trust what it receives.

Do not equate fewer parameters with fewer tokens. Different models tokenize text differently and may reason or answer at different lengths. The only safe claim is what you measure on the same representative workload. Anthropic's model/effort guidance recommends matching configuration to quality needs rather than assuming the smallest setting is always the best value^[3].

A measurement plan that survives next year's model changes

Choose a representative workload and save an unmodified baseline. Record task IDs, source versions, prompts, model identifiers, settings and expected outputs. Use the provider's token accounting for billable usage and a local tokenizer only for clearly labeled estimates or format experiments.

Change one dimension at a time. First trim duplication, then constrain unnecessary output, then trial a format or compressor if structured input remains large. Test caching and Batch separately because they change economics rather than the same volume measure. Add routing only when the acceptance test is stable enough to decide whether escalation was necessary.

For each change, report input/output totals, cached/write categories without double-counting, successful completions, retries, latency and cost. Preserve failed examples. The most useful result may be that a modest reduction is safe while an impressive one is not.

Do not add saving percentages from different experiments. A 30% reduction in input and a 20% reduction in output do not mean the whole workflow fell 50%. Recompute from absolute counts. If a new step uses additional models or tools, include their tokens and other costs too.

What to carry into 2027

The durable practices are measurement, small relevant context, bounded output, verified reuse and explicit quality checks. Specific cache thresholds, model names, format support and discounts will change. Keep them in configuration and recheck documentation instead of baking them into an evergreen promise.

There is no universal token-saving percentage. On our tiny uniform sample, changing a payload format moved the local count substantially. That does not establish equivalent application behavior, and it says nothing about untested data shapes or model output quality.

The goal is fewer unnecessary tokens per useful result. Sometimes the best improvement is a shorter prompt. Sometimes it is avoiding a duplicate request. Sometimes it is accepting the same token volume at a lower rate. Knowing which one happened is what makes the result worth trusting.

FAQ

Does prompt caching reduce context length?

Usually no. It reuses processing for a repeated prefix and can reduce its price. The prefix still contributes to the model's context budget. Use context reduction when fitting the window is the problem.

Is TOON always smaller than JSON?

No. Results depend on the data shape, tokenizer and instructions needed to use the format. Compare against compact JSON and test output correctness, not only the size of an input string.

Should every long prompt go through a compression model?

No. Start by removing irrelevant or duplicate material. A compressor adds latency and compute, and it can lose details that matter. Test it on representative tasks before adoption.

Does Batch use fewer tokens?

Not inherently. The same request can use the same tokens under a cheaper asynchronous tier. Avoiding unnecessary retries is a separate way to reduce total volume.

Why is this a 2027 guide in November 2026?

It is a planning guide for next year's systems using currently verified techniques. It does not claim knowledge of future model releases, prices or feature availability.

What should I measure first?

Measure token usage across the entire accepted task, including output, reasoning where reported, tool loops and retries. Track quality, latency and dollars separately so a discount is not confused with compression.