Home/Writing/Build note

Why my context engine leaves budget unspent

Past a point, adding context makes answers worse while cost and latency keep rising. Most retrieval systems have nothing that ever says stop. I built one that does.

Saad Al-TamariFrom the work: Predictive Context Engine · Jun–Aug 20262 min read
Context engineeringAgentsLLM cost

The Predictive Context Engine decides what an investment agent should actually see for a given task. It is a compiler, not a search box: every compose runs through nine stages and leaves a record you can audit months later.

Classify
Profileversioned
Safety gate
Score value
Allocatebudget
Assemble
Bandtrust
Probe
Emitsnapshot
The nine stages of one compose, each inspectable in the operator console

Pack by value per token, then stop

Candidates are ranked by the value they add per token. Packing continues only while the next item clears a marginal-utility threshold. When it doesn’t, allocation stops, even with budget left over.

Illustrative curve: answer quality rises quickly with the first, most valuable context, flattens, then falls as low-value context is added. The engine stops where the next item's value drops below a threshold.
Illustrative shape, not measured data: the engine stops where the next item adds less than it costs
Allocationsimplified
for item in ranked_by_value_per_token(candidates): if marginal_utility(item) < threshold: break # stop, even with budget left if item.tokens > budget_left: reject(item, "over budget"); continue pack(item)reject_rest(reason="below threshold")

“Why not” is as auditable as “why”

Every item that was left out is recorded with its reason. When an answer is wrong, the question is usually what the agent didn’t see, and that is now a query rather than a guess.

Injection is a boundary, not a filter

Untrusted content is scanned and dropped rather than cleaned up, and low-trust sources are structurally barred from the instruction band. That rule is enforced in code and again by a database constraint, so a bug in one layer doesn’t open the door.

Filter approach

Clean up suspicious text, then pass it along with everything else. One missed pattern and it reaches the instructions.

Boundary approach

Low-trust sources can never enter the instruction band. Enforced in code, and again by a database CHECK constraint.

Two ways to handle prompt injection; the engine uses the second

It never hard-fails

An LLM summary degrades to a deterministic extract, embeddings degrade to lexical similarity, and a timeout returns a valid fallback instead of an error.

A learning loop that can’t approve itself

Every compose is content-addressed and snapshot-backed, so it replays byte for byte. The engine proposes its own policy changes from measured evidence, and is structurally forbidden from applying them. A person approves, and reverting is one click. Before it trusts its own measurements, it calibrates its LLM judge against human labels.

Leaving budget unspent looks like waste. It is the cheapest quality improvement I know.

Saad Al-TamariFull-stack AI engineer in Amman and co-founder of SKAI Group. I build AI products end to end, from the agents and data underneath to the app people use.
About me
Read next · Build noteLet the code decide, let the model explain
Contact

Let's build

Building something like this, or want me to write about your problem next? One email is enough to start.

Book a 30-min call Download CV