Introduction#
In the last post, a code agent did all the talking to the model for me. The two classes on LLM APIs were about doing it yourself. The first class treated a language model as one component in a larger system, an external service that your own backend calls through an API. The second class was about putting that into practice with prompts, structured output, error handling and testing.
Once the model sits behind my own API, I decide what it gets to read and what happens when its answer is wrong. Most of those decisions cost or save tokens.
So this post is about those decisions. I start with system and user prompts and how Mistral and Anthropic each want to be prompted. Then I move on to memory and skills, which decide how much the model has to read on every call. The last part is about the answer itself: getting it back as JSON, and what to do when it goes wrong.
System and User Prompts#
A request to a chat model is a list of messages. The system prompt comes first and sets the rules: who the model is, what its task is and what the answer has to look like. It stays the same from one request to the next. The user prompt carries what changes, like the question or the document of the moment.
Say I were assessing an assignment against a rubric. The system prompt would give the model its role as an assessor, the task and the format of the answer. The user prompt would hold the rubric and the assignment text. That way the rules are written once, and when the output is off, I know which half to look at first.
Mistral and Anthropic both publish guidance on how they want to be prompted. This is what I found in their docs:
- Start with a role and a task:
You are a <role>, your task is to <task>. - Write the prompt for someone with no prior context.
- Avoid blurry words like “things” or “better”, and say exactly what you mean.
- Long system prompts drift into small contradictions. Use a decision tree instead of overlapping rules.
- Only ask for what is strictly necessary, because models read tokens faster than they generate them.
- Treat Claude as a brilliant but new employee. If a colleague would be confused by the prompt, Claude will be too.
- Explain why an instruction matters, not only what it is.
- Give it 3–5 examples, wrapped in
<example>tags. - Put long documents at the top and the question at the end.
- Say what to do instead of what not to do.
Side by side:
| Mistral | Anthropic | |
|---|---|---|
| Role | One line with role and task | In the system prompt, even a single sentence helps |
| Structure | Markdown and/or XML-style tags | XML tags |
| Examples | Example turns, or in the system prompt | 3–5 examples in <example> tags |
| JSON output | Structured outputs from a JSON schema | Structured outputs from a JSON schema |
Most of it is how I would brief a new colleague anyway. What stood out is that Mistral’s guide talks about output tokens directly. It even points out that a bare label is cheaper than JSON, though less reliable. That fits what I wrote in the first post about giving a model only what it needs.
Memory Is a Token Bill#
What surprised me most was memory, or the lack of it. Anthropic’s docs say it plainly: the Messages API is stateless, so you always send the full conversation history. The model itself remembers nothing between two calls. If I want a conversation through a plain chat API, my backend has to keep the history and send all of it again with every new message.
By the tenth question, the request carries the nine questions and nine answers before it, and all of them are paid for as input tokens again. Ragnhild is single-turn, so each question is one call with no history attached. I built it that way to store nothing about visitors, but it turns out to save tokens as well.
Code agents keep project context in instruction files instead. Vibe reads AGENTS.md and Claude Code reads CLAUDE.md when the agent starts, so a project’s rules don’t have to go in every prompt. They work as persistent memory, but they are read in full every time, so it pays to keep them short. Vibe and Claude also have memory features in their apps, but none of that helps when I call the API myself.
Quick rant: Mistral should have kept their old name, “Le Chat”. It has a much better ring to it than Vibe!
Skills#
A skill is a folder with a SKILL.md file, holding reusable instructions for one kind of task. Unlike an instruction file, it loads on demand. According to Anthropic’s docs, only each skill’s name and description sit in the context from the start, at around 100 tokens per skill, and the instructions are read when a task calls for them. Vibe uses the same SKILL.md format, following the Agent Skills specification.
That makes it cheap to give an agent a lot of know-how. Since the Tetris post, I have been looking into Matt Pocock’s skills, but how I use them is for a later post.
Structured Output#
Behind an API, it is my code that reads the answer, and code does not cope well with a friendly paragraph. With structured output, the model answers in JSON that follows a schema, and the backend parses it like any other response. Both providers let you pass that schema with the request.
In the rubric example, the JSON would have an entry for each criterion, with a rating and a line of feedback. Mistral’s docs have a tip for the rating: models do better when they rate on a scale of words than on a scale of numbers. So instead of asking for a score from 1 to 10, I would ask for a level like “low”, “medium” or “high”, and let the backend turn it into a number if it needs one.
When the Answer Goes Wrong#
A model behind an API is an external service. It can time out or fail, and even a successful answer is not always what I asked for. The material for the second class said that unstable behaviour is expected in a first version, and that the job is to find the problems and improve step by step.
So the backend has to treat the answer as untrusted input. It needs a timeout, and it has to check that the JSON matches the schema before using it. A retry is the easy fix, but every retry sends the whole prompt again. If the same thing keeps failing, the prompt or the schema is the better place to fix it.
Conclusion#
In the first post, token efficiency meant calling a model less often and sending it less. These two classes showed me where the tokens actually go, from the history that grows with every message to the output I ask for. The next time a model goes into one of my backends, that is where I will start.
Writing a clear system prompt also felt a lot like writing a spec. The next post picks up there, with this week’s classes on spec-driven development and the company we will work with for the rest of the course.
- Daniel