I argue that all frontier coding harnesses will soon be using only interaction models.
Current frontier models (Opus 4.7, GPT-5.5) that we interact with are non-interaction models. They have one output stream. They are next-token predictors - they write one token at a time, each token conditioned on everything that came before, and have to finish one thought before starting another. To do two things, they have to do one and then the other.
An interaction model runs many channels of input and output in parallel, instead of one stream that takes turns. It can think on one channel while it writes to you on another, while it watches a tool’s output on a third, while it edits a file on a fourth. The streams are concurrent, not sequential.
On the input side, an interaction model is not waiting for a turn either. It reads your messages as you type them, terminal output as it streams, file changes as they happen, webhook events as they arrive. None of these block the others.
The shape comes from how humans work. A person on a coding task is reading the screen, hearing a coworker, typing, thinking about what to do next, and watching a test suite stream - all at the same time. An interaction model is the same shape.
Thinking Machines’ post has video demos that make this concrete - seamless dialog management, visual interjection, simultaneous speech, and simultaneous tool calls and search. The demos are voice-and-video rather than coding, but the same multi-stream idea applies.
You ask the coding agent to add an “Export to CSV” button to your analytics dashboard. For each phase of the session, I describe what it is today, and what it should be.
What it is. You ask the model something like “add Export to CSV to the dashboard”. You can send immediately, or refine your statement in full (which dashboard, what is the time limit1) before sending. You may be dictating through a speech-to-text tool like Wispr Flow, or macOS dictation.2 Whatever you type, delete, or rephrase before sending is invisible to the model. The model only sees the final message when you hit send, and starts working from there.
What it should be. You ask the model something like “add Export to CSV to the dashboard”. When you complete the first phrase, the agent starts to research your code. As you modify and elaborate more, it steers the research in real time. By the time you complete your multi-sentence request, the model has already made significant progress in the research. The agent can already ask useful follow-up questions for your request.
What it is. The agent starts with a general instruction on what to do “add Export to CSV to the dashboard”. Claude Code could decide to search the codebase in parallel with Explore subagents. Claude Code writes the instruction up front for the subagent. One subagent explores the dashboard component. Another subagent explores the data query. Another subagent explores the design components. However, the main agent writes the instruction up front, waits for the subagent to return, and reads only its summary - it does not see what exactly the subagent saw. Claude Code cannot steer subagents mid-flight. Information learnt from one subagent cannot influence the research process of another subagent. This slows the overall research process, and is likely incomplete because subagents do not talk to each other.
What it should be. The agent starts with a general instruction on what to do “add Export to CSV to the dashboard”. The agent immediately starts multiple channels that investigate the different components. One channel explores the dashboard component. You do not waste tokens explaining the situation to the subagent. Another channel, with the same prefix, explores the data query. Another channel, with the same prefix, explores the design components. Information learnt from one channel is immediately shared with another channel. With each channel informed of how the other channels are doing, the research process is faster and more complete. For example, if it is discovered that we have similar data export functions for chat history, this information helps to inform the design components to use and the data queries to make. When the research is done, you also do not waste tokens writing the summary.
What it is. There are design decisions involved in a simple button to export a CSV. Do we give a choice to the user on what to export? Is exporting instantaneous, or is the user required to check back after an hour or so for their data? These are questions you need to ask the user. There is a tradeoff on whether you want to ask the questions early, or whether you want to do your research first before asking the questions. There is a tradeoff on whether to even ask the question, because Claude Code currently does not work in the background when questions need to be answered.
What it should be. The agent should not need to make these tradeoffs. The agent could ask questions as early as they can while working on the research in the background. The agent could retract questions if they have found the answer in the resources (for example, there are data exports that are not instantaneous for data requests of smaller sizes). All responses to the agent will immediately influence the research.
What it is. The agent is already working - it has drafted the button, wired up the export handler, and is running a first export to see the output. Halfway through, you notice that the button text is not visible in dark mode. You type a correction into the chat box. Your message is queued until the next tool boundary - it feels like the agent is stonewalling you. You can interrupt the agent to get your queries immediately answered, but this discards the agent’s current progress.
What it should be. You point out that the text is not visible in dark mode. The model reads your message immediately and acknowledges your comment. The planning channel updates to include the new constraint. The implementation channel will pick it up at the next available opportunity. You do not have to wait for a turn boundary, and you do not feel stonewalled.
What it is. The agent wants to run the export against the production database to validate it on real data, and it needs your approval to do so. The agent halts and surfaces the approval prompt. You approve or deny3. Everything else the agent was doing - drafting the button, type-checking the handler - stops too. The model is single-stream, so a pending approval blocks all the work.
What it should be. The agent surfaces the approval to run the export against production on a dedicated approval channel. Only the export channel pauses. The other channels keep running - the agent continues drafting the button code and refining the export handler while you decide. If you approve, the paused channel resumes and the export runs.
What it is. Testing follows a linear process. Your dashboard has hundreds of existing tests that programmatically test each component. You need to choose a testing setup - do you stop tests on the first failure, or do you continue to run all the tests? If the agent stops tests on the first failure, the agent will not be aware of the other tests that will fail and the agent will need multiple round trips to fix. If the agent lets all the tests run, the agent will not be able to fix the first failure as soon as it can.4
What it should be. The agent starts testing and there is one input channel dedicated to listening for errors. There is an output channel that surfaces testing issues. If there is indeed a failure, the channel working on the code will be informed and it will be expected to investigate and fix the failure. Tests can continue to run so that if there are more test failures, they will be surfaced to the agent. We get both early fixes to failures, and reduced round trips between fixing and testing.
What it is. The inference infrastructure is not wired to generate tokens after a certain context length. Models are also not trained to generate tokens after a certain context length. Agents need to compact their context before more tokens can be generated. 5 The agent cannot do anything when it is compacting.
What it should be. Compacting should be done in parallel. As the agent works on the problem, there should be another channel that decides which information is worth storing and which information should be removed from context. Information removed from context should still be searchable from any channel.
What it is. After you ship your feature, you want to improve your future experience working with the model. You write and improve skills that help you do your work more efficiently. For example, when testing the dashboard, the agent should remember to try both light and dark mode and confirm visibility of every text element. The agent will need to search their history and correctly surface pain points that could have been informed with skills.
What it should be. Reflection happens continuously in a dedicated channel that the agent maintains throughout the session. When the agent finds out that the button text is not visible in dark mode, the reflection channel should note down the issue in parallel. When the feature is shipped, the agent will propose to make improvements to AI instructions.
If interaction models are coming, here is what I think changes for users, builders, and the model market.
I still prefer Claude Code as my main interface.
For most of the work that I do, it is not possible for me to give perfect instructions in the first turn. I also operate with imperfect information, and I do not have all the answers. I need to interact with the agent to understand the problem together.
For me, Claude Code still feels easier to interact with.
I do not really care whether one model is slightly more intelligent than the other. I care about how easy it is for me to communicate with the agent and get things done.
The companies that ship interaction models first will set the floor for what users expect. Going back to a single-stream model will feel like going from a chat app to email.
Coding agents using interaction models should not require you to turn on your webcam and microphone.
You should still be able to talk to your coding agent with chat, just that it is more responsive and effective.
For users, you should continue to be great at using the current text-interface AI coding tools.
The coding agent does not know about your business.
You will still need to teach your coding agent the environment you are working with. Even with interaction models, the model still starts every session afresh.
You will still need to manage instructions and resources for the agent to access. Skills will continue to be written. Resources will still need to be accessed.
Currently the harnesses manage plenty of decisions - for example, whether to compact, whether to auto-approve a command, and the context reset after planning mode.
A lot of the harnesses are built with the assumption of a single-stream model - compaction, monitoring, chain-of-thought. Prompts are written and evaluated. With this, I think most of the harnesses that we use today will be thrown away.
If I am building yet another coding tool, I will make the harness work only with interaction models.
My bet is that the coding model market will collapse to one model size served via API6.
Currently Claude Code by default is served with two models - Opus as the main model and Haiku as the explore model.
With interaction models, everything will be one model, which means one model size.
There will be different knobs that the model can decide to turn for itself.7
Coding is the first killer use case for LLMs.
I think coding will also be the first killer use case for interaction models.
All coding models will be interaction models8.
For some reason all the banks I use make it difficult for me to export all my transaction history at once. ↩
Coding tools should ship with dictation (the human’s voice in) and text-to-speech (the model’s voice out) built in. Today they do not. I had to add a TTS hook to my Claude Code template so the agent can speak its notifications out loud. I do not have voice dictation software. ↩
I am aware that Claude Code has an auto-mode where the agent has a process to automatically decide which commands are safe to run. However, I think interaction models are useful here, there could be one channel where the model decides whether to approve running the command. ↩
Claude Code has this monitor tool where the agent will monitor something in the background. ↩
There are models with millions of tokens of context. In my experience with Opus 4.7, I feel that the model simply forgets a lot of things after the 200,000th token. I would rather the model automatically compact at the 200,000th token. ↩
If there are models of different sizes being developed, I think they are local models that need to be run on device. ↩
We are familiar with the thinking effort as a knob. I think models should be able to tune their thinking effort by prompting themselves. There are other knobs that could be turned if you train the model to do so. Maybe the size of the prefix that you can attend to is tunable. Maybe the number of experts that you can use is also tunable. ↩
I think the first human-level robotics model will also be an interaction model. ↩
The main material available is the pyramid visualization from the Flash Attention paper.
I wanted the visualization to be animated. I did not manage to find a good resource out there 1. So I made mine.
The animation was made with MacOS Keynote, manually. 2. As of writing, it appears to rank second on Google search.
I also wrote a Quora answer explaining how Flash Attention works.
I asked in the GPU Mode Discord for opinions on my work.
gau.nernst replied with the following comments, which I greatly appreciate.
i wanted to comment that the loop order was reversed. but upon checking, turns out FA1 used this loop ordering, but FA2 reversed it (and I only read the FA2 paper lmao)
so in FA2, iterating along K/V is the inner loop, iterating along Q/O is the outer loop, which is implemented as 1 threadblock handling 1 Q/O tile
yea I think FA3 and FA4 also follow the FA2’s general design, but optimized for Hopper and Blackwell respectively
The explanation may not be complete, the details may not be fully correct, but I still hope this makes it slightly easier for you to understand Flash Attention.
The writeup with the links is available on Kaggle.
This is the first time I won prize money from Kaggle competitions.
]]>Think of the most capable person that you have ever worked with.
You hire them into a new company.
There are still things this person could have done more “accurately” and “efficiently”.
Think of what they will need to learn between the first day they join the company until the day where they become effective individual contributors. They will need to learn where to look for information. They will need to know who to approach to access privileged information. They will need to learn the processes necessary to ship things. They will need to learn the pitfalls that they should avoid.
Humans have memory and they could remember things. For AI coding tools however, every time you start a chat, their memory is reset. They know absolutely nothing about your company at the beginning of each session. 1 You need to teach them again.
This teaching process could have been accelerated with skill files.
I want to encourage my colleagues to write and maintain skills to accelerate their work. Here I describe how I am doing it.
Skills consist of three elements at minimum:
Let’s use writing a commit message as an example of a skill.
---
name: write-commit-message
description: Commit message guidelines. Use when writing git commit messages.
---
<skill content>
When the AI coding tool starts a session, it will load the skill name and skill description into the model context.
This is what you see in Claude Code when you run /context.
Skills · /skills
Project
└ write-commit-message: 21 tokens
When the agent needs to write a commit message, it should decide to invoke a skill. When the skill is invoked, the agent will read the skill content. In this case, the skill content contains the information on how to write commit messages.
Skills are needed to instruct the agent on how to do things more accurately and efficiently.
On writing commit messages, the company likely has some standards on how commit messages should be written.
For example, the commit message should contain this information:
If agents want to write commit messages accurately without additional instructions, they will need to figure out these requirements by looking at similar commits, or running the unit tests related to the commit message.
However, if you require the agent to learn the commit message pattern on every session by reading many similar commits, this is not efficient. If the agent skips the learning process, the agent is not being accurate. This will still be the case even as models get better, because not every company has the same commit message standards.
Writing commit messages could have been done more accurately and efficiently2. Skill files allow this. When the agent is going to write commit messages, it will “invoke the skill” and read the skill content.
Initially I placed the commit message standards in CLAUDE.md / AGENTS.md. This was a reasonable place to put the instruction because it is globally relevant. I have been advocating for CLAUDE.md to only include globally relevant information. However, this means the commit message instructions are loaded even when the user is not writing a commit message, for example when asking questions about the codebase. There is room for improvement here, regarding efficiency.
Then I moved the commit message standards to the git commit template. Instead of placing the full instructions in CLAUDE.md, I added a pointer to the git commit template. This is what I have been doing before we had skills. This still follows the progressive disclosure principle, because I load the commit message standards only when I write the commit message.
Even though placing the commit message standards in the git commit template fulfills the principle of progressive disclosure, there is still benefit in making write-commit-message a skill.
We want to centralize our AI instructions instead of scattering them over the codebase.
When we implement telemetry and feedback loops for skills, write-commit-message can also benefit if it is a skill.
If you want a process to be done more accurately and efficiently with AI coding tools, you should write a skill. These are some examples where you should think about writing a skill.
You have a resource that you want your agent to access. The resource could be Notion, Slack, Asana, or any internal pages. Instead of playing telephone between the AI coding tool and the resource, you can write a skill that teaches the agent how to read the resource. However, this assumes that your AI coding tool has access to the resources, which you will have to set up first.
You execute repetitive processes that you want automated. For example, every day I am supposed to check the feed statistics for our recommendation system. This involves looking at dashboards. If there are significant movements in the metrics, I need to explain it by looking at commit logs. This should have been a skill.
You want a process to be done more efficiently in the future. One such process is on-call pages. You might already be handling on-call pages with AI coding tools that have access to dashboards and error logs. In the future, you want to handle this more efficiently. You can write a skill that informs the agent of the resources that it should look at and the dead ends that it should be aware of.
There are cases where you should not write a skill.
plan-mode or clarify-user-questions skill because AI coding tools should already include this in their system prompt.check-commit-message skill, you should not be asking the agent to run checks that could be unit tests.
The agent should not be an expensive linter.
If there is still value in writing check-commit-message to check the qualitative aspects of the commit message, the skill should ask the agent to run the relevant unit tests for the deterministic checks.Start by writing the simplest possible skill that is worth using.
You could look at what you did in the past week and think of:
Then, think whether any of these processes could be done more accurately and efficiently with AI coding tools.
If so, you have found a good candidate for a skill.
Then write your skill. Start simple, with only a SKILL.md file.
---
name: <name>
description: <description>
---
<What exactly the skill does>
Example queries
- <example query>
# Workflow
<step by step process>
Checklist
- [ ] Item 1
- [ ] Item 2
# Pitfalls to avoid
<list them>
After writing your skill, you should test it. When you commit your skill, you should include evidence that it is tested. Unlike unit tests, testing a skill is not deterministic, but you should still provide evidence of testing.
Here are some good forms of evidence:
Your colleagues will review your skill, just as code is reviewed in the codebase.
As hundreds of colleagues commit skills into the codebase, you will soon have hundreds of skills.
This means that you will have hundreds of skill names, and hundreds of skill descriptions. If each skill is 50 tokens, this will be 5000 tokens. Also depending on the quality of your skill descriptions, the agent might invoke skills unnecessarily, or fail to invoke skills when it is needed.
If you look at the skills, there are skills that are company-wide and there are skills that are team-wide. You should only load company-wide skills into context.
This can be done in Claude Code.
For team-wide skills, add disable-model-invocation: true to prevent the skill from being loaded in context.
---
name: investigate-speed-feed
description:
disable-model-invocation: true
---
<skill content>
This will mean that if you go to Claude Code and ask “please investigate feed speed”, the skill will not be invoked.
You need to write /investigate-speed-feed.
This is fine, because people who need to use the skill should know about the skill.
By separating team-wide skills and company-wide skills, and requiring all team-wide skills to have model invocation disabled, you reduce the risk of the agent not invoking necessary skills or invoking unnecessary skills4.
Organize the team-wide skills and company-wide skills into two folders.
skills/
├── company_wide/
│ └── write-commit-message/
│ └── SKILL.md
└── team_wide/
└── investigate-speed-feed/
└── SKILL.md
However, the skill standard requires all skills to be at the same level.
Then symlink every skill folder into skills/all.
skills/
├── all/
├── company_wide/
└── team_wide/
.claude/skills, .cursor/skills, and .codex/skills are soft symlinks to skills/all.
To enforce that skills follow the intended format, you should write unit tests to test skills.
For example, you could have unit tests that check that
As the skills maintainer, you need to define your responsibilities. You should not be reviewing every line of every skill. If there is an issue with a skill, you should direct the complaint to the owner of the skill. You can give advice, but should not be required to.
There are still responsibilities that belong to you as the skills maintainer. For example, if a skill is not invoked when it is supposed to, or unnecessarily invoked, you need to investigate. It might be the issue of how the user queries the model, how the skill name and description is written, or it might be the fault of the model or the AI coding tool. It is still your duty to triage and provide advice here. Periodically, you also need to review how skills are written and identify anti-patterns and legislate against these anti-patterns.
To help your colleagues write skills, you write a skill that helps them write skills.
Initially I used Anthropic’s skill-creator skill to write skills.
However I found out there are many unnecessary parts.
For example, there is no need for the init_skill.py steps that create all the resource directories.
Skills should start simple with just a SKILL.md file.
To help your colleagues improve skills, you again write a skill that helps them improve skills.
There will be a skill-feedback skill where agents can provide feedback on skills.
Feedback should be provided when the agent finds an inaccuracy in the skill file, or pitfalls that the skill has not documented.
The feedback will be stored in some data lake, which should be queried when we iterate on skills.
I hope this helps you write and improve skills for your company, so that the agents can do things more accurately and efficiently.
This sentence was paraphrased from the resource I recommend on how to write CLAUDE.md. ↩
If you have not noticed by now, the keywords are accurately and efficiently. ↩
That said, it might still be reasonable to include a skill that helps to search code if the agent could not find the code. You might have some code that is hard to search, for example you might be searching for a string but the actual string is broken into two pieces with each string defined at different places. However, it should not be expected for the agent to trigger this skill for every search, but only when previous search attempts fail. Of course, the better way to solve this is to avoid writing code that requires doing this, or improving your codebase instead. ↩
The field name is disable-model-invocation: true, which unfortunately is a negative.
It could have been “invokable skills” or “non-invokable skills”, but it is confusing because users can invoke a skill with the slash command.
The more precise term is “model-invokable skills” and “non-model-invokable skills” but that is too long.
I am glad that I have arrived at the terms “team-wide skills” and “company-wide skills”. ↩
Anthropic recommends the gerund form.
However Anthropic is not really following the conventions they recommend.
Currently I only require skill name to be in the format {resource / workflow}-{team name}.
This is a problem I should worry when the monorepo actually has a hundred skills. ↩
These are some evaluation test cases that I think LLMs should no longer be evaluated on.
There are evals that test whether the LLM remembers certain things (MMLU).
This is the README example in MMLU dataset.
{
"question": "What is the embryological origin of the hyoid bone?",
"choices": [
"The first pharyngeal arch",
"The first and second pharyngeal arches",
"The second pharyngeal arch",
"The second and third pharyngeal arches"
],
"answer": "D"
}
I argue that this is an unreasonable memorization task1. LLMs do not need to memorize this to be helpful. Neither do doctors need to memorize this to execute their work.
Even as we do not expect LLMs to always provide the perfect answer to this, we still expect AI systems to solve this task perfectly. AI systems are more than just models. AI systems involve LLMs having access to tools such as web search.
There are currently evals that test whether a model can retrieve a specific piece of information buried in a very long context. Model providers boast that their models could memorize the context well into the millions.
I argue that this is a non-goal. Humans can solve very long problems without the ability to memorize the entire conversation history. A math professor does not need to recall what they did exactly 300 days ago to solve an open math problem that requires a lot of research and trial and error.
The goal here is to solve complex tasks with the minimum compute. I expect the solution to involve an AI system where the LLM manages its own context and queries previous context with tools. The solution does not necessarily require a model that is able to retrieve something within a million tokens.
This means I still expect AI systems to be perfect at super-long context retrieval. LLMs should be able to use tools to query the history and perfectly retrieve any specific piece of information.
Some evaluations test whether an LLM can answer domain-specific questions - legal advice, medical diagnoses, tax regulations.
You cannot expect a lawyer to do well at their job without access to the legal resources. Similarly, you cannot expect the LLM to perform as well as the lawyer without access to the same set of legal resources.
LLMs should have baseline domain knowledge, like a human professional would. We should avoid having evals that simply test the LLMs without providing them access to the necessary legal resources.
Some evaluations test raw computation - multiplying large numbers, performing complex arithmetic, counting characters.
We should not be evaluating LLMs on whether they can do 100-digit multiplication without chain of thought. LLMs should be able to figure out and follow recipes to do 100 by 100 digit multiplication in O(n²) tokens.
LLMs should be evaluated on whether they can discover and execute algorithms, not whether they can compute magically2.
We should not use the same mindset we used to participate in traditional Kaggle competitions to train LLMs.
In traditional Kaggle competitions, you do whatever it takes to maximize your score on the leaderboard, even if it means training on incorrect labels in the training set. We should not carry this mindset to evaluating LLMs.
I expect model providers to remove inconsistent or ambiguous test cases when evaluating their models internally.
I hope we apply higher scrutiny to the test cases we use to evaluate LLMs.
I also hope that you understand the difference in standards that we apply to LLMs and to AI systems. There are tasks we expect AI systems to solve perfectly, but do not expect LLMs to get correct.
So when an LLM fails a benchmark task, ask yourself: is the model wrong, or is the task wrong?
Research shows that LLMs memorize approximately 3.6 bits of information per parameter. Based on this budget, model providers should probably at least have some internal standards on what LLMs are supposed to know and not supposed to know. ↩
It is still an interesting exercise to design weights to a given neural network architecture to multiply integers without a thinking process. ↩
hamel.dev/blog/posts/evals-faq/
Many of the ideas I have been promoting1 align with the Evals FAQ.
The content here does not represent the prevailing opinions or practices of any organization.
These are ideas that I have been promoting. I should have cited the Evals FAQ when promoting my ideas.
On binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales).
Having binary options forces people to make a decision rather than hiding uncertainty in middle values. Binary decisions are also faster to make during error analysis - you don’t waste time debating whether something is a 3 or 4.
For example, you want to classify whether content is adult. You might want to ask for a scale of 1 to 4 instead of a binary classification. On the adult side, you want a label of 1 if the content is CSAM that you have to eliminate from your platform, and a label of 2 if it is merely adult. On the less adult side, you want a label of 3 if it is provocative that you do not want to show to new users, and 4 if the content is perfectly fine. Only users who opted in to adult content will see content with label 2.
I would rather we have three binary classifiers - on whether something is adult (1+2 vs 3+4), on whether something is CSAM (1 vs the rest), and on whether something should be shown to new users (4 vs the rest).
There are some arguments for having one classifier for all 4 labels
I argue that you should think harder about whether having a graduated 1-4 scale is really a better system than having three binary classifiers.
If your systems are uncoupled, you just need to align the parts that you need to align. Sure, it might cost more to run the LLMs three times, but engineering time usually costs even more money.
On versioning and managing prompts
There is an unavoidable tension between keeping prompts close to the code vs. an environment that non-technical stakeholders can access.
My preferred approach is storing prompts in Git. This treats them as software artifacts that are versioned, reviewed, and deployed atomically with the application code.
Prompt management tools are inherently limiting because they can’t easily execute your application’s code. Even when they can, there’s often significant indirection involved, making it difficult to test prompts with your system’s capabilities.
There is indeed a requirement for non-technical stakeholders to read and write prompts. Prompts should already be easily readable if your organization has good AI tooling to point you to the exact prompt.
There is also the case where non-technical stakeholders want to write and test the prompts2 without opening a terminal or Jupyter notebook. It should be possible to keep the prompts in git while they experiment with the different prompts.
On how many people should be annotating LLM outputs
A single expert eliminates annotation conflicts and prevents the paralysis that comes from “too many cooks in the kitchen”. The benevolent dictator can incorporate input and feedback from others, but they drive the process.
I have written exactly on this before.
I argue that democracy is a horrible way to build a dataset for LLM evaluation. Let’s say you want to build dataset to determine whether an advertisement is low quality. You get your team to label the content and treat the labels as immutable ground truth. But you notice the labels provided by your team often disagrees with each other.
If you pass this dataset to your colleague, your colleague is simply guessing what people are voting. Your colleague will likely not perform well. This is what you will be tuning your prompt to.
The FAQ suggests a benevolent dictator to own the prompts, which I agree with.
Evals should only focus on cases you care about. In other words, you do not need a ground truth label for everything.
For example, in adult content classification, you only want to label what is obviously adult and what is obviously not adult. If something is neither obviously adult nor obviously not adult, there is no need for a ground truth label. This means, in production, you are okay with the content being classified either way.
I think some updates have to be made considering models are much more powerful now.
On passing 100% of your evals
Be wary of optimizing for high eval pass rates. If you’re passing 100% of your evals, you’re likely not challenging your system enough. A 70% pass rate might indicate a more meaningful evaluation that’s actually stress-testing your application. Focus on evals that help you catch real issues, not ones that make your metrics look good.
I do not agree that passing 100% of your evals is wrong in itself. Of course, you still need to justify the value of evals that you already score 100% (or any passing rate as well).
When there is a classification failure, either one or multiple parts must be wrong - the classifier (model + prompt), or the label (or the LLM-as-a-judge evaluator). You should find out which one it is. It is possible that the label is wrong3. If the prompt is wrong, you can probably tweak the prompt. If the model is wrong, you likely cannot do anything. My point here is, when there is a mistake, something must be wrong.
Mistakes are mistakes. It is possible to fix all these mistakes to get your eval to 100%. Models are better these days, and now you have AI to help you write and improve prompts. It is possible that you are not able to find any more mistakes with reasonable effort.
There could still be value for evals that pass 100%. For example, you have a classifier that classifies whether something is adult. The requirements are quite loose. You only need to classify correctly if something is obviously adult or obviously not adult. There are a lot of borderline cases where you are okay with the system tagging either way. Models today can perform 100% at this task. There is still value in this eval even though it passes 100%.
You can deploy this classifier and monitor mistakes in production. The classifier is likely to make mistakes (something obviously adult being classified as not adult), and you add the mistakes to the evaluation. Then you can tune the classifier to achieve performance on the mistakes and the initial dataset. The eval that passes 100% is still useful.
You can use the same eval when you migrate models. Models get deprecated, or you found a much cheaper model that is equally performant. The eval that scores 100% still serves as a unit test that is only run once every model migration.
What you should not do is try to add borderline examples to the dataset to make the eval perform at 70%. Similar to how “the goal of evaluations isn’t to pat yourself on the back for a perfect score”, it should also be the case that “the goal of evaluations isn’t to pat yourself on the back for an imperfect score”.
There will continue to be more people working on AI models and products. There will continue to be more compute. Access to compute will be better. You will also have competent AI that helps you improve AI.
The only AI-powered browsing experience that I have tried is the Claude extension on Chrome.
There are a few read-only tasks I want it to do:
The experience is very slow and very inaccurate2.
I see a few problems here:
I hope to be able to trust AI with all my read-only browser tasks. After AI earns my trust with their performance on read-only tasks, I can slowly trust AI with tasks that make changes.
For the user experience, I think context constraints should already be invisible to users of frontier AI products. You should not be hit with an instruction on ChatGPT that your conversation is too long. There are some processes in the background that compact the context.4
For the developer experience though, you need to be aware of the context limits. In 2025, the phrase context engineering was coined.
However, I think developers will no longer need to care about context length. The model API will be shipped with context management,5 similar to how you talk to your friend - you do not need to ask your friend to delete old messages so that you can continue talking to them.
This developer experience should already be achievable with existing models and a suite of scaffolds. There might also be improvements in the model architecture to help.4
We are familiar with the chat interface with AI. You type something, hit enter, and AI replies with something. You are taking turns with AI.
The experience talking to another human is different. You take note of their facial expression and body language. You look up certain information while they are talking. You write notes to yourself. There are no explicit turns to take.
There are some efforts to break this turn-based experience with AI:
There are still some bottlenecks:
It is possible to achieve all this with a suite of scaffolds. There might also be improvements in the model architecture to help.7
Currently we expect models, with their harness, to complete any task with the resources available to them. They are expected to use existing instructions.8 Models are already expected to check their own work before reporting success.
Soon we should expect AI to not just follow processes, but also to improve the processes.
Currently, the human needs to take the initiative to prompt the model to fix the instructions. The instructions may include comments and docstrings in the code that inform future models. The instructions may be skills files that provide guides on how to execute certain processes.
We will see AI products taking the initiative to suggest changes to the process. We will see models that tastefully fix outdated and inaccurate comments in the codebase. We will see skills written, and maintained as they are being used.
See my predictions for 2025 for reference on how inaccurate they were. ↩
I think I can get better results with Claude Code on puppeteer MCP with subagents.
Regarding puppeteer MCP - even though @modelcontextprotocol/server-puppeteer is deprecated, I am not recommending @playwright/mcp because it takes up more tokens (4k vs 14k) and “No vision models needed, operates purely on structured data” is outdated. ↩
AI is kind of blind - you can see that the bottom set of 30 x 30 matrices are misaligned, which should not pass any design review. I think o3 pioneered a method where the model zooms in to a specific section. I look forward to models drawing lines on an image to check whether they aligned the CSS correctly. ↩
One way to scaffold this is to have a parallel summarization process that kicks in at 50k tokens to summarize into 10k tokens, and at 80k tokens, replaces the 50k tokens with the 10k token summary so the conversation continues from 40k tokens. I have an idea where the model discards the KV-cache of parts of the conversation that are no longer relevant, and uses tools to search the conversation history instead. This removes the need to generate 10k tokens every 50k tokens. One prime candidate for decaching is the screenshots when manipulating browsers. ↩ ↩2
OpenAI released the Responses API in 2025 which tracks conversation state server-side.
You do not need to pass in the entire conversation history with each request.
Instead, you pass around an id representing the state of the conversation, and OpenAI keeps it up-to-date for you.
You still need to manually call /responses/compact to compact the context. ↩
For some human-powered helpdesk chatbots, apparently the human operator can see what you typed before you send. ↩
I wrote about how models should be multichannel - humans have multiple input and output channels, and models should be able to converse like humans. ↩
I have written about how instructions are more than just user and system prompts. ↩
Crafting LLM weights to multiply integers without thinking.
It is already expected that frontier models have perfect performance at multiplication without tool use. Frontier models should be able multiply step-by-step, only limited by time and space complexity.
However, I am interested in multiplication without a thought process, the model is expected to reply with the product directly.
I want to try crafting weights to perform multiplication perfectly. I will also need to study the multiplication algorithm itself.
There is this related work on training a model to perform modular addition perfectly.
Understanding segment trees.
Segment trees are an important concept if you want to get a higher rating in competitive programming. I have a template that I do not know how to use. I think I also need to understand the algorithm and the intuition behind segment trees.
Understanding Aho–Corasick algorithm.
LeetCode contests tested this two times.
I solved both problems with other methods.
Tree matching problem with wildcards
You have two rooted trees. Some nodes are wildcards.
The task is to determine whether you can assign each wildcard a subtree so that you can match the two trees.
Please do not ask me where I got this problem.
Memorize the entire architecture of gpt-oss.
gpt-oss-120b is the go-to model for AIMO 3. I want to understand the model architecture very well. I need to be able to derive the parameter count, and also its GPU memory usage. I will also need to compare and explain the differences between 20b and 120b versions.
Watch some films
These are the films I have yet to watch
This post walks through the startup logs from serving GPT-OSS-120B (a 117B parameter MoE model with MXFP4 quantization) on a single GPU via vLLM v0.11.2. Each log line is explained with links to source code and documentation. Logs are from this Kaggle notebook.
INFO 11-28 11:45:51 [scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=2048.
max_num_batched_tokens caps tokens per scheduler step. With chunked prefill enabled (default in V1), the scheduler prioritizes decode requests and batches pending prefills into remaining token budget. Lower values (e.g., 2048) achieve better ITL (inter-token latency) because there are fewer prefills interrupting decodes. Higher values achieve better TTFT (time-to-first-token) as more prefill tokens are processed per batch. Default is 8192 for online serving, 16384 for offline. See vLLM Optimization docs.
This value also affects how much memory is left for KV cache. vLLM allocates all remaining GPU memory to KV cache after loading model weights—controlled by gpu_memory_utilization (default 0.9). Higher max_num_batched_tokens reserves more activation memory during the profiling step, leaving less for KV cache and reducing max_concurrency. Decreasing max_num_batched_tokens or max_num_seqs frees KV cache space for more concurrent requests.
(APIServer pid=90) INFO 11-28 11:45:51 [api_server.py:1977] vLLM API server version 0.11.2
(APIServer pid=90) INFO 11-28 11:45:51 [utils.py:253] non-default args: {'host': '0.0.0.0', 'model': '/kaggle/input/gpt-oss-120b/transformers/default/1', 'max_model_len': 98304, 'served_model_name': ['vllm-model'], 'gpu_memory_utilization': 0.96, 'max_num_seqs': 6}
max_num_seqs caps concurrent sequences per batch (default 1024 in V1, up from 256 in V0). Higher values allow more concurrent requests but require more KV cache space at runtime. Lower values reduce memory pressure and avoid preemption, where requests are evicted and recomputed when KV cache fills. Here, max_num_seqs=6 is set low for this memory-constrained single-GPU setup.
(APIServer pid=90) INFO 11-28 11:46:35 [model.py:631] Resolved architecture: GptOssForCausalLM
vLLM reads architectures from config.json and maps it to the corresponding model class in vllm/model_executor/models/.
(APIServer pid=90) ERROR 11-28 11:46:35 [config.py:307] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/kaggle/input/gpt-oss-120b/transformers/default/1'. Use repo_type argument if needed., retrying 1 of 2(APIServer pid=90) ERROR 11-28 11:46:37 [config.py:305] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/kaggle/input/gpt-oss-120b/transformers/default/1'. Use repo_type argument if needed.
(APIServer pid=90) INFO 11-28 11:46:37 [model.py:1968] Downcasting torch.float32 to torch.bfloat16.
This is about the original checkpoint dtype. Since GPT-OSS uses quantization=mxfp4, weights end up as 4-bit anyway. bfloat16 is used for activations and KV cache.
(APIServer pid=90) INFO 11-28 11:46:37 [model.py:1745] Using max model len 98304
(APIServer pid=90) INFO 11-28 11:46:45 [scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=8192.
Printed twice: first during APIServer config parsing (default 2048), second when EngineCore initializes with computed value (8192). 8192 is the online serving default.
(APIServer pid=90) INFO 11-28 11:46:45 [config.py:272] Overriding max cuda graph capture size to 1024 for performance.
CUDA graphs pre-record GPU operations to eliminate per-kernel CPU launch overhead. Each captured batch size requires memory to store the graph. Limiting to 1024 max balances memory usage against coverage of typical batch sizes. See vLLM CUDA Graphs docs.
(EngineCore_DP0 pid=289) INFO 11-28 11:47:23 [core.py:93] Initializing a V1 LLM engine (v0.11.2) with config: model='/kaggle/input/gpt-oss-120b/transformers/default/1', speculative_config=None, tokenizer='/kaggle/input/gpt-oss-120b/transformers/default/1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=98304, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=mxfp4, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='openai_gptoss', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=vllm-model, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'use_inductor': None, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 528, 544, 560, 576, 592, 608, 624, 640, 656, 672, 688, 704, 720, 736, 752, 768, 784, 800, 816, 832, 848, 864, 880, 896, 912, 928, 944, 960, 976, 992, 1008, 1024], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_cudagraph_capture_size': 1024, 'local_cache_dir': None}
Key settings for GPT-OSS:
quantization=mxfp4: MXFP4 (Microscaling FP4) compresses weights to 4-bit with per-group scaling factorsenable_prefix_caching=True: Automatic Prefix Caching shares KV cache for requests with common prefixescudagraph_mode=FULL_AND_PIECEWISE: FULL_AND_PIECEWISE uses full CUDA graphs for uniform decode batches, piecewise graphs for mixed prefill-decode batchesreasoning_parser='openai_gptoss': GPT-OSS specific reasoning output parser
(EngineCore_DP0 pid=289) INFO 11-28 11:47:31 [parallel_state.py:1208] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.19.2.2:54751 backend=nccl
Single GPU setup. world_size=1 means one process, backend=nccl uses NVIDIA NCCL (NVIDIA Collective Communications Library) for fast GPU-to-GPU communication in multi-GPU setups.
[W1128 11:47:31.402851082 socket.cpp:209] [c10d] The hostname of the client socket cannot be retrieved. err=-3[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
Benign. With one GPU, there are no peers to connect to.
(EngineCore_DP0 pid=289) INFO 11-28 11:47:31 [parallel_state.py:1394] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
Parallelism types (all 0 because single GPU). See How to Parallelize a Transformer from the JAX Scaling Book:
(EngineCore_DP0 pid=289) INFO 11-28 11:47:31 [gpu_model_runner.py:3259] Starting to load model /kaggle/input/gpt-oss-120b/transformers/default/1...
(EngineCore_DP0 pid=289) WARNING 11-28 11:47:32 [mxfp4.py:196] MXFP4 linear layer is not implemented - falling back to UnquantizedLinearMethod.
Some layers lack FP4 kernel implementations and run in bfloat16 instead. See vllm/model_executor/layers/quantization/mxfp4.py.
(EngineCore_DP0 pid=289) WARNING 11-28 11:47:32 [mxfp4.py:208] MXFP4 attention layer is not implemented. Skipping quantization for this layer.
Attention layers (Q, K, V projections) run unquantized. See the same mxfp4.py source.
(EngineCore_DP0 pid=289) INFO 11-28 11:47:32 [cuda.py:377] Using AttentionBackendEnum.TRITON_ATTN backend.
vLLM supports multiple attention backends:
VLLM_FLASH_ATTN_VERSION=3. FlashAttention-3 (paper, vLLM integration) is optimized for Hopper using WGMMA async Tensor Cores and TMA, achieving 75-85% of H100 theoretical max FLOPS (740-850 TFLOPs) vs. 35% for FA2. Requires CUDA ≥12.3. Note that logs don’t distinguish FA2 vs FA3—the version is determined silently based on GPU architecture.Here, TRITON_ATTN is selected.
(EngineCore_DP0 pid=289) INFO 11-28 11:47:32 [layer.py:342] Enabled separate cuda stream for MoE shared_experts
GPT-OSS is a Mixture-of-Experts (MoE) model with shared experts (always active, capture common knowledge) and routed experts (selected by gating). Running shared experts on a separate CUDA stream allows overlapping their computation with the routed expert dispatch/combine operations. The expected overlap pattern is: shared experts compute in parallel with token dispatch to routed experts, then routed experts compute, then results combine. See vLLM RFC #9203 and vLLM MoE implementation.
(EngineCore_DP0 pid=289) INFO 11-28 11:47:32 [mxfp4.py:141] Using Marlin backend
Marlin is a weight-only quantization kernel from IST-DASLab. On GPUs without native FP4 tensor cores, Marlin decompresses FP4 weights on-the-fly during computation. See vLLM Marlin utils.
(EngineCore_DP0 pid=289) Loading safetensors checkpoint shards: 100% Completed 15/15 [09:05<00:00, 36.34s/it](EngineCore_DP0 pid=289) INFO 11-28 11:57:30 [default_loader.py:314] Loading weights took 545.24 seconds
9 minutes is slow. Kaggle storage I/O is the bottleneck. Local NVMe + tensor parallelism would be faster.
(EngineCore_DP0 pid=289) WARNING 11-28 11:57:30 [marlin_utils_fp4.py:204] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
This Kaggle environment’s GPU lacks native FP4 tensor cores. Native FP4 requires Blackwell architecture (SM100+)—Hopper (SM90) only supports FP8. The Marlin kernel provides weight-only compression: weights stored in FP4 (memory savings) but computed in higher precision. See GitHub issue #30135 for more on MXFP4 backend selection.
(EngineCore_DP0 pid=289) INFO 11-28 11:57:33 [gpu_model_runner.py:3338] Model loading took 65.9651 GiB memory and 600.408480 seconds
65.97 GiB is expected. Here’s the math based on GPT-OSS-120B config.json:
Architecture: 36 layers, hidden_size=2880, 128 experts per layer, intermediate_size=2880 per expert
MoE weights (~115B params, MXFP4 quantized):
hidden_size × intermediate_size = 3 × 2880 × 2880 = 24.9M paramsNon-MoE weights (~2B params, BF16):
Total: 59.1 + 4 + ~2-3 GB (CUDA context, buffers) ≈ 65-66 GB
This matches Unsloth’s documentation which recommends “at least 66GB of unified memory” for gpt-oss-120B inference. See also OpenAI’s model card for architecture details.
(EngineCore_DP0 pid=289) INFO 11-28 11:57:52 [backends.py:631] Using cache directory: /root/.cache/vllm/torch_compile_cache/7fcbe477d2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=289) INFO 11-28 11:57:52 [backends.py:647] Dynamo bytecode transform time: 19.07 s(EngineCore_DP0 pid=289) INFO 11-28 11:57:58 [backends.py:251] Cache the graph for dynamic shape for later use(EngineCore_DP0 pid=289) INFO 11-28 11:58:38 [backends.py:282] Compiling a graph for dynamic shape takes 44.90 s(EngineCore_DP0 pid=289) INFO 11-28 11:58:39 [monitor.py:34] torch.compile takes 63.97 s in total
torch.compile is PyTorch’s JIT compiler. TorchDynamo (the frontend) traces Python code and captures computation graphs. TorchInductor (the backend) then fuses multiple operations into optimized CUDA/Triton kernels—this reduces memory bandwidth usage since intermediate results stay in registers instead of being written to and read from global memory. See PyTorch’s torch.compile tutorial for details on how fusion works. vLLM adds custom fusion passes (RMSNorm+quantization, SiLU+quantization) for additional speedups. Compiled artifacts are cached in ~/.cache/vllm/torch_compile_cache for reuse across runs. See vLLM torch.compile docs.
(EngineCore_DP0 pid=289) INFO 11-28 11:58:41 [gpu_worker.py:359] Available KV cache memory: 8.99 GiB(EngineCore_DP0 pid=289) INFO 11-28 11:58:41 [kv_cache_utils.py:1229] GPU KV cache size: 130,960 tokens(EngineCore_DP0 pid=289) INFO 11-28 11:58:41 [kv_cache_utils.py:1234] Maximum concurrency for 98,304 tokens per request: 2.46x
The 2.46x multiplier is calculated in vllm/v1/core/kv_cache_utils.py. GPT-OSS uses an alternating attention pattern: odd layers use sliding window attention (sliding_window=128), even layers use full attention. This creates 2 KV cache groups.
Derivation (block_size=16, page_size=32,768 bytes for GQA with 8 KV heads × 64 head_dim × 2 bytes × 2 K+V):
max_num_batched_tokens (8,192) for chunked prefill bufferWhen KV cache fills: vLLM preempts lower-priority requests via recomputation (V1 default) or swap to CPU. This causes latency spikes for preempted requests.
(EngineCore_DP0 pid=289) 2025-11-28 11:58:41,951 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...(EngineCore_DP0 pid=289) 2025-11-28 11:58:41,973 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends(EngineCore_DP0 pid=289) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 83/83
83 CUDA graphs captured (one per batch size in cudagraph_capture_sizes up to 1024). PIECEWISE mode splits the computation graph at attention operations—attention stays in eager mode while everything else (MLPs, norms) goes into CUDA graphs. This is necessary because attention has variable memory access patterns that are difficult to capture in static graphs.
Why CUDA graphs matter: each kernel launch has ~20μs of CPU overhead for setup and dispatch. For small batches where GPU compute time is short, this overhead dominates total latency. CUDA graphs pre-record the entire workflow once, then replay it with near-constant launch time (~2.5μs + ~1ns per node). See NVIDIA CUDA Programming Guide.
(EngineCore_DP0 pid=289) Capturing CUDA graphs (decode, FULL): 3/3(EngineCore_DP0 pid=289) INFO 11-28 11:58:51 [gpu_model_runner.py:4244] Graph capturing finished in 9 secs, took 0.64 GiB
3 full graphs for decode-only batches (simpler access pattern allows capturing entire forward pass). Uses 0.64 GiB memory.
(EngineCore_DP0 pid=289) INFO 11-28 11:58:51 [core.py:250] init engine (profile, create kv cache, warmup model) took 78.26 seconds
(APIServer pid=90) INFO 11-28 11:58:53 [api_server.py:1725] Supported tasks: ['generate']
(APIServer pid=90) WARNING 11-28 11:58:53 [serving_responses.py:175] For gpt-oss, we ignore --enable-auto-tool-choice and always enable tool use.
(APIServer pid=90) INFO 11-28 11:58:53 [api_server.py:2052] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:38] Available routes are:
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=90) INFO 11-28 11:58:53 [launcher.py:46] Route: /metrics, Methods: GET
vLLM exposes an OpenAI-compatible API:
/v1/chat/completions - Chat completions (conversation format with messages array)/v1/completions - Text completions (raw prompt format)/v1/models - List available models and metadata/tokenize, /detokenize - Encode text to token IDs and decode back/metrics - Prometheus metrics for monitoring (vllm:e2e_request_latency_seconds, vllm:num_requests_running, vllm:kv_cache_usage_perc)/docs - Swagger/OpenAPI interactive documentation
(APIServer pid=90) INFO: Started server process [90](APIServer pid=90) INFO: Waiting for application startup.(APIServer pid=90) INFO: Application startup complete.(APIServer pid=90) INFO: 127.0.0.1:55440 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=90) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=90) INFO: 127.0.0.1:55484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=90) INFO 11-28 11:59:14 [loggers.py:236] Engine 000: Avg generation throughput: 544.8 tokens/s, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.6%
(APIServer pid=90) INFO 11-28 11:59:44 [loggers.py:236] Engine 000: Avg generation throughput: 514.2 tokens/s, GPU KV cache usage: 10.4%, Prefix cache hit rate: 81.6%
(APIServer pid=90) INFO 11-28 12:00:44 [loggers.py:236] Engine 000: Avg generation throughput: 468.0 tokens/s, GPU KV cache usage: 21.5%, Prefix cache hit rate: 81.6%
(APIServer pid=90) INFO 11-28 12:01:44 [loggers.py:236] Engine 000: Avg generation throughput: 426.6 tokens/s, GPU KV cache usage: 31.6%, Prefix cache hit rate: 81.6%
(APIServer pid=90) INFO 11-28 12:02:44 [loggers.py:236] Engine 000: Avg generation throughput: 396.0 tokens/s, GPU KV cache usage: 41.0%, Prefix cache hit rate: 81.6%
(APIServer pid=90) INFO 11-28 12:03:54 [loggers.py:236] Engine 000: Avg generation throughput: 367.2 tokens/s, GPU KV cache usage: 51.1%, Prefix cache hit rate: 81.6%
(APIServer pid=90) INFO 11-28 12:04:54 [loggers.py:236] Engine 000: Avg generation throughput: 348.6 tokens/s, GPU KV cache usage: 59.3%, Prefix cache hit rate: 81.6%
Throughput drops as KV cache fills:
This demonstrates that LLM decoding is memory-bandwidth bound. During each decode step, the GPU must load the entire KV cache from HBM (high-bandwidth memory) to compute attention over all previous tokens. As sequences grow longer, the amount of data transferred per token increases linearly, but the compute per token stays constant (one output token). According to NVIDIA’s LLM optimization guide, “the speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens.”
With 6 concurrent requests generating long outputs, each request’s growing KV cache competes for memory bandwidth. The vLLM architecture blog notes that “decode requests are memory-bandwidth-bound since we still need to load all LLM weights (and KV caches) just to compute one token.”
]]>Training guides
GRPO and variants - I write about my views on GRPO. It is only recently that I found out about this meme which I agree from the left side.
Technical reports - I will read this in the future to look back what models were optimizing for.
Efficiency efforts
Prompting - Now models are trained to effectively run long chains of thoughts without prompting.
Early alignment efforts - I kept this to read in the future to understand what people were thinking.
For an introduction to recommendation systems, I recommend
Value modeling - Recommendation systems calculate P(action) for multiple actions, for each candidate. The candidates are ranked based on a utility function. The utility function takes the action probability as arguments. You need to design a good utility function for your recommendation system. The design also involves deciding how important is each action.
Training multi-task models - We usually use one neural network model to predict multiple action probabilities. The alternative is to use a separate model to predict each action probability. However, sometimes individual models are better at predicting action probabilities than the combined model, even controlling for total parameter count. Hence there is this line of research to bridge the performance gap.
Calibration - When you ship a ranking model (the model that predicts P(action) for multiple actions), you also ship how miscalibrated it is. I think calibration is a very easily misunderstood topic. The concept of calibration should have been taught and tested in schools.
Feature engineering - Manually engineering features does not scale well. Whenever you add a new feature you will need to implement all the feature crosses. It would be great if this process is learnt by the model instead.
Sequence feature modeling - Your sparse features could be a sequence of IDs. You might believe that you can make better predictions on action probabilities by learning from this sequence.
Trainable embeddings - If your model uses sparse features (item IDs, action type is an example of a sparse feature, float values like age is an example of a dense feature), you will need to map each ID to an embedding and train the embeddings. The problem happens when you have too many IDs to train on. It is not a good idea to just use a larger GPU.
Pretrained embeddings - In your neural network model, you can also use embeddings that you do not intend to train. One such embedding is content embeddings, and you can introduce content embeddings to the neural network model, thinking that the model can better predict the action probabilities by knowing more about the content. You still need to prove that these embeddings are useful, and even if you fail to do so, you should be prepared to learn something.
Two tower model - Recommendation systems involve first retrieving thousands of candidates from millions of indexed content. Currently, you index with the item embedding, you retrieve with the user embedding, for items with the largest dot product. The two tower model produces the item and user embedding. You need to train the model.
User interest exploration - A good recommendation system does not just recommend content similar to ones that you have liked. The recommendation system should also appropriately explore what other types of content that you might like.
Item exploration - New content is essential to any recommendation system. To help fresh content succeed, you implement methods to surface it more effectively. However, you still need to prove these methods actually work. This is where a challenge arises: traditional user-split A/B testing tends to show lower engagement for variants that prioritize new content—simply because new content hasn’t yet accumulated the signals that make established content perform well. Hence there is this line of research on how do you both deliver new content effectively and demonstrate that your approach is beneficial.
Recommendation as sequence prediction - Instead of predicting the P(action), there is a line of research where you predict the item directly, similar to how you predict words in a sentence. I think this line of work only starts contributing value when you have systems that are bilingual in semantic IDs and English. Eugene Yan has an open source implementation.
Miscellaneous - These are some papers that I do not manage to classify.
Reinforcement learning - I wrote about reinforcement learning here.
Image models
Attention mechanism - I drew the attention mechanism here.
This just means that I previously printed the papers, and I did not discard the papers as I moved my residence within the Bay Area. There are very impactful papers in the field that I did not print. There are also papers in the list which I had not really read. ↩