Long conversations: compaction

Every turn sends the model the whole path from the first message to the one you are answering. When that path no longer fits the model's context window, the server summarises the older turns, keeps the recent ones word for word, and sends the summary in their place. Nothing is deleted: the original messages stay in the conversation and one click shows them.

grt-nano is a 1024-token model. Without compaction a thread on it is cut off after two or three turns; with it, the thread keeps going.

What you see

• The context meter, under the composer, right-aligned below the Send button: used / window in tokens, amber at 75%, red at 90%. It counts the last turn as the model reported it plus a rough estimate of your draft (characters / 4). A model with no known window shows no meter and never compacts.
• The divider, "Earlier messages were summarised", where the summary sits in the thread. The summary text is collapsed under it. Show original messages renders the covered turns above the divider from the tree the app already holds; nothing is fetched.
• Compact now, in the conversation's actions menu (the … button beside the model picker): folds the older turns immediately, without sending anything. Offered whenever the conversation exists and no answer is streaming. When there is nothing to fold yet (no more than MW_COMPACTION_KEEP_PAIRS user/assistant pairs after the last summary) the server answers 409 and the thread shows "Nothing to compact yet - the last 2 exchanges are always kept"; the number is the server's (keep_pairs on GET .../context), never the client's.
• Slash commands in the composer. A line that is exactly one of these runs it on Enter instead of being sent; none of them reaches the model or is stored as a message. Typing / opens the list (arrow keys or a tap to choose, Tab or Enter to complete, Esc to close).
  • /compact: the same as Compact now. On success the divider appears as after an automatic fold; otherwise the same notices (nothing to fold yet, budget used up, the model could not summarise).
  • /context: the meter's numbers as a line in the thread, "Context: 512 of 1,024 tokens used (50%), 512 left before the next turn", or that the model states no window.
  • /help: lists the commands. An unknown /word is not sent either; the thread says so.
• Continue, on an answer the model stopped because it ran out of room (finish_reason: "length"). The answer is marked truncated; Continue sends "continue" as a new turn.

How the server decides

Before a send, the server estimates the prompt: the previous assistant turn's prompt_tokens + completion_tokens as the model reported them, plus len(text) / 4 for the new user turn. If

estimate + reserve_for_answer > context_window

it compacts first. reserve_for_answer is a share of the window (a quarter by default), so an answer always has room to start. When the kept pairs, a summary at its cap and the new turn would still not leave that room, the fold keeps fewer pairs, never fewer than one. An edit or a regenerate never compacts: when its branch is over the limit it sends a notice and runs uncompacted, and the next send on that branch folds it.

Compaction is one non-streaming completion on the same model with the org's own key. It keeps the last MW_COMPACTION_KEEP_PAIRS user/assistant pairs verbatim (never fewer than one) and summarises everything before them. The summary is capped at MW_COMPACTION_SUMMARY_SHARE of the window. When the turns to fold do not fit the model's own window, the fold runs in slices, each summarising the previous summary plus the next turns.

If the model cannot produce a usable summary (the completion is empty or ends in length), the conversation stays as it was, the turn runs anyway, and the stream carries a notice frame. A failed compaction never loses a message.

Settings

Variable	Default	Meaning
MW_MODEL_CONTEXT_WINDOWS	grt-nano=1024	Fallback windows, model=tokens,..., used only when the gateway reports none for a model. Setting it replaces the default table
MW_COMPACTION_ENABLED	true	false turns automatic and manual compaction off
MW_COMPACTION_RESERVE_SHARE	0.25	Share of the window kept free for the answer
MW_COMPACTION_SUMMARY_SHARE	0.25	Largest share of the window the summary may take
MW_COMPACTION_KEEP_PAIRS	2	Recent user/assistant pairs kept verbatim

The window comes from the gateway first (/api/models reports it as context_window); the table is the fallback.

The summary in the tree

The summary is a message: role: "system", with one part of type summary.

{"type": "summary", "text": "...", "covers_from": "<message id>", "covers_to": "<message id>",
 "tokens_before": 912, "tokens_after": 184}

Its parent is the last message it covers; the next user turn is its child. The covered messages remain its ancestors, so GET /api/conversations/{id}/tree still returns all of them. When the prompt is built, the latest summary on the path replaces its ancestors and its text goes inside the single system message, never as a second one.

Branches follow from the tree:

• Regenerating or editing a turn after the summary keeps the summary on the new branch.
• Editing or regenerating a message the summary covers starts a branch that does not pass through it; that branch is uncompacted and compacts on its own when it needs to.
• Switching branches moves the divider to whichever summary is on the new path, or removes it.
• Forking copies the summary with covers_from / covers_to rewritten to the copied ids.
• Switching to a larger model does not undo a summary; switching to a smaller one may compact again.

The stream

Three frames on the conversation SSE stream (compaction on POST .../messages only; context and notice also on .../edit and .../regenerate):

Event	When	Data
compaction	a compaction ran on this send, before the message frame for the user turn	{"message": <summary message>, "covers_from", "covers_to", "tokens_before", "tokens_after"}
context	beside every done	{"prompt_tokens", "completion_tokens", "context_window"}
notice	compaction was needed but failed, or an edit / regenerate is over the limit; the turn still runs	{"detail", "code": "compaction"}

done keeps its shape. Clients that do not know these events ignore them.

Routes

• GET /api/conversations/{id}/context: the current path's last reported usage and the model's window, the numbers the meter starts from when a conversation opens.
• POST /api/conversations/{id}/compact: runs the same fold without a user turn and returns the summary message. 409 when there is nothing to fold (no more than MW_COMPACTION_KEEP_PAIRS pairs after the latest summary) or the model states no window. Same budget gate as a turn.

Billing

A compaction completion is charged like any turn, at the model's multiplier, and its usage row is tagged kind: "compaction". Settings → Usage shows it as its own line. An org whose budget is exhausted gets the budget error frame and no summary, and so does a member at their own spend cap (code: member_budget_exhausted). Webhooks receive conversation.compacted with ids and counts, never the summary text.

The OpenAI-compatible /v1 routes are stateless and never compact.