A full LLM workspace inside SwarmUI: persistent chats, custom assistants, agentic tool calling, vision, side-by-side model comparison, a floating in-page companion, and deep Generate-tab prompt integration.
It ships its own LLM runtimes, so there is nothing else to install: a pure-C# local GGUF engine and any OpenAI-compatible endpoint (OpenAI, Anthropic Claude, Ollama, LM Studio, vLLM, OpenRouter, …). Pick one under Server > Backends and the tab lights up.
| Version | 2.0.0-alpha.2 |
| Requires | SwarmUI (tested against 0.9.8.2) |
| Local engine | HartsyInference, the latest published version (NuGet, pulled automatically on each build) |
| License | MIT |
- Highlights
- Requirements
- Installation
- Adding an LLM backend
- Quick start
- The interface
- Assistants
- Tool calling
- Built-in tools
- Vision
- Assets
- Companion overlay
- Per-user memory
- Generate-tab integration
- Settings reference
- Permissions
- Multi-user model
- API reference
- Architecture
- Troubleshooting
- Known limitations
- License & credits
- Real chats, stored on the server. Threads are grouped by date, searchable, renameable, exportable (JSON / Markdown / plain text), and auto-titled from the first exchange. Your message is saved before the model is called, so a disconnect never loses it.
- Message branching. Edit a message or regenerate a reply and the old version survives as a switchable sibling branch, so no edit is destructive.
- Side-by-side comparison. Run the same prompt through two models at once, in parallel, then keep the winner.
- Agentic tool calling. 13 built-in tools, up to 8 tool rounds per turn, with per-tool permissions, sandboxing, SSRF protection, rate limits, and an optional audit log. Every provider runs tools through the same loop: the local engine reads the tool-call format from the model's own chat template, and OpenAI-compatible endpoints (Claude included) use their native tool-calling API.
- Custom assistants. Per-mode system prompts, inheritance, per-model prompt variants, parameter overrides, avatars, and a per-assistant tool allowlist.
- Rich rendering. GitHub-flavored Markdown, syntax highlighting, tables, KaTeX math, and Mermaid diagrams, all sanitized through DOMPurify and all served locally. No CDN, no outbound requests.
- Floating companion. An opt-in in-page helper that can critique your last render, suggest a preset, or explain a SwarmUI feature from the bundled docs.
- Prompt integration.
<llmprompt:…>tags in the Generate tab are expanded by the LLM at generation time.
- A working SwarmUI install.
- At least one LLM backend added under
Server > Backends(see below). - For the local engine:
.ggufweights inModels/llm/and, ideally, a GPU: an NVIDIA RTX 30-series or newer through CUDA, or any other GPU through Vulkan (CPU works, slowly). - For OpenAI or Anthropic Claude: each user sets their own API key in the
Usertab. No key is stored server-wide.
-
Clone into SwarmUI's extension folder:
cd /path/to/SwarmUI/src/Extensions/ git clone https://github.com/HartsyAI/SwarmUI-LLMAssistant.git -
Rebuild SwarmUI: run
update-windows.batorupdate-linuxmac.shfrom the SwarmUI root. This restores theHartsyInferenceNuGet package the local engine needs. -
Restart SwarmUI. An LLM Assistant tab appears in the main tab strip.
-
(Local engine only) Drop GGUF weights into
Models/llm/. They show up in SwarmUI's model browser too, since the extension registersLLMas a first-class model type.
Go to Server > Backends. The two LLM backend types appear directly in the "Add new backend" row.
| Backend type | What it is |
|---|---|
| LLM: Local (HartsyInference GGUF) | Runs GGUF models on this machine with the built-in pure-C# engine. No llama.cpp, no Python, no external process. Reads from Models/llm/. |
| LLM: Remote (OpenAI-Compatible) | Any OpenAI-compatible server, such as OpenAI, Anthropic Claude, Ollama, LM Studio, vLLM or OpenRouter. Each user's own key overrides the configured Authorization header: anthropic_api for Anthropic's API (see Claude), openai_api otherwise. |
Each card groups its settings into sections. Basics starts open; Memory and Advanced open by themselves when a setting in them is changed from its default. Each section header has a Guide link, and each ? tooltip ends with a link to the part of this README that explains the setting. The tables below name each setting by its label on the card, with the name it is saved under (and shown in the logs) in brackets.
Dropdowns list only what works here: the local card's Device and GPU come from the engine's own device check, and Default Model comes from the server. A saved value that a dropdown no longer lists stays selected, marked (saved value), so saving the card never changes it behind your back. The device lists are built once per SwarmUI start, so restart SwarmUI after changing a GPU or driver.
The local card has three sections: Basics, Memory and Advanced. A backend you already have keeps every saved value; only a new backend starts on Device: Auto.
| Setting | Default | Notes |
|---|---|---|
Device (Device) |
auto |
Which hardware runs the model. Auto tries CUDA, then Vulkan, then the CPU, and takes the first one that passes the engine's test computation. CUDA is the fast path for NVIDIA RTX 30-series cards and newer. Vulkan runs AMD, Intel and older NVIDIA cards (RTX 20-series, GTX 16-series). CPU needs no GPU but is slow. Only the choices that work on this machine are listed. |
GPU (GPUDeviceId) |
0 |
Each GPU by number and name, for example GPU 0: NVIDIA GeForce RTX 3060 (12 GB). Under CUDA the number is the CUDA ordinal, which is fastest-first and can differ from nvidia-smi's order. Under Vulkan, 0 lets the engine pick the best GPU and any other number is the Vulkan device list's own. The list follows Device. To use more than one GPU, add one backend per GPU. |
VRAM Mode (VramMode) |
Auto |
How hard the engine works to fit models into GPU memory. Auto reads the card's size, then measures free memory, and is right for almost everyone. Performance keeps the model loaded and never frees it on its own. Balanced and Aggressive free more between replies, for sharing the GPU with image generation. Maximum also turns on Keep Weights Compressed and frees the model after every reply. |
| Setting | Default | Notes |
|---|---|---|
Placement (Placement) |
auto |
Where a GGUF model goes when it is too big for this GPU. CUDA only. Auto puts the whole model on this GPU when it fits, else splits its layers across every GPU, else (for a mixture-of-experts model) keeps the most-used experts on this GPU and runs the rest on the CPU. GPU only refuses a model that does not fit instead. Split across GPUs always splits. Offload experts to CPU always offloads, which runs a large mixture-of-experts model on a small card, more slowly. The model list shows where the engine put each loaded model. |
Keep Weights Compressed (LowVramQuant) |
false |
Keeps quantized GGUF weights compressed on the GPU instead of caching a wider copy: less VRAM, slower replies. CUDA, and Vulkan for MOSS-VL. VRAM Mode Maximum turns it on too. |
Weight Precision (WeightPrecision) |
native |
For a MOSS-VL checkpoint folder only: re-quantizes the weights a release leaves in BF16 as the model loads, so it fits a smaller card. Q8_0 suits the NF4 release and Q4_K the FP8 one (see MOSS-VL-Realtime below). A GGUF ignores it, and it applies on the next load: unload the model to change it. |
Unload After Each Reply (AlwaysFreeMemory) |
false |
Unloads the model after every reply, so the next reply loads it again. Unlike VRAM Mode Maximum, it changes nothing else. |
Free RAM Floor (GB) (EvictBelowGb) |
0 |
Before each reply, unloads the loaded model first when free system RAM is below this many GB. 0 turns it off. Set it too high and every reply reloads the model. |
Reuse Chat Memory (ReuseConversationPrefix) |
true |
Keeps each conversation's processed prompt between turns, so long chats start answering sooner (see below). Not used while Unload After Each Reply is on, or with VRAM Mode Aggressive or Maximum. |
Conversation prefix reuse (ReuseConversationPrefix). Each conversation (a chat thread, a voice session) keeps its KV cache on the device between turns, so from turn 2 on the engine only processes the tokens added since, and time to first token stops growing with the history. Replies are equivalent: byte-identical on CPU; on a GPU the reused cache can round slightly differently. The engine bounds what this holds, with settings in ~/.config/hartsyinference/settings.json:
- Shrink to fit. After each turn a conversation's cache shrinks to its length plus
vram.prefixCacheHeadroomTokens(default 256), and grows again by an on-device copy when the next turn needs room. A 1,000-token Qwen3-4B chat holds about 0.34 GiB between turns. - Per-conversation cap. A conversation that would need more than
vram.prefixCacheMaxBytes(default 1.5 GiB: about 5,200 tokens of conversation for Qwen3-4B, 24,300 for Llama-3.2-1B) is not kept at all, so a very long thread runs uncached, as every turn did before this setting existed. - LRU. Each GPU keeps at most
vram.prefixCacheMaxEntries(default 4) conversations within that same byte budget, least recently used first.
Free Memory (including Swarm's idle VRAM clear), switching models or editing the backend releases them all.
| Setting | Default | Notes |
|---|---|---|
Repetition Penalty (RepetitionPenalty) |
1.1 |
Makes tokens the model already used less likely, so small models don't loop. 1.0 turns it off. It applies at temperature 0 too, where it keeps a long reasoning trace from looping. The chat has no override for it, so this is the only place to set it. |
Top K (TopK) |
40 |
Each step, keeps only the K most likely tokens. 0 turns it off. Values from 1 to 64 let Graph Decode run normal chats. |
Min P (MinP) |
0 |
Drops tokens below this fraction of the top token's probability. 0 turns it off; 0.05 is a common choice. |
Graph Decode (GraphDecode) |
false |
Records each decode step once as a CUDA graph and replays it, for faster replies on common dense models (Llama, Qwen, Mistral, Gemma). Works for greedy requests and, while Top K is 1 to 64, for normal chats. CUDA only: Vulkan turns graph decode on by itself for longer greedy replies, so this setting is not sent there. When it is on and a capture fails, the request fails instead of decoding the normal way, so turn it off if replies fail with a graph error. |
Speculative Decode (SpeculativeDecode) |
false |
Prompt-lookup speculative decoding, with no draft model. Only for greedy (temperature 0) requests without JSON mode, and only when Graph Decode does not take the request. Biggest win on output that repeats earlier text. |
Default Loops (RecurrentLoops) |
0 |
For a looped model such as NVIDIA Nemotron Dory: how many passes to run when the chat sets no Loops. Model default keeps the checkpoint's own count; 1 is fastest and 4 most accurate. Every other model ignores it. See looped models. |
| Video Frames Per Second (VideoFps) | 1 | Frames taken each second from an attached video (MOSS-VL). 1 is the model's own default; lower is faster and uses less memory. |
| Video Max Frames (VideoMaxFrames) | 256 | The most frames taken from one attached video. Each frame costs vision memory and time, and replies slow down past about 32 frames held. |
MOSS-VL-Realtime (images and video) . OpenMOSS's 11B vision-language model (engine alpha.420 and later) answers questions about attached images and videos. Put either form in the LLM model folder: the community Q4_K GGUF (Moss-VL-Realtime-q4k.gguf, about 8.4 GB, with tokenizer.json beside it; the vision tower is inside the GGUF, so no mmproj is needed), or an official checkpoint folder (BF16, NF4 or FP8; a folder is listed by its path under the model folder). Both appear in the model list with image and video attachments enabled. A server admin can fetch a variant (gguf-q4k, nf4, fp8, bf16) with LLMAssistantDownloadModel; each folder variant lands in its own moss-vl-realtime-<variant> folder. On a 12 GB card the GGUF runs as is; the NF4 folder with Weight Precision q8_0 decodes at about 28.8 tokens per second and the FP8 folder with q4_k at about 9.7 (Vulkan, RTX 3060). The model answers about the last user message and its media only: it ignores the system prompt and earlier turns, takes no tools, and a message with no image or video gets the engine's refusal. Video needs ffmpeg and ffprobe on the server's PATH, or in the folder the engine setting paths.ffmpegDir names. CUDA has not been run for this model.
Looped models (Dory) . NVIDIA Nemotron Dory 1.0 (engine family dory, HartsyInference alpha.378 and later) re-runs a shared block of layers a chosen number of times per request, trading latency for accuracy without swapping checkpoints. The count is the Loops field: per chat in the parameters popover, per user in the settings modal's Generation section (saved as the loops parameter, 0 = not set), and the backend's Default Loops (RecurrentLoops) when neither is set; a request's own value wins. The chat and voice routes take it as loops (1 to 4); the engine rejects a count above the model's maximum. Calls the extension makes on its own (chat titles, <llm:…> prompt enhancement, the caption tools) use the backend default. The engine keeps a conversation's cached prefix only for requests with the same loop count, so changing it mid-thread re-processes the prompt once. Every other model ignores the field.
Basics holds Address, Authorization Header and Default Model; the rest is under Advanced.
| Setting | Default | Notes |
|---|---|---|
Address (Address) |
(empty) | The server's address, for example http://localhost:11434 (Ollama), http://localhost:1234 (LM Studio), https://api.openai.com, or https://api.anthropic.com for Claude. The backend stays disabled until this is set. |
Authorization Header (AuthorizationHeader) |
(empty) | Only for a server that needs a key, for example Bearer sk-abc123. A user's own key from the User tab replaces it on that user's requests: anthropic_api for Anthropic's API, openai_api otherwise. With this empty, a server that answers "key needed" at startup still counts as running, since each user brings their own. |
Default Model (DefaultModel) |
(empty) | The model used when a request doesn't name one. Once the backend is saved with a working Address, the box suggests the server's models (from its /v1/models, asked with your own key when you have one), and you can still type any name. |
Allow Idle (AllowIdle) |
false |
When the server is offline at startup, the backend waits as idle instead of showing an error, and switches to running after its next successful reply. |
Connect Timeout (seconds) (ConnectionAttemptTimeoutSeconds) |
30 |
How long to wait for the server when connecting and listing models. Lower it (e.g. 5) for a server on your own network. |
Other Headers (OtherHeaders) |
(empty) | Extra headers, one Header: value per line. |
Token Limit Field (TokenLimitParameter) |
auto |
auto picks max_completion_tokens for api.openai.com and max_tokens everywhere else. Change it only if the server rejects the auto choice. |
Claude runs through LLM: Remote (OpenAI-Compatible), pointed at Anthropic's OpenAI SDK compatibility endpoint.
- Add an LLM: Remote (OpenAI-Compatible) backend with Address
https://api.anthropic.com. Leave Authorization Header empty, so each user's own key is used. - Each user sets their Anthropic key in the
Usertab. A key that is not tied to one workspace also needs the workspace: the user's own workspace ID set with the key, or for everyone ananthropic-workspace-id: wrkspc_...line in Other Headers. - Once a user has a key, Default Model and the chat's model picker list the models Anthropic's Models API offers that key, so new models show up without an update.
When the Address is Anthropic's, the backend reads each user's anthropic_api key instead of openai_api, lists models with x-api-key (the Models API does not take a Bearer token), and leaves out the sampling settings Claude rejects: no temperature or top_p for Claude 4.7 and later, and only temperature (capped at 1) for older models.
Anthropic describes the compatibility endpoint as meant for testing and comparing models rather than long-term production use. Through it, Claude's thinking is not returned, prompt caching is off, PDF and file input are refused, and every system message is merged into one system prompt.
Upgrading from the old "LLM: Anthropic Claude" backend. That backend type was removed. On the first start after updating, each saved one becomes a Remote (OpenAI-Compatible) backend with the same title, its Base URL as the Address, the same Default Model, and its Workspace ID as an Other Headers line. Its Reply Timeout has no counterpart and is dropped. The previous file is kept beside it as Data/Backends.fds.before-anthropic-migration.bak.
-
Open the LLM Assistant tab. If you have more than one assistant, you get the gallery; otherwise you drop straight into a chat.
-
Pick a model from the dropdown in the top bar.
-
Type.
Entersends,Shift+Entermakes a newline,/opens the tool picker, and the paperclip (or a drag-and-drop) attaches an image. -
Everything else (parameters, export, settings, the assistant panel) hangs off the top bar and the right-hand panel.
Left sidebar: chats grouped by date, with search, multi-select delete, and F2 to rename inline.
Top bar: thread title, model picker, compare toggle, model refresh, unload-model (frees VRAM), per-thread parameter overrides, export, and settings.
Right panel: the active assistant (with capability badges), live message/token counters, and the thread's Assets index.
Both split bars are draggable and remember their width; double-click resets them.
| Shortcut | Action |
|---|---|
Ctrl/Cmd + K |
Focus chat search |
Ctrl/Cmd + N |
New chat with the active assistant |
Ctrl/Cmd + Shift + F |
Find in the current chat |
F2 |
Rename the focused chat |
↑ / ↓ |
Move through the chat list (when a chat row has focus) |
Esc |
Close popovers, the asset viewer, or settings |
/ |
Tool picker (in the composer) |
Enter / Shift+Enter |
Send / newline (swap with the Enter to Send setting) |
Ctrl/Cmd + Enter |
Send, regardless of the Enter to Send setting |
Click the compare icon next to the model picker, choose a second model, and send once. Both lanes stream in parallel over one socket; each reply is persisted as a sibling of your message, tagged with the device it ran on. Keep this one promotes a lane to the thread's main path.
Editing one of your messages, or regenerating a reply, creates a new branch rather than overwriting. The old version stays reachable through the pager on the message. Under the hood the thread is a tree with an activeLeafId; the conversation you see is the root→leaf path.
An assistant bundles a persona, its system prompts, its parameters, and the tools it's allowed to use.
Open Settings > Assistants to create one from scratch, or clone one of the bundled starter templates (Anime Persona, Code Reviewer, Concise Translator, Story Writer, Vision Analyzer).
Each assistant has:
- Identity: name, category, description, color, and an uploaded avatar.
- Seven instruction modes:
chat,vision,caption,prompt,randomprompt,instructiongen,companion. Each mode is its own system prompt. - Per-model variants: any instruction can carry alternates keyed on the model's
Exactid,Family,Provider,Tag, or aGlobpattern. Specificity order:Exact > Family > Provider > Tag > Glob. - Inheritance (
extends): inherit instructions, parameters, and enabled tools from another assistant, then override selectively. Cycle-safe. - Parameter overrides: temperature / max tokens / top-p, blank to fall back to your global defaults.
- Tool allowlist: a master on/off plus a per-tool checklist, and per-assistant tool config that overrides your account defaults.
- A test runner: run a sample message against the unsaved instruction text before you commit it.
Prompt variables are substituted at request time: {{assistantName}}, {{userName}}, {{userProfile}}, {{currentDate}}.
The built-in Swarmie assistant is a SwarmUI-savvy helper that answers how-to questions by actually reading the bundled docs with the swarm_docs tool and citing them.
The model can call tools, get real results back, and keep going, up to 8 rounds per user turn before the loop is cut off with truncated: true. Every chat turn runs through one loop, and the model's tool calls go through the same permission, rate limit and audit checks whichever provider served the turn.
| Provider | How tools reach the model | How calls come back |
|---|---|---|
| OpenAI-compatible | Native tools, always sent when tools are on, along with saved tool turns as tool_calls and tool messages; the endpoint must support them (an endpoint that does not returns an error, and turning tools off for the chat avoids it) |
Incremental delta.tool_calls[], resolved at finish_reason == "tool_calls" |
| Hartsy-local (GGUF) | Native through the engine's HartsyInference.Tools package. The engine reads the model's chat template: a template with a tools slot renders the tools itself, and a template without one gets the Hermes tool prompt written into the conversation |
The engine parses the model's own call format (Hermes JSON, Qwen3.5 and Qwen3-Coder XML, GLM-4.5 arg pairs, DeepSeek-R1 blocks, Llama 3, Gemma 4, Mistral) and emits a native call event |
A tool result is fed back to the model for its next round, along with the reasoning of the round that made the call, in the same <think> form a stored reply uses (see Muse Spark below), so a template that reads reasoning fields sees it. Saved tool turns are replayed to the model on later turns, so a follow-up can refer to an earlier result. Malformed arguments are not swallowed: when an OpenAI-compatible endpoint (Claude's included) sends a call whose arguments are not valid JSON, the tool does not run and the model gets an error result back and can retry. The local engine cannot parse such a call at all, so it stays in the reply as text.
Muse Spark (engine family muse-spark) is parsed by the engine itself: its own output parser splits each reply into reasoning, answer text and <atem:invoke> tool calls, so its calls run through the same loop, and the reasoning shows in the chat's Reasoning block like a Qwen3 <think> block. It is left out of one-shot replies (voice turns, prompt enhancement, captions), since its template has no enable_thinking switch to turn it off. On later turns a stored reply's <think> blocks go back as reasoning_content instead of answer text, for a template that reads that field and does not split </think> out of the content itself (Muse Spark, Gemma 4); every other template, and a reply that is only reasoning, gets the reply as stored. The reasoning reaches the template from HartsyInference alpha.377 on.
All of these reach the same tool_call / tool_result events on LLMAssistantSendMessageWS. The voice routes use their own shape (native_tool_call / tool_result / done, see Streaming frames), and LLMAssistantVoiceTurnWS runs on any provider with the same loop.
Type / in the composer to get a searchable picker of every tool enabled for the current assistant. Pick one, fill in the form, and it runs directly, with no model in the loop. The call is persisted into the thread exactly like a model-driven one.
The wrench button in the composer turns tool calling on or off for this chat, overriding the assistant's default. With tools off, no tools are sent at all. Small local models get reliably confused by tool descriptions, so turn tools off for a model that answers badly with them.
Thirteen tools ship with the extension. Every one is gated by a permission (see Permissions) and, where relevant, sandboxed and rate-limited.
| Tool | What it does | Enabled by default | Permission |
|---|---|---|---|
generate_image |
Runs SwarmUI's T2I engine. Supports saved presets, raw T2I params, initImage for img2img/inpaint, and aspect shorthand. Returns an image URL. |
✅ | llm_tool_generate_image |
create_image_preset |
Saves a T2I preset to the calling user's account. Also needs SwarmUI's core Manage Presets permission. |
✅ | llm_tool_create_image_preset |
caption_image |
Vision pass over one image in a chosen caption style. | ✅ | llm_tool_vision |
fuse_image_descriptions |
Captions several images with role-specific prompts (style / subject / setting / reference) and merges them into one prompt. | ✅ | llm_tool_vision |
batch_caption_folder |
Captions every image in a folder and writes .txt sidecars, for LoRA dataset prep. Sandboxed to your Output dir. |
❌ | llm_tool_vision |
web_search |
DuckDuckGo HTML scrape. Returns {title, url, snippet}. |
✅ | llm_tool_web_search |
file_read |
Reads a text file inside SwarmUI's Data directory. 64 KB default cap. |
✅ | llm_tool_file_read |
file_write |
Writes a text file into Output/…/llm_assistant/. Extension allowlist (md, json, txt, yaml, yml, csv, log plus your own additions). |
✅ | llm_tool_file_write |
http_request |
GET/POST/PUT/DELETE/HEAD/PATCH. Blocks loopback, RFC1918, link-local, and cloud metadata addresses. 256 KB default response cap. | ✅ | llm_tool_http_request |
shell_exec |
Runs a shell command on the host. Sandboxed working dir, 30 s default timeout, 64 KB output cap. | ❌ | llm_tool_shell_exec |
memory_write |
Saves a fact to the calling user's private memory profile. | ✅ | llm_tool_memory |
memory_read |
Reads the calling user's memory profile. | ✅ | llm_tool_memory |
swarm_docs |
Lists and reads SwarmUI's bundled docs. Sandboxed strictly to docs/. |
✅ | llm_tool_swarm_docs |
shell_execis off by default and its permission defaults toNOBODY. Granting it is equivalent to giving anyone who can chat with the model a shell on your server. Only do it for admins you trust, on models you trust.
- Permissions are keyed on the handler, not the tool id, so defining a custom tool
{ id: "harmless", handlerId: "shell_exec" }can't bypass theshell_execgate. - 60-second execution timeout per tool call, plus each tool's own limits.
- Rate limits: a per-user, per-tool sliding hour window. Defaults:
http_request60/h,web_search30/h,shell_exec20/h,generate_image100/h,batch_caption_folder5/h. Override per tool in its editor;0means unlimited. - Audit log: an opt-in append-only JSONL trail of
shell_exec,file_write, andhttp_requestcalls plus every shared-layer write, atData/LLMAssistant/audit.log. Size-rotated at 10 MB × 5 generations. Toggle it withLLMAssistantSetAuditLogEnabled(admin only).
Settings > Tools > + Create Tool registers a new tool: id, name, description (this is what the model sees), a JSON Schema for the parameters, and a handler id that maps to a registered ToolHandler. There's a Run Test panel for development. Handler types today: builtin; mcp_stdio and mcp_http are reserved for planned MCP support.
Assistants with a vision instruction accept image attachments. Attach with the paperclip or drag-and-drop; the image is uploaded once, downscaled to the configured long-edge cap, and stored as a URL on the message, so the thread blob never carries base64.
A model that also takes video (MOSS-VL-Realtime, see MOSS-VL-Realtime) enables mp4, webm and mov attachments on the same paperclip, up to 64 MB. The video is stored under the thread's uploads like an image (so deleting the thread cleans it up), previews in the composer, and plays in the message. Other models keep the image-only paperclip, and a video dropped on one is refused with a message.
The same vision path backs the caption_image tool, fuse_image_descriptions, batch_caption_folder, the companion's "Critique my last image", and the Generate tab's magic-vision action.
The Vision Image Max Size setting (256–2048 px, default 1536) controls the downscale. Lower it to cut vision-token cost on paid APIs.
Anything worth keeping is promoted to the thread's Assets index in the right panel: fenced code blocks (8+ lines or 400+ bytes; smaller ones stay inline), HTML/SVG snippets, and results from generate_image, file_read, and file_write.
Click one to open the viewer:
From there: Copy, Download, Use as Prompt (text → the Generate tab's prompt box) or Use as Init (image → the Generate tab's init image).
Files written by file_write land in Output/<user>/llm_assistant/<path>, appear in chat as a clickable link, and load their contents lazily in the viewer. An orphan GC sweeps unreferenced uploads and avatars daily with a 24-hour grace period.
An opt-in floating helper that lives over the whole SwarmUI UI, not just the LLM tab.
Turn it on in Settings > Companion.
- Persona: a specific assistant, or "follow my active assistant".
- Position: snaps to a corner; drag for a free offset. Opacity is adjustable.
- Quick actions: ask anything, critique my last image, help with my prompt, suggest a preset, explain a feature, daily tip. Each is individually toggleable.
- Ambient chatter: optional unsolicited messages on session start, after a generation finishes, or when you've been idle. Every trigger has its own switch, plus Quiet Mode as a master mute, quiet hours, and a hard per-session cap.
Replies use the assistant's companion instruction, which asks for one short paragraph; the bubble is small on purpose.
A strictly private profile per user: preferred name, pronouns, bio, current work, preferences, dislikes, and free-form facts. Capped at 50 list entries, 100 facts, and 2000 characters per field.
Swarmie writes to it naturally as a conversation reveals things; nothing is ever asked just to fill a slot. Memory is injected into the system prompt via {{userProfile}}.
Memory is never visible to another user and there is no shared memory layer. Clear it any time with LLMAssistantClearUserProfile.
The extension registers a parameter group called LLM Prompt Processing in the Generate sidebar. Toggle the group on to use it.
| Parameter | Default | Purpose |
|---|---|---|
LLM Use Cache |
true |
Reuse the response for identical prompts within a batch. |
LLM Generate Wildcard Seed |
false |
Generate one shared wildcard seed per batch so <wildcard> picks stay consistent while the LLM is only called once. |
LLM Model ID |
default |
Override which LLM handles prompt processing. |
LLM Instructions |
prompt |
Which instruction set to use: any built-in mode or a custom instruction. |
LLM Assistant ID |
default |
Pin processing to a specific assistant's persona and variants. Default = your active assistant. |
| Tag | Meaning |
|---|---|
<llmprompt:a rough idea> |
At generation time the inner text goes to the LLM under the chosen instructions; the tag is replaced by the reply. |
<llmoriginal> |
Re-inject the original, pre-LLM text into the prompt. |
<llmresponse:…> |
Internal marker for a cached response. |
<mpprompt>, <mpresponse>, and <mporiginal> are accepted as aliases, so prompts written for Hartsy's MagicPrompt extension work unchanged.
Example:
<llmprompt:a cozy cabin in the snow, cinematic>, (highly detailed:1.2)
Settings > General:
| Setting | Default | Notes |
|---|---|---|
| Temperature | 1.0 |
0–2. |
| Max Tokens | 4096 |
Response length cap. |
| Top P | 0.9 |
Nucleus sampling cutoff. |
| Seed | -1 |
-1 = random. Only providers that honor a seed use it. |
| Loops | 0 |
Recurrent-block passes for a looped model such as Nemotron Dory, 1 to 4; 0 = not set (the local backend's Default Loops, RecurrentLoops, applies). Other models ignore it. |
| Context Messages | 0 |
Prior messages included per request; 0 = all. |
| Stream | true |
WebSocket streaming. |
| Vision Image Max Size | 1536 |
Long-edge cap in px, clamped to 256–2048. |
| Markdown Rendering | true |
|
| Enter to Send | true |
Off = Enter makes a newline. Ctrl/Cmd+Enter always sends either way. |
| Show Token Count | true |
Assistant-level parameter overrides beat these; per-thread overrides (the sliders icon in the top bar) beat both.
Per SwarmUI's extension standards, everything this extension can send over the network, and when:
| Connection | When | Off switch |
|---|---|---|
html.duckduckgo.com |
Only when the model (or you, via /) calls the web_search tool |
Disable the tool, or deny llm_tool_web_search |
| Any public URL | Only when the http_request tool is called (SSRF-guarded: loopback/private/link-local/metadata addresses are blocked) |
Disable the tool, or deny llm_tool_http_request |
Your configured Address |
When an OpenAI-compatible backend you added starts, serves a chat request or lists its models (for the model picker, and for the card's Default Model when Server > Backends opens) |
Don't add the backend |
Everything else is local: the HartsyInference engine runs in-process with no network access, and every front-end library (marked, highlight.js, KaTeX, Mermaid, DOMPurify) is bundled, so the UI itself makes zero outbound connections.
Sixteen permissions, all under the LLMAssistant group in Server > Users. Defaults are deliberately conservative.
| Permission | Default | Covers |
|---|---|---|
llm_chat |
POWERUSERS | Send messages, upload chat images, count tokens, run a tool from the picker |
llm_settings |
POWERUSERS | Read/write settings, assistants, tools, instructions |
llm_models |
POWERUSERS | List models, unload models |
llm_threads |
POWERUSERS | Threads, assets, session state |
llm_shared_write |
ADMINS | Write the shared/admin baseline; read the audit log |
llm_companion |
POWERUSERS | The floating overlay |
llm_tool_generate_image |
POWERUSERS (untested) | generate_image |
llm_tool_create_image_preset |
POWERUSERS (untested) | create_image_preset |
llm_tool_vision |
POWERUSERS | caption_image, fuse_image_descriptions, batch_caption_folder |
llm_tool_web_search |
POWERUSERS (untested) | web_search |
llm_tool_file_read |
POWERUSERS (risky) | file_read |
llm_tool_file_write |
POWERUSERS (risky) | file_write |
llm_tool_http_request |
POWERUSERS (risky) | http_request |
llm_tool_shell_exec |
NOBODY (powerful) | shell_exec |
llm_tool_memory |
POWERUSERS | memory_read, memory_write |
llm_tool_swarm_docs |
USER | swarm_docs |
SwarmUI also auto-registers view_extension_tab_llmassistant (default USER) for tab visibility.
Settings live in two layers:
- Shared: the admin-curated baseline: shared assistants, shared tools, default instructions and parameters. Writing to it requires
llm_shared_write. - Personal: each user's own overrides, personal assistants, personal tools, and preferred model.
Reads return the merged view. Assistants and tools are union-merged, personal winning on id collision, and every entry is tagged with a _scope badge (shared / personal) so the UI shows which layer an item lives in and which layer a delete would hit.
Threads, memory, uploads, and session state are always strictly per-user.
Every endpoint follows SwarmUI's standard API conventions: POST /API/<RouteName> with a JSON body that includes a session_id from GetNewSession. Routes marked WS take a WebSocket connection instead.
# Get a session
SID=$(curl -s -H "Content-Type: application/json" -d '{}' \
-X POST http://localhost:7801/API/GetNewSession | jq -r .session_id)
# What models are available?
curl -s -H "Content-Type: application/json" -d "{\"session_id\":\"$SID\"}" \
-X POST http://localhost:7801/API/LLMAssistantGetModels| Route | Method | Request | Returns |
|---|---|---|---|
LLMAssistantCreateThread |
POST | assistantId?, title? |
{success, thread} |
LLMAssistantSendMessage |
POST | message, instructionId?, model?, temperature?, maxTokens?, loops?, enableThinking?, noCache?, assistantId? |
{success, response, reasoning, finish_reason}, one-shot, does not touch threads. response is the answer alone, reasoning what the model thought first (empty if none), finish_reason stop or length (a budget that runs out mid-reasoning is a success with the partial reasoning and an empty response). A failure is {success: false, error, finish_reason: "error"}. enableThinking true/false; omitted keeps the model's template default (on for Dory and Qwen3) |
LLMAssistantSendMessageWS |
WS | threadId, message, userMessageId?, assistantMessageId?, model?, models?, media?, instructionId?, forceToolId?, temperature?, maxTokens?, seed?, loops?, enableThinking? |
streaming frames (below); enableThinking comes from the composer's Thinking toggle (Auto leaves it out) and is read by the edit, regenerate and compare paths too |
LLMAssistantEditMessageWS |
WS | threadId, messageId, content, userMessageId?, assistantMessageId?, … |
streaming frames; forks a new branch |
LLMAssistantRegenerateWS |
WS | threadId, messageId, assistantMessageId?, … |
streaming frames; new sibling reply |
LLMAssistantVoiceTurnWS |
WS | message? | messages?, assistantId?, model?, temperature?, maxTokens?, enableThinking?, loops?, conversationId? |
streaming frames, native shape (below), stateless like LLMAssistantVoiceTurn, runs on any provider (the same tool loop as chat) |
LLMAssistantUploadChatImage |
POST | threadId, messageId, imageData (data URI) |
{success, url, mediaType, width, height, bytesWritten} |
LLMAssistantUploadChatVideo |
POST | threadId, messageId, videoData (data URI; mp4, webm or mov, up to 64 MB) |
{success, url, mediaType, bytesWritten} |
LLMAssistantTestInstruction |
POST | instructionText, sampleInput, model?, assistantName? |
{success, response}, persists nothing |
LLMAssistantCountTokens |
POST | text or messages[] |
{success, count, exact, source} |
Pass a models array of {model, device?, backendId?, assistantMessageId?} with two or more entries to LLMAssistantSendMessageWS and it runs compare mode instead of a single reply.
Each WebSocket frame is one JSON object. In compare mode every frame also carries a lane index.
| Frame | Meaning |
|---|---|
{"chunk": "…"} |
A piece of the reply text. |
{"status": …} |
Backend progress (e.g. model loading). Forwarded verbatim. |
{"iteration": n} |
The agentic loop started round n. |
{"tool_call": {id, name, arguments}} |
A tool call was resolved and is about to run. |
{"tool_result": {id, name, result}} |
That tool finished. |
{"done": true, "full_text": "…", "stopReason": …} |
Generation finished. stopReason: "length" means the token cap cut it off. |
failure on done |
The reply stopped on an error after some text: the message is shown under that text, which stays. The reply still ends with this done frame. |
{"done": true, "truncated": true, "reason": "max_iterations"} |
The 8-round agentic cap was hit. |
{"titleUpdated": "…", "threadId": "…"} |
The chat was auto-titled from its first exchange. |
{"error": "…"} |
A failure: any text already streamed stays, and the message is shown under it. No done frame follows. |
{"lane": n, …} |
Compare mode: routes the frame to a column. |
LLMAssistantVoiceTurnWS uses a different tool shape on the same chunk/done/error frames above, with no iteration, tool_call, status or titleUpdated frames and one frame of its own, the round-limit notice. Its chunk frames carry only the visible text, so a reasoning trace is never spoken; done.full_text is the visible text too:
| Frame | Meaning |
|---|---|
{"native_tool_call": {id, name, arguments}} |
The model made a tool call and it was run. Sent immediately before that call's tool_result; a call that is not run (the round cap, or a round that ended in an error or a stop) is never sent. |
{"tool_result": {id, name, result}} |
That tool finished, same shape as above. |
{"notice": "tool round limit reached …"} |
The 8-round cap was hit while the model still wanted another call; that call was not run. Comes before done. |
{"done": true, "full_text": "…", "toolCalls": [...], "stopReason"?: …} |
Generation finished. toolCalls is the device-action subset, exactly like LLMAssistantVoiceTurn's response. stopReason: "tool_round_limit" means the 8-round cap was hit (see the notice above); this route never sends "tool_call". |
failure on done |
The reply stopped on an error after some spoken text: the message. The spoken text before it is kept, and the reply still ends with this done frame. |
{"error": "…"} |
An engine failure with no spoken answer and no tool result sent: the message, and no done frame follows it. |
Request fields unique to this route: messages? is the full conversation so far (oldest first), as
[{role: system|user|assistant|tool, content, toolCallId?, name?, toolCalls?}, …], a client resuming a
multi-turn voice session sends this instead of message (which still works alone, for a single new turn;
messages takes priority when both are present). The resolved system prompt is prepended as a leading system
turn unless messages already opens with one, so replaying history never double-injects it. An
assistant-role entry's own toolCalls (and a tool-role entry's toolCallId/name) use the same
{id, name, arguments} shape a native_tool_call frame sent earlier in the session, so a client can echo its
own history straight back. enableThinking? (bool) maps directly to the engine's enable_thinking chat-template
toggle (Qwen3-family reasoning-block switch), omit it to keep the model's own default; a voice caller sends
false, since thinking adds hundreds of tokens before the first spoken word. conversationId? (string) names
the conversation a turn belongs to, so a local model reuses its prefix from one turn to the next
(ReuseConversationPrefix); every turn is its own socket, so without it turns are grouped by the SwarmUI session
they were sent with.
| Route | Request | Returns |
|---|---|---|
LLMAssistantGetThreads |
none | {success, threads[]} (index only) |
LLMAssistantGetThread |
threadId |
{success, thread} |
LLMAssistantDeleteThread |
threadId |
{success} |
LLMAssistantRenameThread |
threadId, title |
{success, thread}, also settable with F2 in the sidebar |
LLMAssistantEditMessage |
threadId, messageId, content |
{success, thread} |
LLMAssistantDeleteMessage |
threadId, messageId |
{success, thread} |
LLMAssistantSetActiveLeaf |
threadId, messageId |
{success, thread}, switch branch |
LLMAssistantSetThreadToolsEnabled |
threadId, enabled? |
{success, thread}; omit enabled to clear the override |
LLMAssistantExportThread |
threadId, format (json | markdown) |
{success, filename, content} |
| Route | Request | Returns |
|---|---|---|
LLMAssistantGetAssistants |
none | {success, assistants[], activeAssistantId} |
LLMAssistantGetAssistant |
assistantId |
{success, assistant} |
LLMAssistantGetActiveAssistant |
none | {success, assistant} |
LLMAssistantSaveAssistant |
assistant (object), scope? |
{success, id, scope} |
LLMAssistantDeleteAssistant |
assistantId, scope? |
{success} |
LLMAssistantSetActiveAssistant |
assistantId |
{success} |
LLMAssistantUploadAssistantAvatar |
assistantId, imageData (data URI, ≤2 MB) |
{success, url, bytesWritten} |
LLMAssistantGetStarterTemplates |
none | {success, templates[]} |
| Route | Request | Returns |
|---|---|---|
LLMAssistantGetTools |
none | {success, tools[], canWriteShared} |
LLMAssistantGetTool |
toolId |
{success, tool} |
LLMAssistantSaveTool |
tool (object or JSON string), scope? |
{success, id, scope} |
LLMAssistantDeleteTool |
toolId, scope? |
{success} |
LLMAssistantExecuteTool |
toolId, arguments (object or JSON string), assistantId?, threadId?, callId? |
{success, result, callId} |
LLMAssistantGetToolConfig |
toolId |
{success, config} |
LLMAssistantSetToolConfig |
toolId, config (object or JSON string) |
{success, config} |
LLMAssistantGetImagePresets |
none | {success, presets[]} with a one-line summary per preset |
LLMAssistantExecuteTool still enforces the per-handler llm_tool_* permission, so exposing it at the chat permission level is not an escalation.
| Route | Request | Returns |
|---|---|---|
LLMAssistantGetSettings |
none | {success, settings, canWriteShared} |
LLMAssistantSaveSettings |
settings (object or JSON string), scope? |
{success, settings, scope} |
LLMAssistantResetSettings |
scope? |
{success, settings, scope} |
LLMAssistantGetAuditLog |
max? (default 200, cap 5000) |
{success, enabled, entries[]}, admin only |
LLMAssistantSetAuditLogEnabled |
enabled |
{success, enabled}, admin only |
LLMAssistantGetModels |
none | {success, models[], warnings[]}; providers are queried in parallel under a timeout; a slow backend degrades into a warning instead of hanging the call |
LLMAssistantUnloadModels |
none | {success, freed, providers}; freed counts providers that actually released something |
LLMAssistantDownloadModel |
WebSocket; model (moss-vl-realtime), variant? (gguf-q4k default, bf16, nf4, fp8) |
frames {progress:{file, fraction}}, then {success, path, model} (model is the id to chat with, as the model list shows it); admin only; closing the socket cancels it |
LLMAssistantGetSessionState |
none | {success, state} |
LLMAssistantSetSessionState |
state (patch object; null clears a key) |
{success, state} |
LLMAssistantGetAssets |
threadId |
{success, threadId, assets[]} |
LLMAssistantGetAsset |
threadId, assetId |
{success, asset} |
LLMAssistantDeleteAsset |
threadId, assetId |
{success} |
LLMAssistantGetUserProfile |
none | {success, profile} |
LLMAssistantClearUserProfile |
none | {success} |
LLMAssistantGetCompanionContext |
none | {success, lastImage} (null when you have no generations yet) |
LLMAssistantGetInstructions |
none | {success, instructions[]}, legacy, kept for the T2I prompt tags |
LLMAssistantSaveInstruction |
id, title, content, categories?, tooltip?, scope? |
{success, instruction} |
LLMAssistantDeleteInstruction |
id, scope? |
{success} |
LLMAssistantListGpus |
device? (auto, cuda, vulkan or cpu) |
{kind, values[], names[]}: the local backend card's GPU choices for that Device and the kind it runs on. Needs SwarmUI's View Backends List permission |
LLMAssistantListBackendModels |
backend_id |
{values[], names[], reason?}: Default Model choices for a remote LLM backend's card, from the server's /v1/models (Anthropic's Models API for Claude), asked with your own key. Empty, with the cause in reason, when there is nothing to list. Needs View Backends List |
import asyncio, json, urllib.request
import websockets # pip install websockets
BASE = "http://localhost:7801/API"
def call(route, payload):
req = urllib.request.Request(f"{BASE}/{route}", data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"})
return json.loads(urllib.request.urlopen(req).read())
sid = call("GetNewSession", {})["session_id"]
thread_id = call("LLMAssistantCreateThread", {"session_id": sid, "assistantId": "default"})["thread"]["id"]
async def chat(text):
async with websockets.connect("ws://localhost:7801/API/LLMAssistantSendMessageWS") as ws:
await ws.send(json.dumps({
"session_id": sid,
"threadId": thread_id,
"message": text,
"model": "claude-sonnet-5",
}))
async for raw in ws:
frame = json.loads(raw)
if "chunk" in frame:
print(frame["chunk"], end="", flush=True)
elif "tool_call" in frame:
print(f"\n[calling {frame['tool_call']['name']}]")
elif frame.get("done"):
break
asyncio.run(chat("How do I use wildcards in a prompt?"))SwarmUI-LLMAssistant/
├── LLMAssistantExtension.cs Entry point: assets, model type, tools, API, migrations, GC
├── Constants.cs Instruction ids, feature keys, tool ids, roles
├── LLMs/ The stable seam: no concrete backend referenced here
│ ├── ILLMProvider.cs The interface + LLMProviderRegistry
│ ├── LLMTypes.cs Extension-owned DTOs
│ ├── LLMDispatcher.cs Routes to the provider serving a model; token counting
│ ├── LLMModelLookup.cs Cached ListModels() with a 5-minute TTL
│ ├── LLMModelMatcher.cs Exact / Family / Provider / Tag / Glob / Default matching
│ ├── ExtendedLLMInput.cs Request shape (messages, params, media, tools)
│ ├── LLMStreamHelper.cs Chat entry point: builds the turn context for ToolTurnRunner
│ └── SwarmNativeLLMProvider.cs Optional bridge to native Swarm backends (off)
├── Backends/ REMOVABLE runtime pack
│ ├── LLMProviderBackend.cs AbstractLLMBackend + ILLMProvider base; self-registers
│ ├── HartsyLocalLLMProvider.cs Pure-C# GGUF via HartsyInference.Engine
│ ├── RemoteOpenAILLMProvider.cs Any OpenAI-compatible endpoint, Anthropic's included
│ ├── AnthropicBackendMigration.cs Turns saved backends of the removed Anthropic type into Remote ones
│ ├── LocalDeviceChoices.cs Device and GPU dropdowns for the local card, from the engine's device check
│ ├── BackendCardAPI.cs The two routes the backend cards fill their dropdowns from
│ └── LLMBackendPack.cs One Register() call
├── Services/ Assistants, instructions, threads, tools, media, memory,
│ settings layers, rate limits, audit log, GC, the one tool loop (ToolTurnRunner, ToolTurnSinks)
├── Tools/BuiltIn/ The 13 built-in tool handlers
├── T2I/ Generate-tab parameters and <llmprompt> processing
├── WebAPI/ 51 endpoints across 11 files + permission definitions
├── Tabs/Text2Image/ Tab markup (`LLM Assistant.html`; the filename is the tab label)
└── Assets/ Front-end JS/CSS (llm-backend-cards.js organizes the backend cards), starter templates, vendored libraries
One seam. The whole feature surface (chat, threads, tools, companion, T2I) talks only to ILLMProvider through LLMProviderRegistry. It never names a concrete backend and never references SwarmUI's AbstractLLMBackend. The runtimes live in Backends/, which is deliberately disposable.
Server-authoritative history. The client can't inject fake history. Your message is appended to the stored thread before the model is called; the reply is appended after the agentic loop finishes. Message edits and deletes go through their own endpoints rather than the client re-uploading the thread.
In-tab modal. The settings dialog is position: absolute; inset: 0 inside the tab pane, not fixed to the viewport; a Bootstrap modal would float over the whole SwarmUI UI and break the tab metaphor.
Zero external dependencies at runtime. marked, highlight.js, KaTeX (with fonts), Mermaid, and DOMPurify are vendored under Assets/vendor/ and registered by directory walk, then lazy-loaded on first use. Nothing is fetched from a CDN.
Reused SwarmUI styling. .basic-button, .splitter-bar, and the theme CSS variables (--text, --emphasis, --light-border, …). The tab inherits SwarmUI's theme, including custom ones.
To drop the bundled backends (say, once SwarmUI ships a first-class native LLM API you'd rather use):
- Delete the
Backends/folder. - Remove the
Backends.LLMBackendPack.Register();line fromLLMAssistantExtension.OnInit. - Delete the
HartsyInferencePackageReferenceand theExcludeAssetsblock from the csproj, and putCopyLocalLockFileAssembliesback to its default. - Register a replacement
ILLMProvider. A ready-made bridge to native Swarm backends ships asLLMs/SwarmNativeLLMProvider.cs(disabled by default); callSwarmNativeLLMProvider.Register()fromOnInitinstead of the pack.
Nothing in chat, threads, tools, companion, or the T2I integration changes.
The csproj resolves HartsyInference from NuGet by default. To build against a local engine clone instead:
dotnet build src/Extensions/SwarmUI-LLMAssistant/SwarmUI-LLMAssistant.csproj -p:UseLocalHartsy=trueBuild the engine first (HartsyInference.{LLM,Cpu,Cuda,Vulkan} at net8.0), and override HartsyRepo if your checkout isn't a sibling of the SwarmUI folder. Never commit a -local version pin: end users only have nuget.org.
"No LLM backend is running."
Add one under Server > Backends. The quickest path is LLM: Remote (OpenAI-Compatible) pointed at a local Ollama (http://localhost:11434).
The model list is empty, or shows a warning.
Each provider is queried in parallel under a bounded timeout, and anything that times out is reported in warnings rather than hanging the request. Check the backend's status in Server > Backends and its logs.
"No Anthropic API key set" / OpenAI 401.
Keys are per-user. Set yours in the User tab; the backend config does not carry one. If Claude's model list stays empty after you set a key, Anthropic refused that key: the SwarmUI log shows Failed to list models from https://api.anthropic.com: HTTP Unauthorized. Paste the key again, whole, and save.
Tool calls never fire.
Check, in order: the tool is enabled globally (Settings > Tools); it's checked on the current assistant; the assistant's tool master switch is on; the composer's wrench toggle is on for this chat; and you hold the matching llm_tool_* permission. Small local models often call tools unreliably, try a larger instruct-tuned model, or a provider with native tool calling. A model that answers in prose instead of calling a tool is a model limit, not a broken tool path.
<llmprompt> tags aren't processed.
Toggle the LLM Prompt Processing group on in the Generate sidebar, make sure an LLM model resolves, and check the server log for the [LLMAssistant] prefix.
The chat list is empty after sending.
Threads persist on the first message. If nothing appears, the send failed; check the log for ThreadStorageService errors.
The settings modal renders outside the tab.
The modal is scoped to the tab pane. If it escapes, .llma-container has lost position: relative or the tab pane has lost position: relative; height: 100%.
The companion never appears.
It's opt-in: enable it in Settings > Companion, and make sure you have llm_companion.
"I can't see images."
Either the selected model isn't vision-capable or the assistant has no vision instruction. Switch to a vision model (a Claude model, a vision-tagged Ollama model, a multimodal GGUF with its mmproj alongside, or MOSS-VL-Realtime).
"I can't attach a video", or a video reply says ffmpeg failed.
Only a video-capable model (MOSS-VL-Realtime) enables video; the paperclip's tooltip says what the selected model takes. A video is decoded on the server with ffmpeg and ffprobe: install them on the PATH, or set the engine setting paths.ffmpegDir to the folder that holds them. Videos over 64 MB are refused.
The local LLM stops with an engine error, or CUDA is missing from Device. The engine's CUDA kernels need an NVIDIA RTX 30-series card or newer. Set Device to Auto: on an older NVIDIA card (RTX 20-series, GTX 16-series), or an AMD or Intel GPU, Auto picks Vulkan. The server log says why CUDA was left out. After changing a GPU or driver, restart SwarmUI so the card lists it.
A dropdown on a backend card shows (saved value). The value saved for that backend is not in the list this machine offers, for example a GPU that is gone or CUDA on a card that fails the CUDA check. The backend still uses it. Pick another entry and Save to change it.
Default Model doesn't list any models.
The list appears once the backend is saved with an Address that answers; for Claude it also needs your own Anthropic key in the User tab. The list is asked again each time you open Server > Backends, and until then the field works as before. The SwarmUI log says why a list is empty; for Claude without a key, the browser console does.
- Compare mode doesn't auto-title the chat; single-model turns do.
- The companion bubble renders plain text, not Markdown, by design; its instruction asks the model for one short prose paragraph.
- Shared assistant avatars are served from the uploading user's output folder, so another user needs SwarmUI's
View Others Outputspermission to see them. Personal avatars are unaffected. - Speculative Decode only engages on greedy (temperature 0) requests, so the default chat temperature does not use it. Graph Decode covers normal chats on CUDA only while Top K is 1 to 64.
- The backend cards' device lists are built once per SwarmUI start. Restart after changing a GPU or driver.
- Tool calls on small local models are model-dependent. A small model may answer in prose or end its turn right after a tool-call marker. The engine's own real-weight matrix records which checkpoints call tools reliably on this machine.
ForceToolIdon the local engine states the forced tool in the system prompt and restricts the parser to it; it is not a grammar, so a model can still answer in prose. OpenAI-compatible requests carry the forced choice astool_choice, which the endpoint honours when it supports that field. Claude Opus 5.5, Sonnet 5.5 and Fable 5.1 do not support a forced tool and answer such a request with an error.- Qwen3.5 and the other CUDA-only checkpoints need a CUDA build; their dense weights have no CPU path yet.
- MCP tools (
mcp_stdio/mcp_http) are reserved handler types, not yet implemented.
MIT. See LICENSE.
- SwarmUI and mcmonkey: the platform this builds on.
- HartsyInference: the pure-C# local inference engine.
- marked, highlight.js, KaTeX, Mermaid, DOMPurify: markdown, code, math, diagrams, sanitization. All vendored locally; see
Assets/vendor/FETCH.md.