Bring your own models. Anthropic Claude, OpenAI GPT, DeepSeek, Gemini, Qwen, Grok, GLM, Codestral, Sonar Pro, and Knox's On-Demand model knox/knox-ms are supported out of the box.

Three systems. One loop.

Memory Brain​

Most assistants keep a handful of notes. Knox runs a local cognitive stack — a SQLite brain at ~/.knox/memory/brain.sqlite that spans every session, compresses what it keeps, forgets what it should, and injects only what the next turn needs.

Nothing leaves your machine. There is no cloud sync and no remote memory API. Optional AES-256-GCM export is for local backup only.

What it does for you​

On every chat turn, Knox builds a token-budgeted memory block and injects it as ## Relevant Memory Context. An Injected Memories chip above the input shows what was used and why — pin what matters, forget what does not. If memory is slow, chat continues with a notice instead of blocking.

Memories are extracted automatically after substantial turns (facts, decisions, patterns, error fixes). You can also attach them with @memory.

Five-tab Memory panel​

Open Knox: View Memory or the sidebar brain icon.

How memory thinks​

• 8-phase pipeline — sensory capture → encoding → working memory → consolidation → long-term store → retrieval → sleep → context assembly
• 5-level hierarchy (M₁–M₅) — sensory buffer (~250 ms), working memory (7 slots, 30K tokens, 30 s TTL), short-term, long-term, procedural. Compression theorem: C_effective = Σ |Mᵢ| / rᵢ
• Retrieval without embeddings — FTS5 BM25 + trigram fuzzy + graph spreading (depth 3) + recency + importance. Defaults: threshold θ = 0.6, top-k = 20
• Sleep consolidation — NREM replay, Ebbinghaus decay (λ = 0.03), REM distill. Manual Consolidate or every 24 hours. Pinned memories skip prune
• Knowledge graph — people, files, functions, concepts, and weighted edges. LRU at 5,000 entities
• Modes — full / summarized (default) / selective. Scope — this project (default) or all projects
• Sanitized writes — prompt-injection, credentials, and invisible Unicode are stripped; recalled context is fenced so memories are not treated as instructions

Agent memory tools​

Five first-class tools on the default Agent catalog: builtin_memory, builtin_memory_graph, builtin_memory_sessions, builtin_memory_manage, builtin_memory_learn. Explore/review subagents may recall memory but cannot write it.

/autonomous <goal> runs a local multi-step loop with the memory pipeline on every iteration (default 10 steps).

Agent Mode​

An autonomous agent that plans, executes, and verifies multi-step work — with permissions you control, isolation you can turn on, and a stop button that actually stops.

Switch with the Agent tab or Cmd/Ctrl + Shift + Alt + A. One switch: tools, checkpoints, undo, shadow preview, and verification stay in sync. Chat stays text-only; tools run only in Agent.

Permissions you can feel​

Cycle Ask → Edits → Auto from the Agent tab or Shift+Tab:

Every tool card: Deny / Always (this chat) / Approve. Path and command policy (allow / ask / deny globs) is enforced in Core — deny always wins, including in Auto. Defaults block rm -rf, ~/.ssh, and similar; paths outside the workspace ask.

Presets in Tools permissions: Ask on write and YOLO. AGENTS.md, CLAUDE.md, and .knoxrules merge in, with Knox-specific rules winning.

29 built-in tools​

The agent reads, edits, searches, and verifies with a catalog that is actually implemented — not a list of stubs.

Files & discovery — read, create, StrReplace edit, write, Codex-style apply_patch (multi-file, atomic), glob, directory tree, repo map, ripgrep, git diff

Shell & jobs — persistent cwd, streamed stdout, auto-background after ~30 s, await_shell to poll or kill. Process-group kill so pipelines stop

Git — status, diff, log, commit. No push, force, or amend. Prefer these over shell git

Orchestration — task subagents (explore / review / general), ask_user (never auto-approved), workspace_checkpoint

Intelligence — LSP (definition, references, hover, symbols, call hierarchy), live web search, on-demand skills, generate tests

Memory — the five Memory Brain tools above

Readonly tools run in parallel when the model emits several at once. Writes stay sequential. After mutating tools, Knox checks diagnostics and can attempt an LLM fix (knoxchat.enablePostEditVerification, on by default), with a per-file circuit breaker.

Isolation, not hope​

• Worktree (optional chip) — edits and shell run in a git worktree until you Apply (copy back) or Discard. Session-long isolation
• Shadow preview (knoxchat.enableShadowPreview, off by default) — side-by-side Accept/Reject before a single Apply. Not the same as Worktree
• Undo / redo — real file-byte snapshots. Cmd/Ctrl + Shift + Alt + Z / Y while Agent is active

Stay in control of the loop​

• Activity timeline — thinking → reads → searches → edits → tests/shell. Click a step to jump to the tool card. Checkpoint stamps on the timeline
• Turn meter — steps used / max (default 40), estimated tokens, elapsed time, Stop. Stop stays available while a tool is mid-flight
• Jobs panel — detached shell jobs and in-flight subagents. Inspect output, kill, clear finished
• Doom-loop guard — identical tool+args or a failure streak (default 3) forces a text-only summary
• Ask user — mid-run multiple-choice or short answers. Never auto-approved

Checkpoints​

Git is for commits you meant. Checkpoints are for everything the agent just did — a rewind that does not depend on git reset.

Snapshots live in ~/.knox/checkpoints/. Restore replays the delta chain so the workspace matches that point in time, including files created later.

Two restore modes​

Default is files-only. Memory rewind is opt-in: Shift-click the chat restore button, a timeline cp stamp, or the overlay. After any restore, a system note tells the model not to assume later edits still exist.

When snapshots happen​

• Before the first mutating tool of a turn — edits, patch, tests, git commit, or terminal. Always. Empty trees get a baseline
• After an AI reply when workspace files changed (default on)
• Manual — Command Palette Knox: Create Checkpoint, or the agent tool builtin_workspace_checkpoint
• Worktree Apply — a turn checkpoint before files copy back
• Memory bulk delete — one Memory Brain safety checkpoint (separate from workspace snapshots)

Overlay in chat​

The restore icon in the chat toolbar opens the Checkpoints overlay (list + configuration). Title-bar buttons are gone — you never leave the conversation.

• This session filter (on by default)
• Search by description or ID
• Date groups, file stats, 8-char IDs
• Diffs vs previous checkpoint, current workspace (preview of restore), or any older checkpoint
• Split / unified views, word-level highlights, single-file restore
• Binary assets (images, fonts, PDFs, wasm) when capture is on
• Multi-select bulk delete

Retention & ignore​

Defaults: 1,000 checkpoints, 7-day retention, 5 MB per file, binary capture on, auto-cleanup daily. Tracked extensions cover the usual languages; unknown files are sniffed. .knoxignore, .gitignore, and built-in noisy paths (node_modules/, .git/, dist/, …) are all applied. Knox Checkpoint: Create global .knoxignore ships language presets.

Soul — one session, three systems​

This is the product, not a sidebar feature.

One session.id binds the Agent loop, workspace checkpoints, and the Memory Brain.

• The first mutating tool of a turn always creates a workspace checkpoint (soul-turn-…)
• Mutating tools, deny, doom-loop, max-steps, ask-user, worktree apply/discard, and compaction write SoulEvents as episodic memory
• Compaction pins a brain snapshot linked to the last workspace checkpoint
• Restore can rewind files, or files and what the agent remembered
• Memory rollback offers the linked file checkpoint — disk does not move unless you ask

The model, the disk, and the memory stay honest with each other.

AI Chat​

A sidebar chat streamed in real time, with tool execution displayed inline when you are on the Agent tab.

• Multi-model — Anthropic Claude, OpenAI GPT-5.5 / GPT-5.6, DeepSeek V4, Gemini 3.7, Qwen 3.8, Grok 4.6, GLM 5.3 and knox/knox-ms
• Role-based routing — assign different models to chat, edit, apply, summarize, viewRead, realTimeSearch. Cheaper models for reads; your chat model for writes
• Sessions — multiple tabs, history overlay in the input toolbar, restore
• Rich context — files, folders, selections, images, terminal, git diffs, and @ providers
• Compaction — long threads summarize without dropping memory or plan blocks

Inline Code Editing​

Natural-language edits with visual diff review.

• Cmd/Ctrl + I — describe a change; Knox streams a vertical diff in the editor
• Accept / reject per block — Alt + Cmd/Ctrl + Y / N
• Accept / reject all — Shift + Cmd/Ctrl + Enter / Backspace
• Multi-file batch diffs — review and apply across files in a dedicated panel
• Smart apply — unified diff detection, LLM-assisted lazy apply, or AST fallback

Slash Commands​

Custom commands live in config.yaml or .prompt files.

Context Providers​

Attach context with @. Defaults always available: file, diff, problems, repo-map, terminal, memory.

Context Menu Actions​

Right-click selected code:

• Add as Context — send the selection to Knox
• Write Comments / Write Docstring
• Fix Code / Optimize Code
• Fix Grammar / Spelling — Markdown files

Explorer: Select Files as Context
Terminal: Debug Terminal (Cmd/Ctrl + Shift + R)

Where Knox stores data​

All user data is global in ~/.knox/ — no project-local .knox/ directory is created. Legacy project .knox/ folders migrate automatically on first launch.

Rules System​

Project standards injected into every AI interaction. Later sources win.

Merge order (highest last):

1. ~/.knoxrules and ~/.knox/rules/*.md
2. {workspace}/CLAUDE.md
3. {workspace}/AGENTS.md
4. {workspace}/.knox/AGENTS.md
5. {workspace}/.knoxrules — Knox-specific wins over generic agent files
6. Nested {subdir}/AGENTS.md walking up from the open file

---
applyTo: ["**/*.ts", "**/*.tsx"]
priority: 10
---
- Use `interface` for object shapes, `type` for unions
- Prefer `const` over `let`
- All exports must have JSDoc

Supports applyTo globs, priority, and template variables ({os}, {arch}, {home}). AGENTS.md can declare always / ask / never path and command policy.

Skills System​

Reusable instruction sets that extend Knox.

Discovery:

• ~/.knox/skills/ (global, Knox-native)
• skills/ or skill/ in the workspace
• .claude/skills/, .agents/skills/ (project)
• ~/.claude/skills/ (global)

Each skill is a folder with a SKILL.md:

---
name: react-components
description: React component development patterns
---
## Instructions
- Use functional components with hooks
- Extract shared logic into custom hooks
@src/components/Button.tsx

List loaded skills with /skills. The agent can load a skill body on demand via builtin_skill.

Prompt Files​

Reusable prompts as .prompt files in ~/.knox/prompts/ or project-level .prompts/ / .knox/prompts/. YAML prompts or .prompt files.

---
name: api-endpoint
description: Generate a new API endpoint
---
Create a RESTful endpoint following our patterns.
@src/routes/example.ts
@currentFile
<system>You are an expert backend developer.</system>

Supports @file.ts, @https://..., context providers (@currentFile, @repo-map), system-message blocks, and recursive nesting.

Configuration​

Knox is configured from config.yaml in ~/.knox/ and from the in-app Settings page. Memory preferences live in the Memory panel (not settings.json).

VS Code Settings​

Agent loop caps (experimental.agentMaxSteps default 40, experimental.agentDoomLoopThreshold default 3) and path/command policy are in the in-app Settings / Tools permissions UI.

Keyboard Shortcuts​

Language & File Support​

Knox works across the languages you already use:

JavaScript / TypeScript, Python, Java / Kotlin, C / C++, C#, Go, Rust, PHP, Ruby, Swift, HTML / CSS / SCSS, JSON / YAML, Markdown, SQL — plus custom .prompt files with syntax highlighting.

Internationalization​

• English (default)
• Chinese (中文)

Change language from Settings in the sidebar. Agent UI, permissions, jobs, and tool-error cards are localized.

Requirements​

• VS Code ≥ 1.125.0
• Node.js ≥ 24.19.0
• An API key for at least one supported LLM provider (Anthropic, OpenAI, or Knox)

Documentation & Support​

• Docs: https://docs.knox.chat
• Open VSX: https://open-vsx.org/extension/knoxchat/knoxchat
• GitHub: https://github.com/knoxchat/knoxchat
• Email: support@knox.chat
• Homepage: https://knox.chat

Available in the terminal(coming soon), browser(coming soon), and as a desktop app.

Download KnoxCode

Knox Memory System​

KnoxCode is powered by the Knox Memory System (knox/knox-ms), Knox's purpose-built model for long-running coding work. Unlike a single static LLM, Knox-MS is an orchestration layer that combines planning, task routing, and persistent memory so context can grow far beyond a normal context window.

KnoxCode uses knox/knox-ms exclusively — there is no separate model picker for third-party providers. All agent turns go through Knox-MS on the Knox API (https://api.knox.chat/v1).

How it works​

Knox-MS implements a Plan–Task–Memory architecture on the Knox backend:

1. Plan — Each request is analyzed for complexity. When planning is needed, Knox-MS breaks the goal into tasks with dependencies and picks an execution strategy.
2. Task — Tasks are routed to underlying models by difficulty (easy, medium, hard). Simple work uses faster models; harder reasoning uses more capable ones. Independent tasks can run in parallel.
3. Memory — Conversation history, summaries, extracted knowledge, and vector embeddings are stored per user and session. When building context for the next turn, Knox-MS retrieves the most relevant material instead of sending the full transcript.

For coding agent requests (when tools are present), Knox-MS switches to tool-calling passthrough: it forwards the request directly to a capable underlying model while still attaching session memory and billing through Knox-MS. This keeps OpenAI-compatible tool loops intact for file edits, shell commands, and search.

After each response, the backend runs async post-processing: automatic summarization when context grows large, knowledge-graph extraction, and learning-pattern recording — all without adding latency to your turn.

Memory layers​

Memory is organized in a five-level hierarchy, inspired by human memory:

Hot tiers live in Redis for fast access; older blocks are tiered to S3 and loaded on demand. Memory consolidation runs in the background — strengthening important memories, fading low-value ones, and archiving to long-term storage.

Retrieval combines semantic vector search, lexical search, and knowledge-graph traversal so the agent gets relevant context without stuffing the prompt.

What KnoxCode sends automatically​

KnoxCode wires your local session into Knox-MS on every model call:

• session_id — ties turns to Knox-MS memory so you can resume later
• project_id — scopes memory to the open project when available
• include_reasoning — surfaces reasoning when the model supports it

You do not configure these manually; KnoxCode passes them through its provider layer.

How agent coding works​

KnoxCode runs an agent loop: you send a message, the agent thinks, uses tools on your codebase, and replies. If more work is needed, it keeps going until the task is done or it needs your input.

Each conversation is a session with full history. You can steer the agent while it works, pick up where you left off later, and let long sessions compact automatically so context stays manageable. Knox-MS memory on the backend complements local session history for cross-session recall.

Built-in agents​

KnoxCode ships with specialized agents for different kinds of work:

The Build agent is the default. You can switch agents mid-session when you want exploration without edits, or delegate focused subtasks to another agent.

What the agent can do​

KnoxCode agents use real tools on your machine, not just chat:

• Read and edit source files, with diffs you can review
• Search the codebase by filename pattern or text
• Run shell commands — tests, builds, git, and more
• Browse the web to look up docs or APIs
• Ask you questions when a decision needs your call
• Track progress on multi-step tasks with a built-in todo list

Dangerous or sensitive actions — editing files, running commands, reading secrets — go through permissions. KnoxCode can allow, ask, or block each kind of action, globally or per agent.

In Na'vi, "tsaheylu" is the neural bond — the living link that lets two beings act as one. That is exactly what this does for KnoxChat and your AI coding tools.

Tsaheylu is the AI apps connector built into KnoxChat. From a single page it bonds KnoxChat to every AI coding assistant on your machine — managing their providers, skills, prompts, sessions, and MCP servers in one place, and routing all of them through KnoxChat with one click.

Supported apps​

Claude Desktop shares Claude Code's skills, prompts, and session history; the unified MCP and skills panels can distribute entries to every supported app regardless of which tab you are on.

What it does​

One-click Knox Connect​

Every app panel shows a live connection status — model, routing mode, and whether Knox is the provider currently in use. The Connect dialog generates the right configuration for each app (Anthropic format for Claude, OpenAI-compatible /v1 for Codex, OpenCode, OpenClaw, and Hermes) so your KnoxChat account powers all of them without hand-editing config files.

Provider management and hot switching​

Add providers from curated presets or custom endpoints, test endpoint latency before saving, and switch the active provider per app instantly. Switches that need format conversion or URL rewriting are routed through the built-in local proxy automatically, with a clear explanation of why.

Local routing and automatic failover​

A local proxy can take over an app's traffic to enable:

• Format conversion (for example OpenAI Chat Completions for providers that don't speak an app's native protocol).
• Claude Desktop local routing with per-role model mapping (Sonnet / Opus / Haiku routes).
• Priority-based failover queues (P1 → P2 → …) with a circuit breaker, error-rate thresholds, and automatic recovery — so a failing provider never stalls your session.

Stopping the proxy restores every app's original configuration from backup.

Skills, prompts, sessions, and MCP​

• Skills — install from the discovery catalog or a ZIP, sync across apps, back up and restore, and manage skill repositories.
• Prompts — a shared prompt library applied per app.
• Sessions — browse, search, resume, and batch-manage local session history for Claude Code, Codex, Gemini, and OpenCode, including opening a session directly in your terminal of choice.
• MCP — one unified panel to add, import, validate, and distribute MCP servers across all supported apps.

Safety and housekeeping​

• Environment-variable conflict detection warns when shell exports (such as ANTHROPIC_BASE_URL) would silently override an app's configuration, and offers to remove them.
• Configuration import/export plus optional WebDAV and S3 sync keep machines aligned.
• Deep-link imports let you share providers, skills, prompts, and MCP servers as links.
• Full English and Chinese localization, synced with KnoxChat's language setting.

Requirements​

Tsaheylu manages tools that live on your machine — local config files, SQLite databases, session folders, and the local proxy — so it requires the KnoxChat Desktop app. When the page is opened in a browser it shows a download notice instead; nothing breaks in the web build.

Getting started​

1. Install and open KnoxChat Desktop.
2. Open Tsaheylu from the sidebar.
3. Pick an app (for example Claude Code) and press Connect to bond it to KnoxChat.
4. Optionally enable local routing or failover from the app's topbar, and manage skills, prompts, sessions, and MCP servers from the tabs.

Production verification​

This release was verified end to end before publishing:

Why KnoxChat Desktop?​

KnoxChat Desktop combines enterprise-grade AI capabilities with a native desktop experience, giving you faster access, better privacy, and seamless coding workflows—all in one app.

One Platform, 100+ AI Models​

Access the world's leading AI models through a single unified interface:

• OpenAI — GPT-5.5, GPT-5.4, and more
• Anthropic — Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5
• Google — Gemini 3.1 Pro, Gemini Flash
• DeepSeek — DeepSeek V4 Pro, DeepSeek V4 Flash
• Qwen — Qwen 3.6 Max, Qwen 3.6 Plus
• And 60+ more providers — Mistral, XAI Grok, MiniMax, Perplexity, and beyond

No more juggling multiple subscriptions or switching between apps. One API key, one interface, unlimited possibilities.

Key Features​

Unlimited Context with Knox Memory System​

Traditional AI models have fixed context windows—KnoxChat breaks that barrier.

Knox-MS (Knox Memory System) is a human brain-inspired memory architecture that enables:

• Unlimited conversation length — No more hitting token limits mid-conversation
• Intelligent memory hierarchy — Working memory, short-term, long-term, and permanent storage
• Smart retrieval — Vector search with semantic understanding finds relevant context automatically
• Cross-session learning — The system remembers patterns and preferences across conversations
• Automatic summarization — Long conversations are intelligently compressed without losing key information

Perfect for complex projects, ongoing research, or any task that requires maintaining context over extended interactions.

Intelligent Provider Routing​

KnoxChat doesn't just connect you to AI models—it optimizes every request:

• Automatic failover — If one provider is down, requests seamlessly route to alternatives
• Performance routing — Sub-10ms routing decisions based on real-time latency and success rates
• Cost optimization — Automatically select the most cost-effective provider for your needs
• Quality assurance — Continuous monitoring ensures consistent response quality
• 99.9% uptime — Multi-layer fallbacks guarantee your work is never interrupted

Integrated Code Editor​

A full VS Code experience built right into the app:

• Complete development environment — File explorer, syntax highlighting, extensions support
• **KnoxChat Agent Coding — AI-powered coding assistant available in every project
• Seamless workflow — Ask questions in Chat/Agent, get code suggestions directly in your editor
• Automatic theme sync — Editor theme follows your app preference (Dark/Light)
• Native clipboard support — Full copy/paste functionality between chat and editor

Native Desktop Experience​

Built with performance and usability in mind:

• Fast startup — Native app launches instantly, no browser overhead
• System tray integration — Quick access without cluttering your desktop
• Window state persistence — Remembers your preferred size and position
• Keyboard shortcuts — Familiar shortcuts for power users

Platform Support​

Enterprise-Grade Security​

Your data protection is our priority:

• Granular data policies — Control exactly where your data goes with per-provider controls
• Secure by default — Capability-based permissions with strict CSP policies
• No data retention — Your conversations stay private
• SSRF protection — API health checks restricted to known domains
• SOC2 compliant infrastructure — Enterprise security standards

Use Cases​

For Developers​

• Code generation & review — Get AI assistance while working in the integrated editor
• Debugging assistance — Explain errors, suggest fixes, understand complex codebases
• Documentation writing — Generate clear, consistent documentation from your code
• Multi-model comparison — Test the same prompt across different models instantly

For Researchers​

• Unlimited context analysis — Analyze long documents without context window limitations
• Knowledge persistence — Build up knowledge across multiple research sessions
• Cross-reference insights — The memory system connects related concepts automatically

For Content Creators​

• Long-form writing — Maintain narrative consistency in novels, reports, or articles
• Iterative refinement — Refine content over multiple sessions without losing context
• Multi-format output — Generate text, images, and code in a single workflow

For Teams​

• Consistent experience — Same powerful interface across all platforms
• Centralized billing — One account, unified usage tracking
• API compatibility — Fully compatible with OpenAI API for easy integration

Getting Started​

1. Download — Get the installer for your platform from the releases page
2. Install — Run the installer and follow the on-screen instructions
3. Sign in — Create an account or sign in with GitHub
4. Add credits — Recharge your account to start using AI models
5. Start chatting — Select a model and begin your conversation

Pricing Advantages​

KnoxChat offers significant cost benefits over direct API access:

• Pay-per-use — No monthly subscriptions, pay only for what you use
• Unified billing — One balance works across all 100+ models
• Cost transparency — See real-time cost estimates before sending requests
• Smart routing savings — Automatic provider selection can reduce costs by up to 40%
• No hidden fees — Transparent pricing with no markup on most models

System Requirements​

Support & Resources​

• Documentation — docs.knox.chat
• Support — support@knox.chat

The Vision​

The video production landscape has long been dominated by heavyweight, proprietary editing suites that treat AI as an afterthought — a plugin bolted on, a sidebar feature buried in menus. We asked a different question: What if AI wasn't a feature of the editor, but the co-director?

KnoxStudio is the answer. It's a native macOS application that fuses a professional multi-track video editor with an intelligent AI agent capable of generating videos, images, and audio from natural language — then placing them directly onto the timeline. No browser tabs. No cloud dashboards. No copy-pasting URLs. You describe a scene, and it materializes in your project.

Built entirely in Rust with over 416,000 lines of hand-written code, KnoxStudio is not a thin wrapper around a web API. It's a from-scratch video production environment where every pixel, every frame, and every interaction was designed with one philosophy: the creative intent flows from your words to the screen, uninterrupted.

Why Rust? Why Native?​

Video editing is one of the most demanding categories of desktop software. Frame-accurate playback, real-time canvas compositing, multi-track timeline manipulation, and FFmpeg pipeline orchestration all demand predictable, low-latency performance. Electron was never an option.

We chose Rust for its unique combination of memory safety, zero-cost abstractions, and fearless concurrency. The same language manages our GPU-backed UI rendering, our async AI agent pipeline, SQLite media database, and FFmpeg export orchestration — all without a garbage collector, and with compile-time guarantees that eliminate entire classes of runtime bugs.

The UI framework is egui via eframe — an immediate-mode GUI library that gives us pixel-level control over every widget while running at native speed. The result is a macOS-dark-mode-inspired interface that feels like it belongs on the platform, with smooth 60fps rendering even during complex timeline operations.

Architecture Overview​

KnoxStudio's architecture is organized into clearly separated domains, each responsible for a distinct concern of the video production workflow:

The Core Modules​

• Project System — The heart of the application. A Project contains tracks, clips, markers, transitions, annotations, and canvas settings. Every project element has a UUID, supports full serialization, and can be saved/loaded as a .knoxstudio bundle. The project system supports multi-track video and audio, keyframe animation (position, scale, rotation, opacity), clip speed adjustment, freeze frames, audio ducking, noise reduction, and clip grouping.

• Timeline Engine — A multi-track, non-linear timeline with drag-and-drop clip placement, trimming, splitting, ripple editing, and snapping. Track headers identify video, audio, and overlay tracks. The timeline supports in/out range selection, markers with color coding, and context menus for fine-grained editing control.

• Canvas & Compositing — A real-time preview canvas that maintains aspect ratio, composites multiple video layers with per-clip transforms (position, scale, rotation, opacity), and renders annotations on top. The canvas supports annotation tools including rectangles, ellipses, lines, arrows, freehand drawing, text, callouts, highlights, and numbered steps — each with fade-in/fade-out timing and z-order layering.

• Capture Engine — Native macOS screen recording with full system integration. Uses CoreGraphics for screen capture permissions, AVFoundation (via Swift/Objective-C FFI) for screen, camera, and microphone capture, and system audio loopback. The capture system detects available devices, handles permission requests with user-friendly guidance, and records directly to the timeline.

• Export Pipeline — A sophisticated FFmpeg-based export system supporting multiple resolution presets (480p through 4K), percentage-based scaling, custom resolutions, multiple encoding quality tiers, and format selection. The pipeline handles multi-segment composition, annotation rendering, cursor effect overlays, and audio mixing during export.

• AI Agent System — The crown jewel. A complete AI-powered production assistant that understands natural language, generates media, manages creative assets, and orchestrates complex multi-step workflows. This deserves its own section.

The AI Agent: Your Creative Co-Director​

The AI system in KnoxStudio isn't a simple prompt-to-video button. It's a multi-layered, stateful agent architecture that operates like a virtual production team.

The Agent Hierarchy​

At the top sits the Manager Agent — an LLM-powered orchestrator that acts as the "CEO" of the production pipeline. When you type a message in the chat panel, the Manager interprets your creative intent, decides which tools to invoke, and coordinates the execution across multiple sub-systems.

The Manager has access to a rich set of tools:

• Media Generation — Generate videos, images, and audio from text prompts, with automatic context injection (scene descriptions, character references, generation defaults).
• Memory & Search — Query the local media database to find previously generated assets, check what's been produced for a scene, or locate reference material.
• Planning & Execution — For complex requests ("Generate all shots for Scene 3"), the Manager creates an execution plan, tracks progress, and handles failures gracefully.
• Context Awareness — The Manager has a live snapshot of the current project state: timeline contents, available assets, active roles, and screenplay context.
• User Interaction — When clarification is needed, the Manager can ask the user questions with structured options, displayed inline in the chat.
• Timeline Placement — Generated assets can be automatically placed on the timeline at the correct position.

Below the Manager, sub-model connectors handle the actual media generation. The system supports multiple generation modes:

• Text-to-Video — Describe a scene and generate video from pure text.
• Image-to-Video — Provide a reference image to animate into a video sequence.
• Reference-to-Video — Use a reference video to guide the generation of a new video with consistent style and motion.
• Image Generation — Create still images from text descriptions.
• Audio Generation — Generate music, sound effects, or voiceover.

A Smart Video Router automatically determines which generation mode to use based on the context of your request — if you've attached an image, it routes to image-to-video; if the previous shot's last frame is available, it chains for scene continuity.

The State Machine​

The agent operates through a formal state machine (powered by the statig crate) that tracks the conversation lifecycle: idle, processing, generating, waiting for user input, handling errors. This ensures the UI always knows exactly what the agent is doing and can display appropriate feedback — thinking indicators, generation progress cards, plan execution steps.

Memory & Persistence​

All generated media is stored in a local media library at ~/.knoxmedia/, organized by type (video, audio, image, screenplay, roles). A SQLite database indexes every asset with metadata: the prompt that created it, tags, creation time, source paths, and role associations. This enables the Memory System to provide context-aware generation — the agent knows what you've already created and can build upon it.

The Screenplay System​

One of KnoxStudio's most distinctive features is its built-in screenplay engine. You can write or import a screenplay in either Markdown or Fountain format (the industry-standard plain-text screenplay notation), and KnoxStudio parses it into a structured representation of scenes, action blocks, dialogue, visual descriptions, and transitions.

The Director module then analyzes the screenplay alongside the generation history to suggest which shots to generate next. It tracks coverage — how many of the screenplay's visual descriptions have been realized as generated media — and can batch-generate all shots for a scene in one command.

Visual description blocks (written as blockquotes in Markdown or action lines in Fountain) become generation prompts. Character dialogue markers link to the Roles system. Scene headings become navigation anchors. The entire screenplay becomes an interactive production plan.

Roles: Character Consistency Across Generations​

Maintaining visual consistency for characters across multiple AI-generated shots is one of the hardest problems in AI filmmaking. KnoxStudio addresses this with the Roles system.

A Role represents a character or subject — "Hero," "Sidekick," "The Villain's Lair." Each role carries reference images and reference videos that are uploaded to the cloud asset library and associated with an asset ID. When the agent generates a shot involving a character, the active roles' asset IDs are automatically injected into the generation request, ensuring the output maintains visual consistency with the established references.

The Roles panel in the UI lets you create, manage, toggle, and configure roles with a visual reference gallery. Active roles are automatically included in every relevant generation request.

The Streamdown Markdown Engine​

Rather than pulling in a heavy third-party Markdown library, KnoxStudio includes Streamdown — a custom-built, streaming Markdown parser written from scratch in Rust. Streamdown is designed for real-time rendering of LLM output in the chat panel, handling the incremental token-by-token arrival of streamed responses.

The parser handles headings, bold, italic, inline code, code blocks with syntax labels, blockquotes, ordered and unordered lists, horizontal rules, links, and special inline entities. It's fast, allocation-efficient, and produces parse events that the egui-based Markdown renderer can consume directly — no intermediate AST, no HTML conversion.

Native macOS Integration​

KnoxStudio is a macOS citizen, not a tourist. The native integration layer includes:

• Swift/Objective-C FFI — Screen capture, camera capture, audio capture, video playback, and file handler registration are all implemented in Swift and bridged to Rust via C-compatible FFI. This ensures the app uses Apple's native AVFoundation, ScreenCaptureKit, and CoreGraphics APIs directly.

• System Permissions — The first-run experience walks users through macOS permission grants (screen recording, microphone, camera, system audio) with clear guidance on how to enable each in System Settings.

• File Associations — Double-clicking a .knoxstudio file in Finder opens it directly in the app, thanks to a registered Apple Event handler installed before the NSApplication run loop starts.

• Font Stack — The app loads the macOS system font stack (Arial Unicode, Apple Symbols, Menlo, PingFang for CJK) for proper Unicode, emoji, and symbol rendering, with an embedded Noto Sans Symbols 2 fallback.

• Windowed winit Patch — The project vendors a patched version of winit (the cross-platform window management crate) specifically to preserve macOS-native app activation behavior for proper Info.plist and App Store compatibility.

The UI: macOS Dark Mode, Reimagined​

KnoxStudio's interface draws heavy inspiration from macOS Ventura/Sonoma dark mode aesthetics. The custom theme system defines a meticulous color palette:

• Backgrounds with subtle saturation shifts between panel, timeline, canvas, and elevated surfaces
• Typography with three tiers — primary, secondary, and tertiary text — for clear visual hierarchy
• macOS Blue accent for selections, active controls, and interactive elements
• Teal accent for slider handles and secondary interactive elements
• Semantic colors — a red playhead, green success indicators, yellow warnings, red errors
• Transport controls styled as pill buttons with hover and active states

The layout follows a professional NLE (Non-Linear Editor) paradigm:

• Toolbar at the top with project tools, annotation tools, and recording controls
• Media Browser on the left for browsing imported and generated media
• Canvas in the center for real-time preview with annotation overlays
• Inspector on the right for clip properties, color controls, and transform settings
• Timeline across the bottom with multi-track editing, track headers, and transport controls
• AI Agent Panel as a slide-out chat panel for conversational media generation

Configuration & Flexibility​

KnoxStudio separates sensitive credentials from user preferences using a two-file configuration system:

• ~/.knoxmedia/config.yaml — Contains model configurations, API keys, and active model selections. File permissions are set to 0600 (owner-only read/write) for security.
• ~/.knoxmedia/defaults.yaml — Contains generation defaults (duration, resolution, aspect ratio, audio settings), storage limits, and execution parameters. Safe to share — no secrets.

The config supports multiple providers and models, with a flexible active selection system that lets users switch between different text, video, image, and audio generation backends without changing their workflow.

Export: From Timeline to Final Cut​

The export pipeline is built on FFmpeg and supports a professional range of output options:

• Resolution Presets: 480p, 720p, 1080p, 4K, or percentage-based scaling (25%, 50%, 75%, 100%)
• Quality Tiers: Multiple encoding quality levels for balancing file size and visual fidelity
• Format Support: Common video container formats
• Composition: Multi-segment assembly with proper sequencing and timing
• Overlay Rendering: Annotations and cursor click effects are composited into the final export
• Audio Mixing: Multi-track audio with volume, ducking, and noise reduction per clip

The export runs asynchronously with real-time progress reporting back to the UI.

Annotations & Visual Communication​

For tutorial creators, product demos, and educational content, KnoxStudio includes a complete annotation toolkit:

• Shapes: Rectangles, ellipses, lines, arrows
• Drawing: Freehand draw tool for organic annotations
• Text & Callouts: Text labels and callout boxes with customizable styling
• Highlights: Semi-transparent highlight overlays
• Numbered Steps: Automatically numbered step indicators for sequential tutorials
• Timing: Each annotation has its own start time and duration on the timeline
• Effects: Fade-in and fade-out transitions, rotation, opacity, and z-order layering
• Locking: Annotations can be locked to prevent accidental modification

The Generation Workflow​

Here's what a typical AI-assisted production workflow looks like in KnoxStudio:

1. Write a screenplay in Markdown or Fountain format, or paste one into the Screenplay panel.
2. Define roles for your characters with reference images to ensure visual consistency.
3. Chat with the agent: "Generate the establishing shot for Scene 1." The Manager interprets your intent, reads the screenplay's visual description for Scene 1, checks if any roles are active, and dispatches the generation request.
4. Watch it materialize: A generation card appears in the chat showing real-time progress. When complete, the video downloads to the local media library.
5. Drag to timeline or let the agent place it automatically.
6. Iterate: "Make it more dramatic" or "Continue to the next shot" — the agent chains shots using the last frame of the previous generation for seamless scene continuity.
7. Add annotations, adjust timing, add transitions.
8. Export the final video.

The agent remembers the conversation context, the generation history, and the project state. It can plan multi-step sequences, ask clarifying questions, and recover from failures — all within the same chat interface.

Technical Highlights​

• 416,000+ lines of Rust — No scripting languages, no glue code. Pure Rust from UI to export.
• Rust Edition 2024 — Using the latest Rust edition features and a minimum Rust version of 1.95.
• Async-first AI pipeline — Tokio-powered multi-threaded runtime for concurrent API calls, polling, and downloads without blocking the UI.
• Immediate-mode UI — The entire interface renders at 60fps using egui's immediate-mode paradigm, ensuring the UI is always in sync with application state.
• State machine-driven agent — Formal state machine (via statig) prevents impossible agent states and makes the conversation lifecycle predictable.
• Custom Markdown parser — Purpose-built streaming parser for rendering LLM output in real time.
• SQLite media index — Fast local search and metadata retrieval for the entire media library.
• Smart routing — Context-aware model selection that automatically picks the right generation mode based on available attachments and project state.
• Rate limiting — Built-in rate limiter for API calls to respect provider limits and ensure fair usage.
• FFI to Swift/Objective-C — Native macOS system integration through carefully designed foreign function interfaces.
• Vendored winit — Patched window management for perfect macOS app lifecycle behavior.
• Profile-optimized release builds — LTO, symbol stripping, and optimization level 3 for production builds.

What's Next​

KnoxStudio is version 1.0.2, but the roadmap is ambitious. The foundation — a native, performant, AI-first video editor — is solid. The architecture was designed from day one to support additional providers, new generation modalities, and deeper timeline intelligence.

We believe the future of video production is conversational. Not clicking through menus, but describing what you want and watching it appear. Not switching between five different apps, but having one environment where creative intent flows directly into finished media.

KnoxStudio is that environment. And we're just getting started.

KnoxStudio is built by Knox Core^™. For more information, Please contact us: support@knox.chat

Give your AI a brain that remembers.

Memory Brain is a local-first, SQLite-powered memory system that gives Knox persistent, structured recall across conversations, sessions, and projects. It replaces the "goldfish memory" of stateless LLMs with a multi-layered cognitive architecture inspired by human memory science.

Try KnoxChat VS Code extension with Memory Brain here >>>

Table of Contents​

• Why Memory Brain?
• How It Works
• Architecture Overview
• Memory Types
• The 5-Tier Memory Hierarchy
• Core Features
• All 79 Actions Reference
• Database Schema
• Advantages
• Configuration
• Test Coverage
• Tool Integration

Why Memory Brain?​

Every conversation with an LLM starts from zero. The AI forgets your preferences, past decisions, project context, and the patterns that worked before. You end up repeating yourself, re-explaining your codebase, and losing insights that took hours to reach.

Memory Brain solves this by providing:

How It Works​

The Memory Tool Call Flow​

User or Agent
    │
    ▼
┌─────────────────────┐
│   memory tool call  │  action: "store", params: { title, content, ... }
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   BrainManager      │  dispatch(action, params) → routes to handler
│   (Service Layer)   │  emits events, records audit log
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   BrainStore        │  SQLite CRUD, BM25 search, consolidation
│   (Data Layer)      │  17 tables, 27 indexes, WAL mode
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│   brain.sqlite      │  ~/.knox/memory/brain.sqlite
│   (Local Database)  │  Your data. Your machine. No cloud.
└─────────────────────┘

1. An agent (or user) invokes the memory tool with an action and params.
2. BrainManager.dispatch() routes to one of 79 action handlers.
3. The handler calls BrainStore (SQLite layer) or a specialized module (KnowledgeGraph, LearningEngine, CheckpointManager, etc.).
4. Events are emitted and logged in the audit trail.
5. A human-readable formatted string is returned to the caller.

Automatic Memory Lifecycle​

Memory Brain isn't just storage — it actively manages memory health:

 Store ──► Hot Tier ──► Warm Tier ──► Cold Tier ──► Pruned
           (active)     (1-7 days)   (7-90 days)   (>90 days)
              ▲                                        │
              └─── Promote (frequent retrieval) ◄──────┘

 Background: Ebbinghaus decay, spaced repetition, consolidation

Memories that are frequently accessed get promoted back to hot tier. Memories that decay below importance thresholds are gradually consolidated and eventually pruned. This keeps the system fast and relevant.

Architecture Overview​

Module Map​

core/context/memory/brain/
├── BrainManager.ts        # Service layer — 79-action dispatch, event bus
├── BrainStore.ts          # SQLite data layer — 17 tables, CRUD, search
├── types.ts               # TypeScript interfaces, enums, input/output models
├── ContextBuilder.ts      # Token-budgeted multi-source context assembly
├── KnowledgeGraph.ts      # Entity/edge CRUD, spreading activation traversal
├── LearningEngine.ts      # Pattern recording, Jaccard similarity suggestions
├── AutoMemory.ts          # Rule-based extraction, topic segmentation
├── CheckpointManager.ts   # Snapshots, rollback, event replay, undo
├── BatchOperations.ts     # Bulk store/delete/update, batch audit logging
├── AdvancedFeatures.ts    # SessionDiscovery, FiveTierHierarchy, SpacedRepetition,
│                          # RootCauseAnalyzer, LruCache, MetricsStorage
├── PerformanceMonitor.ts  # MetricsCollector, HealthScorer, PredictiveAnalytics,
│                          # HealingEngine, ConsolidationTracker
├── LlmMemoryService.ts    # LLM-powered entity extraction, summarization, scoring
├── LlmResilience.ts       # CircuitBreaker, rate limiting, decision caching
├── SleepConsolidation.ts  # Background scheduler, Ebbinghaus decay application
├── WorkingMemory.ts       # In-memory session state, short-term buffer
├── test-memory-tool.ts    # 94 integration tests — 100% pass rate
├── test-brain-debug.ts    # 202 ultra-comprehensive integration tests
├── test-brain-debug-modules.ts  # 59 module-level unit tests
└── test-brain-tool-calls.ts     # 88 end-to-end tool call tests

Design Principles​

• Local-first: All data in ~/.knox/memory/brain.sqlite. Nothing leaves your machine.
• Zero-config: Works out of the box with sensible defaults.
• LLM-optional: Every LLM-enhanced feature has a rule-based fallback.
• Audit everything: Every write operation is logged with full undo support.
• Fail gracefully: Circuit breakers, rate limits, and resilient fallback chains.

Memory Types​

1. Semantic Memory — What you know​

Long-term facts, decisions, and knowledge extracted from conversations.

Features: BM25-inspired re-ranking, importance scoring, emotional valence tagging (positive, negative, neutral, surprise, urgency, curiosity), salience weighting, TTL expiration, keyword indexing.

2. Episodic Memory — What happened​

Full conversation history: every user message, assistant response, tool call, and tool result, stored per session.

• Tracks role, token_count, importance_score, emotional_valence
• Links to sessions via foreign key
• Supports cross-session search (search_backlogs)
• Auto-detects topic shifts within sessions (get_session_topics)

3. Knowledge Graph — How things connect​

Entities and their relationships, modeled as a directed graph.

 ┌──────────┐   uses    ┌─────────────┐   depends_on   ┌──────────┐
 │  Knox    ├──────────►│ TypeScript  ├───────────────►│  SQLite  │
 │ (person) │           │ (technology)│                │  (tech)  │
 └──────────┘           └─────────────┘                └──────────┘

13 entity types: person, organization, technology, concept, project, file, function, class, variable, location, event, product, custom.

Graph traversal: Spreading activation — explore from any entity and discover connected nodes weighted by edge strength.

4. Learning Patterns — What works and what doesn't​

Tracks approach success/failure rates by goal type (coding, debugging, analysis, research, etc.). Uses Jaccard similarity to suggest the best approach for new problems based on past outcomes.

5. Procedural Memory — How to do things​

Named, ordered workflows with steps. Tracks execution count and success rate.

{
  "name": "deploy-to-production",
  "steps": ["Run lint", "Run tests", "Build", "Push to registry", "Deploy"],
  "success_rate": 0.92,
  "execution_count": 13
}

6. Working Memory — Right now​

In-memory session state and short-term buffer that flushes to disk. Provides fast access to the current conversation context without DB queries.

The 5-Tier Memory Hierarchy​

Inspired by CPU cache hierarchies and Ebbinghaus forgetting curves.

┌─────────────────────────────────────────────────┐
│  TIER 1: ACTIVE   │ Currently in-session        │ Working Memory
├───────────────────┼─────────────────────────────┤
│  TIER 2: HOT      │ Frequently accessed, recent │ Full fidelity
├───────────────────┼─────────────────────────────┤
│  TIER 3: WARM     │ 1-7 days old, moderate use  │ Full fidelity
├───────────────────┼─────────────────────────────┤
│  TIER 4: COLD     │ 7-90 days old, low access   │ Compressed
├───────────────────┼─────────────────────────────┤
│  TIER 5: FROZEN   │ Archived, rarely accessed   │ Compressed
└───────────────────┴─────────────────────────────┘

Automatic Transitions​

Spaced Repetition​

Memories due for review are surfaced automatically. Each retrieval boosts a memory's importance score, preventing decay — the more you access it, the longer it stays.

Core Features​

1. Context Builder​

Automatically assembles the most relevant memories within a token budget. Used to prime the AI with context at the start of each conversation.

Budget allocation (default 4000 tokens):

memory({ action: "build_context", query: "TypeScript migration", max_tokens: 4000 })

2. Checkpoint & Rollback​

Full memory state snapshots with point-in-time recovery.

• Create: Snapshots all semantic memories, entities, and learning patterns
• Diff: See what changed since a checkpoint (added, removed, modified)
• Rollback: Restore memory to any checkpoint — conversation history preserved
• Compress: gzip compression for space-efficient long-term storage
• Lifecycle cleanup: Automatically prune old checkpoints

Checkpoint strategies:

• manual — Create checkpoints explicitly
• time_based — Auto-create at configurable intervals
• adaptive — Auto-create when change threshold is exceeded
• hybrid — Combine time and adaptive triggers

memory({ action: "create_checkpoint", label: "before-refactor-v2" })
memory({ action: "rollback_checkpoint", checkpoint_id: 4 })

3. Full Audit Trail​

Every write operation is logged with target type, target ID, and details. Supports filtering by action, time range, and target. Enables:

• Undo: Reverse any specific operation
• Event replay: Replay changes from a checkpoint forward
• Forensics: Understand exactly what changed and when

4. Auto-Extract​

Pass raw text (conversation, code review, meeting notes) and Memory Brain automatically extracts:

• Semantic memories (facts, decisions, insights)
• Named entities (people, technologies, projects)
• Emotional valence
• Topic segmentation

Works rule-based by default. LLM-enhanced extraction available when an LLM is configured (with automatic fallback to rules if the LLM is unavailable).

5. Cross-Session Search​

Search across ALL past conversation sessions with filters:

• Query text (full-text search)
• Date range (from_date, to_date)
• Role filter (user, assistant, tool)
• Session ID filter

6. Performance Monitor​

Real-time health monitoring and self-healing capabilities:

• Metrics: Track operation counts, latency, cache hit rates
• Health Score: Weighted composite of fragmentation, stale data, capacity
• Capacity Forecast: Predict when storage or memory limits will be reached
• Auto-Heal: Detect and fix common issues (orphaned records, index fragmentation)
• Trend Analysis: Historical metrics with 24h lookback

7. Batch Operations​

Bulk operations for efficiency:

• batch_store — Store multiple memories in one call
• batch_delete — Delete multiple memories by ID list
• batch_update_importance — Update importance scores in bulk
• batch_move_tier — Move memories between tiers
• batch_audit_log — Log multiple events at once

8. LLM-Enhanced Features (Optional)​

When an LLM is connected, unlock advanced capabilities:

All LLM features include:

• Rate limiting: Configurable calls/tokens per hour
• Circuit breaker: Automatically disables after repeated failures
• Decision cache: Avoids redundant LLM calls for similar inputs

All 79 Actions Reference​

Core Memory (6 actions)​

Session Management (5 actions)​

Knowledge Graph (6 actions)​

Learning Engine (3 actions)​

Procedural Memory (3 actions)​

Tags & Collections (6 actions)​

Associations (1 action)​

Auto-Memory (1 action)​

Maintenance (5 actions)​

Checkpoint & Rollback (5 actions)​

Audit & Undo (4 actions)​

Cross-Session Search (1 action)​

Export / Import (2 actions)​

Checkpoint Strategy (4 actions)​

Batch Operations (5 actions)​

Advanced Features (7 actions)​

Performance & Health (11 actions)​

LLM-Enhanced (4 actions)​

Database Schema​

17 tables with 27 optimized indexes, running on SQLite with WAL mode and foreign key constraints enabled.

brain_sessions           — Conversation sessions
brain_episodic           — Conversation turns (messages, tool calls)
brain_semantic           — Long-term factual memories
brain_associations       — Cross-memory links
brain_entities           — Knowledge graph nodes
brain_graph_edges        — Knowledge graph relationships
brain_learning_patterns  — Success/failure patterns
brain_procedures         — Named workflows
brain_tags               — Memory tags (many-to-many)
brain_collections        — Named memory groups
brain_collection_items   — Collection membership
brain_config             — Configuration key/value store
brain_checkpoints        — Memory state snapshots
brain_rate_limits        — LLM rate limit tracking
brain_audit_log          — Full audit trail
brain_session_topics     — Auto-detected topic shifts
brain_metrics_snapshots  — Performance metrics history

Storage: Single file at ~/.knox/memory/brain.sqlite. Typical size is 1-10 MB after months of active use.

Advantages​

vs. Vector Databases (Pinecone, Chroma, Weaviate)​

vs. File-Based Memory (.md files, JSON)​

vs. No Memory (Stateless LLM)​

Key Pros​

1. Privacy-first — Everything stored locally. No API calls for core features.
2. Self-maintaining — Automatic consolidation, decay, promotion, and healing.
3. Resilient — Circuit breakers, fallbacks, rate limits. No single point of failure.
4. Observable — Full audit trail, health metrics, performance monitoring.
5. Reversible — Checkpoint/rollback for any destructive operation.
6. Token-efficient — Context Builder assembles only what fits in your budget.
7. Structured — Not just text blobs: typed entities, weighted edges, categorized facts.
8. Tested — 443 integration tests across 4 test suites covering all 79 actions at 100% pass rate.
9. LLM-optional — Every feature works without an LLM. Enhanced features degrade gracefully.
10. Portable — Export/import everything as JSON. Move between machines freely.

Configuration​

All settings are stored in brain_config table and can be read/updated via tool calls.

memory({ action: "get_config" })
memory({ action: "update_config", key: "max_hot_memories", value: "1000" })

Test Coverage​

Four test suites exercise 443 integration tests across all 79 actions, calling every action through the actual dispatch pipeline against a real SQLite database.

Suite 1: test-memory-tool.ts — 94 tests​

The original integration test covering all core dispatch actions.

Suite 2: test-brain-debug.ts — 202 tests​

Ultra-comprehensive integration tests exercising every dispatch action with deep output validation, edge cases, and multi-step workflows.

Suite 3: test-brain-debug-modules.ts — 59 tests​

Module-level unit tests for individual classes (BrainStore, KnowledgeGraph, LearningEngine, ContextBuilder, CheckpointManager, BatchOperations, AdvancedFeatures, PerformanceMonitor).

Suite 4: test-brain-tool-calls.ts — 88 tests​

End-to-end tests verifying all 79 tool enum actions work through BrainManager.dispatch(), organized by action tier.

Summary​

Each test suite uses a fresh temporary database, validates output format, and cleans up after itself.

Quick Start​

Store a memory​

memory({ action: "store", category: "decision", title: "Use SQLite for memory",
         content: "Chose SQLite over Postgres for zero-config local storage.",
         keywords: "database,sqlite,architecture", importance: 0.9 })

Recall relevant context​

memory({ action: "recall", query: "database choice", limit: 5 })

Build pre-conversation context​

memory({ action: "build_context", query: "TypeScript migration project",
         max_tokens: 4000 })

Create a safety checkpoint before risky work​

memory({ action: "create_checkpoint", label: "before-major-refactor" })
// ... do risky work ...
memory({ action: "rollback_checkpoint", checkpoint_id: 1 })  // oops, undo

Track what works​

memory({ action: "learn_pattern", goal_type: "debugging",
         description: "Binary search through git history to isolate regression",
         success: true, tokens_used: 1200 })

// Next time:
memory({ action: "suggest_approach", query: "find regression", goal_type: "debugging" })

Tool Integration​

Memory Brain is fully integrated into the Knox tool call system as a first-class built-in tool with its own UI category, interceptor validation, and permission controls.

Registration Architecture​

core/tools/definitions/memory.ts     → Tool definition (schema, 79 actions, params)
core/tools/implementations/memory.ts → Implementation (routes to BrainManager.dispatch)
core/tools/callTool.ts               → Dispatch (BuiltInToolNames.Memory → memoryImpl)
core/tools/builtIn.ts                → Enum (Memory = "builtin_memory")
core/tools/index.ts                  → allTools array (includes memoryTool)
core/config/load.ts                  → Config (tools: [...allTools])

Tool Identity​

How It Appears in the UI​

The memory tool shows up in the Tools panel under the Permissions group:

● Permissions  [12]                              ⬤ ON
  [Files] Read File                        Auto-Approve
  [Files] Create New File                  Auto-Approve
  [Terminal] Run Terminal Command          Auto-Approve
  [Files] View Subdirectory                Auto-Approve
  [Files] View Repo Structure              Auto-Approve
  [Search] Exact Search                    Auto-Approve
  [Search] Web Search                      Auto-Approve
  [Diff] View Diff                         Auto-Approve
  [Files] Read Currently Open File         Auto-Approve
  [Tool] Load Skill                        Auto-Approve
  [Tool] LSP                               Auto-Approve
  [Memory] Memory                          Auto-Approve

File Locations​

Permission States​

Each tool can be in one of three permission states, cycled by clicking:

The entire Permissions group can also be toggled on/off via the group switch.

Tool Call Flow (End-to-End)​

1. LLM generates tool call:
   { name: "builtin_memory", arguments: { action: "store", ... } }

2. ToolCallInterceptor validates:
   - Tool exists in registry ✓
   - Required params present (action) ✓
   - Logs call to history, tracks metrics

3. Permission check:
   - toolSettings["builtin_memory"] === "allowedWithoutPermission"
   - Auto-approved (no user prompt)

4. callTool() dispatches:
   - Matches BuiltInToolNames.Memory
   - Calls memoryImpl(args, extras)

5. memoryImpl() routes:
   - Gets BrainManager singleton
   - Calls brain.dispatch(args.action, args)

6. BrainManager.dispatch() handles:
   - Routes to appropriate handler (store, recall, search, etc.)
   - Records audit trail
   - Emits events
   - Returns formatted string

7. Result returned to LLM as ContextItem[]

Try KnoxChat VS Code extension with Memory Brain here >>>

Memory Brain is part of the KnoxCore platform. All data stays on your machine.

A comprehensive Todo/Task Management System that analyzes complex user requests, breaks them into structured dependency-aware tasks, and tracks progress via tool-call-driven stream synchronization with persistent session management.

Install KnoxChat extension here >>>

The Problem — Unstructured Task Execution​

Previously, complex multi-step requests were handled as monolithic operations:

• No Visibility: Users couldn't see what the AI was working on or what remained
• No Recovery: Failed steps required restarting the entire request
• No Persistence: Progress was lost if the session was interrupted
• No Dependencies: No awareness of task ordering or prerequisites

The Solution — Stream-Sync Task Engine​

A complete task orchestration system spanning the VS Code extension backend (TodoManager), the bidirectional protocol layer, Redux state management (todoSlice + planTaskSlice), tool-call-driven stream synchronization (todoStreamSync), and a React GUI (TodoPanel + PlanTaskStatusPanel).

Core Architecture​

1. Rich Data Model (core/index.d.ts)​

Every task is a TodoItem with 30 fields:

type TodoStatus = "pending" | "in-progress" | "completed" | "failed" | "cancelled" | "blocked";
type TodoPriority = "high" | "medium" | "low";
type TodoComplexity = "simple" | "medium" | "complex";
type TodoCategory = "coding" | "analysis" | "research" | "testing" | "documentation" | "refactoring" | "devops" | "general";

interface TodoItem {
  id: string;
  title: string;
  description: string;
  status: TodoStatus;
  priority: TodoPriority;
  complexity: TodoComplexity;
  category: TodoCategory;
  dependsOn: string[];           // Upstream task IDs
  blockedBy: string[];           // Tasks blocking this one
  filePaths: string[];           // Associated file paths
  toolsNeeded: string[];         // Tools needed (edit_file, run_terminal_cmd, etc.)
  estimatedTimeMs: number;       // Estimated time in milliseconds
  actualTimeMs: number;          // Actual time spent in milliseconds
  retryCount: number;
  maxRetries: number;
  errorMessage?: string;
  resultPreview?: string;        // Human-readable evidence summary
  verified?: boolean;            // true if completion verified by evidence
  evidenceScore?: number;        // 0-1 confidence score
  toolCallCount?: number;        // Number of tool calls that contributed
  subtasks: TodoItem[];          // Recursive nesting
  parentId?: string;             // Parent todo ID (if subtask)
  orderIndex: number;            // Display order
  createdAt: number;
  startedAt?: number;
  completedAt?: number;
  progress: number;              // 0-100
  tags: string[];                // Technology tags (typescript, react, docker, etc.)
}

2. Session Management​

Tasks are grouped into TodoSession objects with full lifecycle:

interface TodoSession {
  id: string;
  title: string;
  description: string;
  prompt: string;                          // Original user request
  todos: TodoItem[];
  status: "idle" | "active" | "paused" | "completed" | "failed";
  progress: number;                        // 0-100
  stats: TodoSessionStats;
  config: TodoSessionConfig;
  createdAt: number;
  startedAt?: number;
  completedAt?: number;
  workspaceDirectory?: string;
  conversationId?: string;
}

Statistics — 11 aggregate counters including verifiedTodos:

interface TodoSessionStats {
  totalTodos: number;
  completedTodos: number;
  verifiedTodos: number;     // Completed todos verified by tool-call evidence
  failedTodos: number;
  cancelledTodos: number;
  inProgressTodos: number;
  pendingTodos: number;
  blockedTodos: number;
  totalEstimatedTimeMs: number;
  totalActualTimeMs: number;
  totalRetries: number;
}

Feature 1: AI-Powered Task Analysis & Decomposition​

Automatically analyzes user messages to detect when task decomposition is beneficial, then uses the ReasoningEngine or a local NLP fallback to break them down.

Smart Detection (shouldCreateTodos()):

• Explicit triggers: /todo or @todo in the message
• Pattern matching: 17 regex patterns for complex tasks (full-stack, CRUD, REST API, CI/CD, auth, GraphQL, microservice, architecture, migration, database schema, component library, testing framework, etc.)
• Multi-clause detection: Messages with 3+ clauses separated by commas, semicolons, and, or then (each >10 chars)
• Threshold: 2+ pattern matches triggers creation

AI Analysis (via ReasoningEngine.performTaskAnalysis()):

User: "Build a REST API with authentication, add unit tests, and set up Docker deployment"

AI Analysis Result:
├── Todo 1: Set up Express.js REST API scaffold     [high, coding, simple]
├── Todo 2: Implement JWT authentication middleware  [high, coding, medium]
├── Todo 3: Create CRUD endpoints                   [medium, coding, medium]
├── Todo 4: Write unit tests with Jest              [medium, testing, medium]  → depends on [1,2,3]
└── Todo 5: Create Dockerfile and docker-compose    [low, devops, simple]     → depends on [1]

Local Fallback Analysis (performLocalAnalysis()):

• Sentence boundary splitting (. ; and then, then)
• Comma splitting fallback for single-sentence inputs
• Automatic category inference via 8 keyword dictionaries (coding, analysis, research, testing, documentation, refactoring, devops, general)
• Complexity: simple (≤2 steps), medium (3–5), complex (>5)
• Priority assignment: first third = high, middle = medium, last third = low
• File path extraction via regex (/path/to/file.ext)
• Technology tag detection: TypeScript, JavaScript, React, Node.js, Python, Rust, API, Database, UI, CSS, HTML, Docker
• Tool inference from action verbs: create → edit_file, test → run_terminal_cmd, search → grep_search/codebase_search, install → run_terminal_cmd, delete → delete_file, list → list_dir

Plan Generation: For complex tasks (>2 steps), the system generates a structured implementation plan via ReasoningEngine.generateImplementationPlan() and writes it to .knox/plans/plan_{sessionId}.md. Plan steps have status markers (❌ → 🔄 → ✅) that update as todos progress.

Feature 2: Sequential Stream-Sync Execution Engine​

Tasks execute one at a time in the stream-sync model. The GUI's todoStreamSync drives advancement — the backend TodoManager.processNextTodos() only marks the next ready todo as in-progress (max 1 concurrent).

Dependency Resolution:

[Task A] ──→ [Task C] ──→ [Task E]
[Task B] ──→ [Task D] ──┘

• Tasks execute sequentially (1 at a time)
• Task B starts only after Task A completes
• Dependency-blocked tasks transition: blocked → pending when upstream completes

Key Mechanisms:

• processNextTodos(): Picks the first ready todo (pending + deps met) and marks it in-progress. Only 1 allowed at a time.
• areDependenciesMet(): Checks all dependsOn todos are completed or cancelled
• unblockDependentTodos(): Transitions blocked → pending when a dependency resolves; clears blockedBy references
• getReadyTodos(): Returns pending todos with all dependencies satisfied
• advanceToNextTodo(): Called from GUI stream-sync; completes current in-progress todo, triggers processNextTodos()
• AbortController: Clean cancellation of in-progress executions on pause/cancel

Execution Model (Stream-Sync):

todoStreamSync.ts (GUI)                        TodoManager.ts (IDE)
────────────────────                           ─────────────────────
onStreamStart()                                
  ├─ Build tracker state                       
  ├─ dispatch(startTodoSession)  ──────────→   startSession()
  │                                              └─ processNextTodos() → mark first todo in-progress
  │                                            
onToolCallEvent() × N                          
  ├─ Count tool calls on current task          
  ├─ Collect affected file paths               
  ├─ If toolCallCount ≥ 3 (AUTO_ADVANCE):     
  │   dispatch(completeTodo)     ──────────→   completeTodo() → unblock deps → processNextTodos()
  │                                            
onStreamEnd() [wrapperDepth === 0 only]        
  ├─ Complete ALL remaining tasks              
  └─ dispatch(completeTodo) for each  ─────→   completeTodo() → checkSessionCompletion()

Feature 3: Auto-Retry with Backoff​

Failed tasks are automatically retried with increasing delays.

Configuration:

• retryOnFailure: Toggle auto-retry (default: true)
• maxRetries: Maximum retry attempts per task (default: 2)

Backoff Formula: 1000 × retryCount ms

• 1st retry: 1 second delay
• 2nd retry: 2 second delay

Behavior:

• If retries remain: Reset status to pending, increment retryCount, schedule retry after backoff
• If retries exhausted: Mark as permanently failed with errorMessage, still process next todos
• Manual retry: Available via UI button, increments retryCount, resets status to pending

Feature 4: Persistent Session Storage​

Sessions are automatically saved to disk and survive editor restarts.

Storage Layout:

.knox/
├── todos/
│   ├── {sessionId}.json         # Full session + all todos (JSON)
│   └── ...
└── plans/
    ├── plan_{sessionId}.md      # Implementation plan (Markdown)
    └── ...

Features:

• Auto-save timer: Configurable interval per session (default 30s), via setInterval
• Event-triggered saves: After persistSession() calls on state changes
• Atomic writes: Via vscode.workspace.fs.writeFile API
• Load on startup: loadPersistedSessions() reads all .json files from .knox/todos/
• Session deletion: Removes both in-memory map entry and on-disk JSON file

Feature 5: Real-Time Progress Tracking UI​

Two React panels with live updates, integrated into the chat interface.

Primary Panel — TodoPanel.tsx (9 sub-components):

Secondary Panel — PlanTaskStatusPanel.tsx:

• Unified streaming progress panel merging Knox MS task metadata and todo progress
• TodoTaskRow sub-component renders individual todo items with status icons and tool call counts
• Falls back to todo-based task list when Knox MS tasks are empty
• Shows todoCurrentTitle from planTaskSlice during streaming

Uses 29 Lucide React icons including Shield, Zap, CheckCircle2, Loader2, Lock, AlertTriangle, Code2, FlaskConical, Globe, FileText, Wrench, Sparkles, Lightbulb, etc.

Panel States:

Collapsed:  [Todo Progress ▼] [75%] ████████░░
Expanded:   Status • 3/4 completed • 75% with full task list, actions, stats
No Session: Create form with prompt input

Feature 6: Bidirectional Protocol Layer​

Fully typed message passing between the VS Code extension and React webview, defined in core/protocol/.

IDE → Webview (5 messages) — ToWebviewFromIdeOrCoreProtocol:

Webview → IDE (13 messages) — ToIdeFromWebviewProtocol:

Feature 7: Redux State Management​

Two Redux Toolkit slices with 12 async thunks covering complete session lifecycle.

todoSlice State:

interface TodoState {
  currentSession: TodoSession | null;
  sessions: TodoSession[];
  isPanelVisible: boolean;
  isPanelCollapsed: boolean;
  isLoading: boolean;
  error: string | null;
  autoCreateEnabled: boolean;
  viewingSessionId: string | null;
}

12 Async Thunks: createTodoSession, startTodoSession, pauseTodoSession, resumeTodoSession, cancelTodoSession, retryTodo, skipTodo, completeTodo, advanceSession, fetchCurrentSession, fetchAllSessions, deleteTodoSession

12 Synchronous Reducers: sessionCreated, sessionUpdated, sessionCompleted, todoUpdated, progressUpdated, togglePanelVisible, togglePanelCollapsed, setPanelCollapsed, setPanelVisible, toggleAutoCreate, setViewingSession, clearError

planTaskSlice Integration: The planTaskSlice tracks streaming metadata (Knox MS + todo sync) with fields:

• hasTodoSync, todoSessionId, todoCurrentIndex, todoCompletedCount
• Updated by todoStreamSync via syncTodoProgress action

Auto-Reset: Todo state resets when a new chat session begins via newSession extra reducer.

Feature 8: Session Configuration​

8 tunable parameters per session (TodoSessionConfig):

Feature 9: Internationalization (i18n)​

Full multilingual support with 107 translation keys across English and Chinese.

Extension (extensions/vscode/src/i18n/locales/en/todo.json): 43 keys covering session lifecycle, task operations, status messages, plan approval, verification rounds

GUI (gui/src/locales/en/todo.json): 64 keys covering panel UI, status labels, action buttons, statistics, empty states, tooltips, verification badges, difficulty levels

Languages: English (en) + Chinese (zh)

Feature 10: VS Code Command Registration​

18 commands registered via vscode.commands.registerCommand in TodoManager.registerCommands():

Feature 11: Tool-Call-Driven Progress Tracking (todoStreamSync)​

Design Principles:

1. Tool calls = progress. Each tool call advances the current task's count. After enough calls, the task is marked done.
2. Sequential model. The AI works through tasks in order. A currentIndex pointer advances forward.
3. No premature completion. Tasks only complete when enough tool calls accumulate OR the entire conversation ends (outermost wrapper exit at wrapperDepth === 0).
4. Every completed task gets the shield icon. No "low evidence" vs "verified" distinction at the tracker level.

Architecture:

callTool.ts          todoStreamSync.ts              TodoManager.ts
───────────         ──────────────────             ──────────────────
ToolCallEvent ──→   onToolCallEvent()
                      ├─ Count on current task
                      ├─ Collect file paths
                      ├─ If count ≥ 3 (AUTO_ADVANCE_THRESHOLD):
                      │   completeCurrentAndAdvance()
                      │     └─ dispatch(completeTodo) ──────→  completeTodo()
                      │                                          ├─ Parse verification metadata
                      │                                          ├─ Update plan file (🔄 → ✅)
                      │                                          ├─ Unblock dependents
                      │                                          └─ processNextTodos()
                      │
                      └─ emitProgress() → syncTodoProgress   planTaskSlice
                                           (Redux action)       → PlanTaskStatusPanel (UI)
                                                                → TodoPanel (UI)

Tracker State:

interface TrackerState {
  sessionId: string;
  todos: TodoEntry[];      // { id, title, completed, toolCallCount, files, chars }
  currentIndex: number;
  totalToolCalls: number;
  totalChars: number;
  finished: boolean;
  roundCount: number;
}

Key Constants:

• AUTO_ADVANCE_THRESHOLD = 3 — Tool calls needed to auto-advance to next task

Wrapper Depth Tracking:

• onWrapperEnter() / onWrapperExit() manage a wrapperDepth counter
• Stream end reconciliation (onStreamEnd()) only fires at wrapperDepth === 0
• Intermediate tool-call rounds (depth > 0) are no-ops

Session Initialization:

• If session exists in Redux and is idle → init tracker + start backend session
• If session is active → attach tracker without re-starting
• If no session → poll every 200ms for up to 8 seconds, then give up
• Early tool calls and chunks are buffered until tracker initializes

Stream End Behavior:

• At outermost exit: complete ALL remaining tasks unconditionally
• Each remaining task gets dispatch(completeTodo) with a result preview (e.g., "Completed with N tool call(s), M file(s)")
• Completion summary dispatched via setCompletionSummary

Verification Metadata Parsing (TodoManager): When completeTodo() receives a resultPreview string, parseVerificationMetadata() extracts:

• verified: true if preview contains [verified], Verified:, text verified, or [tool-evidence]
• evidenceScore: Extracted from score: N%, evidence: N%, keyword match: N%, or (N%); defaults to 0.8 if verified, 0 otherwise
• toolCallCount: Extracted from N tool call pattern

UI Indicators:

• Shield icon (🛡 cyan): Verified completion with tool call evidence
• Checkmark (✓): Standard completion
• Warning (⚠ yellow): Low-evidence completion — may need review
• Zap badge (⚡N): Tool call count indicator
• Percentage badge: Evidence score for low-confidence items
• Result preview tooltip: Full evidence summary on hover

What Is Knox-MS?​

Knox-MS is a custom AI model (knox/knox-ms) that provides truly unlimited context window length through an intelligent memory management system modeled after the human brain. Instead of being constrained by a fixed context window like every other LLM, knox-ms orchestrates multiple underlying models through a sophisticated Plan → Task → Memory architecture, dynamically managing what to remember, what to summarize, and what to forget.

Memory System (Brain) = Core Intelligence
  ↓ manages and updates
Context Cache (LLM) = Working Memory
  ↓ enhanced by
Vector Embeddings & Rerank (Tool) = Information Retrieval

The result: conversations and projects of any scale — from quick questions to multi-month development efforts — without ever losing context or hitting token limits.

1. Autonomous Orchestration Engine​

At the heart of knox-ms is a fully autonomous execution loop driven by LLM reasoning at every stage.

Goal Refinement​

When you send a request, the system doesn't just respond — it thinks. The autonomous core decomposes your input through LLM-driven goal analysis, breaking complex requests into a structured goal hierarchy. If a goal is ambiguous, the engine refines it by asking clarifying sub-questions internally before proceeding.

Multi-Task Planning​

Once the goal is understood, knox-ms generates a structured execution plan with 1 to 8 parallel or sequential tasks. Each task is classified by:

• Type — coding, analysis, research, general
• Difficulty — easy, medium, or hard
• Dependencies — which tasks must complete before others can start

If planning fails or produces a single monolithic task, the system gracefully falls back to a simpler plan rather than failing outright.

Smart Task System​

Tasks aren't static to-do items — they're adaptive execution units with:

• Priority calculation — dynamically computed based on dependencies, difficulty, and goal relevance
• Dependency graphs — tasks are ordered and parallelized based on their relationships
• Adaptive retry with model escalation — if a task fails on a simpler model, it automatically retries with a more capable one
• Difficulty upgrades — failed or low-quality tasks are re-queued at a higher difficulty tier

State Machine Loop Execution​

The autonomous engine runs as a state machine with defined checkpoints. At each iteration:

1. Execute the next batch of ready tasks
2. Self-evaluate quality using LLM-based assessment with confidence scoring
3. Apply corrections if quality is below threshold
4. Check goal completion through LLM-based achievement assessment
5. Refine the plan if the goal evaluation identifies additional needed tasks
6. Create a checkpoint for recovery
7. Repeat until the goal is fully achieved or iteration limits are reached

This architecture means knox-ms can handle complex, multi-step objectives autonomously — planning, executing, evaluating, correcting, and completing without human intervention at each step.

2. The Memory System — A Brain, Not a Buffer​

The Memory System is what makes knox-ms fundamentally different from every other AI model. It provides persistent, organized, self-optimizing memory that works exactly like a human brain.

Human-Like Memory Management​

Just as your brain doesn't store every detail of every conversation, knox-ms intelligently manages information through continuous CRUD operations:

• CREATE — New sessions, plans, task records, and discovered patterns are stored automatically
• READ — Only relevant memory is loaded into the working context for each request
• UPDATE — Task results, conversation history, and context summaries are continuously refined
• DELETE — Temporary files, redundant information, and stale cache entries are cleaned up automatically

Hierarchical Memory Organization​

Memory is organized in a tree-structured filesystem with multiple levels of detail:

Auto Memory Manager​

The automatic memory manager handles the lifecycle of every memory entry:

• Scoring — each memory entry receives a relevance score that decays over time following Ebbinghaus forgetting curves
• Retention policies — configurable retention periods with automatic cleanup of entries below the score threshold
• Deduplication — similar memories are detected and merged to prevent bloat
• Compression — older memories are compressed while preserving essential information
• Configurable limits — maximum entries, memory size caps, and cleanup intervals are all tunable

Summarization Engine​

Knox-ms includes both extractive and structured summarization capabilities:

• Extractive summaries pull out the most important sentences and facts from conversation history
• Structured summaries use LLM calls to produce organized, hierarchical summaries with key decisions, outcomes, and learned patterns
• Summaries are saved alongside raw history, providing both quick-access overviews and full detail when needed

3. Context Management — Unlimited by Design​

The context manager is the bridge between the Memory System and the LLM's working context window.

Multi-Level Context​

Rather than dumping everything into a single prompt, knox-ms maintains multiple context levels:

• Active context — the immediate working set for the current task
• Session context — broader session history and goals
• Cross-session context — knowledge and patterns from previous sessions
• Global knowledge — learned patterns and permanent knowledge base entries

Intelligent Compression​

When context grows beyond thresholds, the system applies progressive compression:

• Older turns are summarized while recent turns remain in full detail
• The compression ratio is configurable (from minimal to aggressive)
• Force-compression is available as a self-healing action when memory pressure is high

Context Cache Optimization​

Knox-ms structures memory to maximize LLM prompt cache hit rates:

1. Stable prefix — frequently-used context is placed in a consistent position to enable caching
2. Dynamic suffix — task-specific context that changes per request is appended after the stable portion
3. Smart invalidation — the cache is only invalidated when memory updates significantly, not on every change

This means you get the cost and speed benefits of prompt caching while maintaining truly unlimited context depth.

4. Knowledge Graph​

Knox-ms builds and maintains a persistent knowledge graph that captures relationships between concepts, code entities, decisions, and patterns.

Graph Structure​

• Knowledge nodes — individual facts, concepts, code patterns, or decisions
• Relations — typed connections between nodes (depends-on, related-to, supersedes, etc.)
• Relevance scoring — each node and relation carries a score that is updated based on access patterns and recency

LLM-Powered Knowledge Extraction​

When tasks complete, knox-ms uses LLM calls to perform structured knowledge extraction:

• Each piece of content is analyzed to produce knowledge entries with title, content, tags, and relevance score (0.0–1.0)
• Extracted knowledge is stored in the graph and made available for future context loading
• The system learns over time which types of knowledge are most useful

5. Vector Embeddings & Semantic Search​

Knox-ms includes a full vector embedding pipeline for semantic search over your project content and conversation history.

Embedding Pipeline​

1. Indexing — project files are chunked (2048 tokens with overlap) and embedded using VoyageAI models (voyage-3.5 for general content, voyage-code-3 for code)
2. Storage — embeddings are persisted to Knox storage with per-user scoping and lazy-loaded on first access
3. Search — queries are embedded and matched against stored vectors using cosine similarity
4. Reranking — top candidates are reranked using VoyageAI's rerank-2.5 model for fine-grained relevance

Resilient Embedding Service​

The embedding pipeline is built for production reliability:

• Retry with exponential backoff — failed API calls retry up to a configurable maximum with exponential delays (500ms × 2^attempt) plus jitter to avoid thundering herd
• Smart retry filtering — 4xx errors (except 429 rate limits) skip retries since they won't succeed on retry
• Circuit breaker — after 5 consecutive failures, the system opens a circuit breaker that fast-fails for 60 seconds, preventing cascading failures. A half-open state allows periodic probing, and the circuit closes after 3 consecutive successes
• Mock fallback — in development environments or when VoyageAI is persistently unavailable, deterministic mock embeddings ensure the system keeps functioning

TTL & Capacity Management​

Vector stores are actively maintained:

• TTL-based eviction — vectors older than the configured maximum age are automatically removed
• Capacity enforcement — per-user vector limits are enforced using LRU eviction (oldest-first)
• Cache cleanup — expired embedding cache entries are pruned during maintenance cycles
• Storage persistence — after any mutation, the cleaned state is persisted to Knox storage asynchronously

6. Self-Healing Infrastructure​

Knox-ms doesn't just recover from errors — it diagnoses and repairs itself through 12 distinct healing action types.

Healing Actions​

How Self-Healing Works​

When the autonomous loop encounters an error:

1. The system analyzes the error reason and selects the most appropriate healing action type
2. The selected action is delegated to the self-manager, which executes real system changes (not simulations)
3. The loop retries the failed operation with the healing applied
4. If healing fails, the system can escalate to more aggressive actions

All healing actions are executed through actual subsystem calls — model switching updates the relay's default model, cache clearing operates on the real context manager, and memory cleanup runs against the actual memory store.

Optimization Engine​

Beyond error recovery, the self-manager also applies 8 optimization types proactively:

• Performance optimizations based on observed execution patterns
• Resource optimizations to reduce memory and token usage
• Quality optimizations to improve output accuracy
• Latency optimizations for faster response times

7. Learning & Pattern Recognition​

Knox-ms learns from every execution and gets smarter over time.

What It Learns​

• Goal classification — recognizes what type of goal is being requested and suggests proven approaches
• Model performance tracking — records which models perform best for which task types and difficulties
• Task type success rates — tracks success/failure rates per task category to improve future planning
• Approach suggestions — before generating a new plan, the system consults learned patterns and applies model preferences and approach hints from past successes

How Learning Integrates​

During the autonomous execution loop:

• Before planning — the system calls the learning service to get approach suggestions based on the current goal, potentially influencing model selection and task decomposition
• After execution — success or failure is recorded with full execution data (all tasks, models used, token counts, latencies)
• Over time — the system builds a database of execution patterns that continuously improve planning quality

Memory Consolidation​

A background consolidation task runs periodically to strengthen long-term memory:

• Ebbinghaus decay — memory entries naturally decay in relevance over time, mimicking human forgetting curves
• Strengthening — frequently accessed or highly relevant memories are reinforced
• Deep summarization — old detailed memories are consolidated into compact summaries
• Knowledge graph updates — new relationships and patterns are integrated into the graph
• Vector store maintenance — TTL eviction, capacity enforcement, and cache cleanup run as part of each consolidation cycle

The consolidation system auto-detects whether knox-ms is available and runs only when the service is initialized — no manual configuration required.

8. Session Management​

Every knox-ms interaction is managed through a robust session system backed by Redis.

Session Features​

• Session state persistence — full session state is stored in Redis with automatic expiry
• Distributed locks — atomic lock acquisition using Lua scripts (SET key value NX EX ttl) prevents race conditions across multiple processes
• Safe lock release — ownership is verified atomically before releasing, preventing one process from accidentally releasing another's lock
• Lock extension — long-running operations can extend their lock TTL with ownership verification
• Atomic metrics — session metrics (iteration counts, task completions, etc.) use atomic Redis operations to prevent lost updates under concurrency

Checkpointing​

The autonomous engine creates checkpoints at configurable intervals during execution:

• Checkpoints are persisted to Knox storage via the storage integration
• They capture the full loop state, completed tasks, and current plan
• On recovery, execution can resume from the last checkpoint rather than starting over
• Checkpoint endpoints accept session-scoped queries with configurable limits

9. Real-Time Event Streaming​

Knox-ms provides full visibility into autonomous execution through Server-Sent Events (SSE).

21 Event Types​

The system streams typed events covering every phase of execution:

• Goal refinement events
• Planning and task creation events
• Task start, progress, and completion events
• Self-evaluation and correction events
• Healing action events
• Checkpoint creation events
• Execution completion events
• And more

Client Integration​

The frontend event service provides:

• Typed event handling — strongly typed handlers for all 21 event types
• Automatic reconnection — exponential backoff with jitter on connection loss
• Last-Event-ID support — seamless reconnection with event replay so no events are missed
• Auto-close — the connection automatically closes when execution_completed is received

10. Execution Analytics & History​

Every autonomous execution is persisted for long-term analytics and review.

What's Recorded​

Each execution stores:

• Execution record — execution ID, session ID, goal, refined goal, goal breakdown, loop state, iterations, config snapshot, final response, timestamps, and cancellation info
• Task records — individual task ID, plan ID, type, difficulty, status, result/error, token usage, and execution time
• Aggregate metrics — 14 metrics across 6 categories: performance, tokens, memory, quality, resilience, and latency

Analytics API​

Two dedicated endpoints provide historical insights:

• Execution history — paginated query with status filtering, returning executions with computed duration
• Aggregated analytics — total/successful executions, success rate, total tasks, per-metric-type averages, and per-task-type success rates

11. Relay Integration & Model Routing​

Knox-ms doesn't use a single model — it dynamically routes requests through the Knox relay infrastructure.

How Routing Works​

1. Channel selection — the system selects the best available channel based on model requirements and availability
2. Adaptor pipeline — requests flow through channel selection → adaptor → convert_request → do_request → do_response
3. Full billing integration — every relay call is metered and billed correctly
4. Dynamic model switching — the default model can be changed at runtime (e.g., by self-healing actions that switch to a more reliable model)
5. Graceful degradation — if the primary relay path fails, fallback mechanisms ensure the request still completes

This means knox-ms can use any model you prefer as its underlying engine while wrapping it with the full memory, planning, and self-healing infrastructure.

12. Storage Architecture​

Knox-ms uses a dual-storage architecture for durability and performance.

Knox Storage (Persistent Storage)​

• User and session-scoped memory files
• Plan and task storage
• Summaries and indexes
• Vector embeddings (per-user stores at knox-ms/vectors/user_{id}/store.json)
• Execution checkpoints
• Task results and execution summaries

Redis (Fast State)​

• Session state and distributed locks
• Set-based data structures using native Redis sets (SADD, SMEMBERS, SREM)
• Atomic operations via Lua scripts for race-free state management
• Metric counters with atomic increment
• Local cache fallback when Redis is unavailable

13. Configuration & Admin Controls​

Every aspect of knox-ms is configurable through validated API endpoints.

Autonomous Engine Config​

Control the execution loop behavior:

• max_iterations (1–1,000) — maximum autonomous loop iterations
• max_execution_time_secs (10–86,400) — execution time limit
• goal_confidence_threshold (0.0–1.0) — minimum confidence to consider a goal achieved
• max_healing_attempts (0–20) — how many self-healing attempts before giving up
• max_parallel_tasks (1–50) — concurrency limit for parallel task execution
• context_window_size (1K–10M) — working context window size
• checkpoint_interval (1–100) — how often to create recovery checkpoints

Context Config​

Fine-tune context management:

• active_context_window (1K–10M tokens)
• compression_ratio (0.01–1.0)
• hierarchy_levels (1–10)
• retrieval_top_k (1–100)
• relevance_threshold (0.0–1.0)
• cross_session_max_age_days (1–3,650)
• max_graph_entities (100–1M)

Memory Config​

Tune the auto memory manager:

• max_context_tokens (1K–10M)
• summarize_trigger_tokens (100–1M)
• knowledge_retention_threshold (0.0–1.0)
• cleanup_threshold_days (1–3,650)
• dedup_similarity_threshold (0.0–1.0)

User Preferences​

Individual users can customize their autonomous execution experience:

• Maximum iterations and time limits
• Confidence thresholds
• Checkpoint intervals
• All validated with the same bounds as admin configs

All configuration is persisted to the database and loaded at startup, so your settings survive restarts.

14. Frontend Experience​

The knox-ms frontend provides a rich, interactive interface for interacting with and managing the memory system.

Visual Architecture​

• Brain Memory Architecture — a visual representation of the memory system's hierarchical structure
• Knox-MS Panel — the main interaction panel available in both Chat and Code views
• Vector Search UI — search and explore your project's semantic embeddings
• Session Manager — view, manage, and switch between active sessions
• Memory Explorer — browse the memory tree, inspect entries, view scores and retention
• Task System UI — monitor active tasks, view plans, and track execution progress

Autonomous Settings​

Users can configure their autonomous execution preferences directly from the UI:

• Settings are loaded on component mount from the backend
• Changes are saved to the backend with real-time validation
• Toast notifications confirm successful saves or report errors

Localization​

Full internationalization support with i18n strings for all knox-ms UI elements.

15. Type Safety Across the Stack​

Knox-ms maintains end-to-end type safety from backend to frontend.

Automated Type Synchronization​

A synchronization script automatically generates TypeScript interfaces from the Rust backend:

• 19 interfaces and 2 union types are generated from 5 backend source files
• Type mappings handle all Rust → TypeScript conversions: String → string, Option<T> → T | null, Vec<T> → T[], HashMap<K,V> → Record<K,V>, and more
• Rust doc comments (///) are preserved as TypeScript JSDoc (/** */)
• The script is configurable — adding new types is as simple as adding entries to a source list

This ensures the frontend and backend never drift out of sync on data structures.

API Quick Reference​

Model Identity​

{
  "id": "knox/knox-ms",
  "object": "model",
  "owned_by": "KnoxChat",
  "context_length": -1
}

A context_length of -1 means unlimited — there is no upper bound.

Special Parameters​

35+ REST Endpoints​

Knox-ms exposes a comprehensive REST API covering:

• Autonomous execution management (start, cancel, status, history, analytics)
• Memory operations (explore, search, cleanup)
• Knowledge graph queries
• Vector search and indexing
• Session management
• Checkpoint operations (list, restore, delete with proper session scoping)
• Configuration management (engine, context, memory, user preferences)
• Real-time event streaming (SSE)

Summary​

Knox-MS is not an incremental improvement — it's a fundamentally new approach to AI interaction. By combining autonomous orchestration, brain-like memory, self-healing infrastructure, continuous learning, and production-grade reliability, knox-ms delivers something no other model can: truly unlimited context with intelligence that grows over time.

Every subsystem is fully operational, battle-tested, and ready for production workloads. Whether you're using knox-ms for a quick question or a months-long development project, the system remembers, learns, adapts, and improves with every interaction.

Start using Knox-MS today — select knox/knox-ms as your model and experience unlimited context for yourself.

>>> Knox-MS Unlimited Context Theorem

Knox Memory System (knox-ms) is a revolutionary custom AI model that provides truly unlimited context window length through an intelligent memory management system that operates like a human brain. Unlike traditional LLMs constrained by fixed context windows (even with context caching), knox-ms breaks through these limitations by orchestrating multiple underlying models through a sophisticated Plan-Task-Memory architecture.

Key Innovation: Unlimited Context​

The core breakthrough of knox-ms is its Memory System - a dynamic, self-updating knowledge base stored in Knox Server System(KSS) that:

• Functions as a Brain: Maintains persistent, organized memory across unlimited conversations
• CRUD Operations: Creates, Reads, Updates, and Deletes memory entries dynamically based on task execution
• Context Cache Integration: Works seamlessly with LLM context caching (prompt caching) to optimize performance
• Intelligent Memory Management: Automatically determines what to remember, update, or forget
• Hierarchical Organization: Structures information from recent (full detail) to long-term (semantic summaries)

Architecture Philosophy​

Memory System (Brain) = Core Intelligence
  ↓ manages and updates
Context Cache (LLM) = Working Memory
  ↓ enhanced by
Vector Embeddings & Rerank (Tool) = Information Retrieval

How It Works:

1. Memory System acts as the persistent brain, storing all session history, plans, and learned patterns
2. LLM Context Cache serves as working memory, optimized by the Memory System's intelligent loading
3. Vector Embeddings provide tools for retrieving relevant user project context when needed

This architecture allows knox-ms to handle conversations and projects of any scale - from short queries to multi-month development projects - without losing context or hitting token limits.

Model Identity​

{
    "id": "knox/knox-ms",
    "object": "model",
    "created": 1735689600,
    "owned_by": "KnoxChat",
    "root": "knox/knox-ms",
    "parent": null
}

Architecture​

Core Components​

1. Plan Model​

• Default Model: Your preferred model
• Purpose: Analyzes input and creates structured execution plans
• Output: Task breakdown with difficulty classification (easy/medium/hard)
• Responsibilities:
  • Parse user intent
  • Decompose complex requests into atomic tasks
  • Assign difficulty levels
  • Define execution order and dependencies

2. Task/TODO System​

Acts as an intelligent task manager that:

• Categorizes tasks by type (coding vs. non-coding)
• Routes tasks to appropriate models based on complexity
• Tracks task progress and status
• Manages task dependencies and execution order
• Records task completion and outcomes

Task Difficulty Matrix:

3. Vector Embeddings & Rerank System​

The Vector Embeddings & Rerank system provides intelligent context retrieval from user projects by:

• Purpose: Converts user project files, code, and documentation into semantic vectors for intelligent retrieval
• Scope: Separate from Memory System - focuses on user's project content, not knox-ms internal state
• Components:
  • Embedding Generation: Processes user project files into vector embeddings
  • Vector Store: Stores embeddings with metadata (file paths, timestamps, types)
  • Semantic Search: Retrieves relevant project context based on user query
  • Rerank Engine: Re-orders search results by relevance using advanced ranking algorithms

Vector Embeddings Pipeline:

1. Indexing Phase (on-demand or background):

   User Project Files → Chunking → VoyageAI Embedding API → Vector Store
   

   • Supported file types: .rs, .py, .js, .ts, .md, .json, .yaml, etc.
   • Chunk size: 2048 tokens with overlap
   • Embedding models: voyage-3.5 (general) or voyage-code-3 (code-focused projects)
   • API: VoyageAI Embeddings API

2. Retrieval Phase (per request):

   User Query → VoyageAI Query Embedding → Semantic Search → Top-K Results → VoyageAI Rerank → Final Context
   

   • Top-K: 20-50 candidates from vector search
   • Rerank model: rerank-2.5 (VoyageAI)
   • Final selection: Top 5-10 most relevant chunks

Integration with Knox-MS:

• Feeds relevant project context to Plan Model for better task decomposition
• Provides code examples and patterns for Code Model Selection
• Keeps project-specific knowledge separate from knox-ms internal memory

Storage Structure:

KSS://knox-ms-vectors/
├── projects/
│   ├── {project_id}/
│   │   ├── metadata.json       # File index and metadata
│   │   ├── chunks/
│   │   │   ├── file_{hash}.json  # Chunked content with embeddings
│   │   └── index/
│   │       └── stats.json      # Indexing statistics
│   └── cache/
│       └── queries/            # Cached query results

4. Memory System​

The Memory System is the brain of knox-ms - providing truly unlimited context through intelligent, dynamic memory management:

Core Concept: Human-Like Memory​

Just as human brains don't store every detail but organize and update memories based on relevance, knox-ms Memory System:

• Persistent Storage: KSS-based tree-structured filesystem for unlimited storage
• Dynamic CRUD Operations: Continuously Creates, Reads, Updates, and Deletes memory based on task execution
• Intelligent Organization: Automatically structures information hierarchically
• Context-Aware Loading: Loads only relevant memory into LLM context cache
• Self-Optimizing: Summarizes, compresses, and reorganizes memory to maintain efficiency

Memory vs. Context Cache vs. Vector Embeddings​

The Memory System manages what goes into the LLM's context cache, while Vector Embeddings serve as a tool to fetch relevant project information.

Dynamic CRUD Operations​

The Memory System performs intelligent CRUD operations based on Task execution:

CREATE (New Information):

• New session initialized → Create session/{session_id}/ directory
• New plan generated → Create plans/plan_{id}.md
• New task started → Create tasks/task_{id}.md
• Pattern discovered → Create entry in knowledge/patterns/

READ (Context Loading):

• Load recent conversation from history.md
• Retrieve relevant plans from plans/
• Fetch task status from tasks/status.md
• Query knowledge base for similar past situations

UPDATE (Memory Refinement):

• Task completed → Update tasks/task_{id}.md with results
• Conversation continues → Append to history.md
• Context grows large → Update context.md with summary
• New pattern matches existing → Update knowledge/patterns/

DELETE (Memory Optimization):

• Old temporary files → Delete after summarization
• Redundant information → Remove duplicates
• Failed/cancelled tasks → Archive or delete
• Stale cache entries → Invalidate and remove

KSS Storage Structure​

• KSS Storage: Stores all knox-ms interactions in a tree-structured filesystem on KSS

• Markdown Files: All content persisted as .md files for human readability

• Dynamic Updates: Automatically updates related files based on Plan and Task system execution

• Separation of Concerns:

  • Memory System (KSS): Knox-ms internal state, session history, task records, plans
  • Vector Embeddings: User project content, code, documentation for context retrieval

• Hierarchical Organization (KSS):

• Hierarchical Organization (KSS):

  KSS://knox-ms-memory/
  ├── sessions/
  │   ├── {session_id}/
  │   │   ├── context.md          # Current conversation context (optimized for cache)
  │   │   ├── history.md          # Full conversation history
  │   │   ├── tasks/
  │   │   │   ├── task_{id}.md    # Individual task records
  │   │   │   └── status.md       # Task status tracker
  │   │   └── plans/
  │   │       └── plan_{id}.md    # Execution plans
  │   └── summary.md              # Session summary
  ├── knowledge/
  │   ├── code/                   # Code-related memory
  │   ├── concepts/               # Learned concepts
  │   └── patterns/               # Recognized patterns
  └── index.md                    # Global memory index
  

Context Cache Integration​

The Memory System intelligently manages LLM context caching (prompt caching):

Optimization Strategy:

1. Cache-Optimized Loading: Structure memory to maximize cache hit rates
2. Stable Prefix: Keep frequently-used context in stable prefix for caching
3. Dynamic Suffix: Append task-specific context that changes per request
4. Cache Invalidation: Smart invalidation when memory updates significantly
5. Token Management: Balance between cache reuse and fresh context

Memory → Cache Flow:

Memory System (KSS) 
  → Load relevant memory 
  → Structure for optimal caching 
  → LLM Context Cache (prompt cache)
  → Execute task
  → Update Memory with results
  → Optimize cache structure

Benefits:

• Cost Efficiency: Reuse cached context across requests (lower input token costs)
• Speed: Faster response times with cached prompts
• Unlimited Scale: Memory System handles what cache cannot (unlimited history)
• Intelligence: Memory System decides what to cache vs. what to summarize

Memory Capabilities​

• Context Optimization: Continuously refines and summarizes past interactions
• Relevant Retrieval: Extracts only necessary context for current tasks
• Loop Integration: Updates continuously during task execution
• KSS Benefits: Scalable, durable, cost-effective storage for unlimited memory
• Cross-Session Learning: Share knowledge across different sessions
• Pattern Recognition: Identify and remember recurring patterns

API Specification​

Model Object​

{
    "architecture": {
        "input_modalities": ["file", "image", "text"],
        "modality": "text+image->text",
        "output_modalities": ["text"],
        "tokenizer": "Claude"
    },
    "context_length": -1,
    "created": 1735689600,
    "id": "knox/knox-ms",
    "object": "model",
    "owned_by": "KnoxChat",
    "parent": null,
    "permission": [
        {
            "allow_create_engine": true,
            "allow_fine_tuning": false,
            "allow_logprobs": true,
            "allow_sampling": true,
            "allow_search_indices": false,
            "allow_view": true,
            "created": 1735689600,
            "group": null,
            "id": "modelperm-knox-ms-001",
            "is_blocking": false,
            "object": "model_permission",
            "organization": "*"
        }
    ],
    "pricing": {
        "completion": "0.000015",
        "image": "0",
        "input_cache_read": "0.0000003",
        "input_cache_write": "0.0000015",
        "internal_reasoning": "0",
        "prompt": "0.000003",
        "request": "0",
        "web_search": "0"
    },
    "root": "knox/knox-ms",
    "supported_parameters": [
        "enable_vector_search",
        "include_reasoning",
        "max_tokens",
        "memory_mode",
        "project_id",
        "reasoning",
        "rerank_threshold",
        "response_format",
        "session_id",
        "stop",
        "structured_outputs",
        "temperature",
        "tool_choice",
        "tools",
        "top_k",
        "vector_top_k",
        "verbosity"
    ],
    "top_provider": {
        "context_length": -1,
        "is_moderated": true,
        "max_completion_tokens": 128000
    }
}

Special Parameters​

Request Format​

{
    "model": "knox/knox-ms",
    "messages": [
        {
            "role": "user",
            "content": "Build a REST API with user authentication"
        }
    ],
    "temperature": 0.7,
    "max_tokens": 8192,
    "stream": true,
    "session_id": "session_abc123",
    "project_id": "proj_xyz789",
    "enable_vector_search": true,
    "vector_top_k": 30,
    "rerank_threshold": 0.5,
    "memory_mode": "full",
    "include_reasoning": true
}