Cortex — Multi-Model Orchestration & Evaluation for DeepSeek-Harness

![npm version] ![npm downloads]

English | 中文 | npm package | GitHub

Cortex is a DeepSeek-Harness plugin that turns the host's primary (strong) model into a high-leverage supervisor. The supervisor plans, decomposes, verifies and accepts; routed executor models (small/cheap models such as Qwen3.8-9B) and tools do the heavy lifting. With a recursive task tree, dynamic routing, four-stage quality gates, budget control, failure recovery, policy reuse and per-dimension model evaluation, Cortex keeps quality stable while driving the unit cost of successful tasks down.

The supervisor is the host primary model that drives the cortex_* tools (e.g. deepseek-v4-pro in the bundled registry), and executors are the models the dynamic router picks per subtask (L1/L2 tiers, e.g. Qwen3.8-9B).

Suggested repository topics (GitHub): dsh-plugin · dsh · cordis · llm-orchestration · multi-model · model-routing · quality-gate

Highlights

Install & Auto-Enable

dsh plugin is the supported way: it installs the package into the profile, and because this package declares dsh.bundle.patch (= cordis.patch.yml), it is automatically appended to dsh.profile.bundles — enabled on next boot with no manual profile edits.

From npm (published — v0.2.0):

# install the plugin package as a dependency
yarn add dsh-cortex            # or: npm install dsh-cortex

# register it into a DSH profile and boot
npx @deepseek-ai/dsh plugin --profile web add dsh-cortex
npx @deepseek-ai/dsh web        # boot — the 13 cortex_* tools are live

From a local checkout (development):

cd <repo-path>                                   # the repo root
npx @deepseek-ai/dsh plugin --profile web add .  # anchors to this dir (or pass the absolute path)
npx @deepseek-ai/dsh web

Remove: npx @deepseek-ai/dsh plugin --profile web remove dsh-cortex.

The shipped bundle patch inserts the cortex row with uiEnabled: true, uiPort: 8788 (console listens on loopback only). stateDir and modelsFile use built-in defaults ($DSH_HOME/storages/cortex and the bundled config/models.yaml) — anything can be overridden by an - id: cortex config row in your profile's own cordis.patch.yml (it applies after bundle layers).

Typical Flow (agent view)

cortex_start → (cortex_strategy_match) → cortex_decompose T0 → cortex_execute [node_ids | recursive]
→ cortex_review (batch accept) → cortex_recover (on failures) → cortex_report (KPI) → cortex_policy (settle)

Operations & Commands

1. Agent tools (13 — the supervisor calls them autonomously)

2. Slash commands (/cortex <subcommand>)

3. Console REST (127.0.0.1:8788)

GET  /api/overview · /api/timeline?hours=all|24|168 · /api/rate · /api/models
     /api/matrix · /api/evaluations · /api/eval-cases · /api/capabilities
     /api/evaluate/status · /api/task/<id> · /api/events (SSE live)
POST /api/models/<id>            {enabled?: boolean, preferred?: boolean, reason?: string}   // reason = preference content (required on the page)
     /api/matrix/reset           (clear evaluation feedback)
     /api/eval-cases             {action: add|remove|save, ...}
     /api/capabilities           {name, description?}
     /api/evaluate               {models?, cases?, use_saved?} (real background run)
     /api/watchdog               {action: add-task|add-goal|pause|resume|remove|run|window-set|window-clear|ws-add, ...} (whitelisted)
GET  /watchdog · /api/watchdog/rules|runs?limit=50|targets|windows   (watchdog page, uses /api/events SSE)

4. Runtime lifecycle

• Mount: apply → registry load (static YAML + persisted overrides + model states) → heartbeat stateDir/mounted.json → optional console → usage-settle timer (60s, zombie sweep every 5 ticks) → dynamic model discovery.
• Per turn: orchestration-discipline context is injected per conversation session (suppressed only when that session has a flow active within 15 min); other sessions' leftover flows never suppress.
• Unmount / restart: effect dispose stops the console (active SSE connections destroyed — no hang) and clears timers; everything durable lives in the state dir (tasks/nodes/policies/profiles/model states/rate) and rebuilds on restart (last-write-wins).
• Self-healing: zombie tasks (running, 4h silent) are finalized automatically; evaluation feedback and drift baselines survive restarts; /cortex reset wipes state files for a fresh start (instant: store cleared, static registry reloaded, then DSH re-discovered); dynamic model discovery also refreshes automatically once a day (modelRefreshHours, default 24, 0=off).

Model Allocation (sub-agent pipeline)

• Sensitivity tiers (driven by qualitySensitiveOf): S (complex planning / arbitration / final review / correction — needs strong reasoning), M (planning / numeric-statistics / key evaluation / visual detail — weak models may repeat & vote), L (regular execution). Decision records carry sensitivity.
• Strongest fallback chain (strongestOf): configured strongestModel (must be active) → else derived from live profiles (tier → defaultQuality → measured quality → exec success rate). If no strongest model is configured, the current best available model IS the strongest — S-tier roles never stall; the mapping shifts dynamically as profiles update.
• Sub-agents per node: planAgent (cortex_decompose auto:true — strongest model generates children), reviewAgent (cortex_review auto:true — threshold auto-accept for regular nodes + review sub-agent for anomalies/high-sensitivity), fixAgent (config.fixAgentAuto — on escalation/upgrade the strongest model may deliver a fix or the corrected answer directly).
• Weak-model voting (repeat_vote, M/S tiers): the same model executes N rounds (2-5) and results are merged by agreement (pass/evaluate/arbitrate) — reliability of a strong model at ~1/100 cost.
• Deterministic statistics (V8.0 §5.1 L0 tier): statistics/counting nodes become "extract (model, vote/early-stop) → majority-merged records → engine computation" — the model only extracts records (EXTRACT_CONTRACT); counts/aggregations are computed by lib/engine/compute.js (zero cost, replayable), eliminating model-count drift; type:"compute" = L0 engine node (decompose accepts compute:{ops:[...]}, inferred from goal by default) — results carry quality_source:"engine" (E tier, auto-accepted without supervisor review).
• Pure-text statistics direct compute: word frequency / char / word / line counts — text is the data: the engine tokenizes and counts the source text directly (zero model calls, exact to each token, reproducible); entity-level stats still go extract+engine.
• V8.0 metrics: cortex_report adds node autonomy rate (autonomyRate), S-tier cost share (cost.sShare) and voting/engine stats (voting: repeat attempts/cost, engine computes, estimated saving); UI report cards updated.
• L0 capability library: AI-generated deterministic tools accumulate at runtime (stateDir/l0_capabilities.json, versioned) — execution checks first (id/name/keyword match → zero-cost engine run; explicit l0:"id" missing + allowed → strongest model generates code → sandbox verification (vm zero-trust / forbidden tokens / timeout / double-run determinism / sample+Gate0) → stored → executed; unverified code never runs); auto-disable after 3 consecutive failures; cortex_l0, /cortex l0, "L0 tools" UI tab.
• Auto mode: autoMode(task) — quality_requirement≥0.9 ∥ complexity≥0.8 ∥ risk≥0.8 → qualityFirst automatically (no manual mode).
• §7.1 KPIs: convergence rate / avg tree depth / cache hit rate (task-level + global, deterministic definitions); attempts now carry fromCache/cacheLevel (cache hits observable); no duplicate cache rows on hit (same-key idempotent).
• §4.13 learning chain ⑤ + §7.1 global model usage: distillation dataset stateDir/distill.jsonl — L3 (strongest tier) high-quality outputs (execution / multi-vote consensus / S-delegated plan·review·fix·L0 generation) auto-collected when quality ≥ distillMinQuality (0.9), capped at 500 rows, observable via report distill{rows,models,lastAt}; globalKpi models gains tierUsage/upgradeRate/fallbackRate.
• Domain-dimension profiles (§4.12 model×domain×task-type): domain_profiles[domain][taskType/cap] — EMA write-back from evaluation/Gate2 (same alpha, cross-restart deep-merge persistence); routing fallback chain domain → type-level → 7-dim mean → default (predictQuality(model, node, env.domain)); cortex_evaluate adds domain param (single-domain case set auto-inherits); cortex_models displays domain profiles.

Model Profiles (how they update)

• Evaluation write-back (cortex_evaluate / one-click UI run): per task-type quality EMA + success rate + cost/latency + 7-dim capability profiles (dims_profile), persisted in profile_overrides.json (drift baselines anchored at first measurement).
• Execution feedback: every real node execution (non-cache) updates deterministic counters — attempts/complete/failed + failure-type & task-type distributions, persisted in the same file; when samples >= 3 (EXEC_PROFILE_MIN_SAMPLES), routing uses the measured success rate instead of the static one (the more it is used, the truer the profile, the better the routing). Quality scores still come from evaluations only (Gate1 self-checks never pollute profiles).
• POST /api/matrix/reset or /cortex models reset restores static baselines (evaluation + execution feedback).
• Measured-only scores: the capability matrix shows a value only for verified capabilities (evaluation/execution write-back keys, tracked in measured); static declarations and defaultQuality never masquerade as verified scores — unverified cells display "no measured data".

Sub-Model Capabilities & Media Support

Executors (routed sub-models) are self-contained worker agents. What they can and cannot do:

Capability dimensions include media tags: image_analysis (declared by vision-capable models — image tasks filter by it) and audio_video_analysis (catalog dimension; no model declares it yet — matrix stays empty until a media-capable model/MCP is configured). Decompose image subtasks with capability: ['image_analysis'] (not document_analysis). Route-time closure: for nodes with image input, routing merges image_analysis into the need and applies an AND-hard-filter for media-exclusive capabilities — text-only models (even strong tiers) are excluded at routing (executor pre-check stays as the second line; force_models/preferred paths remain authoritative and call the user-chosen model).

Example flows: image analysis → cortex_execute on a node with image references (agent mode + read_image/MCP); screenshot/poster generation → Playwright MCP + artifacts declaration (magic-byte validated); audio/video processing → only with a media MCP configured (else explicit diagnosis, classified INPUT_FAILURE).

Preferred-model fallback: when a preferred executor (e.g. a text-only L1 model) is handed an image task, it runs first (attempt is recorded). The executor now pre-checks modality deterministically (modelAcceptsImage via host llm.resolveModelInfo): a model that does not declare image input fails fast with model lacks image modality (no generation — a text model must never be allowed to fabricate read-image evidence; observed in production: invented 800x600 / white-pixel result for a 1x1 red PNG). The recovery engine detects the gap and switches directly to a vision-capable candidate — no unnecessary decompose/upgrade round-trip.

Host prerequisite for vision: the vision model must declare inputModalities: [text, image] in the host's llm-deepseek.models entry (e.g. ~/.dsh/settings.yaml) — otherwise read_image refuses with "model does not declare image input" (the catalog default for undeclared entries is [text]).

Cost Model

• Executor models (sub-models): exact TokenUsage from subagent/chat sessions (billed = input + cache read/write), priced from the registry — true token accounting.
• Supervisor (primary model): real session-model usage via DSH sessionProjections/tokenMeter (the tokenUsage projection returns {totals:{uncachedInputTokens, outputTokens, cacheReadTokens, cacheWriteTokens}}; Cortex normalizes both nested and legacy flat shapes, so outputTokens is captured for accurate A-cost — fallback estimate only when no projection exists), plus a "vs direct primary-model run" baseline saving in cortex_report (85% cache-hit discounted).
• Single ledger (usage.jsonl): task cost / global KPI / timeline all derive from usage rows (multi-vote, sub-agent, Gate2 and supervisor calls included); session-level conversation ledger dedupes detail rows across buckets — including across restarts via the cumulative anchor (no double-count).
• Budget pools cover every cost: supervisor calls map to pools by kind (start/replan→plan, review/fix/evaluate→quality, recover/escalate→emergency), aggregate nodes→exec; the documented 90% high-cost lock is live (usage ≥90% → new L2/L3 excluded in routing, plus an exec-spend guard).
• Peak pricing unified: models.yaml peak windows (×2) apply to A and sub-models alike (peakPricing config, default on; Sundays/weekends off-peak).
• UI-triggered evaluations run without a supervisor (chat path) and are not counted as supervisor calls.

Development

npm test                 # node --test (engine/service/ui/watchdog unit + smoke) — 292 tests
node tools/verify-post-restart.mjs   # post-restart self-check (7 checks, incl. JSONL integrity + policy refs)
node tools/smoke-ui.mjs              # UI script syntax + SSE + model toggle

Repository Layout

app-pkg/index.js   entry forwarder (rollout cache-bypass, same pattern as dsh-tssdp)
lib/index.js       plugin entry (name/inject/Config/apply + lifecycle)
lib/service.js     CortexService facade (high-level operations behind the tools)
lib/tools.js       13 agent tools
lib/cli.js         /cortex command group
lib/ui-server.js   local web console (127.0.0.1:8788)
lib/watchdog.js    cross-workspace scheduled task injection + goal-loop remediation
lib/watchdog-cli.js /cortex watchdog command handlers
lib/engine/*.js    pure engines (fingerprint/router/budget/quality/recovery/store/executor/kpi/granularity/compute/l0/weight-learn/calibration/peak/state-reset)
config/models.yaml model registry (overridable)
test/              node --test suites
tools/             verification/dev scripts

License

MIT