kb — personal information consolidation

Your plans, thoughts and notes are scattered across Claude.ai / ChatGPT web chats, Cursor & Claude Code conversations and plans, and ad-hoc folders. kb pulls all of it into one git-synced raw store (the source of truth) and maintains an LLM-organized, Obsidian-compatible knowledge vault on top — queryable with citations, synced across devices via a private GitHub repo.

sources ──▶ raw/ (append-only, content-addressed)  ──▶ organized/ (Obsidian vault)
            ├─ claude_code   ~/.claude/projects          ├─ SUMMARY.md (every dir)
            ├─ cursor        ~/.cursor/projects           ├─ topic/leaf.md  (full detail,
            ├─ claude_plans  ~/.claude/plans              │   front-matter `sources:` → raw_id)
            ├─ cursor_plans  ~/.cursor/plans              └─ [[wikilinks]] + #tags
            ├─ tracked_dir   any folder you register
            └─ export_zip    Claude/ChatGPT data export

The core invariant: ingest is append-only (commutative across devices → rebases cleanly); organize is destructive (rewrites organized/) and runs only on the designated organizer device (or behind an advisory lock).

Install

pipx install kb-consolidate                 # core: connectors + raw store + sync
pipx install 'kb-consolidate[organize]'     # + organizer & retrieval (litellm, fastembed, lancedb)
# local checkout: pip install -e . / pip install -e '.[organize]'

The core (ingest/sync/status) needs no heavy dependencies. The organizer and ask/search use an embedder + LLM; if those aren't installed or no API key is set, organize falls back to heuristic placement and ask to cited snippets, so the pipeline always runs.

The [organize] extras transitively install keyring. On headless/SSH machines a locked Secret Service could otherwise block; kb guards keyring with a timeout and a KB_NO_KEYRING=1 escape hatch (see Troubleshooting).

Quickstart

kb init --repo git@github.com:you/knowledge.git   # private repo recommended
kb connector add tracked_dir --path ~/code/project/.plans --name project-plans
kb config set-key nim                              # OS keyring or .kb/local.env; never committed
kb config set-model nim meta/llama-3.3-70b-instruct   # default provider is nim

kb ingest            # append new chats/plans/notes to raw/
kb organize --dry-run    # preview the staged vault diff
kb organize --apply      # write organized/
kb ask "what did I decide about the video codec bottleneck?"
kb sync              # pull → ingest → organize → commit → push (also via scheduler)
kb schedule install --interval 12h

How it works

• Raw store (raw/): every source artifact becomes one markdown file with YAML front-matter provenance (raw_id, source, cwd, git_branch, …). A content hash dedups; a stable raw_id lets organized leaves cite their sources. _manifest.jsonl is append-only; a grown chat is appended as a ## continued block, never rewritten.
• Organizer (hybrid): deterministic embedding routing finds candidate leaves; an LLM makes the judgment (append / rewrite / create / no-op), preserving full detail. SUMMARY.md is rolled up bottom-up for dirty branches only. A local cursor (raw_id → content_hash) makes runs diff-only.
• Sync: git pull --rebase → ingest → organize (organizer device) → commit → push. Secrets live in the OS keyring / .kb/local.env (gitignored); the tracked config/kb.yaml (model, schedule, prefs) propagates to all devices on pull.
• Auto-push (auto_push: true, default): a git post-commit hook pushes the store to origin in the background after every commit — sync, config set, or a manual git commit — logging to .kb/last-push.log. Needs a remote; toggle with kb config set auto_push false.
• Scheduling: native per-OS (systemd user timer / launchd / schtasks), so missed runs fire on wake.

Configuration

config/kb.yaml is tracked and propagates across devices:

organizer_device: aurora
llm:
  provider: nim               # nim | openrouter | openai | local  (default: nim)
  model: meta/llama-3.3-70b-instruct
  embedding: { provider: local, model: BAAI/bge-small-en-v1.5 }   # embeddings stay local
organize:
  leaf_max_bytes: 40000
  similarity_threshold: 0.78
  vault_format: obsidian
schedule: { interval_hours: 12 }
connectors:
  - { type: claude_code }
  - { type: cursor }
  - { type: tracked_dir, name: project-plans, path: ~/code/project/.plans, include: ["**/*.md"] }

API keys are never stored here — use kb config set-key <provider>.

Providers (model selection)

The organizer/ask calls go through LiteLLM, so any of these work. Embeddings stay local (fastembed) by default — free and no API round-trips.

NVIDIA NIM (default)

1. Get a key at https://build.nvidia.com — sign in, open a model (e.g. Llama 3.3 70B Instruct), click Get API Key (an nvapi-… key, with free credits).
2. Store it locally (never committed):

   kb config set-key nim          # paste nvapi-... at the hidden prompt

   Equivalent env var: KB_NIM_KEY=nvapi-.... The OpenAI-compatible endpoint (https://integrate.api.nvidia.com/v1) is wired automatically.
3. Choose the model — any id from the catalog, e.g.:

   kb config set-model nim meta/llama-3.3-70b-instruct
   # other good options: nvidia/llama-3.1-nemotron-70b-instruct,
   #                     deepseek-ai/deepseek-r1, qwen/qwen2.5-coder-32b-instruct

   set-model commits the change, so it propagates to your other devices on pull.

OpenRouter / OpenAI work the same way: kb config set-key openrouter then kb config set-model openrouter anthropic/claude-sonnet-4-6 (key env vars KB_OPENROUTER_KEY / KB_OPENAI_KEY).

Verify with kb doctor — it should show api key (nim) ✓ and your model. The organizer/ask need the [organize] extras installed (see Install); without them, organize falls back to heuristic placement and ask to cited snippets.

Shell completion

kb --install-completion        # writes the hook for your shell (bash/zsh/fish)
exec bash                      # reload (or: source ~/.bashrc)

Then <TAB> completes commands and, context-aware: connector types (kb connector add <TAB>), providers (kb config set-model <TAB>), tracked config keys (kb config set <TAB>), configured connector names (kb connector remove <TAB>), and common model ids. kb --show-completion <shell> prints the script without installing it.

Troubleshooting

kb organize looks stuck. On the first run the vault is empty, so the organizer makes one LLM call per ingested item — hundreds of items = many sequential calls. It now shows a live progress bar (item i/N + elapsed) and caps each call at 90s (KB_LLM_TIMEOUT). Do a first pass on a handful with kb organize --dry-run --limit 20, then drop the limit. (No model/key → fast heuristic placement instead.)

kb doctor / any command hangs. The optional keyring package (pulled in by the [organize] extras) can block on the D-Bus Secret Service over SSH or on a headless box with no unlocked keyring. kb now caps keyring calls at 3s and checks .kb/local.env and env vars first, so this shouldn't happen — but you can disable keyring entirely:

export KB_NO_KEYRING=1         # keys then come only from env / .kb/local.env

Your keys in .kb/local.env (gitignored, chmod 600) are used directly, so a blocked keyring never matters.

Command reference

Command	Description
kb init [--repo URL]	Scaffold/clone the store (raw/, organized/, config, gitignore, git).
kb connector add/list/remove	Manage source connectors.
kb ingest	Run connectors; append new items to raw/.
kb organize [--dry-run/--apply] [--rebuild] [--limit N]	Hybrid organizer (live progress); dry-run shows the diff.
kb ask "<q>" / kb search "<q>"	Cited answer / vector search over the base.
kb sync [--no-organize]	pull → ingest → organize → commit → push.
kb config set / set-model / set-key	Tracked config + local secrets.
kb import-export <zip>	Import an official Claude/ChatGPT data-export zip.
kb schedule install/status/uninstall	Native scheduled sync.
kb status / kb doctor	Counts/pending/lock; environment diagnostics.
kb --install-completion / --show-completion	Install/print shell completion.

Status

Phases 1–3 (local connectors, raw store, git sync, hybrid organizer, retrieval) are implemented and verified against real data. Deferred: the experimental web-chat cookie scraper (kb login), age encryption + secret scrubbing, and a LanceDB index backend (the numpy index is a drop-in today).