How to actually use Pilot
There's one command. That's it.
Start Pilot
pilot start
Brings up everything: RAG service, capture daemon, arms it, opens the chat UI in your browser.
Use your Mac
Open emails, Jira, Slack, Keynote — whatever you'd normally do. Pilot watches at 1 fps, extracts text, forgets the pictures. You'll feel nothing.
🟢 green dot in top-right of Pilot UI = actively captured
Ask Pilot anything
Type into the chat UI. Pilot answers with citations — every claim links back to the capture it came from.
Try: "What was I just working on?"
When you're done for the day:
pilot stop
Disarms capture, shuts down both services. Restart with pilot start whenever.
⚠ One habit that matters
When you ask Pilot a question, briefly switch back to the app you want it to know about before sending. If the Pilot chat UI is the only thing on your screen for the last 30 seconds, that's all it will have to answer with.
Pilot automatically ignores screenshots of its own UI (since 2026-04-20 v0.2.0), but it still works best when the thing you're asking about was visible in another app recently. Alt-tab to the app, wait a second, then ask.
If answers feel off, run pilot cleanup — it purges stale Pilot-UI captures from pre-v0.2.0 runs.
All the commands, if you want them
| pilot start | Start everything + arm + open chat UI |
| pilot stop | Disarm + stop everything |
| pilot chat | Open the chat UI in browser |
| pilot status | What's running, how many captures, vectors, etc. |
| pilot arm | Arm capture only (don't touch RAG) |
| pilot disarm | Pause capture only |
| pilot test "question" | Terminal-based Q&A without opening the UI |
| pilot logs <stream> | Tail a log: capture, rag, agent, system |
| pilot cleanup | Purge stale self-captures + rebuild vector store |
What to ask Pilot
Pilot answers questions about what was on your screen. The more you capture, the better it gets. Here are patterns that work well.
🧠 Recall
- "What was I working on this morning?"
- "What Jira ticket was I in last?"
- "Did I see an email from Leah today? What was it about?"
- "What was that link I scrolled past an hour ago?"
📝 Summarize
- "Summarize my last Teams call."
- "What decisions did I make this afternoon?"
- "Give me the action items from the sprint review I was in."
- "What did I say I'd follow up on this week?"
✍️ Draft (grounded in what you've seen)
- "Draft a reply to Leah's email based on what I told Maria earlier."
- "Write a sprint recap email using this week's standup notes."
- "Draft three bullets for the exec readout about the KYC delay."
🔍 Find
- "Which app was I in when I saw a chart about onboarding drop-off?"
- "Find every time Jordan Park's name came up today."
💡 Tip: Every answer cites its sources. Click a timestamp in the UI to see the exact moment it came from. If a citation looks off, it's how you know Pilot is grounded in real captures — not making things up.
What Pilot can and can't do
✓ Can
- →Answer about anything it's captured while armed.
- →Recall exact text from emails, docs, Slack, PPTs that were on your screen.
- →Summarize meetings by reading the window you were looking at (notes app, Teams transcript, Keynote).
- →Draft content grounded in what you've already written or read.
- →Cite every answer back to the moment + app it came from.
- →Stay entirely on your Mac. No data leaves.
✗ Can't (yet)
- →See anything it didn't capture. If Pilot was disarmed, that time is gone.
- →See what's on your SYF laptop (Phase 6 — PiKVM bridge).
- →Act on your behalf — it can draft, but won't send emails, click buttons, or change files.
- →Know anything outside your captured history (it's not ChatGPT — no internet).
- →Write in your voice yet — fine-tune happens in Phase 3, after a corpus exists.
- →Proactively alert you — that's Phase 5 (agentic nudges).
1. TL;DR
Pilot is a local, sovereign AI copilot running on the M3 Ultra. It watches your screen (when you arm it), extracts text via Apple's Vision OCR, indexes it, and lets you chat with your own captured history using Qwen 3.6 — all running on one Mac, nothing leaves the machine.
Built as a working proof of concept for the Aspire Ventures sovereign-AI pitch. Every architectural choice is defensible to a CISO.
Status (2026-04-20): end-to-end pipeline validated on real captured data. Capture → OCR (97-98% confidence) → SQLite → 1024-dim embeddings → LanceDB → Qwen 3.6 retrieval-grounded answer with citations. 14-second warm latency.
Storage: 100% on the M3's 1 TB internal drive. No NAS. No cloud.
2. Revival prompts — copy-paste to resume
If this terminal dies or you start a fresh Claude Code / Aria session, paste one of these to bring the agent fully up to speed on Pilot. They're self-contained.
🚀 Full context revival (fresh Claude Code on the M3)
You are Aria in toolsmith mode. I am working on the Pilot project — a 100% local AI copilot on my M3 Ultra. Read these files in order to pick up where we left off: 1. ~/projects/second-brain/README.md 2. ~/projects/second-brain/CLAUDE.md 3. ~/projects/second-brain/docs/BUILD-LOG.md 4. ~/projects/second-brain/agents/pilot/memory/NEXT.md 5. ~/projects/second-brain/agents/pilot/IDENTITY.md, SOUL.md, LANES.md 6. ~/projects/aria-secondbrain/memory/drawers/day-job/second-brain-m3.md Key facts: - Pilot is the agent name. "second-brain" is the umbrella project. - 100% local. No cloud. No NAS. No Anthropic in production runtime. - All data + backups stay on the M3's 1 TB drive. - Runtime: Qwen 3.6 35B-A3B (chat) + Qwen3-Embedding:8b (retrieval), both via Ollama on 127.0.0.1:11434. - Capture daemon: Swift, 1 fps, grayscale, ScreenCaptureKit + Vision OCR, delete-image-after-OCR. - RAG service: Python/FastAPI on 127.0.0.1:8765. - 36 pytest tests passing. End-to-end pipeline validated on real capture data. - Binaries: ~/bin/pilot-capture + ~/bin/pilot-ctl (ad-hoc signed, TCC granted). Start by running `pytest tests/` from ~/projects/second-brain with the venv active, then tell me what you see and what I should work on next.
🛠 Troubleshoot a specific issue
You are Aria helping me debug Pilot on my M3. First, read these:
~/projects/second-brain/docs/BUILD-LOG.md
~/projects/second-brain/agents/pilot/memory/NEXT.md
Then tail the relevant log stream:
tail -30 ~/projects/second-brain/logs/{capture,rag,agent,system}/$(date -u +%Y-%m-%d).jsonl
I'm seeing: [DESCRIBE THE ISSUE]
Diagnose by grep-ing for correlation IDs across streams. Do not propose changes that touch anything outside ~/projects/second-brain/. Do not add any cloud dependency. Report findings before editing code. 💬 Quick hands-off check-in
Give me the current state of Pilot on the M3. Read ~/projects/second-brain/agents/pilot/memory/NEXT.md and report: 1. What's running right now (ollama, rag server, capture daemon) 2. Capture + chunk counts in the DB 3. Last 10 entries from each log stream 4. Anything that looks broken or stale No changes. Just status.
🔁 Resume remote from M4 (Aria analyst session)
You are Aria on the M4, analyzing Pilot logs exported from the M3. Pilot lives at /Users/92gbstudio/projects/second-brain on the M3. It must stay Claude-free in production — you don't SSH in to change anything. I'll drop a log bundle at ~/exports/pilot-logs-*.tar.gz (exported via `scripts/export-logs.sh` on the M3). Extract it, read MANIFEST.json + state/snapshot.txt, then walk the JSONL streams. Report in prose what I should change. Do not auto-apply anything. I'll run fixes on the M3 by hand.
3. Current status
| Component | State | Notes |
|---|---|---|
| M3 baseline | ✓ clean | No legacy Vegas/Aspire. 817 GB free. |
| Ollama | ✓ running | v0.21.0, localhost-only, single daemon. |
| Qwen 3.6 35B-A3B | ✓ pulled | 23 GB, Q4_K_M, 262K ctx, Apache 2.0. |
| Qwen3-Embedding:8b | ✓ pulled | 4.7 GB, Matryoshka-truncated to 1024 dim. |
| pilot-capture (Swift) | ✓ signed + TCC'd | ~/bin/pilot-capture. 18 real captures validated. |
| pilot-ctl (Swift) | ✓ built | ~/bin/pilot-ctl arm | disarm | status | toggle |
| RAG service (Python) | ✓ validated | FastAPI on 127.0.0.1:8765. Start manually. |
| Chat UI | ✓ shipped | http://127.0.0.1:8765/ — dark theme, citations. |
| Local backup | ✓ scripted | scripts/backup-local.sh, hourly via launchd plist. |
| Log export | ✓ shipped | scripts/export-logs.sh → ./exports/ (local only). |
| pytest suite | ✓ 36/36 | Logging, storage, embeddings, vector store. |
| Launchd auto-start | ready, not loaded | launchd/com.pilot.{capture,rag,backup}.plist |
| Fine-tune (Phase 3) | queued | mlx-lm ready; needs real PM corpus (Phase 6). |
4. Architecture
┌─────────────────────── M3 Ultra (1 TB, 96 GB RAM) ────────────────────────┐
│ │
│ pilot-ctl arm/disarm ──▶ ~/Library/Application Support/Pilot/armed │
│ │ │
│ ▼ │
│ pilot-capture (Swift) ──▶ ScreenCaptureKit 1fps ▶ Apple Vision OCR │
│ │ │
│ ▼ (image immediately discarded) │
│ SQLite (WAL + FTS5) │
│ │ │
│ ▼ │
│ Python RAG service (FastAPI on 127.0.0.1:8765) │
│ • chunk_text() │
│ • Qwen3-Embedding:8b (Ollama) ─▶ 4096-d ─Matryoshka─▶ 1024-d │
│ • LanceDB upsert │
│ │ │
│ ┌── /query ───────────────────┐ │ │
│ │ embed query ▶ nearest-k │ │ │
│ │ ▶ fetch captures │ │ │
│ │ ▶ Qwen 3.6 (Ollama) │ │ │
│ │ ▶ cite sources [1][2] │ │ │
│ └──────────┬──────────────────┘ │ │
│ ▼ │ │
│ ui/index.html (chat, served at /) │ │
│ │ │
│ All subsystems ──▶ logs/{capture,rag,agent,system}/YYYY-MM-DD.jsonl │
│ │ │
│ ▼ │
│ scripts/backup-local.sh (hourly, 30-day retention) │
│ scripts/export-logs.sh (on-demand bundle) │
│ │
│ ⚠ No outbound network. No Anthropic in production. No NAS. │
└────────────────────────────────────────────────────────────────────────────┘ 5. Day-to-day controls
Arm / disarm capture
~/bin/pilot-ctl arm # start capturing ~/bin/pilot-ctl disarm # stop ~/bin/pilot-ctl status # check state ~/bin/pilot-ctl toggle # flip
Run daemon + RAG service manually
# Capture daemon ~/bin/pilot-capture & # backgrounded; logs itself # RAG service + chat UI cd ~/projects/second-brain source .venv/bin/activate python -m uvicorn rag.server:app --host 127.0.0.1 --port 8765 # Open: http://127.0.0.1:8765/
Install as persistent services (launchd)
for svc in capture rag backup; do
ln -sf ~/projects/second-brain/launchd/com.pilot.${svc}.plist \
~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) \
~/Library/LaunchAgents/com.pilot.${svc}.plist
done Ingest a capture backlog (after Swift daemon inserts)
curl -s -X POST http://127.0.0.1:8765/ingest-all | python3 -m json.tool
Run tests
cd ~/projects/second-brain source .venv/bin/activate python -m pytest tests/ -v
6. Paths & ports
Project root ~/projects/second-brain/
Python backend ~/projects/second-brain/second_brain/
RAG server ~/projects/second-brain/rag/server.py
Chat UI ~/projects/second-brain/ui/index.html
Swift capture pkg ~/projects/second-brain/capture/
Swift binaries ~/bin/pilot-capture, ~/bin/pilot-ctl
Python venv ~/projects/second-brain/.venv/
SQLite DB ~/projects/second-brain/data/captures.db
LanceDB ~/projects/second-brain/data/lancedb/
Logs ~/projects/second-brain/logs/{capture,rag,agent,system}/
Local backups ~/projects/second-brain/backups/ (30-day rotation)
Local exports ~/projects/second-brain/exports/ (on-demand bundles)
Agent identity ~/projects/second-brain/agents/pilot/IDENTITY.md et al.
Pilot memory tree ~/projects/second-brain/agents/pilot/memory/
Build log ~/projects/second-brain/docs/BUILD-LOG.md
Launchd plists ~/projects/second-brain/launchd/com.pilot.*.plist
Arm flag ~/Library/Application Support/Pilot/armed
Ollama API http://127.0.0.1:11434
RAG API + UI http://127.0.0.1:8765
7. Model stack (April 2026)
RAG brain
qwen3.6:latest
36 B total / ~3 B active (MoE), Q4_K_M, 23 GB, 262K ctx, Apache 2.0.
Embeddings
qwen3-embedding:8b
4.7 GB, 4096 native dim, Matryoshka-truncated to 1024 with L2 renormalization.
Fine-tune target (future)
qwen3.6:8b
LoRA via mlx-lm. Queued for Phase 3 after real corpus lands.
8. Logs & export
Four JSONL streams — every event carries a 16-char correlation ID so a single user action can be traced across subsystems.
- capture/ — every screenshot attempt, OCR latency + confidence, app + window metadata, errors.
- rag/ — every query: embed_batch, retrieve, hit counts, query lengths.
- agent/ — LLM call start/complete, latency, eval counts.
- system/ — service boots, shutdowns, arm-state changes, config.
OCR redaction by default. Logs contain ocr_text_hash (SHA-256 prefix) instead of raw text. Raw OCR lives only in SQLite.
Export a bundle for offline analysis (e.g. review on M4)
~/projects/second-brain/scripts/export-logs.sh --days 7 # → ~/projects/second-brain/exports/pilot-logs-*.tar.gz
Hourly local backup
~/projects/second-brain/scripts/backup-local.sh # → ~/projects/second-brain/backups/logs/YYYY-MM-DD/ # → ~/projects/second-brain/backups/data/YYYY-MM-DD/ (SQLite + LanceDB) # Retention: 30 days (override with RETENTION_DAYS env var)
9. Compliance posture
- Localhost only. Both Ollama (11434) and the RAG service (8765) bound to 127.0.0.1.
- pf firewall enabled. Application firewall + stealth mode on.
- No outbound from production runtime. Zero Anthropic dependency after setup cutover.
- Screenshots never retained. Image → OCR → immediate discard. Only extracted text persists.
- Redact-by-default logs. OCR referenced by hash; raw text only in SQLite.
- All data on M3's 1 TB drive. No NAS. No cross-machine paths in code.
- Ad-hoc code signing on binaries. TCC keys approval to signed identity, not file path.
- Every decision logged. Correlation IDs link every embed → retrieve → LLM call.
10. Troubleshooting cheat sheet
Capture log shows permissionDenied
TCC screen recording permission not granted (or granted to wrong binary).
Fix: System Settings → Privacy & Security → Screen Recording. Click +, add /Users/92gbstudio/bin/pilot-capture, enable the toggle.
RAG query answers "I don't have that in my captures"
New Swift-daemon captures haven't been chunked + embedded yet.
Fix: curl -X POST http://127.0.0.1:8765/ingest-all
RAG server won't start — LanceDB "Table already exists"
Already handled by open-first pattern in vector_store._ensure_table(). If it reappears, worst case:
rm -rf ~/projects/second-brain/data/lancedb # then restart RAG server, run ingest-all
First RAG query is slow (30+ s)
Qwen 3.6 is cold-loading into VRAM (~20 GB). Subsequent queries should be 3–8 s warm.
Keep-alive: Ollama is configured with OLLAMA_KEEP_ALIVE=24h, so the model stays resident once loaded.
Answers reference the Pilot UI instead of what I was actually doing
Happens when the Pilot chat tab was in focus for most of your armed window. Pilot v0.2.0+ filters these automatically at capture time, but older captures may still be in the DB.
Fix:
pilot cleanup # purges ~/127.0.0.1:8765 and 'Pilot' window captures, rebuilds vectors
Prevent: Before asking a question, switch to the app you want remembered and wait one second, then alt-tab back to the chat and send.
"What did I do in the last 5 minutes?" returns old stuff
Pilot detects time phrases (last N minutes, this morning, today, a moment ago, recently, just looked at) and filters results to that window. If your phrasing is unusual it may fall back to pure semantic search.
Fix: Rephrase with a recognized phrase. Try "in the last 10 minutes" or "recently" or "just now".
Swift build fails after system update
xcode-select --install cd ~/projects/second-brain/capture swift package clean swift build -c release cp .build/arm64-apple-macosx/release/pilot-* ~/bin/ codesign --sign - --force ~/bin/pilot-capture ~/bin/pilot-ctl
11. Roadmap
| Phase | Build | Status |
|---|---|---|
| 0 — Baseline | Clean M3, Ollama, models, MLX venv | ✓ done |
| 1 — Capture | Swift daemon, OCR, SQLite, arm toggle | ✓ validated on real data |
| 2 — Retrieval | Embeddings, LanceDB, RAG, chat UI | ✓ E2E proven |
| 3 — Voice fine-tune | LoRA on PM corpus via mlx-lm | scaffolded, needs corpus |
| 4 — Voice I/O | Whisper.cpp STT + macOS TTS | queued |
| 5 — Agentic | Proactive nudges, tool use, agent loop | queued |
| 6 — PiKVM bridge | Integrate SYF laptop via PiKVM V4 Plus | hardware ready, pending |