Project JARVIS

Problem

My working context is scattered across markdown notes, codebases, project folders, browser tabs and half-finished ideas. A general-purpose AI assistant can answer questions, but it starts cold each time. The real cost is not generating an answer. It is repeatedly explaining what I am working on, where the relevant files live, and why a decision was made.

JARVIS began as an attempt to remove that repeated setup without turning private working material into another hosted data source. The constraint was important: the system had to be useful with personal context while keeping the underlying files and retrieval loop local.

Product thesis

A personal AI system should not behave like a louder chatbot. It should behave like a quiet layer over the work already happening: understand the current project, retrieve the right context, show where that context came from, and get out of the way when it is not needed.

That led to three product requirements:

• Context before conversation. The system should gather relevant project material before asking the user to restate it.
• Local by default. Personal files, notes and code should remain inside the local workflow unless explicitly moved elsewhere.
• Useful under interruption. The retrieval experience should help someone resume work after a break, not only answer a perfectly phrased question.

Architecture

JARVIS is organized as a local ingestion and retrieval loop rather than a single chat screen.

1. Sources. Markdown notes, project documentation, codebase structure and selected local files provide the working material.
2. Ingestion. A file watcher and parser normalize the source material into smaller, searchable documents while preserving paths and basic provenance.
3. Embeddings. Documents are converted into vector representations and stored in a local vector index so related material can be found by meaning, not only by filename.
4. Retrieval. A query is expanded with active project context, then matched against the local index. The system returns the most relevant chunks and their source locations.
5. Generation. A local language model uses the retrieved context to produce an answer, summary or next action. The model is an interpreter of the working set, not the source of truth.
6. Interface. The user can query the system, inspect the context it used, and continue back into the underlying project files.

The separation matters. It makes the system easier to debug because ingestion, retrieval and generation can be tested independently. It also makes the privacy boundary visible instead of hiding everything behind a chat input.

Process

I started by mapping where project context actually lives, then tested local models for a workable balance between response quality, memory use and latency. The first pass focused on markdown and codebase structures because they are consistent enough to index and valuable enough to make resuming work easier.

From there, I built the ingestion path, added local embeddings and connected retrieval to a simple query interface. Each iteration was evaluated against real questions from active projects rather than synthetic prompts. The useful test was not whether the system could produce fluent text. It was whether it could surface the right file, decision or project detail without a long preamble.

Current outcome

The current build can index local project material and retrieve relevant context for active work. It already gives me a faster starting point for research, debugging and planning because I spend less time reconstructing what a project contains before I can work on it.

The result is intentionally not presented as finished. Retrieval quality still depends on consistent source structure, and latency is more noticeable than the base model’s intelligence. The next milestone is an incremental indexing loop with stronger source citations, retrieval benchmarks and a clearer distinction between remembered context and newly retrieved context.

What I learned

The difficult part is not adding a model. It is designing the context boundary around the model.

A useful personal AI system needs disciplined source material, predictable retrieval and enough provenance to let the user trust or challenge an answer. If those pieces are weak, a more capable model mostly produces more confident noise.