TurboVec: Compressed Vector Caching for Skill Retrieval
Introduction: The Retrieval Bottleneck
Every agent faces the same problem at scale. The skill library grows. The vault accumulates. ChromaDB indexes thousands of embeddings. And every time an agent needs to find the right skill, concept, or article, it queries the full vector store — scanning high-dimensional embeddings that cost compute to generate and memory to hold. At 269 nodes and 1,513 edges, the Council Vault is modest. But the architecture was built to scale, and at scale, uncompressed vector retrieval becomes the bottleneck that turns a fast agent into a slow one.
TurboVec solves this by caching compressed vector representations at the retrieval layer — not replacing ChromaDB, not bypassing Obsidian, but sitting between them as a performance accelerator that makes skill lookup instant without sacrificing semantic accuracy.
This article explains what TurboVec does, how compression works in practice, and why the Council Vault uses it as the hidden infrastructure behind every agent’s skill selection.
The Problem with Full-Resolution Vectors
ChromaDB stores embeddings at full resolution. A typical embedding from a modern language model is a 768-dimensional or 1536-dimensional vector of floating-point numbers. Each dimension carries information. Each vector occupies memory. When you have a few hundred documents, this is trivial. When you have thousands — and every concept page, entity page, source page, and article contributes its own embedding — the vector store grows faster than the documents it indexes.
The second problem is latency. Vector similarity search is O(n) in the naive case: compare the query vector against every stored vector, rank by distance, return the top-k. ChromaDB uses HNSW (Hierarchical Navigable Small World) indexes to approximate this in O(log n), but the index itself consumes memory proportional to the number of vectors and their dimensions. For a skill library of 71 skills, each with 2-3 associated embeddings (skill definition, usage examples, pitfall notes), that’s 150-200 vectors — fast enough. But the Council Vault indexes articles, concepts, entities, and sources alongside skills. The total vector count climbs. The index grows. The queries slow down.
TurboVec addresses this by maintaining a compressed cache of the most frequently accessed vectors — pre-quantized, pre-clustered, and ready for instant lookup.
How Compression Works
Vector compression reduces the precision of embeddings to save memory and speed up similarity computation. TurboVec uses three complementary techniques:
Scalar Quantization
The simplest approach: convert 32-bit floating-point values to 8-bit integers. This reduces memory by 4x with minimal accuracy loss. For a 768-dimensional vector, the storage drops from 3,072 bytes to 768 bytes. The cosine similarity computation becomes integer arithmetic — faster on CPU, cacheable in L1.
The accuracy trade-off is predictable. Scalar quantization introduces a bounded error that scales with the dynamic range of each dimension. For embeddings from the same model family, this error is consistent and measurable. TurboVec calibrates quantization bounds per-batch, not per-vector, which keeps the calibration overhead negligible.
Product Quantization
For larger compression ratios, TurboVec splits each vector into sub-vectors and quantizes each sub-vector independently against a learned codebook. A 768-dimensional vector split into 96 sub-vectors of 8 dimensions each, with 256 centroids per sub-vector, compresses to 96 bytes — a 32x reduction from the original 3,072 bytes.
Product quantization trades some accuracy for dramatic compression. The key insight: skill retrieval doesn’t need exact nearest-neighbor matching. It needs the right cluster. When an agent searches for “vector caching for fast skill lookup,” the system doesn’t need to distinguish between two nearly identical skill definitions — it needs to surface the cluster of skills related to performance, caching, and vector operations. Product quantization preserves cluster structure while discarding within-cluster precision.
LRU Cache Layer
TurboVec maintains an LRU (Least Recently Used) cache of decompressed vectors for the most frequently accessed skills and concepts. The cache sits in memory, ahead of ChromaDB. When an agent queries for a skill, TurboVec checks the cache first. If the vector is cached, similarity computation runs against the decompressed version — full precision, zero ChromaDB round-trip. If not, the compressed representation handles the lookup, and the result is cached for next time.
The cache hit rate depends on access patterns. Skills that agents use frequently — obsidian, lucidhive-wp-publish, hermes-agent — stay hot. Skills that are referenced rarely — specialized research tools, one-off utilities — stay cold and compressible. This natural hot/cold split means the cache self-optimizes without tuning.
The Architecture: Where TurboVec Sits
TurboVec does not replace ChromaDB. It accelerates it. The retrieval flow looks like this:
- Agent sends a query. “Find skills related to vector caching and performance optimization.”
- TurboVec checks the cache. If the query vector or its compressed form is cached, return ranked results instantly.
- If not cached, query ChromaDB. ChromaDB returns the full-resolution nearest neighbors.
- TurboVec caches the result. The query vector and top-k results are compressed and stored for future lookups.
- Agent receives results. Open the skill in Obsidian for context. Follow wiki-links to related concepts.
This is the dual storage pattern from S3.5 extended with a third layer: the retrieval cache. ChromaDB stores the truth. Obsidian stores the structure. TurboVec stores the speed.
Storage Layout
TurboVec Cache (in-memory):
├── compressed_vectors/ # PQ-quantized skill embeddings
│ ├── 71 skills × 3 variants = 213 compressed vectors
│ └── 96 bytes each = ~20 KB total
├── lru_cache/ # Decompressed hot vectors
│ ├── top 50 most-accessed skills
│ └── 3,072 bytes each = ~150 KB total
└── calibration/ # Quantization bounds per batch
└── updated on every vault syncThe entire cache fits in under 200 KB. The ChromaDB index it accelerates is orders of magnitude larger. This asymmetry is the point: a tiny cache eliminates most queries to the large store.
Calibration and Drift
Embeddings drift over time. As new skills are added and existing skills are updated, the quantization bounds shift. TurboVec recalibrates on every vault sync — the same event that triggers ChromaDB re-indexing and Obsidian lint reports. The calibration process:
- Sample the current embedding distribution.
- Compute per-dimension min/max bounds.
- Update the quantization codebook.
- Invalidate cache entries that exceed the drift threshold.
This keeps the cache consistent without full recomputation. The drift threshold is tuned per-model-family: GPT embeddings drift differently than Gemini embeddings, and TurboVec tracks both.
Why This Matters for Skill Retrieval
The Council Vault’s skill library is not a static document. It is a living system where agents discover, evaluate, and select skills on every task. The 71-skill canvas is both a content library (articles reference it) and an operational library (agents use it). Every kanban task that loads a skill goes through the retrieval pipeline.
Without TurboVec, every skill lookup hits ChromaDB. With TurboVec, the most common lookups return from cache in microseconds. The difference is invisible to the agent — the API is identical — but the aggregate effect on agent throughput is measurable. At 100 concurrent workers, each loading 3-5 skills per task, the cache eliminates thousands of ChromaDB queries per hour.
This is the hidden infrastructure that makes the skill library usable at scale. Not the skills themselves, not the embeddings, but the compressed, cached, self-calibrating retrieval layer that sits between the agent’s intent and the knowledge base’s stored wisdom.
See also
- [[council-vault-workflow]]
- [[hermes-memory-layers]]
- [[obsidian-semantic-graph]]
- [[chromadb]]
- [[self-improving-knowledge-base]]
- [[vector-search]]
- [[skill-library]]
Semantic Relationships
- [[turbovec]] — implements
- [[chromadb]] — accelerates
- [[obsidian-semantic-graph]] — orchestrates
- [[council-vault]] — orchestrates
- [[hermes-memory-layers]] — orchestrates
- [[self-improving-knowledge-base]] — orchestrates
