An arxiv paper drops at 8:00 PM Eastern. By 8:04 AM the next morning, it has been found, read, summarized, cross-referenced against three existing wiki concepts, grounded with inline citations, and fed into a decision about whether the Council’s research direction needs to shift. No human read the paper. No human needed to. The pipeline did the work — and the pipeline is the subject of this article.
S10.1 established prediction markets as sensors: prices that absorb information before headlines publish. S10.3 will cover blogwatcher as the RSS early-warning layer. This article sits between them, addressing the middle layer of the research stack: how raw academic papers become actionable intelligence through a structured pipeline of discovery, synthesis, and decision.
The discovery layer: arxiv as signal source
Arxiv is not a library. It is a firehose. More than 25,000 new papers appear on arxiv every month, spanning machine learning, physics, economics, quantitative biology, and a dozen other categories. The signal-to-noise ratio is brutal: for every paper that changes the state of a field, hundreds are incremental, duplicative, or simply wrong. The challenge is not accessing the papers — arxiv is open. The challenge is filtering them.
The arxiv skill handles this mechanically. A search query runs against the arxiv API. Results come back with title, authors, abstract, categories, and submission date. The skill does not editorialize. It returns what matches the query, sorted by relevance or recency, and leaves the interpretation to the pipeline. This is the same design principle as blogwatcher: the tool is mechanical; the curation is intellectual.
For the Council’s research pipeline, the discovery step runs on a schedule. Queries target specific domains — prediction markets, agent orchestration, knowledge graphs, sovereign infrastructure — and return the latest papers in each area. The output is not a reading list. It is a structured feed: title, abstract, categories, URL. The next step decides what to do with it.
The synthesis layer: from abstract to grounded claim
Most papers die at the abstract. A researcher reads the first paragraph, decides it is not relevant, and moves on. The pipeline does something different: it reads the abstract, extracts the core claim, and checks that claim against the existing knowledge base before deciding whether the full paper warrants deeper analysis.
This is where the grounded-citations skill earns its name. Every claim the pipeline produces must be attributed to a verifiable source. When a paper claims that “prediction market prices precede polling shifts by up to 14 days,” the pipeline does not repeat that claim as fact — it records the source, the methodology, and the confidence level. The citation is not decoration. It is the mechanism that separates intelligence from opinion.
The synthesis step operates on three levels:
Level 1 — Abstract triage. The pipeline reads the abstract and extracts the core claim, the methodology, and the key finding. If the claim is relevant to the Council’s research focus, the paper advances to Level 2. If not, it is archived with a brief note on why it was excluded.
Level 2 — Wiki grounding. The extracted claim is checked against the llm-wiki knowledge base. Does this paper confirm, contradict, or extend an existing wiki concept? If the paper’s findings align with an existing concept, the concept is updated with the new citation. If the paper introduces a genuinely new finding, a new wiki node is created. If the paper contradicts existing knowledge, the contradiction is flagged for human review.
Level 3 — Cross-reference. The pipeline checks whether the paper’s claims have been corroborated by other sources — blog posts caught by blogwatcher, prediction market movements captured by polymarket, or previous arxiv papers in the same research line. A claim backed by a single paper is interesting. A claim backed by three independent sources is actionable.
The synthesis layer is where the pipeline earns its value. A human researcher might read twenty papers a day. The pipeline can process two hundred. But the pipeline’s advantage is not speed — it is consistency. Every paper gets the same three-level treatment. Every claim gets grounded. Every citation gets recorded. The human researcher’s synthesis is brilliant but sporadic. The pipeline’s synthesis is mechanical but reliable.
The decision layer: from knowledge to action
Synthesis without decision is scholarship. The Council is not an academic institution. The pipeline exists to inform action: should we shift our research focus? Should we publish an article on this topic? Should we adjust our competitive positioning? Should we alert a client?
The decision layer takes the grounded syntheses from Level 3 and routes them through a simple framework:
If the finding confirms existing direction — log it, update the wiki, move on. No action required. The knowledge base is stronger; the next article will benefit from the additional citation.
If the finding extends existing direction — flag it for the next article in the relevant series. The extension becomes a supporting data point in an upcoming piece. The pipeline does not write the article; it creates the note that the article writer will use.
If the finding contradicts existing direction — route to human review. The pipeline does not override strategic decisions. It surfaces the contradiction, presents the evidence, and waits. The human decides.
If the finding opens a new direction — create a research card on the kanban board. The card includes the paper’s claim, the grounding citations, and a suggested research thread. The card enters the backlog. The orchestrator decides when to act on it.
This framework is deliberately simple. Complex routing logic creates failure modes that are hard to debug. Simple routing creates failure modes that are easy to spot. The pipeline favors legibility over sophistication.
The loop: why the pipeline is not a funnel
A funnel has a wide mouth and a narrow exit. Papers go in; decisions come out. The research pipeline is not a funnel. It is a loop.
Every decision feeds back into the discovery layer. When the pipeline flags a new research direction, that direction generates new arxiv queries. When a wiki concept is updated with a new citation, the next synthesis pass has more context to work with. When a contradiction is flagged and resolved, the resolution becomes a new wiki node that future synthesis can reference.
The loop is what makes the pipeline self-improving. The first time the pipeline runs, it has no knowledge base. Every paper is novel. Every claim is ungrounded. The second time, it has a handful of wiki nodes. Some papers confirm them. Some contradict them. The tenth time, the knowledge base is dense enough that most papers can be quickly triaged — they confirm what is already known, and only the genuinely novel findings require deep synthesis.
This is the llm-wiki architecture at work: a persistent, queryable knowledge base that grows with every pipeline cycle. The wiki is not a document store. It is a graph — concepts linked to sources, sources linked to claims, claims linked to evidence. The pipeline reads the graph to contextualize new papers. The pipeline writes to the graph to capture new findings. The graph is both the input and the output.
What this means for the S10 series
The S10 series maps the market sensing stack: prediction markets for consensus signals, the research pipeline for academic rigor, blogwatcher for independent speed. Each layer operates on a different timescale and a different credibility model. The research pipeline is the slowest layer — arxiv papers take months from submission to publication — but it is the most rigorous. When a claim is grounded in a peer-reviewed paper, cited inline, and cross-referenced against the wiki graph, it carries a weight that a blog post or a market price cannot match.
The pipeline does not replace human judgment. It replaces human drudgery. The three-level synthesis — triage, grounding, cross-reference — is work that a human researcher would do if they had time. Most do not. The pipeline does it mechanically, consistently, and at scale. The human researcher’s job shifts from reading papers to making decisions about what the papers mean. That is a better use of human intelligence than scanning abstracts.
The research pipeline is the engine that turns arxiv’s firehose into the Council’s knowledge base. Discovery. Synthesis. Decision. Repeat.
Sources
[1] https://info.arxiv.org/about/reports/statistics_24.html
[2] https://arxiv.org/html/2605.00493v2
*Grounded in wiki concepts arxiv, grounded-citations, and llm-wiki. Skills: arxiv, grounded-citations, llm-wiki. Second article in the S10 series on Market Sensing & Research. Design notes on a running system, not a sales pitch.*
Semantic Relationships
– [[arxiv]] — orchestrates
– [[grounded-citations]] — orchestrates
– [[llm-wiki]] — orchestrates
