Literature & novelty graph for the Phase-178 autonomous-research stack (TODOS/phase-178 §178.1).
Components#
| Module | Summary |
|---|---|
http.py |
Pluggable HttpTransport (stdlib UrllibTransport live, FakeTransport for tests), a token-bucket rate-limiter with exponential backoff, and a two-tier cache (memory + SQLite). |
adapters.py |
Semantic Scholar (search/paper/citations/references/embedding/bulk), OpenAlex (works/concepts + TSV-snapshot ingestion), arXiv (Atom parse), Crossref, PubMed (E-Utilities), CORE, bioRxiv — all into one Paper model with cross-source dedup_key for @sophia/crawling interop. (§178.1.1) |
graph.py |
Citation-graph traversal — bounded BFS with node budget, dead-end detection, and relevance attenuation by hop distance × publication-year gap. (§178.1.1.5) |
novelty.py |
NoveltyScorer (max cosine, second-neighbour gap, citation distance, year spread), PriorArtDetector (top-3 threshold + novel/overlap diff), CitationQualityRanker (venue tier + citation velocity + recency + reviewer agreement), method/dataset/claim extractor. (§178.1.2) |
survey.py |
ScholarQA citation-grounded survey with a hallucination gate (non-obvious claims must cite a resolvable paper id), STORM outline-then-expand, and a deep-research orchestrator (fail-loud external backends + a self-hosted corpus baseline, per-tenant budget, citation-quality floor). (§178.1.3) |
Live API fetches need network; adapter request-building and response-parsing are
verified against recorded payloads through FakeTransport. Everything
downstream of the corpus — graph traversal, novelty math, citation quality, the
survey hallucination gate — is fully local and tested. Swap HashingEmbedder
for Gemini-Embeddings-2 and the LLM extractor/section-generator seams for real
models to go to production.
Testing#
bash
python3 -m pytest tests/ -q # 24 tests
python3 -m ruff check .