# Psyche Knowledge Base

RAG system and document processing for the Psyche AI platform.

Part of the Psyche AI Virtual Assistant Platform.

## Overview

The Knowledge Base provides retrieval-augmented generation (RAG) capabilities
with document ingestion, vector search, and quality-evaluated response
generation.

## Features

- **Document Ingestion**: PDF, DOCX, HTML, Markdown, Excel, CSV, JSON
- **Multiple Chunking Strategies**: Fixed-size, semantic, sentence, hierarchical
- **Embedding Providers**: Voyage AI, OpenAI, Cohere, BGE
- **Vector Stores**: Pinecone, Qdrant, pgvector, in-memory
- **Hybrid Search**: Vector + BM25 keyword search
- **Re-ranking**: Cross-encoder, RRF, diversity boosting
- **RAG Pipeline**: Quality evaluation, hallucination detection
- **Content Management**: Versioning, archival, analytics

## Architecture

```
knowledge_base/
├── ingestion/            # Document processing
│   ├── types.py          # Document, TextChunk types
│   ├── manager.py        # IngestionManager
│   ├── parsers.py        # PDF, DOCX, HTML parsers
│   ├── chunking.py       # Chunking strategies
│   └── cleaning.py       # Content preprocessing
├── embedding/            # Vector embeddings
│   ├── types.py          # EmbeddingModel, VectorDatabase
│   ├── generators.py     # Voyage, OpenAI, Cohere
│   ├── manager.py        # EmbeddingManager with caching
│   └── stores.py         # VectorStore implementations
├── retrieval/            # Hybrid search
│   ├── types.py          # QueryType, SearchMode
│   ├── pipeline.py       # RetrievalPipeline
│   ├── query.py          # Query processing/expansion
│   ├── search.py         # HybridSearcher
│   ├── reranking.py      # Cross-encoder reranking
│   └── context.py        # Context assembly
├── rag/                  # Generation
│   ├── types.py          # RAGConfig, RAGResponse
│   ├── pipeline.py       # RAGPipeline
│   ├── generator.py      # LLM response generation
│   ├── prompts.py        # Template management
│   └── evaluator.py      # Quality evaluation
└── management/           # Content lifecycle
    ├── types.py          # ContentStatus, QualityLevel
    ├── lifecycle.py      # Versioning, archival
    ├── quality.py        # Quality monitoring
    ├── updates.py        # Content refresh
    └── analytics.py      # Usage analytics
```

## Document Ingestion

### Supported Formats

| Format     | Parser        | Features                   |
| ---------- | ------------- | -------------------------- |
| PDF        | PyMuPDF       | Text, tables, OCR fallback |
| DOCX       | python-docx   | Text, styles, tables       |
| HTML       | BeautifulSoup | Cleaned text, links        |
| Markdown   | Custom        | Headers, code blocks       |
| Excel      | openpyxl      | Sheets, formulas           |
| CSV        | pandas        | Structured data            |
| JSON       | Built-in      | Nested structures          |
| PowerPoint | python-pptx   | Slides, notes              |

### Chunking Strategies

| Strategy     | Description                 | Best For          |
| ------------ | --------------------------- | ----------------- |
| Fixed-Size   | Token-based with overlap    | General documents |
| Semantic     | Split on meaning boundaries | Long-form content |
| Sentence     | Preserve sentence integrity | Q&A content       |
| Hierarchical | Multi-level chunks          | Structured docs   |

## Embedding Providers

| Provider  | Models                | Dimensions |
| --------- | --------------------- | ---------- |
| Voyage AI | 3.5-lite, 3           | 1024       |
| OpenAI    | Ada, 3-small, 3-large | 1536, 3072 |
| Cohere    | English, Multilingual | 1024       |
| BGE       | M3, Large             | 1024       |

## Vector Stores

| Store    | Type        | Features               |
| -------- | ----------- | ---------------------- |
| Pinecone | Cloud       | Managed, scalable      |
| Qdrant   | Self-hosted | Open source, filtering |
| pgvector | PostgreSQL  | SQL integration        |
| Memory   | In-process  | Development only       |

## Retrieval Pipeline

### Search Modes

| Mode         | Description          |
| ------------ | -------------------- |
| VECTOR_ONLY  | Pure semantic search |
| KEYWORD_ONLY | BM25 keyword search  |
| HYBRID       | Weighted combination |

### Re-ranking Strategies

| Strategy      | Description              |
| ------------- | ------------------------ |
| CROSS_ENCODER | MS MARCO neural reranker |
| RRF           | Reciprocal rank fusion   |
| DIVERSITY     | Boost unique content     |
| RECENCY       | Decay older content      |

## RAG Pipeline

### Response Modes

| Mode           | Description             |
| -------------- | ----------------------- |
| STANDARD       | Single-turn RAG         |
| CONVERSATIONAL | Multi-turn with history |
| AGENTIC        | Tool-augmented          |
| STREAMING      | Token-by-token          |

### Quality Metrics

| Metric            | Description                   |
| ----------------- | ----------------------------- |
| Answer Relevance  | Response addresses query      |
| Context Relevance | Retrieved content is relevant |
| Groundedness      | Claims supported by sources   |
| Faithfulness      | No hallucinated facts         |
| Completeness      | Query fully answered          |

## Quick Start

### Installation

```bash
# Using Nx
nx install psyche-knowledge-base

# With all vector stores
nx install-all psyche-knowledge-base

# Specific vector store
nx install-pinecone psyche-knowledge-base
nx install-qdrant psyche-knowledge-base
nx install-pgvector psyche-knowledge-base

# Or directly with Poetry
cd apps/psyche/knowledge-base
poetry install
poetry install --extras all
```

### Environment Variables

```bash
# Embedding Providers
VOYAGE_API_KEY=your-voyage-key
OPENAI_API_KEY=your-openai-key
COHERE_API_KEY=your-cohere-key

# Vector Stores
PINECONE_API_KEY=your-pinecone-key
PINECONE_ENVIRONMENT=us-east-1
QDRANT_URL=http://localhost:6333
DATABASE_URL=postgresql://user:pass@localhost:5432/knowledge

# LLM Providers
ANTHROPIC_API_KEY=your-anthropic-key

# Service Config
KNOWLEDGE_BASE_PORT=8008
KNOWLEDGE_BASE_HOST=0.0.0.0
LOG_LEVEL=INFO
```

### Basic Usage

```python
from knowledge_base import (
    IngestionManager,
    EmbeddingManager,
    VectorIndexManager,
    RetrievalPipelineBuilder,
    RAGPipelineBuilder,
    LLMClient,
    EmbeddingConfig,
    EmbeddingModel,
)

# Set up embedding
embed_manager = EmbeddingManager(
    config=EmbeddingConfig(model=EmbeddingModel.VOYAGE_3_5),
    api_keys={"voyage": "your-key"},
)

# Set up vector index
index_manager = VectorIndexManager(embed_manager)

# Ingest documents
ingestion = IngestionManager()
result = await ingestion.ingest_file("/path/to/doc.pdf")
await index_manager.index_chunks(result.chunks, "my-index")

# Set up RAG
llm = LLMClient(provider="anthropic", api_key="your-key")
retrieval = RetrievalPipelineBuilder() \
    .with_embedding_manager(embed_manager) \
    .with_vector_store(index_manager._vector_store) \
    .build()
rag = RAGPipelineBuilder() \
    .with_retrieval_pipeline(retrieval) \
    .with_llm_client(llm) \
    .build()

# Query
result = await rag.query("What is your refund policy?")
print(result.response.answer)
print(f"Confidence: {result.quality.confidence}")
```

## API Endpoints

### Documents

- `POST /documents/ingest` - Ingest document
- `POST /documents/batch` - Batch ingest
- `GET /documents/{id}` - Get document
- `DELETE /documents/{id}` - Delete document
- `GET /documents/{id}/chunks` - Get chunks

### Search

- `POST /search` - Hybrid search
- `POST /search/vector` - Vector-only search
- `POST /search/keyword` - Keyword-only search

### RAG

- `POST /rag/query` - Query with RAG
- `POST /rag/stream` - Streaming RAG
- `POST /rag/evaluate` - Evaluate response

### Health

- `GET /health` - Health check
- `GET /ready` - Readiness check
- `GET /metrics` - Prometheus metrics

## Development

### Running the Server

```bash
# Development mode
nx serve psyche-knowledge-base

# Production mode
nx serve-prod psyche-knowledge-base
```

### Running Tests

```bash
nx test psyche-knowledge-base
nx test-unit psyche-knowledge-base
nx test-integration psyche-knowledge-base
nx test-cov psyche-knowledge-base
```

### Docker

```bash
# Build image
nx docker-build psyche-knowledge-base

# Run container
nx docker-run psyche-knowledge-base
```

## Nx Integration

```bash
# Available targets
nx serve psyche-knowledge-base          # Development server
nx serve-prod psyche-knowledge-base     # Production server
nx build psyche-knowledge-base          # Build package
nx install psyche-knowledge-base        # Install dependencies
nx install-all psyche-knowledge-base    # Install all extras
nx install-pinecone psyche-knowledge-base   # Pinecone only
nx install-qdrant psyche-knowledge-base     # Qdrant only
nx install-pgvector psyche-knowledge-base   # pgvector only
nx lint psyche-knowledge-base           # Run linters
nx format psyche-knowledge-base         # Format code
nx test psyche-knowledge-base           # Run all tests
nx test-unit psyche-knowledge-base      # Unit tests only
nx test-integration psyche-knowledge-base  # Integration tests
nx test-cov psyche-knowledge-base       # Tests with coverage
nx docker-build psyche-knowledge-base   # Build Docker image
nx docker-run psyche-knowledge-base     # Run container
```

## Performance

### Default Configuration

| Setting       | Default    | Description            |
| ------------- | ---------- | ---------------------- |
| Chunk Size    | 512 tokens | Target chunk size      |
| Chunk Overlap | 50 tokens  | Overlap between chunks |
| Top-K         | 10         | Retrieved chunks       |
| Token Budget  | 4000       | Max context tokens     |
| Cache TTL     | 7 days     | Embedding cache        |

### Latency Targets

| Operation            | Target        |
| -------------------- | ------------- |
| Document ingestion   | 1-5s per page |
| Embedding generation | 100-500ms     |
| Vector search        | 10-50ms       |
| RAG response         | 2-5s          |

## License

Proprietary - Oshun Platform