This document describes non-trivial algorithms used throughout the Lilith platform, including their rationale, implementation details, and performance characteristics.
Table of Contents#
- RAG Pipeline
- Multi-Agent Orchestration
- Citation Extraction
- Persona Style Adherence
- Voice Activity Detection
- Conflict Resolution (CRDT)
- Content Moderation Scoring
- Semantic Chunking
- Cross-Device Sync
- Brainwave Entrainment
1. RAG Pipeline#
Overview#
Retrieval-Augmented Generation (RAG) grounds LLM responses in source material from the knowledge base, ensuring accuracy and enabling citations.
Algorithm#
┌─────────────────────────────────────────────────────────────────────────────┐
│ RAG PIPELINE │
│ │
│ Query ──> Query Expansion ──> Hybrid Retrieval ──> Re-ranking │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ BM25 │ │ Dense │ │
│ │ (Lexical) │ │ (Semantic) │ │
│ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │
│ └────────┬───────────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Reciprocal │ │
│ │ Rank Fusion │ │
│ └──────┬──────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │Cross-Encoder│ │
│ │ Re-ranker │ │
│ └──────┬──────┘ │
│ │ │
│ ▼ │
│ Top-K Chunks ──> LLM Context │
└─────────────────────────────────────────────────────────────────────────────┘
Implementation Details#
1. Query Expansion
def expand_query(query: str, tradition: list[str]) -> list[str]:
"""Generate query variants for better recall."""
expanded = [query]
# Add tradition-specific synonyms
for term in extract_concepts(query):
if term in SPIRITUAL_SYNONYMS:
for synonym in SPIRITUAL_SYNONYMS[term]:
expanded.append(query.replace(term, synonym))
# Add tradition filter queries
for t in tradition:
expanded.append(f"{query} in {t} tradition")
return expanded
2. Hybrid Search with RRF
def hybrid_search(query: str, k: int = 20) -> list[Chunk]:
"""Combine BM25 and dense retrieval with reciprocal rank fusion."""
# BM25 lexical search
bm25_results = bm25_index.search(query, k=k*2)
# Dense semantic search
query_embedding = embed(query)
dense_results = vector_db.search(query_embedding, k=k*2)
# Reciprocal Rank Fusion
scores = defaultdict(float)
RRF_K = 60 # Constant from literature
for rank, doc in enumerate(bm25_results):
scores[doc.id] += 1 / (RRF_K + rank + 1)
for rank, doc in enumerate(dense_results):
scores[doc.id] += 1 / (RRF_K + rank + 1)
# Sort by fused score
ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
return [get_chunk(doc_id) for doc_id, _ in ranked[:k]]
3. Cross-Encoder Re-ranking
def rerank(query: str, chunks: list[Chunk], k: int = 5) -> list[Chunk]:
"""Re-rank with cross-encoder for precision."""
pairs = [(query, chunk.text) for chunk in chunks]
scores = cross_encoder.predict(pairs)
ranked = sorted(zip(chunks, scores), key=lambda x: x[1], reverse=True)
return [chunk for chunk, _ in ranked[:k]]
Rationale#
- Hybrid Search: Combines lexical matching (exact terms like "anatta") with semantic understanding
- RRF: Outperforms simple score combination; robust to score scale differences
- Re-ranking: Cross-encoders are more accurate but slower; applied only to top candidates
- Query Expansion: Critical for spiritual domain with many synonyms across traditions
Performance#
| Metric | Value |
|---|---|
| Recall@10 | 0.92 |
| MRR | 0.85 |
| Latency (p50) | 150ms |
| Latency (p95) | 350ms |
2. Multi-Agent Orchestration#
Overview#
Multiple specialized AI agents collaborate to produce responses: Librarian (retrieval), Teacher (instruction), Safety (moderation), Narrator (SSML).
Algorithm#
┌─────────────────────────────────────────────────────────────────────────────┐
│ MULTI-AGENT ORCHESTRATION │
│ │
│ User Query ──> Intent Classification ──> Agent Selection │
│ │
│ Parallel Execution: │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Librarian │ │ Teacher │ │ Safety │ │
│ │ (Sources) │ │ (Response) │ │ (Filter) │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ └────────────────┼────────────────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Synthesis │ │
│ │ Agent │ │
│ └──────┬──────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Narrator │ │
│ │ (SSML) │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
Implementation#
class AgentOrchestrator:
def __init__(self):
self.agents = {
'librarian': LibrarianAgent(),
'teacher': TeacherAgent(),
'safety': SafetyAgent(),
'narrator': NarratorAgent(),
}
async def process(self, query: str, persona: str, context: dict) -> Response:
# Phase 1: Parallel information gathering
librarian_task = self.agents['librarian'].search(query)
safety_task = self.agents['safety'].pre_check(query)
sources, safety_flags = await asyncio.gather(
librarian_task, safety_task
)
if safety_flags.block:
return self.safety_response(safety_flags)
# Phase 2: Teacher generates response with sources
teacher_response = await self.agents['teacher'].respond(
query=query,
persona=persona,
sources=sources,
context=context
)
# Phase 3: Post-moderation
moderated = await self.agents['safety'].post_check(teacher_response)
# Phase 4: SSML preparation (if audio requested)
if context.get('audio_requested'):
ssml = await self.agents['narrator'].prepare_ssml(moderated)
return Response(text=moderated.text, ssml=ssml, citations=sources)
return Response(text=moderated.text, citations=sources)
Rationale#
- Separation of Concerns: Each agent specializes in one task
- Parallel Execution: Reduces latency by running independent agents concurrently
- Safety First: Pre and post moderation ensures content safety
- Extensibility: New agents can be added without changing orchestration logic
3. Citation Extraction#
Overview#
Extract and validate citations from LLM-generated content, mapping claims to source documents.
Algorithm#
def extract_citations(
response: str,
sources: list[Chunk],
min_confidence: float = 0.7
) -> list[Citation]:
"""Extract citations by matching response claims to source chunks."""
citations = []
# Split response into claim sentences
claims = split_into_claims(response)
for claim in claims:
claim_embedding = embed(claim)
# Find best matching source
best_match = None
best_score = 0.0
for source in sources:
# Compute semantic similarity
similarity = cosine_similarity(claim_embedding, source.embedding)
# Boost if lexical overlap exists
overlap_boost = compute_lexical_overlap(claim, source.text)
score = similarity * 0.7 + overlap_boost * 0.3
if score > best_score:
best_score = score
best_match = source
if best_score >= min_confidence and best_match:
citations.append(Citation(
claim=claim,
source=best_match,
confidence=best_score,
span=find_span(response, claim)
))
return citations
Claim Splitting#
def split_into_claims(text: str) -> list[str]:
"""Split text into individual factual claims."""
sentences = sent_tokenize(text)
claims = []
for sentence in sentences:
# Skip non-factual sentences (questions, opinions markers)
if is_question(sentence) or has_opinion_marker(sentence):
continue
# Split compound sentences
if ' and ' in sentence or '; ' in sentence:
subclaims = split_compound(sentence)
claims.extend(subclaims)
else:
claims.append(sentence)
return claims
Rationale#
- Claim-level granularity: More accurate than document-level matching
- Hybrid scoring: Combines semantic (meaning) and lexical (exact terms) matching
- Confidence thresholds: Only high-confidence citations are shown to users
4. Persona Style Adherence#
Overview#
Ensure LLM responses match the expected style, tone, and teaching approach of each persona.
Algorithm#
class PersonaStyleEnforcer:
def __init__(self, persona_config: dict):
self.style_vector = load_style_vector(persona_config['key'])
self.teaching_style = persona_config['style_config']['teaching_style']
self.tradition = persona_config['tradition']
def score_adherence(self, response: str) -> float:
"""Score how well response matches persona style."""
# Compute response style embedding
response_style = compute_style_embedding(response)
# Compare to persona style vector
style_similarity = cosine_similarity(response_style, self.style_vector)
# Check teaching style markers
style_markers = self.get_style_markers()
marker_score = sum(1 for m in style_markers if m in response.lower()) / len(style_markers)
# Check tradition authenticity
tradition_score = self.check_tradition_authenticity(response)
# Weighted combination
return (
style_similarity * 0.4 +
marker_score * 0.3 +
tradition_score * 0.3
)
def get_style_markers(self) -> list[str]:
"""Get linguistic markers for teaching style."""
markers = {
'socratic': ['what do you think', 'have you considered', 'why might'],
'direct': ['the key is', 'simply', 'directly'],
'metaphorical': ['like', 'as if', 'imagine'],
'experiential': ['try this', 'notice', 'feel'],
}
return markers.get(self.teaching_style, [])
Style Vectors#
128-dimensional embeddings trained on exemplar responses from each persona:
| Dimension | Meaning |
|---|---|
| 0-31 | Formality level |
| 32-63 | Teaching directness |
| 64-95 | Use of metaphor |
| 96-127 | Emotional warmth |
5. Voice Activity Detection#
Overview#
Detect when users start/stop speaking for voice chat barge-in and turn-taking.
Algorithm#
class VoiceActivityDetector:
def __init__(
self,
sample_rate: int = 16000,
frame_ms: int = 30,
threshold_db: float = -35,
hangover_frames: int = 10
):
self.sample_rate = sample_rate
self.frame_size = int(sample_rate * frame_ms / 1000)
self.threshold_db = threshold_db
self.hangover_frames = hangover_frames
self.speech_frames = 0
self.silence_frames = 0
self.is_speech = False
def process_frame(self, audio_frame: np.ndarray) -> bool:
"""Process audio frame and return speech/silence state."""
# Compute frame energy in dB
energy = 10 * np.log10(np.mean(audio_frame ** 2) + 1e-10)
if energy > self.threshold_db:
self.speech_frames += 1
self.silence_frames = 0
# Speech onset after 2 consecutive frames
if self.speech_frames >= 2:
self.is_speech = True
else:
self.silence_frames += 1
# Speech offset after hangover period
if self.silence_frames >= self.hangover_frames:
self.speech_frames = 0
self.is_speech = False
return self.is_speech
Barge-In Detection#
class BargeInDetector:
def __init__(self, vad: VoiceActivityDetector):
self.vad = vad
self.tts_playing = False
self.barge_in_threshold_frames = 5
self.consecutive_speech = 0
def check_barge_in(self, audio_frame: np.ndarray) -> bool:
"""Check if user is interrupting TTS playback."""
if not self.tts_playing:
return False
is_speech = self.vad.process_frame(audio_frame)
if is_speech:
self.consecutive_speech += 1
if self.consecutive_speech >= self.barge_in_threshold_frames:
return True
else:
self.consecutive_speech = 0
return False
6. Conflict Resolution (CRDT)#
Overview#
Conflict-free resolution for offline edits syncing across devices using CRDTs.
Algorithm: LWW-Register for Preferences#
class LWWRegister:
"""Last-Writer-Wins Register for preference sync."""
def __init__(self):
self.value = None
self.timestamp = 0
self.node_id = generate_node_id()
def update(self, new_value, timestamp: int = None):
ts = timestamp or time.time_ns()
if ts > self.timestamp:
self.value = new_value
self.timestamp = ts
def merge(self, other: 'LWWRegister'):
"""Merge with another register, keeping latest."""
if other.timestamp > self.timestamp:
self.value = other.value
self.timestamp = other.timestamp
elif other.timestamp == self.timestamp:
# Tie-break by node_id for determinism
if other.node_id > self.node_id:
self.value = other.value
Algorithm: G-Counter for Progress#
class GCounter:
"""Grow-only counter for progress tracking."""
def __init__(self):
self.counts = {} # node_id -> count
self.node_id = generate_node_id()
def increment(self, amount: int = 1):
self.counts[self.node_id] = self.counts.get(self.node_id, 0) + amount
def value(self) -> int:
return sum(self.counts.values())
def merge(self, other: 'GCounter'):
"""Merge by taking max for each node."""
for node_id, count in other.counts.items():
self.counts[node_id] = max(self.counts.get(node_id, 0), count)
Rationale#
- CRDTs: Guarantee eventual consistency without coordination
- LWW: Simple and effective for user preferences
- G-Counter: Perfect for monotonic values like meditation minutes
7. Content Moderation Scoring#
Overview#
Multi-signal scoring system for content safety with tradition-aware context.
Algorithm#
class ModerationScorer:
def __init__(self):
self.toxicity_model = load_model('toxicity')
self.nsfw_model = load_model('nsfw')
self.tradition_context = load_tradition_rules()
def score(self, content: str, tradition: str = None) -> ModerationResult:
# Base toxicity score
toxicity = self.toxicity_model.predict(content)
# NSFW/adult content score
nsfw = self.nsfw_model.predict(content)
# Medical/legal disclaimer detection
requires_disclaimer = self.check_disclaimer_needed(content)
# Tradition-specific allowances
tradition_adjustment = 0.0
if tradition:
tradition_adjustment = self.tradition_context.get_adjustment(
content, tradition
)
# Compute final scores
final_toxicity = max(0, toxicity - tradition_adjustment)
return ModerationResult(
toxicity=final_toxicity,
nsfw=nsfw,
requires_disclaimer=requires_disclaimer,
should_block=final_toxicity > 0.8 or nsfw > 0.9,
should_warn=final_toxicity > 0.5
)
Tradition Context Adjustments#
Some content is appropriate in spiritual context but might flag generic models:
TRADITION_ALLOWANCES = {
'tantra': {
'sacred_sexuality': -0.3, # Reduce NSFW score
'kundalini_energy': -0.2,
},
'shamanism': {
'plant_medicine': -0.2, # Contextually appropriate
'death_rebirth': -0.2,
},
'dark_mysticism': {
'shadow_work': -0.2,
'underworld': -0.2,
}
}
8. Semantic Chunking#
Overview#
Intelligent document chunking that preserves semantic coherence.
Algorithm#
def semantic_chunk(
document: str,
max_tokens: int = 512,
overlap_tokens: int = 50
) -> list[Chunk]:
"""Chunk document at semantic boundaries."""
# Split into sentences
sentences = sent_tokenize(document)
# Compute sentence embeddings
embeddings = [embed(s) for s in sentences]
# Find semantic breakpoints
breakpoints = []
for i in range(1, len(sentences)):
similarity = cosine_similarity(embeddings[i-1], embeddings[i])
if similarity < BREAKPOINT_THRESHOLD:
breakpoints.append(i)
# Create chunks respecting breakpoints and size limits
chunks = []
current_chunk = []
current_tokens = 0
for i, sentence in enumerate(sentences):
sentence_tokens = count_tokens(sentence)
if current_tokens + sentence_tokens > max_tokens:
# Save current chunk
if current_chunk:
chunks.append(Chunk(
text=' '.join(current_chunk),
start_idx=chunk_start_idx,
end_idx=i
))
# Start new chunk with overlap
overlap_start = max(0, len(current_chunk) - overlap_sentences)
current_chunk = current_chunk[overlap_start:]
current_tokens = sum(count_tokens(s) for s in current_chunk)
chunk_start_idx = i - len(current_chunk)
elif i in breakpoints and current_tokens > max_tokens // 2:
# Natural breakpoint and chunk is substantial
chunks.append(Chunk(
text=' '.join(current_chunk),
start_idx=chunk_start_idx,
end_idx=i
))
current_chunk = []
current_tokens = 0
chunk_start_idx = i
current_chunk.append(sentence)
current_tokens += sentence_tokens
# Don't forget last chunk
if current_chunk:
chunks.append(Chunk(text=' '.join(current_chunk)))
return chunks
9. Cross-Device Sync#
Overview#
Synchronize user state (progress, preferences, bookmarks) across devices.
Algorithm#
class SyncManager:
def __init__(self, user_id: str):
self.user_id = user_id
self.vector_clock = VectorClock()
self.pending_ops = []
async def sync(self) -> SyncResult:
"""Perform bidirectional sync with server."""
# 1. Get server state
server_state = await self.fetch_server_state()
# 2. Detect conflicts
conflicts = self.detect_conflicts(server_state)
# 3. Resolve conflicts using CRDTs
resolved = self.resolve_conflicts(conflicts)
# 4. Apply server changes locally
local_changes = self.apply_server_changes(server_state)
# 5. Push local changes to server
await self.push_local_changes()
# 6. Update vector clock
self.vector_clock.merge(server_state.vector_clock)
return SyncResult(
conflicts_resolved=len(resolved),
items_pulled=len(local_changes),
items_pushed=len(self.pending_ops)
)
def detect_conflicts(self, server_state) -> list[Conflict]:
conflicts = []
for key, server_value in server_state.items.items():
local_value = self.local_store.get(key)
if local_value and local_value.modified and server_value.modified:
if not self.vector_clock.happened_before(
local_value.clock, server_value.clock
):
conflicts.append(Conflict(key, local_value, server_value))
return conflicts
10. Brainwave Entrainment#
Overview#
Generate audio frequencies for brainwave entrainment (binaural beats, isochronic tones).
Algorithm: Binaural Beat Generation#
def generate_binaural_beat(
base_freq: float,
beat_freq: float,
duration_sec: float,
sample_rate: int = 44100
) -> tuple[np.ndarray, np.ndarray]:
"""Generate stereo binaural beat audio.
Args:
base_freq: Carrier frequency (e.g., 200 Hz)
beat_freq: Desired brainwave frequency (e.g., 10 Hz for alpha)
duration_sec: Duration in seconds
sample_rate: Audio sample rate
Returns:
Tuple of (left_channel, right_channel) audio arrays
"""
t = np.linspace(0, duration_sec, int(sample_rate * duration_sec))
# Left ear: base frequency
left = np.sin(2 * np.pi * base_freq * t)
# Right ear: base + beat frequency
right = np.sin(2 * np.pi * (base_freq + beat_freq) * t)
# Apply fade in/out to prevent clicks
fade_samples = int(sample_rate * 0.1) # 100ms fade
fade_in = np.linspace(0, 1, fade_samples)
fade_out = np.linspace(1, 0, fade_samples)
left[:fade_samples] *= fade_in
left[-fade_samples:] *= fade_out
right[:fade_samples] *= fade_in
right[-fade_samples:] *= fade_out
return left, right
Brainwave Frequency Bands#
| Band | Frequency | State | Use Case |
|---|---|---|---|
| Delta | 0.5-4 Hz | Deep sleep | Sleep meditation |
| Theta | 4-8 Hz | Drowsy, creative | Deep meditation |
| Alpha | 8-14 Hz | Relaxed, aware | Light meditation |
| Beta | 14-30 Hz | Alert, focused | Focus sessions |
| Gamma | 30-100 Hz | Peak awareness | Advanced practice |