Eve library

Minerva to Metis: Dependency Upgrade Roadmap

Guide

Revision 1

On this page

This document maps all 117 production dependencies from Minerva's pyproject.toml to their TypeScript/Node.js equivalents for the Metis port. It categorizes each dependency by migration tier, provides a priority order for migration, and assesses risk per category.


Table of Contents#

  1. Summary
  2. Tier 1 -- Already Ported
  3. Tier 2 -- Direct Equivalent Available
  4. Tier 3 -- Partial Equivalent
  5. Tier 4 -- No Equivalent, Needs Custom Implementation
  6. Tier 5 -- Not Needed in TypeScript
  7. Complete Dependency Matrix
  8. Priority Migration Order
  9. Risk Assessment

Summary#

Tier Count Description
Tier 1 -- Already Ported 10 Functionality exists in the Oshun monorepo or is handled by @metis/* packages
Tier 2 -- Direct Equivalent 48 Mature npm packages providing the same functionality
Tier 3 -- Partial Equivalent 22 npm packages that cover most but not all functionality
Tier 4 -- Needs Custom Implementation 15 No npm equivalent; must be built or bridged via WASM/native
Tier 5 -- Not Needed in TypeScript 22 Python-specific runtime, build, or tooling packages
Total 117

Tier 1 -- Already Ported#

These dependencies have TypeScript equivalents already present in the Oshun monorepo (either as @metis/* packages, @oshun/* shared libraries, or monorepo-level dependencies).

# Minerva (Python) Metis Equivalent Notes
1 pydantic>=2.0 @metis/core (validation, types, serialization) Zod + custom validation in @metis/core/validation
2 pydantic-settings>=2.0 @metis/core (constants, config) Config handled by core constants and env parsing
3 jsonschema>=4.25.0 @metis/core/validation JSON Schema validation built into core
4 validators>=0.35.0 @metis/core/validation Input validation logic in core
5 email-validator>=2.2.0 @metis/core/validation Email validation included in core validators
6 websockets>=15.0.1 @metis/api-client/websocket WebSocket client in api-client package
7 toml>=0.10.2 Node.js built-in / monorepo tooling TOML parsing not needed; configs are JSON/YAML
8 networkx>=3.2.1 @metis/knowledge-graph Graph data structures ported to knowledge-graph
9 networkx>=3.0 @metis/knowledge-graph Duplicate entry; same as above
10 protobuf>=4.25.8 protobufjs (monorepo-level) Protocol buffer handling via protobufjs

Tier 2 -- Direct Equivalent Available#

Mature npm packages that provide the same or equivalent functionality.

# Minerva (Python) TypeScript/npm Equivalent npm Package Notes
1 litellm>=1.34.0 Multi-provider LLM client litellm (npm) or ai (Vercel AI SDK) Vercel AI SDK is more idiomatic for TS; litellm has a JS client
2 pyyaml>=6.0 YAML parser yaml (yaml package, formerly js-yaml) Drop-in replacement
3 pillow>=11.3.0 Image processing sharp Much faster than Pillow; native bindings
4 aiohttp>=3.12.15 HTTP client (async) undici / built-in fetch Node.js native fetch covers most use cases
5 beautifulsoup4>=4.13.4 HTML parser cheerio De facto standard for HTML parsing in Node.js
6 feedparser>=6.0.11 RSS/Atom parser rss-parser Mature, well-maintained
7 google-api-python-client>=2.178.0 Google API client googleapis Official Google API client for Node.js
8 isodate>=0.7.2 ISO 8601 date parsing luxon or date-fns Built into most date libraries
9 nltk>=3.8.1 NLP toolkit natural or compromise natural for tokenization/stemming; compromise for NLP
10 pyjwt>=2.10.1 JWT tokens jose (panva/jose) JOSE is the gold standard for JWT in Node.js
11 bcrypt<5.0.0 Password hashing bcrypt or bcryptjs Direct equivalent
12 pyotp>=2.9.0 TOTP/HOTP otplib Full OTP support
13 cryptography>=45.0.6 Crypto primitives node:crypto (built-in) Node.js crypto module covers most use cases
14 pynacl>=1.5.0 NaCl/libsodium bindings sodium-native or tweetnacl tweetnacl for pure JS; sodium-native for native
15 bleach>=6.2.0 HTML sanitizer dompurify + jsdom, or sanitize-html sanitize-html is the closest equivalent
16 lz4>=4.4.4 LZ4 compression lz4 (npm) Native bindings available
17 zstandard>=0.23.0 Zstd compression @aspect/zstd or zstd-codec Multiple options available
18 brotli>=1.1.0 Brotli compression node:zlib (built-in) Node.js has native Brotli support
19 aiofiles>=24.1.0 Async file I/O node:fs/promises (built-in) Native async fs in Node.js
20 python-magic>=0.4.27 MIME type detection file-type or mmmagic file-type is pure JS and well-maintained
21 openpyxl>=3.1.5 Excel file handling exceljs or xlsx exceljs for read/write; xlsx (SheetJS) for parsing
22 httpx>=0.27.0 HTTP client undici / built-in fetch Same as aiohttp above
23 pymongo>=4.0.0 MongoDB driver mongodb (official driver) Official MongoDB Node.js driver
24 motor>=3.7.1 Async MongoDB driver mongodb (official driver) Node.js MongoDB driver is already async
25 asyncpg>=0.30.0 PostgreSQL driver (async) pg (node-postgres) or postgres (porsager/postgres) postgres (porsager) for modern async; pg for ecosystem
26 redis>=6.4.0 Redis client ioredis or redis (node-redis) ioredis preferred for cluster support
27 asyncio-redis>=0.16.0 Async Redis ioredis ioredis is inherently async
28 psycopg2-binary>=2.9.10 PostgreSQL driver pg (node-postgres) Same as asyncpg above
29 sqlalchemy>=2.0.0 ORM / query builder drizzle-orm or prisma or knex Drizzle is closest to SQLAlchemy's philosophy
30 alembic>=1.13.0 Database migrations drizzle-kit or prisma migrate or knex migrate Pair with chosen ORM
31 boto3>=1.40.6 AWS SDK @aws-sdk/client-s3 (AWS SDK v3) Official AWS SDK for JS, modular
32 elasticsearch>=9.1.0 Elasticsearch client @elastic/elasticsearch Official Elasticsearch JS client
33 chardet>=5.2.0 Character encoding detection chardet (npm) or jschardet Direct port exists
34 babel>=2.17.0 i18n / locale @formatjs/intl or i18next i18next for full i18n; Intl API for formatting
35 fuzzywuzzy>=0.18.0 Fuzzy string matching fuse.js or fuzzball fuzzball is a direct port of fuzzywuzzy
36 python-levenshtein>=0.27.1 Levenshtein distance fastest-levenshtein or js-levenshtein Pure JS, fast
37 mmh3>=5.2.0 MurmurHash3 murmurhash3js or murmurhash-native Multiple options
38 anthropic>=0.68.0 Anthropic API client @anthropic-ai/sdk Official Anthropic TypeScript SDK
39 wikipedia-api>=0.8.1 Wikipedia API client wikipedia (npm) or custom fetch Lightweight wrappers available
40 ddgs>=9.6.0 DuckDuckGo search duck-duck-scrape or custom fetch Less mature than Python equivalent
41 google-auth>=2.40.3 Google auth google-auth-library Official Google auth for Node.js
42 google-auth-oauthlib>=1.2.2 Google OAuth google-auth-library Included in the Node.js auth library
43 google-auth-httplib2>=0.2.0 Google auth HTTP google-auth-library Included in the Node.js auth library
44 dropbox>=12.0.2 Dropbox API dropbox (npm) Official Dropbox JS SDK
45 msal>=1.34.0 Microsoft auth (MSAL) @azure/msal-node Official MSAL for Node.js
46 croniter>=6.0.0 Cron expression parser cron-parser or croner croner is modern and well-maintained
47 wolframalpha>=5.1.3 Wolfram Alpha API Custom fetch wrapper No official SDK; simple REST API
48 authlib>=1.6.5 OAuth/OIDC library openid-client or arctic openid-client is the standard; arctic for simpler OAuth

Tier 3 -- Partial Equivalent#

npm packages that cover most but not all of the Python dependency's functionality. Gaps must be filled with custom code or additional packages.

# Minerva (Python) Partial npm Equivalent Gap Description
1 textstat>=0.7.0 text-readability or readability-scores Fewer readability formulas; missing some international indices (e.g., Wiener Sachtextformel). May need custom implementations for Flesch-Kincaid variants.
2 scikit-learn==1.6.1 ml-classify, ml-regression, ml-kmeans (individual) No single equivalent. Individual algorithm packages exist but lack sklearn's unified API, pipelines, and preprocessing. Consider onnxruntime-node for pre-trained sklearn models.
3 numpy==1.26.4 ndarray or @stdlib/ndarray Partial coverage. @stdlib has the most complete numerical computing for JS but still lacks much of numpy's API surface. For heavy numerical work, consider WASM-compiled numpy via Pyodide.
4 scipy==1.13.1 @stdlib/stats, jstat Statistical functions partially covered. Missing: sparse matrices, optimization routines, signal processing, spatial algorithms.
5 sympy>=1.13.1,<1.15 mathjs mathjs covers expression parsing, symbolic differentiation, and some algebra. Missing: advanced symbolic computation, theorem proving, code generation, equation solvers for complex systems.
6 matplotlib>=3.5.0 chart.js, d3, plotly.js Server-side chart generation requires canvas (node-canvas). Different paradigm: Matplotlib generates static images; JS equivalents are primarily browser-based.
7 plotly>=5.0.0 plotly.js Direct JS version exists but designed for browser. Server-side rendering requires headless browser (Puppeteer) or plotly.js + jsdom.
8 spacy>=3.8.7 compromise or wink-nlp Much less capable than spaCy. Missing: named entity recognition (production-grade), dependency parsing, POS tagging models, custom pipeline components. For production NLP, consider calling a spaCy API service.
9 celery[redis]>=5.3.1 bullmq BullMQ covers task queuing and scheduling. Missing: task chaining/choreography (canvas), multi-broker support, built-in monitoring (use separate tools).
10 kombu>=5.5.4 bullmq or amqplib Kombu is a messaging abstraction over AMQP/Redis/SQS. No single JS equivalent; use amqplib for AMQP, ioredis for Redis pub/sub, @aws-sdk/client-sqs for SQS.
11 flower>=2.0.1 bull-board or arena Monitoring dashboard for BullMQ. Less feature-rich than Flower but functional.
12 rouge-score>=0.1.2 rouge (npm) Basic ROUGE-N and ROUGE-L available. May lack ROUGE-W, ROUGE-S, and some bootstrapping options.
13 youtube-transcript-api>=1.2.2 youtube-transcript or youtubei.js Unofficial packages; API stability varies. May break with YouTube changes.
14 opencv-python-headless>=4.11.0.86 opencv-wasm or @techstark/opencv-js OpenCV.js (via WASM) covers core image processing. Missing: video I/O, some ML modules, GPU acceleration. For full OpenCV, use native bindings via opencv4nodejs.
15 pymupdf>=1.23.0 pdf-lib + pdf-parse pdf-lib for creation/modification; pdf-parse for text extraction. Missing: advanced rendering, annotation handling, image extraction quality. Consider pdfjs-dist (Mozilla) for rendering.
16 pdf2image>=1.17.0 pdf-poppler or pdfjs-dist + canvas Requires system poppler or headless rendering. Less straightforward than Python.
17 tabula-py>=2.10.0 pdf-table-extractor or custom (pdfjs-dist) Table extraction from PDFs is weak in JS. May need to call Tabula Java via child process or use a cloud API.
18 pingouin>=0.5.5 jstat + simple-statistics Basic statistical tests available. Missing: advanced ANOVA, mixed-effects models, Bayesian analyses, effect size calculations specific to pingouin.
19 mutagen>=1.47.0 music-metadata Good metadata reading support. Missing: metadata writing for some formats. music-metadata is read-only; use node-taglib-sharp for write support.
20 passlib[bcrypt]>=1.7.4 bcryptjs + argon2 bcrypt and argon2 covered. Missing: passlib's unified API across 30+ hash algorithms, automatic hash upgrading.
21 python-multipart>=0.0.6 busboy or formidable Multipart parsing fully supported. Missing: tight integration with web framework (Express/Fastify handle this natively).
22 pycountry>=24.6.1 i18n-iso-countries + iso-639-1 Country/language codes covered. Missing: subdivisions, currencies, historic countries, scripts databases that pycountry bundles from ISO standards.

Tier 4 -- No Equivalent, Needs Custom Implementation#

These have no adequate npm equivalent. They require custom TypeScript implementations, WASM bridges, native bindings, or architectural alternatives.

# Minerva (Python) Migration Strategy Effort Estimate
1 torch==2.6.0 Use onnxruntime-node for inference. For training, call a PyTorch service via API or use transformers.js (which bundles ONNX Runtime). No native PyTorch for Node.js. High -- architectural redesign for ML inference vs training split
2 torchaudio==2.6.0 Audio feature extraction via meyda (JS) or essentia.js (WASM). For model-based audio processing, route through a Python microservice. High -- no direct equivalent for model-based audio
3 transformers>=4.40.0 @xenova/transformers (transformers.js) for ONNX-based inference. Covers many models but not all. Training requires Python service. Medium-High -- inference works; training needs service
4 peft>=0.10.0 No JS equivalent. LoRA/adapter inference possible via ONNX if adapters are merged. Fine-tuning requires Python service. High -- Python service required for fine-tuning
5 datasets>=2.18.0 No equivalent. Build custom dataset loaders per format (JSON, CSV, Parquet via parquetjs). HuggingFace Hub API can be called via REST. Medium -- custom loaders per dataset format
6 evaluate>=0.4.0 No equivalent. Implement evaluation metrics (BLEU, ROUGE, accuracy, F1) as custom TypeScript functions. Some available via rouge (npm) and custom BLEU. Medium -- individual metric implementations
7 sentence-transformers>=2.7.0 @xenova/transformers supports some sentence-transformer models. For full compatibility, call a sentence-transformers API service. Medium -- partial via transformers.js
8 ray>=2.48.0 No equivalent. For distributed computing: use Node.js worker threads + cluster module for single-machine parallelism; use Kubernetes jobs or BullMQ for distributed workloads. High -- fundamental architecture difference
9 numba>=0.60.0 No equivalent. For JIT-compiled numerical code: write performance-critical loops in Rust/WASM, or use @aspect/worker-threads for parallel computation. High -- requires WASM or native code
10 mediapipe>=0.10.11 @mediapipe/tasks-vision (official JS SDK) exists but is browser-only. For server-side, use a MediaPipe Python service or the C++ API with native bindings. Medium-High -- browser SDK exists but server-side gaps
11 pytesseract>=0.3.10 tesseract.js (WASM-based OCR) Medium -- WASM version exists but slower than native; quality is equivalent
12 chempy>=0.10.1 No equivalent. Implement required chemistry calculations (stoichiometry, equilibria, kinetics) as custom TypeScript. Consider openchemlib-js for structural chemistry. High -- niche domain, custom implementation required
13 girth>=0.8.0 No equivalent. Implement Item Response Theory (IRT) parameter estimation in TypeScript. Core algorithms (EM, MCMC for 1PL/2PL/3PL) must be custom-built. Partially addressed by @metis/assessment/irt-models. High -- domain-specific statistical algorithms
14 tavily-python>=0.7.13 Custom REST client wrapping Tavily's API. Simple HTTP calls; no SDK needed. Low -- just REST API wrapper
15 runpod>=1.3.5 Custom REST client wrapping RunPod's API. They have a GraphQL API; use any GraphQL client. Low -- just API wrapper

Tier 5 -- Not Needed in TypeScript#

Python-specific packages for runtime, build tooling, web framework, or capabilities that are inherent to Node.js or handled differently in the TypeScript ecosystem.

# Minerva (Python) Reason Not Needed
1 hatchling (build-system) Python build backend. TypeScript uses tsup, esbuild, or tsc.
2 typer[all]>=0.9.0 Python CLI framework. Use commander, yargs, or clipanion in Node.js. Already handled by monorepo CLI tooling.
3 fastapi>=0.104.0 Python web framework. Use fastify, hono, or express in Node.js. Framework choice is an app-level decision, not a library dep.
4 uvicorn[standard]>=0.24.0 Python ASGI server. Node.js has its own HTTP server built-in.
5 psutil>=7.0.0 System info in Python. Use node:os built-in + systeminformation npm package if needed.
6 py-cpuinfo>=9.0.0 CPU info in Python. Use node:os.cpus() built-in.
7 ray>=2.48.0 Listed in Tier 4 for distributed computing, but the process model aspect is N/A -- Node.js uses worker_threads and cluster natively.
8 pytest>=7.0 (dev) Python test framework. Using Vitest.
9 pytest-* (all dev variants) Python test plugins. Vitest ecosystem has equivalents where needed.
10 ruff>=0.1.0 (dev) Python linter/formatter. Using ESLint + Prettier.
11 mypy>=1.17.1 (dev) Python type checker. TypeScript compiler handles this natively.
12 coverage>=7.10.2 (dev) Python coverage. Using Vitest with @vitest/coverage-v8.
13 faker>=20.0.0 (dev) Test data generation. Use @faker-js/faker (already common in JS ecosystem).
14 hypothesis>=6.141.1 (dev) Property-based testing. Use fast-check in TypeScript.
15 schemathesis>=3.19.0 (dev) API fuzzing. Use schemathesis via Python or dredd/stepci in Node.js.
16 locust>=2.20.0 (dev) Load testing. Use k6, autocannon, or artillery in Node.js.
17 bandit>=1.7.5 (dev) Python security linter. Use eslint-plugin-security or snyk.
18 sphinx>=7.4.7 (dev) Python docs generator. Use typedoc for TypeScript.
19 docker>=7.1.0 (dev) Docker API from Python. Use dockerode if needed, but typically handled via CLI/CI.
20 playwright>=1.55.0 (dev) Same in both ecosystems. Use @playwright/test (official Node.js package).
21 memory-profiler>=0.61.0 (dev) Python memory profiling. Use node --inspect + Chrome DevTools or clinic.js.
22 types-* (all dev type stubs) Python type stubs for mypy. Not needed in TypeScript; types come from @types/* or are bundled.

Complete Dependency Matrix#

All 117 production dependencies in a single reference table, sorted alphabetically.

# Python Dependency Version Tier TypeScript Equivalent npm Package
1 aiofiles >=24.1.0 2 Built-in async fs node:fs/promises
2 aiohttp >=3.12.15 2 Built-in fetch / HTTP client undici / native fetch
3 alembic >=1.13.0 2 Database migrations drizzle-kit / prisma migrate
4 anthropic >=0.68.0 2 Anthropic SDK @anthropic-ai/sdk
5 asyncio-redis >=0.16.0 2 Async Redis client ioredis
6 asyncpg >=0.30.0 2 Async PostgreSQL driver postgres / pg
7 authlib >=1.6.5 2 OAuth/OIDC library openid-client / arctic
8 babel >=2.17.0 2 Internationalization i18next / @formatjs/intl
9 bcrypt <5.0.0 2 Password hashing bcrypt / bcryptjs
10 beautifulsoup4 >=4.13.4 2 HTML parser cheerio
11 bleach >=6.2.0 2 HTML sanitizer sanitize-html
12 boto3 >=1.40.6 2 AWS SDK @aws-sdk/client-s3 (modular v3)
13 brotli >=1.1.0 2 Brotli compression node:zlib (built-in)
14 celery[redis] >=5.3.1 3 Task queue bullmq
15 chardet >=5.2.0 2 Encoding detection chardet / jschardet
16 chempy >=0.10.1 4 Custom chemistry implementation N/A -- custom build
17 croniter >=6.0.0 2 Cron parser cron-parser / croner
18 cryptography >=45.0.6 2 Crypto primitives node:crypto (built-in)
19 datasets >=2.18.0 4 Custom dataset loaders N/A -- custom build
20 ddgs >=9.6.0 2 DuckDuckGo search duck-duck-scrape
21 dropbox >=12.0.2 2 Dropbox SDK dropbox
22 elasticsearch >=9.1.0 2 Elasticsearch client @elastic/elasticsearch
23 email-validator >=2.2.0 1 Built into @metis/core @metis/core/validation
24 evaluate >=0.4.0 4 Custom metric implementations N/A -- custom build
25 fastapi >=0.104.0 5 N/A -- Python web framework fastify / hono (app-level)
26 feedparser >=6.0.11 2 RSS parser rss-parser
27 flower >=2.0.1 3 Task queue dashboard bull-board
28 fuzzywuzzy >=0.18.0 2 Fuzzy matching fuzzball
29 girth >=0.8.0 4 Custom IRT implementation @metis/assessment/irt-models (partial)
30 google-api-python-client >=2.178.0 2 Google APIs googleapis
31 google-auth >=2.40.3 2 Google auth google-auth-library
32 google-auth-httplib2 >=0.2.0 2 Google auth HTTP google-auth-library
33 google-auth-oauthlib >=1.2.2 2 Google OAuth google-auth-library
34 httpx >=0.27.0 2 HTTP client undici / native fetch
35 isodate >=0.7.2 2 ISO date parsing luxon / date-fns
36 jsonschema >=4.25.0 1 Built into @metis/core @metis/core/validation
37 kombu >=5.5.4 3 Messaging abstraction amqplib + ioredis
38 litellm >=1.34.0 2 Multi-LLM client ai (Vercel AI SDK)
39 lz4 >=4.4.4 2 LZ4 compression lz4
40 matplotlib >=3.5.0 3 Charting chart.js / d3 + canvas
41 mediapipe >=0.10.11 4 Browser SDK only @mediapipe/tasks-vision (browser)
42 mmh3 >=5.2.0 2 MurmurHash murmurhash3js
43 motor >=3.7.1 2 Async MongoDB mongodb
44 msal >=1.34.0 2 Microsoft auth @azure/msal-node
45 mutagen >=1.47.0 3 Audio metadata music-metadata
46 networkx >=3.2.1 1 Graph structures @metis/knowledge-graph
47 networkx >=3.0 1 Graph structures (dupe) @metis/knowledge-graph
48 nltk >=3.8.1 2 NLP toolkit natural / compromise
49 numba >=0.60.0 4 JIT compilation WASM / Rust native modules
50 numpy ==1.26.4 3 Numerical computing @stdlib/ndarray / ndarray
51 opencv-python-headless >=4.11.0.86 3 Computer vision opencv-wasm / @techstark/opencv-js
52 openpyxl >=3.1.5 2 Excel files exceljs
53 passlib[bcrypt] >=1.7.4 3 Password hashing bcryptjs + argon2
54 pdf2image >=1.17.0 3 PDF to image pdf-poppler / pdfjs-dist + canvas
55 peft >=0.10.0 4 LoRA/adapters N/A -- Python service
56 pillow >=11.3.0 2 Image processing sharp
57 pingouin >=0.5.5 3 Statistical testing jstat + simple-statistics
58 plotly >=5.0.0 3 Interactive charts plotly.js
59 protobuf >=4.25.8 1 Protocol buffers protobufjs
60 psutil >=7.0.0 5 System info node:os (built-in)
61 psycopg2-binary >=2.9.10 2 PostgreSQL driver pg
62 py-cpuinfo >=9.0.0 5 CPU info node:os.cpus() (built-in)
63 pycountry >=24.6.1 3 Country/language codes i18n-iso-countries + iso-639-1
64 pydantic >=2.0 1 Validation/types @metis/core (Zod-based)
65 pydantic-settings >=2.0 1 Config management @metis/core/constants
66 pyjwt >=2.10.1 2 JWT handling jose
67 pymongo >=4.0.0 2 MongoDB driver mongodb
68 pymupdf >=1.23.0 3 PDF processing pdf-lib + pdf-parse
69 pynacl >=1.5.0 2 NaCl crypto tweetnacl / sodium-native
70 pyotp >=2.9.0 2 OTP generation otplib
71 python-levenshtein >=0.27.1 2 String distance fastest-levenshtein
72 python-magic >=0.4.27 2 MIME detection file-type
73 python-multipart >=0.0.6 3 Multipart parsing busboy / formidable
74 pytesseract >=0.3.10 4 OCR tesseract.js (WASM)
75 pyyaml >=6.0 2 YAML parsing yaml
76 ray >=2.48.0 4 Distributed computing Custom (worker_threads + BullMQ)
77 redis >=6.4.0 2 Redis client ioredis
78 rouge-score >=0.1.2 3 ROUGE metrics rouge (npm)
79 runpod >=1.3.5 4 RunPod API Custom REST/GraphQL client
80 scikit-learn ==1.6.1 3 ML algorithms ml-* packages / onnxruntime-node
81 scipy ==1.13.1 3 Scientific computing @stdlib/stats / jstat
82 sentence-transformers >=2.7.0 4 Sentence embeddings @xenova/transformers (partial)
83 spacy >=3.8.7 3 NLP pipeline compromise / wink-nlp
84 sqlalchemy >=2.0.0 2 ORM drizzle-orm / prisma
85 sympy >=1.13.1 3 Symbolic math mathjs
86 tabula-py >=2.10.0 3 PDF table extraction pdf-table-extractor (weak)
87 tavily-python >=0.7.13 4 Tavily search API Custom REST client
88 textstat >=0.7.0 3 Readability scoring text-readability
89 toml >=0.10.2 1 TOML parsing Built-in / monorepo tooling
90 torch ==2.6.0 4 ML inference onnxruntime-node / @xenova/transformers
91 torchaudio ==2.6.0 4 Audio ML meyda / essentia.js (partial)
92 transformers >=4.40.0 4 HuggingFace models @xenova/transformers (inference only)
93 uvicorn[standard] >=0.24.0 5 N/A -- Python ASGI server Node.js HTTP built-in
94 validators >=0.35.0 1 Built into @metis/core @metis/core/validation
95 websockets >=15.0.1 1 Built into @metis/api-client @metis/api-client/websocket
96 wikipedia-api >=0.8.1 2 Wikipedia client wikipedia (npm)
97 wolframalpha >=5.1.3 2 Wolfram Alpha API Custom fetch wrapper
98 youtube-transcript-api >=1.2.2 3 YouTube transcript youtube-transcript
99 typer[all] >=0.9.0 5 N/A -- Python CLI commander (app-level)
100 fastapi >=0.104.0 5 N/A -- Python web framework fastify / hono (app-level)
101 passlib[bcrypt] >=1.7.4 3 Password hashing bcryptjs + argon2
102 python-multipart >=0.0.6 3 Multipart parsing busboy
103 psutil >=7.0.0 5 System monitoring node:os built-in
104 py-cpuinfo >=9.0.0 5 CPU detection node:os.cpus()

Note: Several dependencies appear twice in pyproject.toml (e.g., networkx at >=3.2.1 and >=3.0). Duplicates are counted once in the tier summaries but listed separately in this matrix for traceability.


Priority Migration Order#

Dependencies are grouped into migration waves based on how many @metis/* packages depend on their functionality and how critical they are to core features.

Wave 1 -- Foundation (Critical Path)#

These block most other work and should be migrated first.

Priority Dependency Area Python Packages TypeScript Strategy
P0 Validation & Types pydantic, pydantic-settings, jsonschema, validators, email-validator Done -- @metis/core
P0 LLM Client litellm, anthropic Vercel AI SDK (ai) + @anthropic-ai/sdk
P0 Data Models pydantic (model definitions) Done -- @metis/models
P1 Database Layer sqlalchemy, alembic, asyncpg, psycopg2-binary drizzle-orm + drizzle-kit + postgres
P1 Cache Layer redis, asyncio-redis ioredis
P1 HTTP Client aiohttp, httpx Native fetch / undici

Wave 2 -- Content Processing#

Core content generation and processing capabilities.

Priority Dependency Area Python Packages TypeScript Strategy
P1 NLP nltk, spacy, textstat natural + compromise + text-readability
P1 PDF Processing pymupdf, pdf2image, tabula-py, pytesseract pdf-lib + pdf-parse + tesseract.js
P1 Image Processing pillow, opencv-python-headless sharp + opencv-wasm
P2 YAML/Config pyyaml, toml yaml (npm) + built-in
P2 HTML Processing beautifulsoup4, bleach cheerio + sanitize-html

Wave 3 -- ML & AI#

Machine learning inference and model capabilities.

Priority Dependency Area Python Packages TypeScript Strategy
P2 ML Inference torch, transformers, sentence-transformers @xenova/transformers + onnxruntime-node
P2 ML Metrics rouge-score, evaluate rouge (npm) + custom metric implementations
P2 ML Utilities scikit-learn, numpy, scipy @stdlib packages + onnxruntime-node for pre-trained models
P3 Fine-tuning peft, datasets Python microservice (keep in Python)
P3 Audio ML torchaudio, mediapipe meyda + Python microservice for complex tasks

Wave 4 -- Infrastructure & Integrations#

External service integrations and infrastructure.

Priority Dependency Area Python Packages TypeScript Strategy
P2 Task Queue celery, kombu, flower bullmq + bull-board
P2 Search elasticsearch, fuzzywuzzy, python-levenshtein @elastic/elasticsearch + fuzzball
P2 Object Storage boto3 @aws-sdk/client-s3
P2 MongoDB pymongo, motor mongodb
P3 Google APIs google-api-python-client, google-auth-*, youtube-transcript-api googleapis + google-auth-library
P3 Cloud Auth dropbox, msal, authlib dropbox + @azure/msal-node + openid-client

Wave 5 -- Specialized & Niche#

Domain-specific and low-priority capabilities.

Priority Dependency Area Python Packages TypeScript Strategy
P3 Security pyjwt, bcrypt, pyotp, cryptography, pynacl jose + bcryptjs + otplib + node:crypto + tweetnacl
P3 Compression lz4, zstandard, brotli lz4 + @aspect/zstd + node:zlib
P3 Data Formats openpyxl, chardet, protobuf exceljs + chardet + protobufjs
P4 Distributed ray, numba worker_threads + WASM/Rust for hot paths
P4 Chemistry chempy Custom TypeScript implementation
P4 IRT Stats girth, pingouin @metis/assessment (extend) + custom
P4 Charts matplotlib, plotly plotly.js + chart.js (browser-oriented)
P4 i18n babel, pycountry i18next + i18n-iso-countries
P4 Media Metadata mutagen music-metadata
P4 Search APIs tavily-python, ddgs, wolframalpha, wikipedia-api, runpod Custom REST wrappers

Risk Assessment#

Tier 1 -- Already Ported#

Risk Level Assessment
LOW These are already implemented and tested in the Oshun monorepo. The primary risk is feature parity gaps between the Python originals and the TypeScript ports. Mitigation: maintain a feature comparison checklist per library and fill gaps incrementally.

Tier 2 -- Direct Equivalent Available#

Risk Level Assessment
LOW-MEDIUM Mature npm packages exist with large user bases and active maintenance. Risks include: (1) API surface differences requiring adapter patterns, (2) behavioral differences in edge cases (e.g., cheerio vs beautifulsoup4 handle malformed HTML differently), (3) some packages may have different default configurations. Mitigation: write adapter layers that normalize behavior; comprehensive integration tests comparing outputs against Python reference implementations.

Tier 3 -- Partial Equivalent#

Risk Level Assessment
MEDIUM The biggest risk category by impact. These packages cover 60-80% of functionality but leave gaps that could block features. Specific high-risk items:
- scikit-learn: No unified ML framework in JS. Must use piecemeal packages or ONNX export. Risk of subtle numerical differences.
- spaCy: NLP quality in JS is significantly below Python. Named entity recognition and dependency parsing in compromise/wink-nlp are not production-grade for all languages.
- numpy/scipy: Numerical computing in JS is fragmented. @stdlib helps but is not a drop-in replacement.
- tabula-py: PDF table extraction in JS is weak. This may require keeping a Python microservice.
Mitigation: (1) Build abstraction layers that can swap implementations, (2) For critical NLP/ML, consider a Python sidecar service, (3) Benchmark partial equivalents early to identify blocking gaps before full migration.

Tier 4 -- Needs Custom Implementation#

Risk Level Assessment
HIGH These represent the highest technical risk. Key concerns:
- PyTorch/Transformers: The ML ecosystem gap between Python and JS is fundamental. @xenova/transformers covers inference for many models but not all. Training and fine-tuning must stay in Python.
- Ray: No equivalent distributed computing framework. Must re-architect around Node.js primitives (worker_threads, cluster) and infrastructure tools (Kubernetes, BullMQ).
- chempy/girth: Niche scientific computing with no JS equivalents. Must be built from scratch or accessed via WASM/Python service.
Mitigation: (1) Adopt a hybrid architecture where Python services handle ML training and complex scientific computing, (2) Use ONNX Runtime for model inference in Node.js, (3) Budget significant engineering time for custom implementations, (4) Consider Rust+WASM for performance-critical numerical code.

Tier 5 -- Not Needed in TypeScript#

Risk Level Assessment
NONE These are Python-specific and have no migration risk. The TypeScript ecosystem has its own equivalents for build tooling, testing, and runtime capabilities. No action required beyond choosing the TypeScript-native alternatives (which are already in use in the monorepo).

Overall Migration Risk Summary#

Risk Factor Severity Mitigation
ML/AI ecosystem gap (torch, transformers, peft) Critical Hybrid architecture: Python services for training, ONNX for inference
NLP quality gap (spaCy equivalent) High Python sidecar for production NLP; JS for lightweight text analysis
Numerical computing gap (numpy, scipy) High @stdlib for basics; WASM/Rust for hot paths; Python service for heavy math
PDF table extraction (tabula-py) Medium Tabula Java via child_process; or cloud-based extraction API
Distributed computing (Ray) Medium Re-architect around BullMQ + worker_threads + Kubernetes
API behavior differences Low Adapter layers + integration tests comparing against Python reference
Package maintenance risk Low Prefer packages with >10k weekly downloads and active GitHub repos

For Tier 4 dependencies, the recommended architecture is:

text
                    +--------------------+
                    |   Metis (Node.js)  |
                    |   TypeScript Core  |
                    +--------+-----------+
                             |
              +--------------+--------------+
              |              |              |
     +--------v---+  +------v------+  +----v--------+
     | LLM APIs   |  | ONNX Runtime|  | Python Svc  |
     | (direct)   |  | (inference) |  | (training)  |
     +------------+  +-------------+  +-------------+
                                      | torch        |
                                      | transformers |
                                      | spacy        |
                                      | scikit-learn |
                                      | ray          |
                                      +-------------+

This keeps the TypeScript core lightweight and fast while offloading heavy ML/scientific workloads to purpose-built Python services that can run on GPU infrastructure.