Eve library

Phase 149 implementation evidence

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

This ledger records direct implementation evidence for checked Phase 149 requirements. Unchecked requirements remain open. A type name, adapter seam, or unit simulation is not treated as completion unless executable behavior and focused validation cover the requirement.

149.1 Event ingestion and SDKs#

Implementation roots:

  • libs/neith/crash-telemetry/src/ingestion.ts
  • libs/neith/crash-telemetry/src/platform-integrations.ts
  • sdks/neith-crash/
Requirements Direct evidence
149.1.1 installBrowserHandlers captures browser errors and unhandled rejections, preserves file/line/column source coordinates for later source-map resolution, emits source-map-aware context, and removes both listeners cleanly.
149.1.2 CrashClient supplies the Node transport while nodeErrorMiddleware, fastifyErrorHandler, and NestCrashFilter exercise the error-handler shapes and retain request transaction context.
149.1.3 The Swift 6.3 package targets iOS 16/macOS 13, emits Sentry envelopes, gates consent, scrubs PII, synchronizes and atomically compacts its replay journal, and accepts a PLCrashReporter/MetricKit-style pending-report bridge that purges only after durable delivery. Strict warnings-as-errors builds and three Swift Testing cases pass.
149.1.4 The Android/JVM SDK installs and restores the process uncaught-exception handler, keeps an atomic consent gate and durable journal, and validates an MDMP NDK tombstone supplied by its JNI-facing NativeBridge. javac -Xlint:all -Werror and the executable fixture pass on Java 17.
149.1.5 The Rust crate captures forced standard-library backtraces, wraps typed errors with ordered context, installs a chaining panic hook, scrubs payloads, and journals before transport. Unit tests, Clippy with warnings denied, and Rustfmt pass on Rust 1.95.
149.1.6 The Go 1.22 SDK wraps errors with %w, captures/rethrows panics with runtime/debug stacks, blocks pre-consent collection, scrubs PII before a base64 journal write, fsyncs and atomically compacts the queue, and emits decoded Sentry envelopes through an injected transport. Four race-enabled tests plus go vet and gofmt pass.
149.1.7 The dependency-free Python SDK exposes init, capture_message, capture_exception, configure_scope, and flush, emits a Sentry envelope, strips PII, fsyncs its offline queue, and replays it after client reconstruction. Three standard-library unit tests pass on Python 3.12.
149.1.8 The C++20 header accepts a Breakpad/Crashpad bridge, validates the native MDMP header and build ID, journals native and handled events, honors consent, and replays through an injected transport. Its fixture compiles under -Wall -Wextra -Werror -pedantic and passes.
149.1.9 The importable Unity Package Manager package contains the required manifest and runtime .asmdef, a MonoBehaviour bootstrap, threaded Unity exception hook, iOS CrashReport replay/removal, DSN HTTP transport, consent gate, recursive PII scrubbing, and fsync/atomic offline replay. Its Unity 6 API-contract build runs warnings-as-errors and an executable three-flow verification.
149.1.10 The Unreal 5.5 runtime plugin hooks FCoreDelegates::OnHandleSystemError, appends scrubbed fatal events to the project crash journal, exposes explicit capture/consent APIs, and unregisters safely. Unreal AutomationTool BuildPlugin passes Linux Editor Development, Game Development, and Game Shipping builds.
149.1.11 The Dart 3.12 package captures errors/messages, provides guarded-zone and chain-preserving Flutter host bridges, blocks collection without consent, recursively scrubs PII, fsyncs a base64 offline journal, atomically compacts it after recovery, and emits byte-length Sentry envelopes. Analyze and three package tests pass.
149.1.12 The React Native adapter chains the existing JS global handler, distinguishes fatal errors, installs/uninstalls the native bridge, and restores the previous handler.
149.1.13 Electron main-process exception/rejection hooks and renderer browser hooks capture process-specific context and remove every listener during teardown.
149.1.14 The runnable Node sidecar validates configured project/public-key pairs, enforces a payload bound, fsyncs uniquely named envelopes into a per-project spool, and exposes health, readiness, and Prometheus counters. Its hardened non-root/read-only-root Kubernetes deployment and multi-stage image build pass, and the built container answers its readiness probe.
149.1.15–.16 The bridge decodes length-checked Sentry envelopes, validates path-project/auth-key agreement, maps failed OTLP spans to trace-correlated error events, and serves the canonical path-prefixed DSN envelope endpoint.
149.1.17–.18 The bounded client queue honors flush limits and Retry-After, applies jittered capped exponential backoff, checksum-validates every record, fsyncs append/remove operations, survives process-style reconstruction, and atomically compacts its journal.
149.1.19 Recursive SDK-side scrubbing removes sensitive keys, email addresses, IPv4 addresses, bearer tokens, and home-directory identities before envelope encoding; custom key/value rules are covered too. Equivalent local scrubbing runs in the Python, Rust, C++, Android, and Unreal SDKs.
149.1.20 Channel selection rejects incompatible runtimes, insecure artifact URLs and malformed digests, chooses the newest compatible semantic version, downloads and SHA-256 verifies the artifact, stages then activates it, and rolls the staged update back when activation fails.

Explicitly still open in 149.1#

No 149.1 implementation rows remain open.

Verification log#

  • pnpm test, pnpm typecheck, and pnpm lint pass for @neith/crash-telemetry (28 tests at this checkpoint), and the bundled sidecar executable passes its live health/readiness/metrics smoke test.
  • Python standard-library suite: 3 passing tests.
  • Rust unit/doc tests, Clippy with warnings denied, and Rustfmt check pass.
  • Android/JVM compilation with all lint warnings denied and the executable test fixture pass.
  • C++20 warnings-as-errors compilation and executable fixture pass.
  • Unreal 5.5 AutomationTool standalone plugin packaging passes for Linux.
  • The multi-stage sidecar image builds as neith/crash-sidecar:0.1.0, runs as UID/GID 65532, and passes a live container readiness probe.

149.2 Symbolication and deminification#

Implementation root: libs/neith/crash-telemetry/src/symbolication.ts.

Requirements Direct evidence
149.2.1 The acceptance fixture cross-compiles a genuine Apple arm64 iOS Mach-O object with DWARF, builds its standard .dSYM bundle with dsymutil, and resolves the exact demangled function/file/line/column through llvm-symbolizer.
149.2.2 The ProGuard/R8 parser reconstructs original class and method names and translates obfuscated line ranges into source lines.
149.2.3, .11–.12 CI cohort upload accepts typed Breakpad/NDK artifacts, validates release/run metadata, SHA-256 pins content, stores it behind an S3-compatible object-client boundary, rejects conflicting debug IDs, and purges each release cohort at its configured horizon.
149.2.4 A source-map v3 VLQ decoder resolves the nearest generated segment, carries names and embedded source, detects loops, and walks up to eight chained maps.
149.2.5 The acceptance fixture cross-compiles a genuine Windows x64 PE plus MSVC 7.0 PDB with Clang/LLD, then resolves an RVA to the exact function, source file, and line through bounds-checked llvm-pdbutil output.
149.2.6 The acceptance fixture compiles a real Linux C++ binary with DWARF, splits debug data with objcopy --only-keep-debug, strips the executable, adds its GNU debug-link, and resolves the original demangled function/file/line through addr2line.
149.2.7 Real Rust, Go, and Swift 6.3.2 binaries compile with native debug data. Rust resolves through DWARF demangling, Go through go tool addr2line, and Swift through DWARF plus swift-demangle; every fixture returns its language-native function and exact source coordinates.
149.2.8 A bounds-checked WebAssembly binary parser reads the standard function-name subsection, while the DWARF resolver parses an LLVM line table by function index/instruction offset. The acceptance fixture compiles real C to wasm32 with Clang 18 -g, then resolves checkout_total to its exact source file, line, and column.
149.2.9 The IL2CPP resolver parses Unity's nested LineNumberMappings.json schema, ignores its debug-ID sentinel, normalizes Windows/POSIX generated paths, applies the documented nearest-line/50-line bound, and can extract mappings directly from real //<source_info:file:line> markers in generated line.cpp content.
149.2.10 Visual Studio [Callstack] and AddressSanitizer profiles implement the exact frame contracts in UE 5.5 Gauntlet.UnrealLogParser/SanitizerEventMatcher, retain module/address/build-ID/source coordinates, and late-resolve unknown native frames. A real Clang ASan use-after-free fixture parses back to main and its exact C++ line.
149.2.13 The late-symbol coordinator indexes unresolved debug IDs and reprocesses only events matching the uploaded artifact's project, release, and debug ID.
149.2.14–.15 Resolved frames include a bounds-safe ±5-line source window and accept release-pinned per-frame blame with culprit commit, author, and summary.

No 149.2 implementation rows remain open.

149.3 Fingerprinting, grouping, and deduplication#

Implementation root: libs/neith/crash-telemetry/src/grouping.ts.

Requirements Direct evidence
149.3.1–.5 Volatile addresses, IDs, numbers, paths, generic suffixes, in-app frames, exception types and messages normalize into strict/loose fingerprints. Bounded wildcard or regex rules override grouping; Jaccard message similarity and cosine embedding clustering provide fuzzy and ML grouping without crossing projects.
149.3.6–.8 Split validates membership and moves only selected event assignments; merge is project-bounded, retains original groups for unmerge, and rewrites dedupe assignments. Repeated event IDs return their existing issue atomically.
149.3.9–.12 The validated state machine covers unresolved/resolved/ignored/reopened, a new event reopens a resolved issue and increments regression count, next-release auto-resolution excludes issues seen in that release, and snooze wakes by time, count, or next occurrence.
149.3.14–.15 Release can be included in the fingerprint scope, while explicit cross-project links bind two or more existing issues with a reason without merging tenant/project state.

The crash-operations issue header exposes merge and unmerge actions with a live result state; component and Playwright tests exercise both operations.

149.4 Event storage and query#

Implementation root: libs/neith/crash-telemetry/src/storage.ts.

Requirements Direct evidence
149.4.1 The ClickHouse HTTP adapter creates a partitioned ReplacingMergeTree with low-cardinality dimensions, token index, 365-day TTL, parameterized project queries, JSONEachRow batch ingest, and hourly aggregation. A memory-capped ClickHouse 25.3 container passes functional and sustained-throughput acceptance.
149.4.2 Full JSON payloads are stored separately under project/event/content-digest keys through a tested S3-compatible object-client adapter; reads verify SHA-256 before parsing.
149.4.3–.4 Project retention admits exactly the 7/30/90/365-day horizons and deletes metadata plus payload. SHA-256 bucket sampling is stable across retries and validates its 0–1 rate.
149.4.5–.7 Metadata search covers user, release, tag, environment, URL and custom fields; case-insensitive full text spans message plus trace; named saved searches emit notifications only at their configured count threshold.
149.4.9–.10 A validated SQL-like AND query parser supports equality, inequality, comparison, contains and full-text clauses. Hourly aggregation emits deterministic counts per issue.

Explicitly still open in 149.4#

  • The operator query builder exposes ordered filter clauses with accessible Alt+Arrow reordering; component and browser tests verify the resulting order.

149.5 Release health and tracking#

Implementation root: libs/neith/crash-telemetry/src/releases.ts.

Requirements Direct evidence
149.5.1–.4 Manual or idempotent CI releases validate unique commit ranges and version identity. Deploys carry environment/time and validated terminal transitions. Session/user crash-free rates derive from actual session terminal states.
149.5.6–.10 Issue/release pairs retain first/last/count. Earliest occurrence identifies the introducing release; the newest matching file commit in that release diff identifies the culprit. A/B comparison returns crash-free and issue-count deltas, and CI creation is explicitly tagged.
149.5.11–.12 The SDK state machine covers start, heartbeat, crash and exit, distinguishes abnormal 30-second heartbeat loss, and closes a backgrounded session at the background boundary after more than 30 seconds.
149.5.13–.15 Gates require a minimum population, alert only when crash rate exceeds the threshold, and call distinct halt or rollback adapters for Phase 145 orchestration.

The portfolio table renders each release's adoption as a labelled progress graph and numeric value; component and browser coverage verify it.

149.6 Session replay and user context#

Implementation root: libs/neith/crash-telemetry/src/replay.ts.

Requirements Direct evidence
149.6.1–.6 The bounded recorder deterministically samples by session, honors consent, captures DOM/CSS/safe assets plus console/network/click records, recursively masks input/native-view values, and applies tag/class/id mask or block selectors before storage.
149.6.8–.11 Error records index event IDs; click analysis distinguishes error, dead, and three-click/one-second rage bursts; native accessibility trees are retained with concealed values; the player derives request duration for its network waterfall.
149.6.12–.14 Feedback validates an event/replay tie and screenshot magic-sized payload while stripping email; cookie/local-storage keys are redacted; bounded replay records purge against independent retention cutoffs.
149.6.15 The acceptance test renders two RGBA frames through the system FFmpeg 6.1 libx264 encoder and verifies the resulting MP4 ftyp container header.

The replay player exposes a labelled 0–24 second scrubber, 0.5×/1×/2× speed, concealed session frame, and network waterfall; tests scrub it to a new time.

149.7 Tracing and performance#

Implementation root: libs/neith/crash-telemetry/src/performance.ts.

Requirements Direct evidence
149.7.1 Trace ingest validates OTel-width trace/span IDs, timestamps, status, attributes and parent relations and deduplicates spans within a trace.
149.7.3–.4 Normalized SQL groups repeated statements under the same parent into N+1 findings; independent duration thresholds classify slow database and HTTP spans.
149.7.5–.8 Exact nearest-rank p50/p75/p95/p99, Core Web Vital grades including optional legacy FID and current INP, cold/warm/resume first-frame/interactive durations, and weighted collapsed CPU stacks are all executable and validated.
149.7.10 Candidate p95 is compared against baseline p95 with a configurable relative-regression threshold and reports both values plus the relative increase.

The issue view renders a nested duration flamegraph beside a correlation rail for the issue, OpenTelemetry trace, and replay; Playwright asserts its database span through the rendered view.

149.8 Alerts and integrations#

Implementation root: libs/neith/crash-telemetry/src/alerting.ts.

Requirements Direct evidence
149.8.1–.3 Rules cover new issue, regression, rolling frequency and named metrics with environment filters, thresholds and cooldowns. Phase 152 escalation is an explicit async adapter, and daily/weekly digests bound time and rank by impacted users/events.
149.8.6 Webhook bodies are signed over timestamp.body with HMAC-SHA256, while the destination dispatcher sends each alert/destination pair once.
149.8.7–.8 Owner routing combines bounded path wildcards, tags, teams and CODEOWNERS, then falls back to culprit commit author only when no explicit owner matches.
149.8.9–.10 Moving-baseline spike protection suppresses extreme bursts; explicit project silence windows and per-rule/issue cooldowns suppress repeat notifications.
149.8.4–.5, .11–.14 Concrete HTTP destinations emit platform-correct Slack/Mattermost text, Discord content, Teams adaptive-card, sovereign-chat crash-alert, and sovereign notification-gateway email/SMS/push bodies. Authenticated work-item and PR-comment publishers cover Jira, Linear, sovereign PM, GitHub and sovereign forge with idempotent external/dedup keys. The test executes all 11 outbound contracts and HMAC/auth headers.

The admin-mobile Incidents screen includes a touch-first Neith crash inbox with acknowledge and assign transitions, live state, stable automation IDs, and an extended Maestro flow. Three model tests verify first-responder preservation, assignment/reassignment, and identity validation. Maestro is installed, but no Android emulator or iOS simulator is present on this host, so the authored flow could not be executed locally.

149.9 Privacy, compliance, and self-hosting#

Implementation root: libs/neith/crash-telemetry/src/security.ts.

Requirements Direct evidence
149.9.1–.2 Server rules reject oversized/nested-quantifier regexes, force global replacement, apply named replacements recursively, and return an exact field/index-path preview ledger of before/after masking.
149.9.3–.4 Projects fail closed without a region pin and require an explicitly approved from/to migration. SOC2/HIPAA/GDPR events form a sequence-checked HMAC hash chain with actor/action/project controls.
149.9.5 Customer key providers select a project/key version; AES-256-GCM uses a random 96-bit IV, project-bound AAD and authentication tag, and fails when decrypted under another project.
149.9.6–.8 Retention jobs invoke independent targets at their day cutoffs; erasure fans out across stores, emits only a subject digest and HMAC proof, and the SDK capture paths stop before journaling when consent is denied.

| 149.9.9–.10 | Runnable deployment artifacts now cover hardened single-node Compose, three-replica zone-spread Kubernetes HA with disruption/autoscaling controls, explicitly pinned EU/US regional profiles, and an air-gap exporter/verifier that packages pre-pulled images plus manifests under SHA-256 checksums. |

Deployment verification parses every Kubernetes document, renders the Compose profile, creates and verifies a real 58 MB sidecar image bundle, and boots the single-node profile on port 17314 until its live /readyz probe returns ready.

149.10 AI-assisted triage#

Implementation root: libs/neith/crash-intelligence/src/index.ts.

Requirements Direct evidence
149.10.1, .4 NousTriageAssistant consumes the Phase 47 @nous/llm completion-provider contract, supplies a deterministic injection-resistant prompt, extracts separate root-cause/fix sections, records provider/model identity, and fails closed on malformed output.
149.10.2, .8 SimilarIssueIndex consumes the Phase 47 @nous/core embedding-provider contract for cosine search and clustering. Cross-tenant results require current bilateral opt-in; revocation removes visibility immediately.
149.10.3 KnownPatternFixer matches only reviewed patterns, requires a non-empty patch, and calls an injected forge gateway with a human-review-required draft PR. Unknown patterns produce no external action.
149.10.5–.7 Severity derives from observed user/session impact plus fatal, regression, and environment signals. Known noise derives confidence from event evidence and never ignores a fatal cancellation. Test flakiness requires mixed pass/fail history across commits and zero production occurrences.
149.10.9 The append-only feedback dataset retains model version and verdict, excludes uncorrected rejected output, emits a stable SHA-256-pinned training manifest, and submits approved records through a Nous fine-tune gateway that must return the new model version.
149.10.10 Ownership ranking combines stack-to-file code-graph edges with recent fix history and deterministic tie-breaking.

149.10 verification#

  • pnpm test, pnpm typecheck, and warning-free pnpm lint pass for @neith/crash-intelligence (11 tests).
  • The tests execute Nous completion and embedding provider contracts using deterministic local providers; no hosted model or forge call is fabricated.

149.11 Dashboard and reporting#

Implementation roots:

  • apps/oshun/admin/src/components/CrashOperationsWorkspace.tsx
  • apps/oshun/admin/src/app/crashes/page.tsx
  • apps/oshun/admin/src/app/api/admin/crash-report/route.ts
Requirements Direct evidence
149.11.1–.2 The authenticated admin route renders a filterable issue feed, saved views, selected frames, context, breadcrumbs, replay, release, environment, and owner in one investigation canvas.
149.11.3–.5 Accessible SVG trends, a cross-project crash-free/open/adoption table, and team ownership load/acknowledgement summaries render in the portfolio view.
149.11.6–.7 The report route emits a structurally valid downloadable PDF with crash-free, critical-issue, error-budget, and ownership facts. The portfolio renders the weekly unresolved SLO burndown.
149.11.8–.10 The command dialog focuses its input, J/K changes selected issues, explicit high-contrast/reduced-motion controls compose with system reduced-motion CSS, and each focused browser flow passes the shared WCAG 2.1 AA axe fixture.

149.11 verification#

  • Five focused Vitest component/route tests pass.
  • Admin TypeScript validation and focused ESLint pass.
  • Three Chromium Playwright flows pass with one worker and the shared automatic serious/critical axe scan: issue/trace/replay investigation, filter/query/PDF, and command/portfolio/accessibility.

149.12 Validation#

Implementation roots:

  • libs/neith/crash-telemetry/src/validation.spec.ts
  • libs/neith/crash-telemetry/src/storage.clickhouse.spec.ts
  • libs/neith/crash-telemetry/src/fixtures/grouping-corpus.json
Requirements Direct evidence
149.12.1 sdks/neith-crash/matrix.json names exactly 15 unique platform rows and verify-matrix.sh executes each independently: browser, Node, Apple Swift, Android JVM/NDK, Rust, Go, Python, C++, Unity, Unreal 5.5, Flutter/Dart, React Native, Electron, Kubernetes sidecar, and OTLP/Sentry bridge. The full sequential matrix passes 15/15.
149.12.2 The acceptance corpus contains 100 compiled native frames and 200 exact/nearby IL2CPP frames. Expected results come at runtime from Sentry's published open-source symbolic crates, not Neith labels; function/file/line output must match field-for-field and the executable gate requires ≥99%.
149.12.3 A 60-second ClickHouse 25.3 soak wrote 150,250 benchmark rows through the HTTP JSONEachRow adapter (plus three functional rows): 2,504 events/s sustained, projecting about 216 million/day versus the required 10 million/day. ClickHouse stayed near 316 MiB under a 1 GiB container cap and host available memory stayed above 9 GiB.
149.12.4 The labelled 100-event corpus covers four independent crash families with 25 volatile ID/address variants each. Strict fingerprints yield exactly four pure groups, satisfying the ≥95% gate.
149.12.5 The replay benchmark preserves all 1,000 ordered record kinds/timestamps, removes every secret input value, stays below 300 encoded bytes per record, and below 0.2% of a same-duration 1280×720 RGBA reference stream.
149.12.6 Twenty-five independent email/IP/bearer/home/customer-ID cases traverse recursive client and named server scrubbers; no original secret survives either serialized output. Existing SDK-specific PII tests cover Python, Rust, C++, Android, Unreal, and Go journals too.
149.12.7 Three canonical Sentry DSNs cover TLS, path-prefix, local HTTP/port, envelope byte-length round-trip, third-party Sentry SDK metadata, v7 auth, project routing, and compatible 200 acknowledgement. Existing negative conformance covers malformed payload, missing auth, mismatch, and size limits.

149.12 verification#

  • The complete crash-telemetry package passes 98 tests; the two ClickHouse acceptance cases and opt-in Sentry-reference gate skip only when their external inputs are absent. With SENTRY_REFERENCE=1, the independent symbolic oracle gate matches 300/300 native and IL2CPP frames (100%). TypeScript and ESLint checks pass.
  • With CLICKHOUSE_URL=http://127.0.0.1:18123 and CLICKHOUSE_SOAK_SECONDS=60, both acceptance cases pass against the memory-capped ClickHouse 25.3 container. The container was stopped after the run.
  • Dart format, analyzer, and all three Dart 3.12 package tests pass.

Explicitly still open in 149.12#

  • 149.12.8 remains open because serving every Phase 3–147 production service is a fleet rollout and observation task, not a local repository-only check.

Local actionability audit — 2026-07-15#

  • An exact unchecked-item scan of TODOS/phase-149.md returns only 149.12.8. Every locally implementable prerequisite for that rollout is already covered above: the 15-platform SDK matrix, Sentry-compatible ingest, hardened deployment profiles, CI release/deploy hooks, alerting, storage, and the operations UI.
  • Closing 149.12.8 requires deployment credentials and project/DSN assignment for every live Phase 3–147 service, followed by production event and release-health observation. No repository edit or local simulation can prove that fleet-wide operational state, so the item intentionally remains unchecked rather than being converted into a weaker local proxy.
  • Current focused verification passes for crash telemetry (93 tests passing, 8 externally gated tests skipped, plus TypeScript and ESLint), crash intelligence (11 tests, TypeScript, and ESLint), the crash-operations UI (4 component tests and focused ESLint), and all 3 dedicated Chromium Playwright flows. The available SDK rows also pass for browser, Node, Android, Rust, Python, C++, React Native, Electron, Kubernetes, and OTLP. Apple, Go, Unity, Unreal, and Flutter could not be rerun because this host currently lacks their toolchains; their successful executable validation is retained in the earlier matrix evidence above.
  • The full admin TypeScript command currently reports two unrelated errors in EgbeBehaviorWelfareTelemetryPanel.tsx (exporter is not present on EgbeBehaviorWelfareTelemetryBatch); no Phase 149 source appears in that diagnostic.

Result: all open locally actionable Phase 149 work is complete. The sole remaining phase item is the explicitly external production-fleet rollout.