Eve library

phase-131.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 131: Kalika Reproducible Scientific Compute Operations — Environment Capture, License-Aware Scheduling, Artifact Stores, Workflow Replay & Cross-Site Research Governance#

The workflow and provenance layers added in prior Kalika phases are necessary but not sufficient for the standards expected by premier computational groups. At that level, reproducibility fails not only because inputs are missing, but because runtime environments drift, code licenses are misallocated, containers are inconsistent across clusters, artifact stores are incomplete, large jobs cannot be replayed months later, and cross-site collaborations lose operational coherence. This phase turns Kalika into a true research-compute operating platform by making environments, artifacts, execution policies, and replayable runtime state first-class, auditable objects.


131.1 Scaffolding & Core Runtime Objects (@kalika/research-ops)#

131.1.1 Project Setup#

  • 131.1.1.1 Create libs/kalika/research-ops/ with Nx scaffolding, package.json, project.json, tsconfig.json, and Rust helpers for content hashing, manifest validation, and artifact transfer
  • 131.1.1.2 Create sub-libraries: @kalika/environment-capture, @kalika/license-scheduler, @kalika/artifact-registry, @kalika/workflow-replay, and @kalika/site-governance
  • 131.1.1.3 Integrate the research-ops stack with Phases 103, 111, 127, 128, 129, and 130 so every serious simulation and experimental campaign can be replayed with environment fidelity

131.1.2 Operational Type System#

  • 131.1.2.1 Define ExecutionEnvironment type: OS image, compiler stack, MPI implementation, BLAS/LAPACK/FFTW/CUDA versions, Python/Rust/Node dependencies, environment variables, loaded modules, and container digest
  • 131.1.2.2 Define LicensedResource type: code/package identifier, license token pool, queue restrictions, approved users/sites, and checkout policy
  • 131.1.2.3 Define ArtifactManifest type: input files, binaries, scripts, checkpoints, logs, parsed outputs, hashes, encryption status, retention policy, and storage location
  • 131.1.2.4 Define ReplayBundle type: workflow DAG, exact environment, artifacts, scheduler parameters, secrets references, and deterministic replay expectations/tolerances

131.2 Environment Capture & Portable Runtime Packaging (@kalika/environment-capture)#

131.2.1 Environment Specification#

  • 131.2.1.1 Implement environment snapshotting: capture loaded modules, package manifests, compiler versions, shared library fingerprints, GPU driver/runtime versions, and cluster metadata at submission time
  • 131.2.1.2 Implement portable environment specs: Conda/Mamba lockfiles, pyproject/uv metadata, Cargo lock capture, Node package lock capture, and site-specific module recipes in a unified schema
  • 131.2.1.3 Implement Spack environment export/import for HPC-native scientific software stacks and compiler/MPI matrix management
  • 131.2.1.4 Implement Nix-style immutable environment descriptors for research workflows requiring strongest reproducibility guarantees

131.2.2 Containerization#

  • 131.2.2.1 Implement container build recipes for Docker/Podman for local/cloud execution and Apptainer/Singularity for HPC clusters
  • 131.2.2.2 Implement multi-architecture scientific images: x86_64, aarch64, CUDA-enabled, ROCm-enabled, and CPU-only variants with digest-pinned manifests
  • 131.2.2.3 Implement environment-to-container compiler: given a Kalika environment spec, generate a reproducible container recipe, build it, scan it, and register the resulting digest
  • 131.2.2.4 Implement container validation suite: verify expected binaries, library ABI compatibility, MPI launch behavior, GPU visibility, and benchmark smoke tests

131.3 License-Aware Scheduling & Research Infrastructure Policy (@kalika/license-scheduler)#

131.3.1 Licensed Software Operations#

  • 131.3.1.1 Implement license pool registry for commercial scientific codes such as VASP and other restricted packages: token counts, borrowing policy, site entitlement, and queue restrictions
  • 131.3.1.2 Implement license-aware job placement: do not launch a job unless required licenses are available and reserved for its expected duration
  • 131.3.1.3 Implement fair-share and priority rules across users, projects, grants, and publication-critical deadlines without violating license or facility policies
  • 131.3.1.4 Implement token contention resolution: backfill, hold-and-release, preemption rules where permitted, and advance reservations for beamtime-aligned or campaign-critical runs

131.3.2 Cluster & Site Policies#

  • 131.3.2.1 Implement site policy abstraction: queue names, maximum walltime, node architecture, file-system constraints, internet restrictions, security posture, and allowed container runtimes
  • 131.3.2.2 Implement cluster capability registry: CPUs, GPUs, interconnect, memory classes, scratch storage, and preferred workload classes for each site
  • 131.3.2.3 Implement policy-aware workflow planner: route jobs to the site/environment combination that satisfies technical, budget, and license constraints with full justification captured

131.4 Artifact Registry, Data Logistics & Long-Run State (@kalika/artifact-registry)#

131.4.1 Artifact Capture#

  • 131.4.1.1 Implement immutable artifact registry with content-addressed storage for inputs, outputs, checkpoints, logs, binary dependencies, and derived analysis products
  • 131.4.1.2 Implement tiered storage management: local scratch, parallel file system, object storage, cold archive, and DOI-ready publication deposit targets
  • 131.4.1.3 Implement artifact promotion rules: decide what remains ephemeral, what is retained for reproducibility, and what is elevated to benchmark/reference status

131.4.2 Checkpointing & Restart Governance#

  • 131.4.2.1 Implement checkpoint manifest standard: checkpoint format version, code version, environment hash, expected restart semantics, and compatibility matrix across code upgrades
  • 131.4.2.2 Implement periodic checkpoint policy engine: schedule checkpoint cadence based on job length, queue preemption risk, output volume, and restart overhead
  • 131.4.2.3 Implement restart validation: after resumed execution, compare key state metrics against uninterrupted runs within allowed numerical tolerances
  • 131.4.2.4 Implement artifact integrity verification: hashes, signatures, corruption detection, replication status, and repair workflows for damaged or incomplete archives

131.5 Workflow Replay, Determinism & Cross-Version Comparability (@kalika/workflow-replay)#

131.5.1 Replay Infrastructure#

  • 131.5.1.1 Implement replay bundle generator: package workflow DAG, environment manifest, artifacts, scheduler settings, license requirements, and expected outputs into a replayable unit
  • 131.5.1.2 Implement replay executor: instantiate the original environment on local, cluster, or cloud targets, restore artifacts/checkpoints, and rerun the workflow with controlled differences documented
  • 131.5.1.3 Implement deterministic and tolerance-based replay modes: exact replay where possible, and scientifically acceptable replay where floating-point, scheduler, or stochastic nondeterminism is unavoidable

131.5.2 Comparability & Drift Detection#

  • 131.5.2.1 Implement environment drift detector: identify when a replay environment differs materially from the original in compilers, MPI, BLAS, GPU runtime, container digest, or code revision
  • 131.5.2.2 Implement replay difference reports: energy/force/property deltas, runtime changes, memory changes, and attributable root-cause annotations
  • 131.5.2.3 Implement upgrade qualification workflows: before adopting a new code version or environment, rerun benchmark suites and quantify scientific equivalence or meaningful regressions

131.6 Cross-Site Governance, Secrets, Security & Collaboration (@kalika/site-governance)#

131.6.1 Cross-Site Research Operations#

  • 131.6.1.1 Implement federated site registry: institutions, clusters, storage endpoints, legal constraints, approved software, and collaboration boundaries
  • 131.6.1.2 Implement cross-site artifact replication with provenance: primary/secondary copies, egress accounting, replication lag, and disaster recovery objectives
  • 131.6.1.3 Implement collaborative execution handoff: transfer a workflow or replay bundle to another site while preserving hashes, environment manifests, and audit trails

131.6.2 Secrets & Security#

  • 131.6.2.1 Implement secure secret references for cluster credentials, API tokens, license-server endpoints, and encrypted storage keys without embedding secrets in workflow manifests
  • 131.6.2.2 Implement signed provenance and artifact attestations for sensitive or high-value research programs
  • 131.6.2.3 Implement access-control policies for pre-publication, export-controlled, proprietary, or restricted collaboration data

131.7 Benchmarks, Validation & Kalika Integration#

131.7.1 Validation#

  • 131.7.1.1 Implement environment capture round-trip tests: snapshot -> materialize -> benchmark -> compare across local and HPC targets
  • 131.7.1.2 Implement license-aware scheduler simulations: verify no token oversubscription, correct queue behavior, and fair-share enforcement under contention
  • 131.7.1.3 Implement artifact-registry resilience tests: corruption injection, partial-upload recovery, replication failure, and cold-archive restoration
  • 131.7.1.4 Implement replay benchmark suite: select representative DFT, GW, MD, phase-field, and closed-loop campaign workflows and verify replay fidelity over time

131.7.2 Platform Hooks#

  • 131.7.2.1 Surface environment and replay state in Phase 128 project and calculation views so researchers can see whether a result is merely reproducible in principle or replay-validated in practice
  • 131.7.2.2 Require new flagship workflow templates in Phases 127, 129, and 130 to emit replay bundles and artifact manifests by default
  • 131.7.2.3 Add notebook and CLI affordances for capture-environment, build-replay-bundle, replay-workflow, and qualify-upgrade
  • 131.7.2.4 Add final acceptance criterion for Kalika flagship methods: no method is considered production-ready until its workflow is replayable with documented environment, artifacts, and variance bounds

Phase 131 Implementation Evidence#

  • Implemented @kalika/research-ops at libs/kalika/research-ops/ with Nx project metadata, package exports, TypeScript configs, Vitest coverage, and Rust helper assets for digest validation, manifest validation, and artifact transfer planning.
  • Added submodule entry points and workspace aliases for @kalika/environment-capture, @kalika/license-scheduler, @kalika/artifact-registry, @kalika/workflow-replay, and @kalika/site-governance.
  • Added typed runtime objects for execution environments, licensed resources, artifact manifests, replay bundles, scheduler parameters, workflow DAGs, secret references, retention policy, and provenance state.
  • Added implementation coverage for environment snapshotting, Spack/Nix descriptors, container recipe generation and validation, license-aware placement and fair-share scoring, site policy routing, immutable artifact registration, retention, replication, integrity repair planning, checkpoint cadence, restart validation, replay bundle generation, drift reports, upgrade qualification, federated site handoff, secret references, signed attestations, access policy checks, and flagship workflow replay-readiness gates.
  • Verification run: pnpm nx test @kalika/research-ops --verbose; pnpm nx build @kalika/research-ops --verbose; pnpm nx lint @kalika/research-ops --verbose; pnpm exec prettier --check libs/kalika/research-ops/project.json libs/kalika/research-ops/package.json libs/kalika/research-ops/tsconfig.json libs/kalika/research-ops/tsconfig.lib.json libs/kalika/research-ops/tsconfig.spec.json libs/kalika/research-ops/vitest.config.ts libs/kalika/research-ops/src/*.ts tsconfig.base.json; cargo test --manifest-path libs/kalika/research-ops/rust/Cargo.toml; git diff --check.

Phase 131 Completion Audit (2026-07-14)#

  • Replaced metadata-only claims with executable operations for digest-pinned container build/scan/registration, multi-architecture manifests, Spack import, pinned Nix descriptors, resumable artifact uploads, verified replica repair, cold restoration, checkpoint compatibility, contention resolution, cross-site routing, replay DAG execution, and keyed provenance attestations.
  • Added an executable kalika-research-ops command and wired capture-environment, build-replay-bundle, replay-workflow, and qualify-upgrade through the top-level Kalika CLI.
  • Added direct Phase 127, 129, and 130 replay-bundle emission hooks and tests; Phase 128 calculation/project models now distinguish uncaptured, captured-in-principle, replay-validated, and diverged results.
  • Expanded environment evidence with lockfile families, scheduler metadata, code revision, GPU driver capture, shared-library fingerprints, and material drift reporting. Expanded placement evidence with atomic stale reservation rejection, backfill, advance reservation, policy-permitted preemption, memory/architecture/runtime constraints, budget, and license entitlement checks.
  • Added verified Rust SHA-256 and artifact-copy helpers with post-transfer validation, plus failure-path coverage for corrupt artifacts, incomplete uploads, failed replay nodes, stale reservations, export restrictions, and forged provenance.