## Phase 131: Kalika Reproducible Scientific Compute Operations — Environment Capture, License-Aware Scheduling, Artifact Stores, Workflow Replay & Cross-Site Research Governance

The workflow and provenance layers added in prior Kalika phases are necessary
but not sufficient for the standards expected by premier computational groups.
At that level, reproducibility fails not only because inputs are missing, but
because runtime environments drift, code licenses are misallocated, containers
are inconsistent across clusters, artifact stores are incomplete, large jobs
cannot be replayed months later, and cross-site collaborations lose operational
coherence. This phase turns Kalika into a true research-compute operating
platform by making environments, artifacts, execution policies, and replayable
runtime state first-class, auditable objects.

---

### 131.1 Scaffolding & Core Runtime Objects (`@kalika/research-ops`)

#### 131.1.1 Project Setup

- [x] **131.1.1.1** Create `libs/kalika/research-ops/` with Nx scaffolding,
      `package.json`, `project.json`, `tsconfig.json`, and Rust helpers for
      content hashing, manifest validation, and artifact transfer
- [x] **131.1.1.2** Create sub-libraries: `@kalika/environment-capture`,
      `@kalika/license-scheduler`, `@kalika/artifact-registry`,
      `@kalika/workflow-replay`, and `@kalika/site-governance`
- [x] **131.1.1.3** Integrate the research-ops stack with Phases 103, 111, 127,
      128, 129, and 130 so every serious simulation and experimental campaign
      can be replayed with environment fidelity

#### 131.1.2 Operational Type System

- [x] **131.1.2.1** Define `ExecutionEnvironment` type: OS image, compiler
      stack, MPI implementation, BLAS/LAPACK/FFTW/CUDA versions,
      Python/Rust/Node dependencies, environment variables, loaded modules, and
      container digest
- [x] **131.1.2.2** Define `LicensedResource` type: code/package identifier,
      license token pool, queue restrictions, approved users/sites, and checkout
      policy
- [x] **131.1.2.3** Define `ArtifactManifest` type: input files, binaries,
      scripts, checkpoints, logs, parsed outputs, hashes, encryption status,
      retention policy, and storage location
- [x] **131.1.2.4** Define `ReplayBundle` type: workflow DAG, exact environment,
      artifacts, scheduler parameters, secrets references, and deterministic
      replay expectations/tolerances

---

### 131.2 Environment Capture & Portable Runtime Packaging (`@kalika/environment-capture`)

#### 131.2.1 Environment Specification

- [x] **131.2.1.1** Implement environment snapshotting: capture loaded modules,
      package manifests, compiler versions, shared library fingerprints, GPU
      driver/runtime versions, and cluster metadata at submission time
- [x] **131.2.1.2** Implement portable environment specs: Conda/Mamba lockfiles,
      `pyproject`/`uv` metadata, Cargo lock capture, Node package lock capture,
      and site-specific module recipes in a unified schema
- [x] **131.2.1.3** Implement Spack environment export/import for HPC-native
      scientific software stacks and compiler/MPI matrix management
- [x] **131.2.1.4** Implement Nix-style immutable environment descriptors for
      research workflows requiring strongest reproducibility guarantees

#### 131.2.2 Containerization

- [x] **131.2.2.1** Implement container build recipes for Docker/Podman for
      local/cloud execution and Apptainer/Singularity for HPC clusters
- [x] **131.2.2.2** Implement multi-architecture scientific images: x86_64,
      aarch64, CUDA-enabled, ROCm-enabled, and CPU-only variants with
      digest-pinned manifests
- [x] **131.2.2.3** Implement environment-to-container compiler: given a Kalika
      environment spec, generate a reproducible container recipe, build it, scan
      it, and register the resulting digest
- [x] **131.2.2.4** Implement container validation suite: verify expected
      binaries, library ABI compatibility, MPI launch behavior, GPU visibility,
      and benchmark smoke tests

---

### 131.3 License-Aware Scheduling & Research Infrastructure Policy (`@kalika/license-scheduler`)

#### 131.3.1 Licensed Software Operations

- [x] **131.3.1.1** Implement license pool registry for commercial scientific
      codes such as VASP and other restricted packages: token counts, borrowing
      policy, site entitlement, and queue restrictions
- [x] **131.3.1.2** Implement license-aware job placement: do not launch a job
      unless required licenses are available and reserved for its expected
      duration
- [x] **131.3.1.3** Implement fair-share and priority rules across users,
      projects, grants, and publication-critical deadlines without violating
      license or facility policies
- [x] **131.3.1.4** Implement token contention resolution: backfill,
      hold-and-release, preemption rules where permitted, and advance
      reservations for beamtime-aligned or campaign-critical runs

#### 131.3.2 Cluster & Site Policies

- [x] **131.3.2.1** Implement site policy abstraction: queue names, maximum
      walltime, node architecture, file-system constraints, internet
      restrictions, security posture, and allowed container runtimes
- [x] **131.3.2.2** Implement cluster capability registry: CPUs, GPUs,
      interconnect, memory classes, scratch storage, and preferred workload
      classes for each site
- [x] **131.3.2.3** Implement policy-aware workflow planner: route jobs to the
      site/environment combination that satisfies technical, budget, and license
      constraints with full justification captured

---

### 131.4 Artifact Registry, Data Logistics & Long-Run State (`@kalika/artifact-registry`)

#### 131.4.1 Artifact Capture

- [x] **131.4.1.1** Implement immutable artifact registry with content-addressed
      storage for inputs, outputs, checkpoints, logs, binary dependencies, and
      derived analysis products
- [x] **131.4.1.2** Implement tiered storage management: local scratch, parallel
      file system, object storage, cold archive, and DOI-ready publication
      deposit targets
- [x] **131.4.1.3** Implement artifact promotion rules: decide what remains
      ephemeral, what is retained for reproducibility, and what is elevated to
      benchmark/reference status

#### 131.4.2 Checkpointing & Restart Governance

- [x] **131.4.2.1** Implement checkpoint manifest standard: checkpoint format
      version, code version, environment hash, expected restart semantics, and
      compatibility matrix across code upgrades
- [x] **131.4.2.2** Implement periodic checkpoint policy engine: schedule
      checkpoint cadence based on job length, queue preemption risk, output
      volume, and restart overhead
- [x] **131.4.2.3** Implement restart validation: after resumed execution,
      compare key state metrics against uninterrupted runs within allowed
      numerical tolerances
- [x] **131.4.2.4** Implement artifact integrity verification: hashes,
      signatures, corruption detection, replication status, and repair workflows
      for damaged or incomplete archives

---

### 131.5 Workflow Replay, Determinism & Cross-Version Comparability (`@kalika/workflow-replay`)

#### 131.5.1 Replay Infrastructure

- [x] **131.5.1.1** Implement replay bundle generator: package workflow DAG,
      environment manifest, artifacts, scheduler settings, license requirements,
      and expected outputs into a replayable unit
- [x] **131.5.1.2** Implement replay executor: instantiate the original
      environment on local, cluster, or cloud targets, restore
      artifacts/checkpoints, and rerun the workflow with controlled differences
      documented
- [x] **131.5.1.3** Implement deterministic and tolerance-based replay modes:
      exact replay where possible, and scientifically acceptable replay where
      floating-point, scheduler, or stochastic nondeterminism is unavoidable

#### 131.5.2 Comparability & Drift Detection

- [x] **131.5.2.1** Implement environment drift detector: identify when a replay
      environment differs materially from the original in compilers, MPI, BLAS,
      GPU runtime, container digest, or code revision
- [x] **131.5.2.2** Implement replay difference reports: energy/force/property
      deltas, runtime changes, memory changes, and attributable root-cause
      annotations
- [x] **131.5.2.3** Implement upgrade qualification workflows: before adopting a
      new code version or environment, rerun benchmark suites and quantify
      scientific equivalence or meaningful regressions

---

### 131.6 Cross-Site Governance, Secrets, Security & Collaboration (`@kalika/site-governance`)

#### 131.6.1 Cross-Site Research Operations

- [x] **131.6.1.1** Implement federated site registry: institutions, clusters,
      storage endpoints, legal constraints, approved software, and collaboration
      boundaries
- [x] **131.6.1.2** Implement cross-site artifact replication with provenance:
      primary/secondary copies, egress accounting, replication lag, and disaster
      recovery objectives
- [x] **131.6.1.3** Implement collaborative execution handoff: transfer a
      workflow or replay bundle to another site while preserving hashes,
      environment manifests, and audit trails

#### 131.6.2 Secrets & Security

- [x] **131.6.2.1** Implement secure secret references for cluster credentials,
      API tokens, license-server endpoints, and encrypted storage keys without
      embedding secrets in workflow manifests
- [x] **131.6.2.2** Implement signed provenance and artifact attestations for
      sensitive or high-value research programs
- [x] **131.6.2.3** Implement access-control policies for pre-publication,
      export-controlled, proprietary, or restricted collaboration data

---

### 131.7 Benchmarks, Validation & Kalika Integration

#### 131.7.1 Validation

- [x] **131.7.1.1** Implement environment capture round-trip tests: snapshot ->
      materialize -> benchmark -> compare across local and HPC targets
- [x] **131.7.1.2** Implement license-aware scheduler simulations: verify no
      token oversubscription, correct queue behavior, and fair-share enforcement
      under contention
- [x] **131.7.1.3** Implement artifact-registry resilience tests: corruption
      injection, partial-upload recovery, replication failure, and cold-archive
      restoration
- [x] **131.7.1.4** Implement replay benchmark suite: select representative DFT,
      GW, MD, phase-field, and closed-loop campaign workflows and verify replay
      fidelity over time

#### 131.7.2 Platform Hooks

- [x] **131.7.2.1** Surface environment and replay state in Phase 128 project
      and calculation views so researchers can see whether a result is merely
      reproducible in principle or replay-validated in practice
- [x] **131.7.2.2** Require new flagship workflow templates in Phases 127, 129,
      and 130 to emit replay bundles and artifact manifests by default
- [x] **131.7.2.3** Add notebook and CLI affordances for `capture-environment`,
      `build-replay-bundle`, `replay-workflow`, and `qualify-upgrade`
- [x] **131.7.2.4** Add final acceptance criterion for Kalika flagship methods:
      no method is considered production-ready until its workflow is replayable
      with documented environment, artifacts, and variance bounds

---

### Phase 131 Implementation Evidence

- Implemented `@kalika/research-ops` at `libs/kalika/research-ops/` with Nx
  project metadata, package exports, TypeScript configs, Vitest coverage, and
  Rust helper assets for digest validation, manifest validation, and artifact
  transfer planning.
- Added submodule entry points and workspace aliases for
  `@kalika/environment-capture`, `@kalika/license-scheduler`,
  `@kalika/artifact-registry`, `@kalika/workflow-replay`, and
  `@kalika/site-governance`.
- Added typed runtime objects for execution environments, licensed resources,
  artifact manifests, replay bundles, scheduler parameters, workflow DAGs,
  secret references, retention policy, and provenance state.
- Added implementation coverage for environment snapshotting, Spack/Nix
  descriptors, container recipe generation and validation, license-aware
  placement and fair-share scoring, site policy routing, immutable artifact
  registration, retention, replication, integrity repair planning, checkpoint
  cadence, restart validation, replay bundle generation, drift reports, upgrade
  qualification, federated site handoff, secret references, signed attestations,
  access policy checks, and flagship workflow replay-readiness gates.
- Verification run: `pnpm nx test @kalika/research-ops --verbose`;
  `pnpm nx build @kalika/research-ops --verbose`;
  `pnpm nx lint @kalika/research-ops --verbose`;
  `pnpm exec prettier --check libs/kalika/research-ops/project.json libs/kalika/research-ops/package.json libs/kalika/research-ops/tsconfig.json libs/kalika/research-ops/tsconfig.lib.json libs/kalika/research-ops/tsconfig.spec.json libs/kalika/research-ops/vitest.config.ts libs/kalika/research-ops/src/*.ts tsconfig.base.json`;
  `cargo test --manifest-path libs/kalika/research-ops/rust/Cargo.toml`;
  `git diff --check`.

### Phase 131 Completion Audit (2026-07-14)

- Replaced metadata-only claims with executable operations for digest-pinned
  container build/scan/registration, multi-architecture manifests, Spack import,
  pinned Nix descriptors, resumable artifact uploads, verified replica repair,
  cold restoration, checkpoint compatibility, contention resolution, cross-site
  routing, replay DAG execution, and keyed provenance attestations.
- Added an executable `kalika-research-ops` command and wired
  `capture-environment`, `build-replay-bundle`, `replay-workflow`, and
  `qualify-upgrade` through the top-level Kalika CLI.
- Added direct Phase 127, 129, and 130 replay-bundle emission hooks and tests;
  Phase 128 calculation/project models now distinguish uncaptured,
  captured-in-principle, replay-validated, and diverged results.
- Expanded environment evidence with lockfile families, scheduler metadata, code
  revision, GPU driver capture, shared-library fingerprints, and material drift
  reporting. Expanded placement evidence with atomic stale reservation
  rejection, backfill, advance reservation, policy-permitted preemption,
  memory/architecture/runtime constraints, budget, and license entitlement
  checks.
- Added verified Rust SHA-256 and artifact-copy helpers with post-transfer
  validation, plus failure-path coverage for corrupt artifacts, incomplete
  uploads, failed replay nodes, stale reservations, export restrictions, and
  forged provenance.
