V2: NIC RDMA transport (feature-gated) + shard directory + G4 cold path

.github/workflows/ci.yml

@@ -24,14 +24,36 @@ jobs:
         run: cargo run -p gpucache-transport --bin fault-demo --quiet
       - name: Paired baseline stub
         run: cargo run -p gpucache-bench --bin paired-baseline --quiet
-      - name: Reject accelerator crates in V1
+      - name: Shard and cold demos
         run: |
-          if grep -RInE 'cuda|doca|nixl|rdma|spdk' crates/*/Cargo.toml; then
-            echo "V1 crates must not depend on GPU or DPU stacks"
-            exit 1
-          fi
+          cargo run -p gpucache-transport --bin shard-demo --quiet
+          cargo run -p gpucache-cold --bin cold-demo --quiet
+      - name: Optional RDMA feature still skips hardware
+        run: cargo test -p gpucache-transport --all-targets --features rdma-nixl
+      - name: Reject accelerator linkage and erasure coding
+        run: |
+          python3 - <<'PY'
+          import pathlib, re, sys
+          bad = re.compile(
+              r"^(cuda|doca|nixl|rdma|spdk|libnixl)([-_a-z0-9]*)?\s*=",
+              re.I,
+          )
+          failed = False
+          for path in pathlib.Path("crates").glob("*/Cargo.toml"):
+              in_deps = False
+              for lineno, line in enumerate(path.read_text().splitlines(), 1):
+                  stripped = line.split("#", 1)[0].strip()
+                  if stripped.startswith("[") and stripped.endswith("]"):
+                      in_deps = "dependencies" in stripped
+                      continue
+                  if in_deps and bad.match(stripped):
+                      print(f"{path}:{lineno}: accelerator dependency is not allowed")
+                      failed = True
+          if failed:
+              sys.exit(1)
+          PY
           if grep -RInE 'reed_solomon|erasure_coding|parity_shard' crates --include '*.rs'; then
-            echo "V1 must not add erasure coding"
+            echo "erasure coding must not be added"
             exit 1
           fi
       - name: Keep the cold tier off the hot path

AGENTS.md

@@ -4,11 +4,11 @@ GPUCache is the G3.5 shared KV tier. RustFS is the colder G4 backend. Read [READ
 
 ## Phase
 
-The active phase is **V1 milestone C**. Do not implement V2 RDMA or V3 DPU work in a V1 C change. Do not publish crates, cut releases, deploy, or edit a website unless asked.
+The active phase is the **V2 first slice** (explicitly authorized). V1 A/B/C stay in tree and must keep passing. Do not start V3 DPU, DOCA, SPDK, or GDS work. Do not publish crates, cut releases, deploy, or edit a website unless asked.
 
 ## Honesty rules
 
-- Do not claim DPU, DOCA, RDMA, SPDK, or CUDA acceleration. V1 is CPU only. An aarch64 build is not a BlueField test.
+- Do not claim DPU, DOCA, RDMA, SPDK, or CUDA acceleration. The default build is CPU and TCP. Feature `rdma-nixl` does not link a device. An aarch64 build is not a BlueField test. GDS and BlueField stay in development / `NOT_RUN_HARDWARE`.
 - Do not invent vLLM, LMCache, or NIXL results. A real engine KV export/import needs a GPU and stays `NOT_RUN_HARDWARE` until that run exists.
 - Do not treat a KV block as model-agnostic long-term memory. A miss falls back to recompute. A wrong-tenant or wrong-layout hit is a defect.
 - Do not add erasure coding, parity, or durability replicas. A lost or failed hot block is a clean miss, then recompute and repopulate. Cold durability belongs to RustFS. An extra hot-prefix copy, if it is ever added, is a latency hint and not a durability promise.
@@ -21,11 +21,11 @@ The active phase is **V1 milestone C**. Do not implement V2 RDMA or V3 DPU work
 | `gpucache-core-types` | `sha2` only |
 | `gpucache-protocol` | `gpucache-core-types` |
 | `gpucache-storage` | the two crates above |
-| `gpucache-transport` | the three crates above. TCP only |
+| `gpucache-transport` | the three crates above. TCP correctness path. Optional feature `rdma-nixl` must add no crate |
 | `gpucache-cold` | `gpucache-core-types` and `sha2`. Not a dependency of storage or transport |
 | `gpucache-bench` | nothing. Prints `NOT_RUN_HARDWARE` baseline rows and must not invent timings |
 
-No crate may add CUDA, DOCA, NIXL, RDMA, SPDK, erasure coding, or replica placement. `unsafe` is forbidden. Registered buffers whose completion is unknown stay quarantined until drain or a newer epoch; do not free them only because a lease timer fired. The cold crate stays off the hot path.
+No crate may add a CUDA, DOCA, NIXL, RDMA, SPDK, erasure-coding, or replica-placement dependency. The feature name `rdma-nixl` is a slot only. `unsafe` is forbidden. Registered buffers whose completion is unknown stay quarantined until drain or a newer epoch; do not free them only because a lease timer fired. The cold crate stays off the hot path.
 
 ## Checks
 

COMPATIBILITY.md

@@ -13,11 +13,11 @@ Recorded for V1 milestones A, B, and C. This file is not a performance acceptanc
 | GPU / driver | None in this environment | `NOT_RUN_HARDWARE` |
 | vLLM | Not pinned; no engine commit | `NOT_RUN_HARDWARE` |
 | LMCache | Not a dependency | `NOT_RUN_HARDWARE` (route undecided) |
-| NIXL / UCX / RDMA | Not a dependency | Not in V1 |
-| DOCA / BlueField BFB / firmware | Not present | `NOT_RUN_HARDWARE` |
-| RustFS server / client | No endpoint authorized in this environment. Contract mock covers Put, Head, RangeGet, Delete, pack, and manifest | Live probe `NOT_RUN_HARDWARE`. Mock `PASS` |
+| NIXL / UCX / RDMA | Not linked. Candidate recorded: `nixl-sys` 1.3.1 (crates.io, updated 2026-07-07; docs.rs also lists 1.5.0), Apache-2.0, repo `ai-dynamo/nixl`. Feature `rdma-nixl` has no crate dependency. `stub-api` aborts on call, so it is not used. Mooncake Transfer Engine is Apache-2.0 and was not linked (in-tree `unsafe` FFI, not a pinned crates.io crate) | Live RDMA `NOT_RUN_HARDWARE`. `NixlTransport::open` returns `rdma not linked` with and without the feature |
+| DOCA / BlueField BFB / firmware / GDS | Not present. Wording stays in development / `NOT_RUN_HARDWARE` | `NOT_RUN_HARDWARE` |
+| RustFS server / client | No endpoint authorized in this environment. Contract mock covers Put, Head, RangeGet, Delete, pack, manifest, async sink, and cost-bounded refill | Live probe `NOT_RUN_HARDWARE`. Mock `PASS` |
 | TCP / file store | Loopback TCP and a temp directory on the build host | `PASS` for `cargo test --workspace --all-targets` on x86_64. Not an NVMe format and not a DPU run |
-| Direct Rust deps | `sha2` 0.10.8; dev `proptest` 1.4.0 | Apache-2.0 OR MIT; see `Cargo.lock`. V1 C adds no direct crate. `gpucache-bench` has an empty dependency list |
+| Direct Rust deps | `sha2` 0.10.8; dev `proptest` 1.4.0 | Apache-2.0 OR MIT; see `Cargo.lock`. The V2 slice adds no direct crate. `gpucache-bench` has an empty dependency list. Optional feature `rdma-nixl` adds none |
 | License of this repo | Apache-2.0 | See `LICENSE` |
 
 CI (see `.github/workflows/ci.yml`) builds and runs the same tests on x86_64 and on aarch64 (GitHub ARM runner, plus qemu-user cross execution). An aarch64 result is a CPU result. It does not certify BlueField-3, DOCA, or NVMe-on-DPU.

README.md

@@ -8,9 +8,9 @@ GPUCache does not do erasure coding and does not keep durability replicas. A los
 
 ## Status
 
-**V1 milestones A, B, and C are implemented. DPU acceleration is not.**
+**V1 milestones A, B, and C are implemented. The V2 first slice is in progress. DPU acceleration is not.**
 
-V1 runs the cache contract on CPU. The long-term deployment target is a storage-side BlueField-3, but this repository does not link DOCA, CUDA, RDMA, or SPDK, and it does not claim that any of those paths exist. An aarch64 build is a CPU build. It is not a BlueField functional test.
+V1 runs the cache contract on CPU. V2 adds a transport trait, a simulated registered-buffer pool, a shard-local directory, and a stricter G4 cold contract. The long-term deployment target is a storage-side BlueField-3, but this repository does not link DOCA, CUDA, RDMA, or SPDK, and it does not claim that any of those paths exist. Feature `rdma-nixl` does not link libnixl. An aarch64 build is a CPU build. It is not a BlueField functional test. GDS and BlueField offload are in development / `NOT_RUN_HARDWARE`.
 
 | Milestone | Status |
 | --- | --- |
@@ -19,7 +19,12 @@ V1 runs the cache contract on CPU. The long-term deployment target is a storage-
 | V1 B: TCP service, file backend, LRU admission, tenant quotas, two-client reuse | Implemented in this tree |
 | V1 B: RustFS live Put/Head/RangeGet/Delete | `NOT_RUN_HARDWARE` (no authorized endpoint). Contract mock passes |
 | V1 C: fuzz, fault injection, synthetic token checks, paired-baseline stubs | Implemented in this tree. Real engine output is `NOT_RUN_HARDWARE` |
-| V2: NIC RDMA (NIXL or Mooncake) plus a real RustFS cold path | Not started; blocked on V1 |
+| V2: `KvTransport`, TCP default, feature `rdma-nixl` | In progress. The feature compiles and `NixlTransport::open` refuses. Live RDMA is `NOT_RUN_HARDWARE` |
+| V2: bounded registered buffer pool (simulated pin, reuse, quarantine, backpressure) | Implemented in this tree. Not a NIC registration |
+| V2: shard-local directory, epoch fencing, single-flight, capacity credits | Implemented in-process. One process is one shard. No cross-shard quorum |
+| V2: G4 pack/manifest, async sink, cost-bounded refill | Implemented against the mock. Live RustFS is `NOT_RUN_HARDWARE` |
+| V2: P/D, G3.5, and G4 hop list | Documented in [docs/V2_DATA_PATHS.md](docs/V2_DATA_PATHS.md). Not a zero-copy claim |
+| V2: paired TTFT/TPOT against native NIXL | `NOT_RUN_HARDWARE` |
 | V3: BlueField-3 data-path offload and production hardening | Not started; not purchased and not measured. Not an erasure-coding milestone |
 | Erasure coding and durability replicas | Will not be built. A lost or failed hot block is a clean miss, then recompute and repopulate. Cold durability is RustFS G4 |
 
@@ -43,7 +48,7 @@ The batch API is Lookup, Reserve/Put, Commit, Get, Release/Cancel, and Invalidat
 
 Lookup can return `PrefixHit` when a committed block's hash chain is a strict prefix of the query and the layout, dtype, shard, and model identity match. The hit is the whole stored block. `Get` must use `identity_for_prefix_hit`. A shorter `valid_length` on the same chain is an incompatible miss. The service does not slice inside a block.
 
-`gpucache-cold` packs blocks into an object plus a manifest and can Put, Head, RangeGet, and Delete through a trait. The TCP hot path does not call it. Without `GPUCACHE_RUSTFS_*` variables the live probe is `NOT_RUN_HARDWARE`. The mock still runs.
+`gpucache-cold` packs blocks into an object plus a manifest and can Put, Head, RangeGet, and Delete through a trait. The TCP hot path does not call it. Without `GPUCACHE_RUSTFS_*` variables the live probe is `NOT_RUN_HARDWARE`. The mock still runs. V2 adds a bounded async sink and a refill that returns no bytes when the manifest or the payload range exceeds its budget.
 
 Lifecycle:
 
@@ -87,6 +92,22 @@ Paired GPU baselines stay `NOT_RUN_HARDWARE` and print no timings:
 cargo run -p gpucache-bench --bin paired-baseline
 ```
 
+One shard, simulated registration, and the TCP staging path:
+
+```bash
+cargo run -p gpucache-transport --bin shard-demo
+```
+
+Success is `status=SHARD_OK`, `registration=SIMULATED`, and `rdma=NOT_RUN_HARDWARE`.
+
+Mock G4 pack, sink, and budget-capped refill:
+
+```bash
+cargo run -p gpucache-cold --bin cold-demo
+```
+
+Success is `status=COLD_CONTRACT_OK` and `gds=NOT_RUN_HARDWARE`. `rustfs_live=NOT_RUN_HARDWARE` when no endpoint is configured.
+
 `kv-block-xfer import <credential-hex>` reads one export frame on stdin and imports it into a fresh `MemoryStore`. It does not talk to an engine.
 
 ## Layout
@@ -95,14 +116,14 @@ cargo run -p gpucache-bench --bin paired-baseline
 crates/core-types    identity, keys, lifecycle, errors
 crates/protocol      batch messages, replies, version negotiation
 crates/storage       MemoryStore, FileStore, kv-block-xfer
-crates/transport     TCP server, client, share-demo, fault-demo
-crates/cold          RustFS contract, pack/manifest, live probe
+crates/transport     TCP server, client, transport trait, shard directory, share-demo, fault-demo, shard-demo
+crates/cold          RustFS contract, pack/manifest, async sink, refill, cold-demo
 crates/bench         paired-baseline stub (no timings)
 docs/adr             decision records
 COMPATIBILITY.md     environment manifest
 ```
 
-Design notes: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/PROTOCOL.md](docs/PROTOCOL.md), [docs/SECURITY.md](docs/SECURITY.md), [docs/OPERATIONS.md](docs/OPERATIONS.md), [docs/BENCHMARKS.md](docs/BENCHMARKS.md), [docs/CORRECTNESS.md](docs/CORRECTNESS.md), [docs/RISKS.md](docs/RISKS.md), [docs/V1C_TEST_MATRIX.md](docs/V1C_TEST_MATRIX.md).
+Design notes: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/PROTOCOL.md](docs/PROTOCOL.md), [docs/SECURITY.md](docs/SECURITY.md), [docs/OPERATIONS.md](docs/OPERATIONS.md), [docs/BENCHMARKS.md](docs/BENCHMARKS.md), [docs/CORRECTNESS.md](docs/CORRECTNESS.md), [docs/RISKS.md](docs/RISKS.md), [docs/V1C_TEST_MATRIX.md](docs/V1C_TEST_MATRIX.md), [docs/V2_DATA_PATHS.md](docs/V2_DATA_PATHS.md), [docs/V2_TEST_MATRIX.md](docs/V2_TEST_MATRIX.md).
 
 ## License
 

THIRD_PARTY_NOTICES.md

@@ -1,6 +1,6 @@
 # Third-party notices
 
-GPUCache V1 A, V1 B, and V1 C are Apache-2.0. See [LICENSE](LICENSE). V1 C adds no direct dependency.
+GPUCache is Apache-2.0. See [LICENSE](LICENSE). The V2 first slice adds no direct dependency. NIXL (`nixl-sys`, Apache-2.0) and Mooncake (Apache-2.0) were audited and not linked. See [docs/adr/0004-nixl-transport.md](docs/adr/0004-nixl-transport.md).
 
 Direct dependencies:
 

crates/bench/src/lib.rs

@@ -20,6 +20,13 @@ pub fn paired_baseline_report() -> String {
         "scenario_hbm_within=NOT_RUN_HARDWARE",
         "scenario_hbm_cliff=NOT_RUN_HARDWARE",
         "write_cost=NOT_RUN_HARDWARE",
+        "nixl_pd=NOT_RUN_HARDWARE",
+        "shared_dram=NOT_RUN_HARDWARE",
+        "shared_nvme=NOT_RUN_HARDWARE",
+        "rustfs_refill=NOT_RUN_HARDWARE",
+        "ttft=NOT_RUN_HARDWARE",
+        "tpot=NOT_RUN_HARDWARE",
+        "goodput=NOT_RUN_HARDWARE",
         "samples=NONE",
         "numeric_tolerance=NOT_APPLICABLE",
         "bypass_when_no_reuse=MISS_RECOMPUTE",
@@ -50,6 +57,13 @@ mod tests {
             "scenario_multi_turn=NOT_RUN_HARDWARE",
             "scenario_hbm_within=NOT_RUN_HARDWARE",
             "scenario_hbm_cliff=NOT_RUN_HARDWARE",
+            "nixl_pd=NOT_RUN_HARDWARE",
+            "shared_dram=NOT_RUN_HARDWARE",
+            "shared_nvme=NOT_RUN_HARDWARE",
+            "rustfs_refill=NOT_RUN_HARDWARE",
+            "ttft=NOT_RUN_HARDWARE",
+            "tpot=NOT_RUN_HARDWARE",
+            "goodput=NOT_RUN_HARDWARE",
             "samples=NONE",
         ] {
             assert!(report.contains(key), "missing {key}");

crates/cold/Cargo.toml

@@ -8,6 +8,10 @@ authors.workspace = true
 description = "Optional RustFS G4 cold-tier contract. Not on the GPUCache hot path."
 publish = false
 
+[[bin]]
+name = "cold-demo"
+path = "src/bin/cold_demo.rs"
+
 [dependencies]
 gpucache-core-types.workspace = true
 sha2.workspace = true

crates/cold/src/bin/cold_demo.rs

@@ -0,0 +1,25 @@
+// Copyright 2024-2026 RustFS, Inc.
+// SPDX-License-Identifier: Apache-2.0
+//! Mock G4 pack, async sink, and cost-bounded refill.
+//!
+//! Success is `status=COLD_CONTRACT_OK`. Live RustFS stays
+//! `NOT_RUN_HARDWARE` unless `GPUCACHE_RUSTFS_*` is set and the probe passes.
+//! This is not a GDS run.
+
+use std::process::ExitCode;
+
+use gpucache_cold::run_cold_demo;
+
+fn main() -> ExitCode {
+    match run_cold_demo() {
+        Ok(report) => {
+            print!("{report}");
+            ExitCode::SUCCESS
+        }
+        Err(err) => {
+            eprintln!("status=COLD_FAILED");
+            eprintln!("error={err}");
+            ExitCode::from(2)
+        }
+    }
+}

crates/cold/src/demo.rs

@@ -0,0 +1,104 @@
+// Copyright 2024-2026 RustFS, Inc.
+// SPDX-License-Identifier: Apache-2.0
+
+//! CPU contract for the G4 pack, sink, and refill path.
+//!
+//! A missing `GPUCACHE_RUSTFS_*` environment is `NOT_RUN_HARDWARE`. The mock
+//! result is not a measured RustFS or GDS run.
+
+use core::fmt;
+
+use gpucache_core_types::{content_checksum, synthetic_identity, CacheError};
+
+use crate::http::{live_probe, ProbeStatus};
+use crate::pack::{contract_smoke, pack_blocks, PackedBlock};
+use crate::sink::{refill_generation, AsyncSink, ColdJob, RefillBudget, RefillOutcome};
+use crate::MockObjectStore;
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ColdReport {
+    pub live: &'static str,
+}
+
+impl fmt::Display for ColdReport {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        writeln!(f, "status=COLD_CONTRACT_OK")?;
+        writeln!(f, "mock=PASS")?;
+        writeln!(f, "async_sink=DRAINED")?;
+        writeln!(f, "refill=CHECKSUM_OK")?;
+        writeln!(f, "over_budget=NO_BYTES")?;
+        writeln!(f, "rustfs_live={}", self.live)?;
+        writeln!(f, "gds=NOT_RUN_HARDWARE")?;
+        Ok(())
+    }
+}
+
+pub fn run_cold_demo() -> Result<ColdReport, CacheError> {
+    let mut store = MockObjectStore::default();
+    contract_smoke(&mut store)?;
+    let blocks = demo_blocks()?;
+    let pack = pack_blocks("demo", &blocks)?;
+    let mut sink = AsyncSink::new(2, 1024 * 1024)?;
+    sink.enqueue(ColdJob::from_pack(&pack))?;
+    if sink.drain(&mut store, 1)? != 1 {
+        return Err(CacheError::Storage {
+            reason: "cold sink",
+        });
+    }
+    let budget = RefillBudget::new(1024 * 1024, 16)?;
+    match refill_generation(&store, &pack.manifest_key, 2, &budget)? {
+        RefillOutcome::Loaded { payload } if payload == blocks[1].payload => {}
+        _ => {
+            return Err(CacheError::ChecksumMismatch);
+        }
+    }
+    let tight = RefillBudget::new(1, 16)?;
+    if refill_generation(&store, &pack.manifest_key, 2, &tight)? != RefillOutcome::OverBudget {
+        return Err(CacheError::Malformed {
+            reason: "refill budget",
+        });
+    }
+    let live = match live_probe().status {
+        ProbeStatus::NotRunHardware { .. } => "NOT_RUN_HARDWARE",
+        ProbeStatus::Passed => "PASSED",
+        ProbeStatus::Failed { .. } => {
+            return Err(CacheError::Storage {
+                reason: "object store",
+            })
+        }
+    };
+    Ok(ColdReport { live })
+}
+
+fn demo_blocks() -> Result<Vec<PackedBlock>, CacheError> {
+    let mut out = Vec::new();
+    for (generation, byte) in [(1u64, 9u8), (2, 8u8)] {
+        let payload = vec![byte; 16];
+        let mut identity = synthetic_identity(byte, 1);
+        identity.prefix_chain[0].0[0] = byte;
+        out.push(PackedBlock {
+            generation,
+            checksum: content_checksum(&payload),
+            payload,
+            identity,
+        });
+    }
+    Ok(out)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn demo_stays_a_mock_without_an_endpoint() {
+        let report = run_cold_demo().unwrap();
+        let text = report.to_string();
+        assert!(text.contains("status=COLD_CONTRACT_OK"));
+        assert!(text.contains("over_budget=NO_BYTES"));
+        assert!(text.contains("gds=NOT_RUN_HARDWARE"));
+        if std::env::var_os("GPUCACHE_RUSTFS_ENDPOINT").is_none() {
+            assert!(text.contains("rustfs_live=NOT_RUN_HARDWARE"));
+        }
+    }
+}

crates/cold/src/lib.rs

@@ -9,11 +9,18 @@
 
 #![forbid(unsafe_code)]
 
+mod demo;
 mod http;
 mod mock;
 mod pack;
 mod sigv4;
+mod sink;
 
+pub use demo::{run_cold_demo, ColdReport};
 pub use http::{live_probe, ProbeReport, ProbeStatus};
 pub use mock::MockObjectStore;
-pub use pack::{contract_smoke, pack_blocks, ObjectMeta, ObjectStore, PackedBlock, PayloadRange};
+pub use pack::{
+    contract_smoke, pack_blocks, read_manifest, Manifest, ObjectMeta, ObjectStore, PackedBlock,
+    PayloadRange,
+};
+pub use sink::{refill_generation, AsyncSink, ColdJob, RefillBudget, RefillOutcome};