V3: BlueField feature path + production hardening (hardware NOT_RUN)

.github/workflows/ci.yml

@@ -28,8 +28,14 @@ jobs:
         run: |
           cargo run -p gpucache-transport --bin shard-demo --quiet
           cargo run -p gpucache-cold --bin cold-demo --quiet
+      - name: V3 DPU contract and paired gate
+        run: |
+          cargo run -p gpucache-dpu --bin dpu-demo --quiet
+          cargo run -p gpucache-bench --bin v3-paired --quiet
       - name: Optional RDMA feature still skips hardware
         run: cargo test -p gpucache-transport --all-targets --features rdma-nixl
+      - name: Optional DPU feature still skips hardware
+        run: cargo test -p gpucache-dpu --all-targets --features dpu-bf3
       - name: Reject accelerator linkage and erasure coding
         run: |
           python3 - <<'PY'
@@ -62,6 +68,10 @@ jobs:
             echo "RustFS cold tier must not be a hot-path dependency"
             exit 1
           fi
+          if grep -n 'gpucache-dpu' crates/storage/Cargo.toml crates/transport/Cargo.toml crates/protocol/Cargo.toml crates/core-types/Cargo.toml crates/cold/Cargo.toml; then
+            echo "DPU crate must stay off the hot path and off the cold crate"
+            exit 1
+          fi
 
   test-aarch64:
     name: test (aarch64 runner)

AGENTS.md

@@ -4,14 +4,15 @@ GPUCache is the G3.5 shared KV tier. RustFS is the colder G4 backend. Read [READ
 
 ## Phase
 
-The active phase is the **V2 first slice** (explicitly authorized). V1 A/B/C stay in tree and must keep passing. Do not start V3 DPU, DOCA, SPDK, or GDS work. Do not publish crates, cut releases, deploy, or edit a website unless asked.
+The active phase is **V3** (explicitly authorized). V1 A/B/C and the V2 first slice stay in tree and must keep passing. Do not start work past V3. Do not purchase hardware. Do not publish crates, cut releases, deploy, or edit a website unless asked.
 
 ## Honesty rules
 
-- Do not claim DPU, DOCA, RDMA, SPDK, or CUDA acceleration. The default build is CPU and TCP. Feature `rdma-nixl` does not link a device. An aarch64 build is not a BlueField test. GDS and BlueField stay in development / `NOT_RUN_HARDWARE`.
-- Do not invent vLLM, LMCache, or NIXL results. A real engine KV export/import needs a GPU and stays `NOT_RUN_HARDWARE` until that run exists.
+- Do not claim DPU, DOCA, RDMA, SPDK, GDS, or CUDA acceleration. The default build is CPU and TCP. Feature `rdma-nixl` does not link a device. Feature `dpu-bf3` does not link DOCA, SPDK, or an NVMe device. An aarch64 build is not a BlueField test. Public wording for BlueField-3, GDS, and DPU stays in development / `NOT_RUN_HARDWARE` / optional feature.
+- Do not invent vLLM, LMCache, NIXL, TTFT, or TCO results. A real engine KV export/import needs a GPU and stays `NOT_RUN_HARDWARE` until that run exists. Cost stays `TCO_UNKNOWN` until a paired NIC-RDMA vs DPU run exists. TCP is not the DPU win baseline.
 - Do not treat a KV block as model-agnostic long-term memory. A miss falls back to recompute. A wrong-tenant or wrong-layout hit is a defect.
 - Do not add erasure coding, parity, or durability replicas. A lost or failed hot block is a clean miss, then recompute and repopulate. Cold durability belongs to RustFS. An extra hot-prefix copy, if it is ever added, is a latency hint and not a durability promise.
+- Do not call the tree PB-scale verified or production-ready. Arm process start is not a completed offload. Host GPUNetIO NIC-mode samples are not a DPU Arm storage daemon.
 - Docs in this repo are English.
 
 ## Crates
@@ -23,9 +24,10 @@ The active phase is the **V2 first slice** (explicitly authorized). V1 A/B/C sta
 | `gpucache-storage` | the two crates above |
 | `gpucache-transport` | the three crates above. TCP correctness path. Optional feature `rdma-nixl` must add no crate |
 | `gpucache-cold` | `gpucache-core-types` and `sha2`. Not a dependency of storage or transport |
+| `gpucache-dpu` | `gpucache-core-types` only. Optional feature `dpu-bf3` must add no crate. Not a dependency of storage, transport, protocol, or cold |
 | `gpucache-bench` | nothing. Prints `NOT_RUN_HARDWARE` baseline rows and must not invent timings |
 
-No crate may add a CUDA, DOCA, NIXL, RDMA, SPDK, erasure-coding, or replica-placement dependency. The feature name `rdma-nixl` is a slot only. `unsafe` is forbidden. Registered buffers whose completion is unknown stay quarantined until drain or a newer epoch; do not free them only because a lease timer fired. The cold crate stays off the hot path.
+No crate may add a CUDA, DOCA, NIXL, RDMA, SPDK, erasure-coding, or replica-placement dependency. The feature names `rdma-nixl` and `dpu-bf3` are slots only. `unsafe` is forbidden. Registered buffers whose completion is unknown stay quarantined until drain or a newer epoch; do not free them only because a lease timer fired. The cold crate stays off the hot path. The DPU crate stays off the hot path until a measured offload exists.
 
 ## Checks
 
@@ -35,4 +37,4 @@ cargo clippy --workspace --all-targets -- -D warnings
 cargo test --workspace --all-targets
 ```
 
-`cargo test --workspace --all-targets` is the one-command CPU repro.
+`cargo test --workspace --all-targets` is the one-command CPU repro. It must pass with default features and without DOCA or libnixl.

COMPATIBILITY.md

@@ -1,6 +1,6 @@
 # Compatibility manifest
 
-Recorded for V1 milestones A, B, and C. This file is not a performance acceptance report and not a hardware inventory of a BlueField node.
+Recorded for V1 milestones A, B, and C, the V2 first slice, and the V3 CPU slice. This file is not a performance acceptance report and not a hardware inventory of a BlueField node.
 
 | Component | Value | Result |
 | --- | --- | --- |
@@ -14,12 +14,23 @@ Recorded for V1 milestones A, B, and C. This file is not a performance acceptanc
 | vLLM | Not pinned; no engine commit | `NOT_RUN_HARDWARE` |
 | LMCache | Not a dependency | `NOT_RUN_HARDWARE` (route undecided) |
 | NIXL / UCX / RDMA | Not linked. Candidate recorded: `nixl-sys` 1.3.1 (crates.io, updated 2026-07-07; docs.rs also lists 1.5.0), Apache-2.0, repo `ai-dynamo/nixl`. Feature `rdma-nixl` has no crate dependency. `stub-api` aborts on call, so it is not used. Mooncake Transfer Engine is Apache-2.0 and was not linked (in-tree `unsafe` FFI, not a pinned crates.io crate) | Live RDMA `NOT_RUN_HARDWARE`. `NixlTransport::open` returns `rdma not linked` with and without the feature |
-| DOCA / BlueField BFB / firmware / GDS | Not present. Wording stays in development / `NOT_RUN_HARDWARE` | `NOT_RUN_HARDWARE` |
+| DOCA / BlueField BFB / firmware / GDS | Not present. Feature `dpu-bf3` is empty. Planned mode, BFB, OS, DOCA, firmware, and IOMMU values are unset targets in `docs/V3_DPU_TARGETS.md`. Wording stays in development / `NOT_RUN_HARDWARE` / optional feature | `NOT_RUN_HARDWARE` |
 | RustFS server / client | No endpoint authorized in this environment. Contract mock covers Put, Head, RangeGet, Delete, pack, manifest, async sink, and cost-bounded refill | Live probe `NOT_RUN_HARDWARE`. Mock `PASS` |
 | TCP / file store | Loopback TCP and a temp directory on the build host | `PASS` for `cargo test --workspace --all-targets` on x86_64. Not an NVMe format and not a DPU run |
-| Direct Rust deps | `sha2` 0.10.8; dev `proptest` 1.4.0 | Apache-2.0 OR MIT; see `Cargo.lock`. The V2 slice adds no direct crate. `gpucache-bench` has an empty dependency list. Optional feature `rdma-nixl` adds none |
+| Direct Rust deps | `sha2` 0.10.8; dev `proptest` 1.4.0 | Apache-2.0 OR MIT; see `Cargo.lock`. V2 and V3 add no direct crate. `gpucache-bench` and `gpucache-dpu` add no third-party crate. Optional features `rdma-nixl` and `dpu-bf3` add none |
+| SPDK | Not linked. `AffineQueue` is a reactor-id check with no thread | `NOT_RUN_HARDWARE` |
+| mTLS | Not linked. Boundary is the trusted fabric in ADR 0007 | `NOT_RUN` (library hooks for ACL, replay, and seals are CPU tests) |
 | License of this repo | Apache-2.0 | See `LICENSE` |
 
 CI (see `.github/workflows/ci.yml`) builds and runs the same tests on x86_64 and on aarch64 (GitHub ARM runner, plus qemu-user cross execution). An aarch64 result is a CPU result. It does not certify BlueField-3, DOCA, or NVMe-on-DPU.
 
-The integration decision between a native vLLM KV connector and LMCache is pending. No engine commit is pinned because the spike did not run.
+The integration decision between a native vLLM KV connector and LMCache is pending. No engine commit is pinned because the spike did not run. SGLang / HiCache is an evaluation note only (ADR 0008) and is not a dependency.
+
+| Path | Default build | Optional feature | Hardware |
+| --- | --- | --- | --- |
+| TCP + file store | CPU tests | none | CPU `PASS` on the V1/V2 suite. Not an NVMe format |
+| `rdma-nixl` | `open` refuses | empty feature, still refuses | `NOT_RUN_HARDWARE` |
+| `dpu-bf3` | stubs refuse | empty feature, still refuses | `NOT_RUN_HARDWARE` |
+| aarch64 | same Rust core | none | CPU only. Not a BlueField test |
+| RustFS | mock | `GPUCACHE_RUSTFS_*` | Live `NOT_RUN_HARDWARE` |
+| NIC RDMA vs DPU paired gate | `v3-paired` text | none | `NOT_RUN_HARDWARE`, `TCO_UNKNOWN` |

Cargo.toml

@@ -6,6 +6,7 @@ members = [
     "crates/storage",
     "crates/cold",
     "crates/transport",
+    "crates/dpu",
     "crates/bench",
 ]
 
@@ -24,6 +25,7 @@ gpucache-protocol = { path = "crates/protocol" }
 gpucache-storage = { path = "crates/storage" }
 gpucache-cold = { path = "crates/cold" }
 gpucache-transport = { path = "crates/transport" }
+gpucache-dpu = { path = "crates/dpu" }
 
 [workspace.lints.rust]
 unsafe_code = "forbid"

README.md

@@ -8,9 +8,9 @@ GPUCache does not do erasure coding and does not keep durability replicas. A los
 
 ## Status
 
-**V1 milestones A, B, and C are implemented. The V2 first slice is in progress. DPU acceleration is not.**
+**V1 milestones A, B, and C are implemented. The V2 first slice is in this tree. V3 is in progress: a CPU contract for a storage-side BlueField path, plus production hooks that do not need a DPU. DPU acceleration is not measured.**
 
-V1 runs the cache contract on CPU. V2 adds a transport trait, a simulated registered-buffer pool, a shard-local directory, and a stricter G4 cold contract. The long-term deployment target is a storage-side BlueField-3, but this repository does not link DOCA, CUDA, RDMA, or SPDK, and it does not claim that any of those paths exist. Feature `rdma-nixl` does not link libnixl. An aarch64 build is a CPU build. It is not a BlueField functional test. GDS and BlueField offload are in development / `NOT_RUN_HARDWARE`.
+V1 runs the cache contract on CPU. V2 adds a transport trait, a simulated registered-buffer pool, a shard-local directory, and a stricter G4 cold contract. V3 adds an optional `dpu-bf3` feature that does not link DOCA or SPDK, a simulated DRAM-slab storage queue, and CPU-side authz, replay, quota, and cold-ops hooks. The long-term deployment target is a storage-side BlueField-3. The GPU client keeps an ordinary ConnectX. This repository does not link DOCA, CUDA, RDMA, or SPDK, and it does not claim that any of those paths ran. Feature `rdma-nixl` does not link libnixl. Feature `dpu-bf3` does not link a device. An aarch64 build is a CPU build. It is not a BlueField functional test. GDS, DOCA, and BlueField offload are in development / `NOT_RUN_HARDWARE` / optional feature. They are not "already implemented" and not "deployed".
 
 | Milestone | Status |
 | --- | --- |
@@ -25,10 +25,17 @@ V1 runs the cache contract on CPU. V2 adds a transport trait, a simulated regist
 | V2: G4 pack/manifest, async sink, cost-bounded refill | Implemented against the mock. Live RustFS is `NOT_RUN_HARDWARE` |
 | V2: P/D, G3.5, and G4 hop list | Documented in [docs/V2_DATA_PATHS.md](docs/V2_DATA_PATHS.md). Not a zero-copy claim |
 | V2: paired TTFT/TPOT against native NIXL | `NOT_RUN_HARDWARE` |
-| V3: BlueField-3 data-path offload and production hardening | Not started; not purchased and not measured. Not an erasure-coding milestone |
+| V3: `dpu-bf3` feature slot, simulated DRAM slab + storage queue, SPDK affinity check | In progress. Devices are not linked. `open_doca` / `open_nvme_device` / `open_spdk` refuse. `dpu-demo` prints `NOT_RUN_HARDWARE` |
+| V3: one candidate offload (storage queue) and ablation stubs | Interface only. P2P, DPA, compression, and crypto offload stay `NOT_RUN_HARDWARE` when enabled. Hot-tier EC cannot be enabled |
+| V3: authz, replay window, integrity seal, rate limit, secret rotation, audit | CPU hooks in this tree. Not mTLS. Trusted-fabric boundary is [docs/adr/0007-trusted-fabric.md](docs/adr/0007-trusted-fabric.md) |
+| V3: rebalance, rolling upgrade, fault-domain isolation | In-process stubs on the V2 shard directory. They fence fills. They do not copy a replica |
+| V3: degrade matrix (DPU restart, GPU reset, NIC flap, storage loss, NVMe timeout, control-plane loss, RustFS slow/down) | Decisions are clean misses. No payload, no wrong hit, no permanent block. Not a hardware fault injection |
+| V3: G4 retention, manifest version, reclaim, delete verification, tenant cold keys, backfill budget | Implemented against the mock. Live RustFS is `NOT_RUN_HARDWARE` |
+| V3: paired NIC-RDMA baseline vs DPU | `NOT_RUN_HARDWARE`. `tco=TCO_UNKNOWN`. `go_gate=NOT_EVALUATED`. TCP is not the DPU baseline |
+| V3: second engine (SGLang / HiCache) | Not added. Evaluation note only. See [docs/adr/0008-second-engine.md](docs/adr/0008-second-engine.md) |
 | Erasure coding and durability replicas | Will not be built. A lost or failed hot block is a clean miss, then recompute and repopulate. Cold durability is RustFS G4 |
 
-The integration route (native vLLM KV connector or LMCache) is **not chosen**. See [docs/adr/0001-integration-route.md](docs/adr/0001-integration-route.md). Hot-tier failure policy is [docs/adr/0002-no-erasure-coding.md](docs/adr/0002-no-erasure-coding.md).
+The integration route (native vLLM KV connector or LMCache) is **not chosen**. See [docs/adr/0001-integration-route.md](docs/adr/0001-integration-route.md). Hot-tier failure policy is [docs/adr/0002-no-erasure-coding.md](docs/adr/0002-no-erasure-coding.md). The V3 offload candidate is [docs/adr/0006-storage-queue-offload.md](docs/adr/0006-storage-queue-offload.md). It is a hypothesis pending a paired measurement, not a measured bottleneck.
 
 ## What V1 A does
 
@@ -108,6 +115,26 @@ cargo run -p gpucache-cold --bin cold-demo
 
 Success is `status=COLD_CONTRACT_OK` and `gds=NOT_RUN_HARDWARE`. `rustfs_live=NOT_RUN_HARDWARE` when no endpoint is configured.
 
+In-process DPU contract. This does not open a device:
+
+```bash
+cargo run -p gpucache-dpu --bin dpu-demo
+```
+
+Success is `status=DPU_CONTRACT_OK`, `offload_active=false`, and `hardware=NOT_RUN_HARDWARE`. `doca=NOT_LINKED`. Arm process start is not an offload.
+
+Optional feature, still with no SDK:
+
+```bash
+cargo test -p gpucache-dpu --all-targets --features dpu-bf3
+```
+
+V3 paired gate. No timings, no TCO, TCP is not the baseline:
+
+```bash
+cargo run -p gpucache-bench --bin v3-paired
+```
+
 `kv-block-xfer import <credential-hex>` reads one export frame on stdin and imports it into a fresh `MemoryStore`. It does not talk to an engine.
 
 ## Layout
@@ -117,13 +144,14 @@ crates/core-types    identity, keys, lifecycle, errors
 crates/protocol      batch messages, replies, version negotiation
 crates/storage       MemoryStore, FileStore, kv-block-xfer
 crates/transport     TCP server, client, transport trait, shard directory, share-demo, fault-demo, shard-demo
-crates/cold          RustFS contract, pack/manifest, async sink, refill, cold-demo
-crates/bench         paired-baseline stub (no timings)
+crates/cold          RustFS contract, pack/manifest, async sink, refill, retention, cold-demo
+crates/dpu           BlueField feature slot, simulated storage queue, dpu-demo. Not linked by transport
+crates/bench         paired-baseline and v3-paired stubs (no timings)
 docs/adr             decision records
 COMPATIBILITY.md     environment manifest
 ```
 
-Design notes: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/PROTOCOL.md](docs/PROTOCOL.md), [docs/SECURITY.md](docs/SECURITY.md), [docs/OPERATIONS.md](docs/OPERATIONS.md), [docs/BENCHMARKS.md](docs/BENCHMARKS.md), [docs/CORRECTNESS.md](docs/CORRECTNESS.md), [docs/RISKS.md](docs/RISKS.md), [docs/V1C_TEST_MATRIX.md](docs/V1C_TEST_MATRIX.md), [docs/V2_DATA_PATHS.md](docs/V2_DATA_PATHS.md), [docs/V2_TEST_MATRIX.md](docs/V2_TEST_MATRIX.md).
+Design notes: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/PROTOCOL.md](docs/PROTOCOL.md), [docs/SECURITY.md](docs/SECURITY.md), [docs/OPERATIONS.md](docs/OPERATIONS.md), [docs/BENCHMARKS.md](docs/BENCHMARKS.md), [docs/CORRECTNESS.md](docs/CORRECTNESS.md), [docs/RISKS.md](docs/RISKS.md), [docs/V1C_TEST_MATRIX.md](docs/V1C_TEST_MATRIX.md), [docs/V2_DATA_PATHS.md](docs/V2_DATA_PATHS.md), [docs/V2_TEST_MATRIX.md](docs/V2_TEST_MATRIX.md), [docs/V3_DPU_TARGETS.md](docs/V3_DPU_TARGETS.md), [docs/V3_TEST_MATRIX.md](docs/V3_TEST_MATRIX.md), [docs/V3_FAILURES.md](docs/V3_FAILURES.md).
 
 ## License
 

THIRD_PARTY_NOTICES.md

@@ -1,6 +1,16 @@
 # Third-party notices
 
-GPUCache is Apache-2.0. See [LICENSE](LICENSE). The V2 first slice adds no direct dependency. NIXL (`nixl-sys`, Apache-2.0) and Mooncake (Apache-2.0) were audited and not linked. See [docs/adr/0004-nixl-transport.md](docs/adr/0004-nixl-transport.md).
+GPUCache is Apache-2.0. See [LICENSE](LICENSE). The V2 first slice and the V3 CPU slice add no direct dependency. NIXL (`nixl-sys`, Apache-2.0) and Mooncake (Apache-2.0) were audited and not linked. See [docs/adr/0004-nixl-transport.md](docs/adr/0004-nixl-transport.md).
+
+## DOCA, SPDK, and GDS (not linked)
+
+This tree does not vendor or link DOCA, SPDK, or GDS. Feature `dpu-bf3` has an empty dependency list. No DOCA binary ships with this source tree.
+
+DOCA is NVIDIA software. Its license is NVIDIA's, not Apache-2.0. This change did **not** review the DOCA 3.5 EULA or any performance-disclosure clause, and it does not state a legal conclusion. Before anyone links DOCA or publishes a number from a DOCA run, a human has to read the exact license for the version they installed. Do not assume every NVIDIA hardware result is unpublishable, and do not publish a restricted result. That review is `NOT_RUN_LICENSE` here.
+
+SPDK is a candidate for a future reactor-backed NVMe queue. It is not a dependency. This notice does not pin an SPDX identifier for a version we did not vendor.
+
+No model weights are included. Engine licenses stay out of scope until ADR 0001 selects a route.
 
 Direct dependencies:
 

crates/bench/Cargo.toml

@@ -12,5 +12,9 @@ publish = false
 name = "paired-baseline"
 path = "src/bin/paired_baseline.rs"
 
+[[bin]]
+name = "v3-paired"
+path = "src/bin/v3_paired.rs"
+
 [lints]
 workspace = true

crates/bench/src/bin/v3_paired.rs

@@ -0,0 +1,16 @@
+// Copyright 2024-2026 RustFS, Inc.
+// SPDX-License-Identifier: Apache-2.0
+//! V3 paired-experiment placeholder.
+//!
+//! The required baseline is a normal NIC RDMA path, not TCP. This process
+//! prints `NOT_RUN_HARDWARE`, `TCO_UNKNOWN`, and `go_gate=NOT_EVALUATED`.
+//! It does not measure TTFT, watts, or token goodput.
+
+use std::process::ExitCode;
+
+use gpucache_bench::v3_paired_report;
+
+fn main() -> ExitCode {
+    print!("{}", v3_paired_report());
+    ExitCode::SUCCESS
+}