[frankmcsherry]

Summary

Compute key and value field demand backward through scopes and loops to a fixed point. Mask dead producer fields with () while preserving tuple positions and the external schema. Corgi removes work feeding the masked fields. distinct, min, collect, and reduce lambdas keep their complete inputs. This PR does not move work through joins or change the physical plan.

The pass now uses Corgi acff1dfe and its typed demand_transform/demand_projection instead of parsing explain text. Install validates the original program first, creates generated sources, then tries the optional rewrite and rechecks its shapes and exports. Any analysis or shape failure falls back to the original program. Composed lambda source is documented as descriptive when its graph has been rewritten. The self-optimizer and demand examples are in draft #1002.

SF0.1 results

One release server binary (SHA-256 prefix fc5ff230243a), 41 SNB reads installed together, four workers, one measured mixed-mode round. Naive queries use snb/queries.py from b97d3d47 before #996; curated queries use #996. /usr/bin/time -l reports whole-server instructions and maximum RSS. All four runs passed; all 248 recorded answer hashes and effective bindings matched. Python reference checking was disabled for this timed comparison. These are single runs, so RSS has run-to-run variation.

Query plans	Pass	Instructions	Max RSS
Naive	Off	405.92 billion	7.81 GiB
Naive	On	302.38 billion	5.10 GiB
Curated	Off	301.98 billion	6.18 GiB
Curated	On	278.09 billion	4.76 GiB

The pass saves 25.5% of instructions on naive plans and 7.9% on curated plans.

Validation

• The Corgi pin alone passed cargo test --workspace --all-targets with the pass off.
• DDIR_PROJECTION=1 cargo test --workspace --all-targets passed after the hook and demand API changes.
• The named server_shapes, server_install, and server_columnar_sources tests passed with the flag on.
• cargo clippy -p interactive -p ddir-server --all-targets completed with existing repository warnings.
• A fresh DIFFTEST_SEED=0 DIFFTEST_CASES=5000 DDIR_PROJECTION=1 release sweep passed 5,000/5,000 cases, zero refused or failed. The coordinator reported about 5.2M overnight cases on the earlier pass without a wrong answer.

Landing and follow-up

I recommend landing this opt-in and making it the default, removing DDIR_PROJECTION, in a small next PR. Physical removal of () positions with Corgi compact_unit_fields is separate. Curated IC12's request-first class path and BI15's edge semijoin restrict rows and require planner choices beyond automatic field demand. The explain question remains with Frank.

[frankmcsherry]

I reviewed projection.rs and the server hook. The pass looks sound, the numbers are strong, and the points below are about landing it.

What I checked, and found right

• Barriers. distinct, min, collect and reduce lambdas are opaque: their output demands every input field, and the dead fields of their output are masked by a Project inserted after them (rewrite_scope).
• count is not a barrier, correctly. It's the sum of a key's diffs, so merging rows that differ only in dead values leaves it unchanged.
• Joins. Merged rows add their diffs before the product, (a1 + a2)·b = a1·b + a2·b, so masking a join input's dead values is exact. Both keys are always demanded.
• Presence. Every fact of a filter's or flat map's output demands what the predicate or list reads, so rows are never kept or dropped differently.
• Root exports are all seeded in pushdown. Published traces keep their full schema, and imported traces are never rewritten.
• The explain-text probe is conservative against today's printer. I checked corgi's explain at the pinned fdbb5e2:
  • a tuple prints only node ids, (%a %b), so field positions can't shift;
  • nested bodies are indented, so they're skipped and their op counts as reading all of its inputs;
  • anything unparsed falls back to "all fields".

Before landing

1. Enabling the pass should never make an install fail.
   • pushdown returns Err when shape inference reports conflicts or unknowns before the rewrite, and build_scope errors propagate too. The server hook ?s both.
   • The post-rewrite check already falls back to the original program. Do the same up front: return Ok(0) and install the original. Then the pass is purely optional.
   • Likewise, the unreachable! in half_join, and the panic!s in PlannedScope::at, would be better as that same fallback.
2. The lambda's text and graph disagree after masking.
   • mask_lambda and mask_half_join build text with Lambda::compose, then replace .graph with a hand-built graph. The hand-built graph anchors dead fields' Unit to the input, so dead upstream work is dropped.
   • So the text no longer parses to the graph that runs. Text is what dedup compares and what serde writes, so a round trip brings the dead work back.
   • It would be better if the composed text alone gave the lean graph: for example, corgi's optimize folding a field of a tuple through the let. Failing that, a comment at Lambda saying the text is no longer authoritative.
   • Separately, mask_half_join repeats most of mask_lambda and could share it.
3. Replace the text probe with corgi #112's API when it merges.
   • demand_projection answers exactly "which input fields does this output read", structurally.
   • That deletes parse_graph, input_paths and most of graph_fields, about 150 lines, and the dependence on explain's format. Your module doc already says this.
4. Split the PR.
   • The pass is projection.rs, the server hook, and suite.py --skip-check.
   • examples/self_opt, about 780 lines, is the #919 self-optimizer port, checked against Rust on the 78 .ddp programs. It's a different thing to review, and could be its own PR, together with demand.ddp and examples/demand.rs.
5. Default on, later. This is a certain optimization, so the end state is on by default, without DDIR_PROJECTION. A 50,000-case sweep with the pass on, using the widened harness (half joins, frames, feed-from), is running overnight. If it's clean, I'd make the pass the default, in this PR or the next.
6. Small: the per-install eprintln! on worker 0 should go, or move behind the same flag.

A question. How does the pass interact with explain? The fallback comment says some explanation plans "require the producer's original shape". Does explaining a masked program explain the masked program or the original one? Either is defensible, but it should be one, written down.

🤖 Generated with Claude Code