Skip to content

LDBC: reduce query intermediates and profile additional memory - #996

Merged
frankmcsherry merged 7 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ldbc-query-memory-pr
Oct 22, 2026
Merged

frankmcsherry merged 7 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ldbc-query-memory-pr

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Large intermediate rows and collect-based integer aggregates made the SF1 update trials expensive, while whole-process peaks obscured how much memory each query added. Project unused fields before large joins, lower integer counts/sums through differential weights (preserving nonempty zero-sum groups), restrict IC12/BI15/BI17 earlier, and release private Python reference intermediates after their last consumer.

Use bounded Corgi ranking by default for every finite limit. The ordinary compiler now emits the shared reducer in snb/topk.py, supports arbitrary sort-key arities and limits, and retains complete tie-break values and bag multiplicities. IC2 is flat; IC5/IC9 use the measured 1024/32/1 hierarchies (forum identity for IC5, full sort value for IC9). Keep bounded_topk.py as a bucket/hash tuning tool using that same compiler. Remove the profiler's opt-in flag. Unbounded ranking and nested collection outputs still use collect.

Add a memory profiler that settles shared inputs before installing a query and compares each fresh query process with a separate input-only replay. Keep active memory, full-job peaks, retirement and resource failures separate. interactive/server/bench/ldbc/MEMORY.md records all 41 observed SF1 results in query order, exact bindings, plan/binary/journal hashes and the method; the compact CSV/JSON are committed, while raw logs and data remain investigation artifacts.

Observed on clean source 6801cb3a, one worker, one fixed request per query, 521 update epochs:

  • Input-only final RSS: 2.180 GiB; a separate repeat differed by 0.625 MiB.
  • Largest final active additions: BI15 2.213 GiB, BI4 1.294 GiB, IS7 1.277 GiB, IS2 1.205 GiB. These are single process-RSS observations, including allocator retention, for triage.
  • 39 queries complete the full lifecycle. IC9 exceeds the 6-GiB server cap during the extra post-retirement settling ticks; BI17 exceeds it while binding. Preserve both failures and censored peaks.
  • All 40 queries reaching capture match saved, previously Python-validated initial/final answers. This profile performs no fresh Python evaluation and does not certify intervening epochs.

The current IC2/IC5/IC9 default plans match those measured plans byte for byte. The table remains historical for 25 other queries whose limited rankings now use bounded reduction; they have not been remeasured at SF1.

Validation: all 41 default query plans match Python at all 49 update-fixture states and retire at one/four workers; the full small-witness catalogue also passes mixed mode at one/four workers and maintained mode at four workers. Bounded ranking checks cover limits 0/1/10/20/100, varying sort-key arities, composite groups, duplicates, signed diffs, deletions and replacement candidates; default and experimental IC2/IC5/IC9 plans match Python through all 49 states at one/four workers. Aggregate arithmetic/retraction tests pass at one/four workers, the memory-profiler fixture checks late installation/endpoint comparisons/retirement/portable summaries, and harness/SNB/update unit tests pass. Local server-backed checks and observations use the pinned release binary from 854ade89; CI builds the current server and runs these checks alongside the existing release suites.

@frankmcsherry
frankmcsherry merged commit 53f3f63 into TimelyDataflow:master-next Oct 22, 2026
7 checks passed
@frankmcsherry
frankmcsherry deleted the ldbc-query-memory-pr branch October 22, 2026 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant