Repository navigation
LDBC: reduce query intermediates and profile additional memory - #996
Merged
frankmcsherry merged 7 commits intoOct 22, 2026
Merged
frankmcsherry merged 7 commits into
frankmcsherry merged 7 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large intermediate rows and collect-based integer aggregates made the SF1 update trials expensive, while whole-process peaks obscured how much memory each query added. Project unused fields before large joins, lower integer counts/sums through differential weights (preserving nonempty zero-sum groups), restrict IC12/BI15/BI17 earlier, and release private Python reference intermediates after their last consumer.
Use bounded Corgi ranking by default for every finite limit. The ordinary compiler now emits the shared reducer in
snb/topk.py, supports arbitrary sort-key arities and limits, and retains complete tie-break values and bag multiplicities. IC2 is flat; IC5/IC9 use the measured 1024/32/1 hierarchies (forum identity for IC5, full sort value for IC9). Keepbounded_topk.pyas a bucket/hash tuning tool using that same compiler. Remove the profiler's opt-in flag. Unbounded ranking and nested collection outputs still use collect.Add a memory profiler that settles shared inputs before installing a query and compares each fresh query process with a separate input-only replay. Keep active memory, full-job peaks, retirement and resource failures separate.
interactive/server/bench/ldbc/MEMORY.mdrecords all 41 observed SF1 results in query order, exact bindings, plan/binary/journal hashes and the method; the compact CSV/JSON are committed, while raw logs and data remain investigation artifacts.Observed on clean source
6801cb3a, one worker, one fixed request per query, 521 update epochs:The current IC2/IC5/IC9 default plans match those measured plans byte for byte. The table remains historical for 25 other queries whose limited rankings now use bounded reduction; they have not been remeasured at SF1.
Validation: all 41 default query plans match Python at all 49 update-fixture states and retire at one/four workers; the full small-witness catalogue also passes mixed mode at one/four workers and maintained mode at four workers. Bounded ranking checks cover limits 0/1/10/20/100, varying sort-key arities, composite groups, duplicates, signed diffs, deletions and replacement candidates; default and experimental IC2/IC5/IC9 plans match Python through all 49 states at one/four workers. Aggregate arithmetic/retraction tests pass at one/four workers, the memory-profiler fixture checks late installation/endpoint comparisons/retirement/portable summaries, and harness/SNB/update unit tests pass. Local server-backed checks and observations use the pinned release binary from
854ade89; CI builds the current server and runs these checks alongside the existing release suites.