Repository navigation
DDIR server: profile, per-operator time labelled by program statement - #989
Open
frankmcsherry wants to merge 1 commit into
Open
frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
With DDIR_PROFILE=1 each worker records, from Timely's logs, every operator's name, address, and scheduling time, and keeps them after its dataflow is dropped, so a program loaded and dropped each epoch adds up. `profile` answers each operator's own time (a scope less its children), summed over workers and installs, grouped by program, statement label, operator, and address; `profile reset` forgets the times. It takes Timely's logger, so it is exclusive with DDIR_DIAGNOSTICS=1. Labels: lowering records, per scope item, the statement it came from (the `let`, `var`, or `export` whose expression built it, or a child scope's name) in `Scope::labels`, which `optimize` keeps aligned. `render_tree` logs to `"ddir/labels"`, when a logger is registered there, the range of operator identifiers each item built. A row then reads `label="best" op=CorgiReduce` rather than an address to match against the program by hand. Scopes built without labels (explain's rewrites) render as before, unlabelled. The txn bench takes `--profile N` (reset after warmup, print the top N), and its README's operator table is now labelled. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A per-operator readout that survives short-lived programs and names operators by the program statement that built them. It grew out of #983's benchmark, where the resolver is installed and dropped every epoch and the diagnostics dataflow loses the names of operators that live under its 100 ms window.
profile(server,DDIR_PROFILE=1). Each worker records, from Timely's logs, every operator's name, address, and scheduling time, and keeps them after its dataflow is dropped.profileanswers onedataline per operator, largest first:msis the operator's own time (a scope's less its children's), summed over workers and over every install of the program under that name.profile resetforgets the times. It takes Timely's"timely"logger, as the diagnostics dataflow does, so the two are exclusive; with both set, diagnostics win.Labels (library). Lowering records, for each scope item, the statement it came from (the
let,var, orexportwhose expression built it, or a child scope's name) inScope::labels;optimizekeeps it aligned through compaction.render_treelogs to"ddir/labels", when a logger is registered there, the range of operator identifiers each item built (LabelEvent). An arrangement a join or reduce needs is its statement's. Scopes built without labels (explain's rewrites) render unlabelled, as before. With no logger registered, nothing is logged.Bench.
bench_txn.py --profile Nresets the profile after warmup and prints the topNoperators over the measured rounds; the txn README's operator table now uses the labels. Overhead with profiling on is about 2% of a cycle.Checks.
cargo test -p interactive -p ddir-server(192 tests), includingtests/labels.rs: every item is labelled afteroptimize, and rendering logs the operators each label built.Possible follow-ons, not here: one fan-out logger so profiling and diagnostics coexist; keeping operator names in the diagnostics dataflow for short-lived dataflows; labels through explain's rewrites; per-round counts inside recursions; record counts next to times.
🤖 Generated with Claude Code