Repository navigation
Conversation
…ent_ids) The serve-from-repository-version file lookup (handler.py:963) and its pass-through twin (handler.py:897) filtered ContentArtifact with content__in=version.content, which compiles to content_id IN (SELECT ... unnest(content_ids) FROM core_repositoryversion ...) The planner cannot estimate unnest() row counts (hardcoded rows=10), so on every file request it materialized the version's whole content_ids array (hundreds of thousands of UUIDs on prod versions) and probed core_content and core_contentartifact per id, before the selective relative_path filter could apply. Work was O(version size), not O(result size). Compute membership from core_repositorycontent instead (version_added/removed interval, same semantics as _content_relationships), which has real table statistics and real indexes, and skips the redundant core_content existence hop. Generated-by: GLM 5.3 (EnMaaS)
…ISTS Even with the repositorycontent membership subquery, content_id IN (SELECT ...) still forces the planner to evaluate the membership side of the semi-join up front (O(version size)) before the LIMIT 1 file lookup can be satisfied, instead of starting from the selective relative_path index on core_contentartifact. Rewrite both file-lookup call sites as a correlated EXISTS on content_id: the planner can now scan core_contentartifact by relative_path (typically a handful of rows across the whole table) and probe membership per candidate row through the unique (repository_id, content_id, ...) index on core_repositorycontent. Work becomes O(matching contentartifacts), not O(version size). Generated-by: GLM 5.3 (EnMaaS)
…tent_ids) Replace the content__in=repo_version.content filter (per-request unnest of the whole content_ids array + core_content hop + rows=10 planner misestimate) with a correlated EXISTS over _content_relationships(), same pattern as the file-lookup fix (d516451, c1337df). Lets the planner drive the scan from the relative_path filter and probe repositoryversion membership per candidate row instead of materializing version membership up front. Generated-by: GLM 5.3 (EnMaaS)
A plain btree cannot serve LIKE 'prefix%' under a non-C collation (RDS default en_US.UTF-8), so directory-listing queries (relative_path__startswith) could not be index-driven on the relative_path filter. text_pattern_ops also supports equality, so it covers both the listing and the file-lookup path. Built CONCURRENTLY (25M+ rows on prod), following the 0148 precedent. Generated-by: GLM 5.3 (EnMaaS)
Two fixes for list_directory() on large repository versions: - Bound the number of entries rendered to DIRECTORY_LISTING_LIMIT (default 10000, 0 disables). Root listings of huge versions previously fetched and rendered every content artifact of the version; the queryset now scans at most limit+1 rows and a '…' marker entry signals truncation. - The last-modified dates loop iterated the entire version membership in Python to find the handful of listed contents; filter _content_relationships() by the listed content ids instead, so the query is bounded by the listing size, not the version size. Generated-by: GLM 5.3 (EnMaaS)
django.contrib.postgres.indexes exposes BTreeIndex (capital T), not BtreeIndex; the wrong name broke django.setup() at app startup. Generated-by: GLM 5.3 (EnMaaS)
PostgreSQL identifiers are capped at 32 characters for Django index names; core_contentartifact_relative_path_like (40) failed the system check. Rename to core_conten_relativ_like_idx (28), following Django's truncated-name convention. Generated-by: GLM 5.3 (EnMaaS)
The index.html existence probe runs before every serve-from-repo-version request. For file URLs like ".../a-1.0.0.jar" the probe key is ".../a-1.0.0.jar/index.html", which can never exist, so on the hot file path the probe is a pure per-request DB round trip (prod: 2673 execs, 2.14 s avg against the read-only replica). Skip the probe when the request path has no trailing slash and its last segment contains a dot (file-ish). Directory URLs still probe, so the serve-index.html-instead-of-listing-directory behaviour is unchanged. A dotted directory name without a slash now takes the trailing-slash redirect from list_directory() and serves index.html on the redirected request - one extra redirect, same content. Generated-by: GLM 5.3 (EnMaaS)
For directory URLs the index.html existence probe still used content__in=repo_version.content, i.e. the unnest(content_ids) subquery: per-request materialization of the whole content_ids array and a hash semi-join against core_contentartifact, O(version size) on big repo versions. Rewrite as a correlated EXISTS on _content_relationships() (same pattern as the file-lookup call sites): the planner scans core_contentartifact by the relative_path index for "<dir>/index.html" and probes core_repositorycontent membership per candidate row. Generated-by: GLM 5.3 (EnMaaS)
Adjust directory metadata tests for repository membership querysets. Assisted-by: pi coding agent
Assisted-by: pi coding agent
Contributor
Author
|
I compared this with Larry's #8249. The indexes are attribute-equivalent: B-tree on I added the shared |
1 of 7 tasks
6 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Draft for review/testing of the read-only content query optimizations documented by Jan Hutar.
TL;DR: index tradeoff
The new
relative_path text_pattern_opsindex is similar to existing index, but the existing index iscontent_id-leading and cannot efficiently serve directory-prefix lookups.It adds one index entry and corresponding write/WAL/storage overhead for each
ContentArtifactinsert or delete, including batch Maven ingestion and index-page generation, but does not add work to normal content GETs.ContentArtifactrows are predominantly append-only, while directory/file serving is a high-frequency read path; the expected tradeoff is favorable for this workload. Migration0162usesAddIndexConcurrentlyto avoid a blocking table lock.Included
EXISTSrepository-membership checks instead ofunnest(content_ids)at content-serving call sites.relative_pathpattern index migration.index.htmlprobes for file-like paths.The branch is based on Jan Hutar's
4-readonly-speedupbranch.Validation
Combined OCI agent tests:
GABI validation not quite pratical right now because requires the index. Asking in app-sre channel if possible to install hypopg plugin to test hypothetical indexes
Assisted-by: pi coding agent