[configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279

Problem

The [configured resource]-draw workflow recorded all_attempts_exhausted with reason provider_unavailable again. The incident signature is unchanged since its first occurrence about two hours after the [configured resource]-draw re-route merged; the count rose from 11 to 13 over about 53.6 hours, so two new exhaustions occurred since the prior scoped analysis, under the same deployed revision. This occurrence failed both of the workflow's candidates at the same second on the same single draw provider with the identical reason. Our client classifies HTTP 429 or >=500 as provider_unavailable, so this is a provider-response class, not an internal error path. A re-route change merged minutes before this analysis replaces the disabled previous primary model's assignments with a new model family on the same provider, but it is not deployed yet and does not remove the single-provider dependency. No code defect is observed; the leading hypothesis remains provider-side 429/5xx bursts, with load concentration from the re-route and retry-cadence amplification as secondary hypotheses requiring provider-side and production-row evidence the scoped tools cannot reach.

External cause: possible. Code defect: not_observed.

Observations

• Incident snapshot: event_type all_attempts_exhausted, reason provider_unavailable, workflow image_generation_boogu_turbo ([configured resource] draws); count 13, first_seen 2026-10-21T15:52:21Z (about 2 h after the [configured resource]-draw re-route merged), last_seen 2026-10-23T21:30:37Z; the count was 11 at the prior scoped analysis, so two new occurrences ran since, about 12 h apart.
• Route-attempt evidence for this occurrence: two attempt_failed events at the same timestamp, one per candidate of the workflow (the previous primary image model and the newer turbo model inserted as fallback by the routing migration), both provider_unavailable on the same single draw provider; references attempt:1 and attempt:2.
• provider_unavailable for this provider class is a provider-response classification: HTTP 429 or >=500 becomes provider_unavailable, other statuses become request_failed (crates/openplotva-app/src/image_jobs.rs, boogu_http_status_error, verified at repository HEAD).
• Both candidates are endpoint models of the single draw provider; no draw workflow has a cross-provider fallback (migrations 210 and 212 both keep every draw slot on that provider), so exhaustion under a provider-wide 429/5xx burst is configured behavior.
• Deployed revision is the merge before the repository base: the running build contains the [configured resource]-draw re-route (primary switched to the turbo model, previous model inserted as fallback with breaker threshold 5 and 30 s cooldown, migration 210) but not the newer re-route (migration 212).
• The newer re-route (merged 2026-10-23T21:51:06Z, merge commit equals the repository base, awaiting deploy) re-points every assignment of the disabled-by-it previous primary model to the new model family on the same provider, including this workflow's fallback row, and adds a third parallel VIP slot on the same provider.
• Runtime healthy at evidence time: running, restart_count 0, oom_killed false, moderate CPU, ample memory and disk; database available with 0 waiting locks; queues progressing (image-regular 4 failed vs 2154 completed in the windowed counters, image-vip 0 failed vs 114 completed).
• Both the [configured resource] lane and the VIP lane use the same provider, service and generator code path; the windowed failure counters concentrate the failures in the highest-volume lane.
• Existing unit tests cover provider_unavailable fallback events and all-slots-failed behavior for this queue in crates/openplotva-app/src/image_jobs.rs, and the observed event sequence matches those tested paths.
• A repository-level guard treats a single-attempt exhaustion with a retryable reason as non-actionable, but this occurrence recorded 2 failed attempts, so it surfaced as an incident as designed (crates/openplotva-app/src/runtime_routing.rs).

Hypotheses

• H1 (leading, external): the single draw backend intermittently returns 429/5xx for all endpoints in short bursts (capacity, overload or worker restarts); both distinct endpoint models fail identically at the same instants and exhaustion follows as designed; recurrence over 53.6 h at count 13 fits intermittent bursts rather than a hard outage.
• H2 (needs provider-side evidence, related to our changes): the [configured resource]-draw re-route concentrated the highest-volume image lane onto the turbo endpoint, and parallel VIP slots added on the same provider raised concurrent slot demand enough to provoke 429 responses; deploy-to-incident timing and shared-endpoint topology support checking this but do not prove it.
• H3 (design question, weaker): the job-retry cadence against the 30 s fallback breaker cooldown lets jobs re-exhaust quickly during a burst, amplifying the incident count per burst; configured behavior to confirm as intended rather than defective.
• H4 (weakly contradicted): a client-side request defect specific to the [configured resource] path provoking 5xx responses; weakened because the VIP lane shares the provider, the service and the generator code and completes with ~0 failures in the window, but not excluded without raw provider responses.

Supporting

• Two distinct endpoint models failed with the identical reason on the same provider at the same timestamp, consistent with a provider-level rather than model-level condition.
• provider_unavailable is a provider-response classification (429/5xx) in our client code, not an internal error path.
• Host, runtime, database and non-draw queues were healthy at evidence time, ruling out app-wide resource failure or crash-looping.
• Failures remain low-volume and concentrated in the highest-volume image lane while sibling image queues on the same provider completed, consistent with intermittent bursts hitting the lane with the most traffic.
• The incident persisted identically across the redeploy to the current running revision, which contains no draw-routing change beyond the [configured resource]-draw re-route, so a build-specific client fault is unlikely.
• The fallback candidate appears in the chain at all, which the routing migration only inserts after switching the primary, so the deployed database rows reflect the re-route rather than the pre-change single-candidate chain.

Contradicting

• No raw provider response bodies, status codes or transport errors are available in the scoped evidence, so a client-side misclassification or request defect (H4) cannot be fully excluded from observation alone.
• If the provider were hard-down, the sibling VIP draw workflow on the same provider should fail in the same window; its windowed failure counter is 0 versus 114 completed, favoring bursts over a sustained outage, but windowed counters cannot align failures to the exact incident instants.
• The attempt evidence lists the previous primary model as attempt:1 and the newer turbo model as attempt:2; with identical timestamps the ordering does not prove which assignment ran first, so the deployed primary/fallback order cannot be confirmed from the attempt list alone.
• The [configured resource]-draw re-route about 2 h before first_seen remains a correlated change that no current evidence rules out as a contributing cause (H2).

Related changes

• PR Free draws on Krea 2; VIP draws Krea 2 + MOSS-Image #274 (merged 2026-10-21T13:52:47Z, deployed before first_seen): switched this workflow's primary to the newer turbo model and inserted the previous model as fallback (migration 210, breaker 5 / 30 s); both candidates in every failed chain of this incident, including the current one, come from this change.
• PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278 (merged 2026-10-23T21:51:06Z, awaiting deploy; its merge commit is the repository base): replaces every assignment of the previous primary model with a new model family on the same provider (migration 212), disables the old model, and adds a third parallel VIP slot on the same provider; once deployed it changes this workflow's chain and adds endpoint load, but leaves the single-provider dependency intact.
• PR Give VIP draws a third picture from MOSS-Image #266 (merged 2026-10-19): expanded VIP draws to additional parallel picture slots on the same single draw provider, increasing concurrent endpoint pressure that the load-aggravation hypothesis (H2) needs evaluated.
• PR Fix image model routing for free generation and editor #92 (merged 2026-07-31): historical fix of image model routing for [configured resource] generation in the same workflow area; precedent that this workflow's routing has regressed before.

Missing

• Raw provider responses for the failed attempts: exact HTTP status codes or transport errors, to separate 429 capacity from 5xx failure or connection-level faults and to exclude client-side misclassification.
• Draw backend health and logs for each burst window (slot residency, load, restarts) to confirm or refute a provider-side condition; not accessible from the scoped evidence tools.
• Deployed workflow_assignments rows for this workflow (primary/fallback roles, weights, breaker settings) to verify production routing matches migration 210's intent, since the attempt ordering in the evidence is ambiguous.
• Per-window failure records for sibling workflows on the same provider aligned to the exact burst instants, to confirm provider-wide correlation rather than [configured resource]-lane-specific failures.
• Timestamps of all 13 exhaustion events to establish whether they cluster into few bursts or recur steadily across the 53.6 h span.
• Post-deploy behavior of the pending re-route (PR Route FLUX.3 work and a third VIP picture to Fibo 2.0 #278): whether the changed chain and added VIP slot raise or lower the incidence, observable only after the controller deploys it.

Acceptance

• Deep stage: obtain the classified status codes or transport errors behind the provider_unavailable attempts for a burst window.
• Deep stage: align sibling-workflow failures on the same provider with the burst instants; provider-wide correlation confirms H1, [configured resource]-lane-specific failures point back at our request path (H4).
• Deep stage: verify the workflow_assignments rows for the [configured resource]-draw workflow match migration 210's intent (turbo primary, previous model fallback with markers) in production, not just in the repository.
• Deep stage: review whether the retry cadence versus the 30 s breaker cooldown should space [configured resource]-draw retries during provider bursts, as a deliberate resilience decision rather than a defect fix.
• After the pending re-route deploys, re-check whether the incidence persists under the changed chain before proposing routing changes of our own.
• Any code fix requires a reproduced, verifiable behavioral defect in classification, fallback selection or breaker handling; none is observed so far.

Impact

13 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.