[DO NOT MERGE] Spike having a federation layer on top of multiple instances of ServiceControl

[SzymonPobiega]

ServiceControl aggregating proxy: design proposal

Status: draft, reviewed once. An initial implementation exists in src/ServiceControl.Federation (see §14).

Decisions from the first review:

• Failed-message id collisions across installations are not expected, so passing ids through unchanged (§5.1) is accepted.
• The same monitored endpoint in several installations is not expected in normal setups. The merge rules in §10 remain as a safety net.
• Hosting. The proxy is an optional feature of a primary instance, in the same way as the integrated ServicePulse. The code stays in its own library project, and the primary turns it on with a setting and configures the members with a second setting in the style of RemoteInstances (§14). An earlier standalone host was dropped.

1. Problem and goal

One ServicePulse instance today talks to exactly one ServiceControl primary (error) instance and one ServiceControl.Monitoring instance. Organisations that run several independent ServiceControl installations (per region, per business unit, per broker) need one ServicePulse per installation.

Goal: a C# component, called the aggregating proxy below, that:

• presents the same HTTP API as a ServiceControl primary (/api/...) and as ServiceControl.Monitoring, so ServicePulse is not modified;
• fans each request out to N member installations and aggregates the answers, or routes the request to the member that owns the addressed resource;
• degrades gracefully when a member is slow or down, using the partial-results signal ServicePulse already understands.

Non-goals:

• Replacing the primary-to-audit aggregation that already exists inside each primary. The proxy treats each member primary as a black box that already includes its own audit instances.
• Merging persisted data or providing a global store. The proxy has no database.
• Making the licensing usage report a single signed artefact (see §9.6).

Terminology

Term	Meaning
Member	One ServiceControl installation: a primary URL, an optional Monitoring URL, and a short stable member key
Lead member	The member configured to answer instance-wide questions that cannot be aggregated (for example the endpoint connection snippet)
{SC} / {MU}	The ServiceControl and Monitoring base URLs ServicePulse is configured with. The proxy serves both.
Owner	The member, or members, that hold a record addressed by id

2. Inputs to this design

This proposal is based on reading the code, not on running anything.

• ServiceControl primary route list: src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.HttpApiRoutes.approved.txt. That file omits the /api prefix, api/authentication/configuration, api/maintenance/retention/purge, and the licensing component's api/licensing/* routes. Those were taken from the controllers.
• Audit and Monitoring route lists: the matching APIApprovals.HttpApiRoutes.approved.txt files.
• Existing primary-to-audit aggregation: docs/multipleservicecontrolinstancescommunication.md, src/ServiceControl/CompositeViews/Messages/ScatterGatherApi.cs, and how ETags are combined in docs/data-versioning-design.md.
• ServicePulse frontend: src/Frontend/src in the ServicePulse repository. All HTTP calls go through components/serviceControlClient.ts and components/monitoring/monitoringClient.ts.

Key facts that shape the design

1. ServicePulse knows only two base URLs: one {SC} (ending in /api/) and one {MU}. Every call is {SC} or {MU} plus a relative suffix. Putting the proxy at those two URLs is enough.

   • ServicePulse's own .NET host already reverse-proxies /api/** and /monitoring-api/** with YARP (src/ServicePulse/ReverseProxy.cs), so the proxy can sit behind it.

2. There is no push channel. SignalR was removed from ServiceControl. ServicePulse polls everything, mostly every 5 s and down to 1 s during operations (§10.1).

3. ServicePulse already handles partial results. It parses X-Particular-Incomplete-Results: <instanceId>:<timeout|unavailable|error>, … on messages2 (components/incompleteResults.ts).

4. Only two server-provided URLs are followed:

   • my_routes_url from the root document, which is absolute.
   • body_url, which is relative, /messages/{id}/body?instance_id=…, and resolved against {SC}.

   All other URLs in the root document are stored but never fetched.

5. One header drives every feature gate: X-Particular-Version on GET {SC}. ServicePulse requires at least 6.6.0, and features switch on at 5.2, 5.4, 5.9, 6.6 and 6.19.

6. ServicePulse reads Total-Count for counts and paging. It reads it on errors, recoverability/groups/{id}/errors, customchecks, eventlogitems, redirects and messages2. It reads no Link or ETag headers. The browser default HTTP cache still honours ETag.

7. Many ids are derived from content, so they are the same on every member (DeterministicGuid.MakeId, which is MD5 of the concatenated strings):

   Id	Derived from	Effect across members
   Failure group id	MakeId(classifier, title)	Identical on every member by design, which makes it a natural merge key
   Redirect id	MakeId(fromPhysicalAddress)	Collides whenever two members redirect the same address
   Endpoint instance id	MakeId(name, hostId)	Same instance, same id
   Custom check id	MakeId(endpoint, hostId, checkId)	Same instance, same id
   Failed message unique id	MakeId(messageId, endpoint)	Collides only if the same message failed in two installations
   Retry/archive operation ids	e.g. FailureGroup/{groupId}, "All"	Collide
   Instance id (instance_id on body_url)	base64 of the configured URL	Collides when two members use the same default URL, e.g. localhost:33333
   Event log item id (EF Core persisters)	Database identity column: 1, 2, 3, …	Collides between unrelated items. RavenDB ids are unique.

8. Commands on unknown ids are not safe to broadcast. ArchiveMessageHandler dereferences GetFailedMessage(...) without a null check, and the EF Core store returns FailedMessage?. An archive for an id that a member does not hold therefore fails in that member's message handler. Commands addressed by id must go to the owner (§5.2).

9. The existing aggregation is only exact for page 1. ScatterGatherApi asks every instance for the same page and per_page, unions the results, re-sorts them and keeps per_page rows.

   • Page 2 and beyond can skip rows. Total-Count overcounts after duplicates are removed.
   • The proxy must not copy this paging approach (§6.4).

10. Ids appear inside payloads, not only in URLs. Examples: edit_of, event log related_to: ["/message/{id}"] (ServicePulse takes substring(9)), headers such as ServiceControl.Retry.UniqueMessageId, conversation messages, and Monitoring instance ids passed as classifierFilter values to recoverability/groups/Endpoint Instance. Rewriting ids would mean rewriting all of these consistently.

3. Architecture

                  ┌───────────────────────────── Aggregating proxy ──────────────────────────────┐
 ServicePulse     │  ASP.NET Core                                                                │
 {SC} = /api/ ───►│  Route table (mirrors SC primary routes)  ──► Operation handlers            │
 {MU} = /mon/ ───►│  Route table (mirrors Monitoring routes)        │                            │
                  │                                                 ▼                            │
                  │      Fan-out engine ── Member clients (HttpClient per member,                │
                  │       │                 timeouts, health, circuit breaker)                   │
                  │       ├─ Aggregators (union, sum, key-merge, sorted page merge, first hit)   │
                  │       ├─ Owner resolver (+ in-memory owner cache)                            │
                  │       └─ Response composer (Total-Count, Link, ETag, X-Particular-*)         │
                  └───────┬──────────────────────────┬──────────────────────────┬──────────────┘
                          ▼                          ▼                          ▼
                 Member A primary (+audits)  Member B primary (+audits)  Member C ...
                 Member A monitoring         Member B monitoring

• Hosting. Inside a primary instance, enabled by configuration (§14).

  • It serves both surfaces on the primary's port under /federation/api/ and /federation/monitoring/. The primary's own /api/ is unchanged, because the federation calls members' /api/, including this instance's when it is a member.
  • The host provides forwarded headers, CORS, authentication and authorization. Routes that members serve anonymously are marked anonymous so the primary's authorization fallback policy does not apply to them.

• Statelessness. The proxy keeps no durable state. It holds three kinds of in-memory cache, all rebuildable:

  • an owner cache (id → members);
  • member health;
  • short-lived response coalescing.

  Running several replicas is safe.

• No database and no transport. The proxy speaks HTTP only. It never sends NServiceBus messages; members do that when the proxy forwards commands.

3.1 Configuration

Two settings on the hosting primary, read like every other setting (environment variable, then App.config, then registry):

Setting	Environment variable	Value
ServiceControl/EnableFederation	SERVICECONTROL_ENABLEFEDERATION	true to host the federated API (default false)
ServiceControl/FederatedInstances	SERVICECONTROL_FEDERATEDINSTANCES	JSON array of members, in the style of RemoteInstances

[
  { "api_uri": "http://sc-eu:33333/api/", "monitoring_uri": "http://mon-eu:33633/", "key": "eu", "name": "Europe" },
  { "api_uri": "http://sc-us:33333/api/", "monitoring_uri": "http://mon-us:33633/", "key": "us", "name": "Americas" }
]

• Only api_uri is required. The first member is the lead member.
• key is short and stable. It appears in the synthesised instance_id values (§5.3), so changing it breaks links ServicePulse has cached. When it is omitted, it is derived from the host and port of api_uri, for example sc-us-33333.
• To include the hosting instance itself, list its own API URL. Members are always called over HTTP.
• Timeouts and paging limits keep their defaults (MemberTimeout 4 s, DeepPagingMaxRows 1000) and are not configurable yet.
• The member set is static configuration in version 1.

3.2 Choosing between typed contracts and JSON trees

Recommendation: handle payloads as System.Text.Json.Nodes.JsonNode and read only the fields needed to merge, sort or route (for example id, time_of_failure, count).

• Typed DTOs taken from ServiceControl.Persistence would silently drop fields that newer members add.
• They would also tie the proxy to one ServiceControl version.

Unknown fields pass through untouched. The few fields the proxy computes, such as group count and operation_*, are written back into the node.

4. Request handling model

Every proxied route is assigned exactly one strategy. The route table is explicit. A request for an unknown route goes to the lead member and is logged at warning level, so new ServiceControl routes keep working without silently returning wrong aggregates. A test (§12) catches routes that have no assigned strategy.

Strategy	Meaning	Used for
U union	Concatenate member arrays; optionally de-duplicate by key	endpoints, known endpoints, remotes, classifiers
S sum	Add numeric totals (body and/or Total-Count)	HEAD errors, heartbeats/stats, errors/summary, disconnected count
K merge by key	Group by key and combine fields with per-field rules	failure groups, queue addresses, audit counts, monitored endpoints
P sorted page merge	Exact k-way merge of sorted, paged lists (§6.4)	errors, group errors, custom checks, event log, redirects
W top-N window	Fan out unchanged, merge, keep N (exact without offsets)	messages2, search, conversation, endpoint messages
F first hit	Ask members, return the first 2xx (or merge if several hit)	errors/last/{id}, errors/{id}, groups/id/{id}, sagas/{id}
R route to owner	Resolve the owner(s) of an id and forward only there	per-id commands, body fetch
B broadcast	Send to all members; merge status codes	group commands, retry all, settings writes
L lead	Answer from the lead member only	connection, edit/config, licensing report, auth configuration
X synthesise	Proxy builds the response itself	root document, my/routes, OPTIONS

5. Identity and routing

5.1 Ids are passed through unchanged

Decision: the proxy does not rewrite resource ids. The one exception is instance_id, covered in §5.3.

Rewriting is attractive because a prefix such as eu~<guid> would make every id unique and routing stateless. It was rejected for three reasons:

• Ids appear inside payloads in many places (§2, fact 10). Every one of them would need consistent rewriting, and ServicePulse parses some of them, for example related_to.substring(9).
• Several ServiceControl routes constrain ids to GUIDs. The proxy could relax this on its own routes, but any member-produced string containing an id would still be unprefixed.
• Ids that are equal on every member by design, such as group ids, are exactly what the UI needs to show one merged group. Prefixing them would break merging.

The cost of passing ids through is that the proxy must work out who owns an id (§5.2), and that a rare real collision of failed-message ids is acted on in both installations (§13).

5.2 Owner resolution

Commands addressed by id (errors/retry [ids], errors/archive [ids], errors/unarchive [ids], edit/{id}, pendingretries/retry [ids], pendingretries/resolve {uniquemessageids}, customchecks/{id}, endpoints/{id}, redirects/{id}) must reach only the owner, because members do not tolerate unknown ids (§2, fact 8). Resolution has two steps:

1. Owner cache. Every list or detail response the proxy relays records id → member for the rows it contains. This covers failed messages, custom checks, endpoints and redirects. The cache is bounded (LRU), expires entries after a TTL, and can be rebuilt at any time.

   • In practice ServicePulse always lists a record before acting on it, so the cache answers nearly every command.

2. Probe on a miss. The proxy issues a cheap existence check to every member in parallel:

   Record type	Existence check
   Failed message	GET errors/last/{id}, 200 or 404
   Custom check, endpoint	The member's list, filtered client-side
   Redirect	GET redirects

   The command is then forwarded only to the members that answered 200.

Batch bodies, which are arrays of ids, are partitioned by owner. One request goes to each owning member with only its ids. The response is 202 if every partition was accepted. Ids with no owner are dropped and logged.

5.3 Synthesised instance_id

body_url carries ?instance_id=<memberInstanceId>. That value is a base64 URL and is not unique across members. For each message row in W/F results the proxy rewrites it to:

instance_id = {memberKey}.{originalInstanceId}

• If the member's body_url has no instance_id (the member's local body store), the proxy appends instance_id={memberKey}..
• On GET messages/{id}/body?instance_id=eu.abc the proxy:
  1. splits on the first .;
  2. forwards to member eu with instance_id=abc, or with no instance_id if abc is empty;
  3. streams the response through with YARP's HttpForwarder, unchanged, keeping Content-Type.
• The same rewrite applies to:
  • POST errors/{id}/retry?instance_id=;
  • the instance ids in X-Particular-Incomplete-Results that members return;
  • the instance_id field on message rows, if present.

Base64url never contains ., so splitting on the first . is unambiguous.

When ServicePulse falls back to messages/{message_id}/body with no instance_id (saga diagram), the proxy uses strategy F: it asks members in turn and returns the first 200.

6. Read aggregation details

6.1 Fan-out engine

The engine is modelled on ScatterGatherApi, with these changes:

• Parallel requests. One HttpClient per member, sent in parallel, each with its own timeout (default 4 s, so it fits inside ServicePulse's 5 s poll) and cancelled if the caller goes away.

• Error classification is the same as ScatterGatherApi.FetchAndParse:

  Member outcome	Treated as
  404	Empty result
  401 or 403	error
  504 or timeout	timeout
  Connection failure	unavailable, and the member is marked temporarily down

• Member health. A member that is marked down is skipped until a background probe (GET {member} every 15 s) succeeds. This saves timeouts on every 5 s poll.

• Every aggregated response adds X-Particular-Incomplete-Results with {memberKey}:{reason} entries for the failed members.

  • ServicePulse surfaces this only on messages2 today. On other routes it is harmless and useful in diagnostics.

• If no member answers and at least one timed out, the proxy returns 504 with a problem+json body, as the primary does. If all members are unavailable it returns 503.

• Authorization header. The caller's Authorization header is forwarded to every member (§8).

6.2 Union (U), sum (S) and key merge (K): field rules

Endpoint	Strategy	Rule
GET endpoints	U	Concatenate. Record the owner of each id.
GET endpoints/known	U	Concatenate, de-duplicate by id. Total-Count is the row count.
GET endpointssettings	K by name	track_instances from the lead member wins on conflict. The default entry name: "" comes from the lead member.
GET heartbeats/stats	S	Sum active and failing.
GET errors/summary	S per key	For each of Endpoints, Hosts and Message types, sum the counts per name.
HEAD errors, HEAD recoverability/groups/{id}/errors	S	Total-Count is the sum. The ETag is combined (§7).
GET errors/queues/addresses	K by physical_address	Sum failed_message_count. Sort by count descending (the member sort order). Page in the proxy.
GET recoverability/classifiers	U	Order-preserving distinct; the lead member's order wins.
GET recoverability/groups/{classifier}	K by id	See §6.3. The classifierFilter query parameter is forwarded.
GET errors/groups/{classifier} (archived)	K by id	Sum count, take the minimum first and maximum last; for comment, see §6.3.
GET recoverability/groups/id/{id}, GET archive/groups/id/{id}	F, then K	Merge every hit as in §6.3. 404 only if all members return 404.
GET recoverability/history	U	Concatenate historic_operations and sort by completion_time descending. Keep the depth limit of a single member. Concatenate unacknowledged_operations.
GET endpoints/{endpoint}/audit-count	K by date	Sum count per utc_date.
GET configuration/remotes	U + X	Concatenate every member's remotes (audit instances), de-duplicated by api_uri. Also add each non-lead member primary as an entry: {api_uri, version, status, configuration} from its GET configuration. ServicePulse already models remote-error instances in platform health (PlatformModelStore.ts), so members appear there without UI changes.

6.3 Merging failure groups

Groups are the most important merge, because the group id is the same on every member by design. For each group id, combine every member's GroupOperation:

• title, type: taken from any member; they are identical.

• count: sum. first: minimum. last: maximum.

• comment: if all non-empty comments are equal, use that value. Otherwise use the lead member's comment, or the first non-empty one, and log the conflict. Writes are broadcast (§6.5), so comments converge after the next edit.

• Operation state. Each member reports its own state for the group.

  Field	Combination rule
  operation_status	The "most active" state across members. Precedence: Preparing/ArchiveStarted > Forwarding/ArchiveProgressing > Waiting > Completed/ArchiveCompleted > None. A group is complete only when every participating member is complete.
  operation_progress	Average weighted by each member's total for the operation (operation_remaining_count + operation_messages_completed_count)
  operation_remaining_count, operation_messages_completed_count	Sum
  operation_failed	OR
  need_user_acknowledgement	OR
  operation_start_time	Minimum
  operation_completion_time, last_operation_completion_time	Maximum, and only once every member has finished

• Pseudo-groups that a member adds for unacknowledged completed operations use the same group id, so the same rules apply.

• Sort by last descending, which is the member order.

Limit on group lists. Each member returns at most 200 groups (PR #5899). The merged list is cut at 200 after sorting, to keep the contract.

6.4 Sorted page merge (P)

This strategy applies to:

• GET errors and GET endpoints/{name}/errors
• GET recoverability/groups/{id}/errors
• GET customchecks
• GET eventlogitems
• GET redirects

The request carries page, per_page, sort and direction, plus filters such as status, modified and queueaddress.

Exact algorithm. The rows for page p of size s are rows (p-1)·s … p·s-1 of the global order. Each member contributes at most p·s of them. So:

1. Ask each member for page=1&per_page=p·s with the same sort and filters.
2. Merge the results k-way using the same comparator.
3. Skip (p-1)·s rows and take s.
4. Set Total-Count to the sum of the members' Total-Count values.

The merge is exact and stateless. Its cost grows linearly with page depth, so the proxy limits it with DeepPagingMaxRows (default 1000 rows per member).

What happens beyond the limit:

• The proxy falls back to the existing primary behaviour: forward page=p&per_page=s unchanged, union, sort, take s.
• It adds X-Particular-Incomplete-Results: proxy:approximate-paging so the approximation is visible.

ServicePulse usage makes deep pages rare:

• Failed messages use 50 per page.
• Custom checks use the member default of 50, although the UI assumes 10 per page (a ServicePulse bug, not something the proxy can fix).
• The event log uses at most 75 per page.

Comparator. The comparator must reproduce each member's sort for the sort keys ServicePulse sends.

List	Sort keys	Tie-breaker
Failed messages	time_of_failure, message_type, time_of_retry_request, modified/last_modified	id
Event log	raised_at descending	id (string)
Custom checks	Member order (status, then reported_at)	id
Redirects	sort and direction query parameters (from_physical_address, to_physical_address, last_modified)	message_redirect_id

• An unknown sort key falls back to the member ordering and uses a stable member-interleaving merge.
• Before coding: confirm each persister's tie-breaking. Without a total order, page boundaries may jitter between polls, though that is no worse than today.

Smaller choices:

• Paging links. Link is rebuilt from the merged totals, like the primary's WithPagingLinks but using ceil(sum/s). ServicePulse ignores Link, but ServiceInsight-style clients do not.
• Event log ids from EF Core persisters collide (§2, fact 7). They are rendered only as Vue keys. The proxy rewrites id on event log rows to {memberKey}.{id} for those rows. This is the one id rewrite besides instance_id, and it is safe because no API accepts an event log id.

6.5 Message searches and conversations (W)

GET messages2 pages with a time window and page_size, and has no offset. Forward the request unchanged, merge by the requested sort, and take page_size. This is exact.

The other searches:

• GET messages
• GET messages/search, GET messages/search/{keyword}
• GET endpoints/{e}/messages, GET endpoints/{e}/messages/search[/{keyword}]
• GET conversations/{id}

These are scatter-gather endpoints on each member already. The proxy applies P to the ones that are paged and plain union to conversations/{id}, which returns the whole conversation. It also de-duplicates rows by receiving_endpoint.name + message_id, as ScatterGatherApiMessageView does, giving priority to the lead member.

ServicePulse uses these as follows:

• messages/search/{message_id} is used to load a message's details. The UI picks the row whose id matches, so merging all hits is correct.
• messages/search?q={sagaId} is used to find saga message bodies.

GET sagas/{id}: if one member answers, return its result. If several answer, merge the changes arrays ordered by finish_time descending, as GetSagaByIdApi does today.

6.6 Root document and configuration

GET {SC} (X). The proxy synthesises RootUrls with its own base URL. It builds my_routes_url, errors_url and the other URLs from the incoming request, respecting X-Forwarded-*. Field values:

• name: a configured DisplayName, defaulting to "ServiceControl (federated)".
• license_status: the worst status across members.
• Response header X-Particular-Version: the lowest member version, so ServicePulse never enables a feature that one member lacks.
  • If a member is unreachable, its last known version is used.
  • If any member is below ServicePulse's minimum (6.6.0), the proxy logs an error and reports the low version, so the existing incompatibility toast appears.
• platform_health_version: the version of the member that answered, normally the lead. ServicePulse shows it in platform health in preference to X-Particular-Version, but its feature gates read only the header, so the header keeps the lowest version. The other member primaries have their own rows in configuration/remotes.

GET configuration (and GET instance-info). The lead member's document, with these fields changed:

Field	Value
data_retention.error_retention_period	The shortest across members. ServicePulse uses it to warn "scheduled for deletion", so the conservative value is correct.
mass_transit_connector	The lead member's value. If any member has one, merge error_queues and logs. Its presence hides the flow, sequence and saga tabs in ServicePulse, so this needs a product decision (§13).

GET connection (L). Answered by the lead member.

• The snippet configures an endpoint to report to one installation, and choosing an installation is a deployment decision rather than a UI one.
• Alternative: merge the settings with suffixes (ErrorQueue, ErrorQueue1, …), reusing the duplicate-key convention of PlatformConnectionDetails. This produces a confusing snippet, so it is not recommended.

7. Headers and HTTP caching

Header	Proxy behaviour
Total-Count	Recomputed for each strategy (sum, row count, or merged count)
Link	Recomputed for P (§6.4); omitted elsewhere, as on members
ETag	W/"…" built from DataVersion.Combine-style hashing of (memberKey, memberETag) pairs, sorted by member key. If any member returned no ETag, or failed, no ETag is emitted (the "absence never answers 304" rule in docs/data-versioning-design.md). For P routes the member ETags cover different per_page windows than the client asked for; that is still a sound validator, because a member's window ETag moves whenever its rows within the window move.
If-None-Match	Answered by the proxy against the combined ETag; returns 304 if it matches. Not forwarded to members, because the client's tag is a composite.
Conditional requests to members (optional)	The proxy keeps the last (memberKey, upstream URL) → ETag + body per URL in a small memory cache and sends If-None-Match upstream. A member 304 lets it reuse the cached body. This cuts member load, since ServicePulse polls the same URLs every 5 s.
X-Particular-Version	On every response: the lowest member version, as for the root (§6.6). On forwarded responses (body, routed commands), the member's own value passes through.
X-Particular-Incomplete-Results	Member-prefixed entries (§6.1, §5.3)
X-Particular-Reason	Passed through from the member that produced the response. For broadcast writes, taken from the first failing member.
Content-Type, Content-Disposition	Passed through on streamed responses
Cache-Control: private, max-age=0, Vary: Accept	Emitted as members do
CORS	Same policy as the primary: exposes ETag, Last-Modified, Link, Total-Count, X-Particular-Version, Content-Disposition, Request-Id, X-Particular-Incomplete-Results, plus Allow on the Monitoring surface

8. Authentication and authorization

• Token passthrough. ServicePulse obtains an OIDC access token and sends it as a bearer token to {SC} and {MU}. The proxy forwards the Authorization header unchanged to members.
  • Requirement: every member must trust the same authority and audience. This is already required today between a primary and its audit instances.
  • The proxy may validate the token itself, to reject early and to log the subject, using the same OpenIdConnectSettings shape as ServiceControl (ServiceControl.Hosting/Auth).
  • Members remain the enforcement point; the proxy never grants more than members do.
• GET authentication/configuration (anonymous, strategy L). Answered by the lead member. At startup the proxy checks that every member returns the same enabled, authority and audience, and logs an error if not.
• GET my/routes (X). The proxy calls each member's my/routes with the caller's token and returns:
  • routes: the intersection of route keys. An action is offered only if the user may perform it on every member, which matches how broadcast commands behave. Routes are expressed as /api/... templates, which is the shape ServicePulse normalises in routeMatching.ts.
  • roles: the union.
  • Members that are down are left out of the intersection. ServicePulse gating fails open, and members still enforce on the actual call.
• Monitoring GET api/my/routes. Same approach over the member Monitoring instances.
• Anonymous routes stay anonymous: root, configuration, instance-info, configuration/remotes and authentication/configuration.

9. Commands (writes)

All member commands are asynchronous: they return 202 and send a local NServiceBus message. Progress shows up later in polled reads.

9.1 Status-code merging for broadcast writes

Member results	Proxy returns
All 2xx	The lowest success code (202 if any member returned 202)
Some 2xx, some 404	2xx (the resource did not exist on some members)
All 404	404
Any 4xx other than 404	That 4xx (first by member order) with its body and X-Particular-Reason
Any 5xx or timeout, at least one success	2xx plus X-Particular-Incomplete-Results
No success	502, or 504 if timeouts

9.2 Command routing table

Route	Strategy	Notes
POST errors/retry [ids]	R, partitioned	§5.2
POST errors/{id}/retry[?instance_id]	R	Owner from instance_id if present (§5.3), else from the owner resolver
PATCH/POST errors/archive [ids], PATCH errors/unarchive [ids]	R, partitioned	Must not broadcast (§2, fact 8). The trailing-slash form errors/archive/ is accepted too.
PATCH/POST errors/{id}/archive	R
PATCH errors/{from}...{to}/unarchive	B	Range by time; every member applies it to its own data
POST errors/retry/all	B
POST errors/{endpoint}/retry/all	B	Members without that endpoint do nothing
POST errors/queues/{queue}/retry	B	Members without that queue do nothing
POST edit/{id}	R	The member that owns the message re-sends it; the edited message fails back into that member's error queue
POST pendingretries/retry [ids], PATCH pendingretries/resolve {uniquemessageids}	R, partitioned
POST pendingretries/retry {from,to}, PATCH pendingretries/resolve {from,to}, `.../queues/retry	resolve {from,to,queueaddress}`	B
`POST recoverability/groups/{id}/errors/retry	archive	unarchive`
POST/DELETE recoverability/groups/{id}/comment	B	Keeps comments converged
DELETE recoverability/unacknowledgedgroups/{id}	B	200 if any member returned 200
DELETE customchecks/{id}	R
PATCH endpoints/{id} {monitor_heartbeat}, DELETE endpoints/{id}	R
PATCH endpointssettings/{name?}	B	The default entry (name empty) applies everywhere
POST/PUT/DELETE redirects[...]	R or B, configuration choice	See §9.3
POST notifications/email, .../toggle	B	GET notifications/email comes from the lead member. authentication_password is returned in clear by members; the proxy passes it through unchanged and does not log bodies.
POST notifications/email/test	B	Fails if any member fails; X-Particular-Reason names the member
POST license/detailsUpload	L, configurable as B
POST licensing/endpoints/update, POST licensing/settings/masks/update	B
POST maintenance/retention/purge	B	Status: running if any member is running; last_outcome is the worst

9.3 Redirects

A redirect maps a failed queue address to another address on one member's broker.

• If members share a broker, redirects should apply everywhere: use strategy B. Ids are equal on every member because they are derived from from_physical_address.
• If members use different brokers, a redirect belongs to the member whose errors/queues/addresses list contains the source queue. Strategy R then resolves the owner by from_physical_address, and GET redirects unions the members' lists.
• The default is R by address, which is correct in both cases. It degrades to "applies only where the queue has failures".

10. Monitoring surface ({MU})

Route	Strategy	Rule
GET /	X	The answering member's document, with platform_health_version set to its own version and platform_health_status set to the worst health across members (at least degraded when one does not answer). X-Particular-Version is the lowest member Monitoring version. ServicePulse has one Monitoring row and no remote Monitoring role, so other members' Monitoring instances have no row of their own.
OPTIONS /	X	Allow is the intersection of the members' Allow values (so it advertises DELETE only if all members support it)
GET monitored-endpoints?history=N	K by Name (default)	See below
GET monitored-endpoints/{name}?history=N	K	Instances: concatenate. MessageTypes: merge by Id (type identity), combining their metrics. Digest: recompute from the merged series. MetricDetails.Metrics: merge index by index with TimeAxisValues from the lead member.
GET monitored-endpoints/disconnected	K	Recomputed: the number of merged endpoint names whose instances are all stale. A sum would be wrong once names are merged. Computed from the same cached history=1 fan-out.
DELETE monitored-instance/{name}/{instanceId}	R	The owner comes from the cached merged list (EndpointInstanceIds per member)
GET connection	L	{ Metrics: { Enabled, MetricsQueue, Interval } } from the lead member
GET license	X	Worst status
GET api/my/routes	X	Intersection (§8)

Merging monitored-endpoints by name (recommended).

• Every member uses the same history period. Each series has 60 buckets aligned to wall-clock UTC, so points can be combined index by index.
• Combination rules:
  • Throughput, Retries and QueueLength: sum.
  • ProcessingTime and CriticalTime: average weighted by the members' Throughput points at the same index. Use a plain average when throughput is 0.
  • IsStale: AND (the merged endpoint is stale only if every member says so).
  • ConnectedCount and DisconnectedCount: sum.
  • EndpointInstanceIds: concatenate.
• Why merge. ServicePulse looks up failure groups for an endpoint with recoverability/groups/Endpoint Name?classifierFilter={name}, and the proxy merges those groups by id (§6.3). Merging monitored endpoints by name is therefore consistent with the failed-message counts shown beside them.
• Option to keep names separate. Setting MergeMonitoredEndpointsByName=false suffixes names as {name} ({memberKey}) and strips the suffix on monitored-endpoints/{name}. This breaks the group lookup above, so it is not the default.

history must be one of 1, 5, 10, 15, 30 or 60. Members throw 500 on other values, so the proxy validates up front and returns 400.

10.1 Load profile

The proxy multiplies member load by roughly N.

On every page, every 5 s, each ServicePulse tab triggers about ten calls:

• root, configuration, configuration/remotes
• customchecks?status=fail
• errors?status=unresolved (count only)
• endpoints, endpointssettings
• monitored-endpoints/disconnected
• eventlogitems (dashboard)
• capability probes

Mitigations:

1. Coalescing. Identical concurrent GETs for the same member URL and auth subject share one upstream call.
2. Micro-cache. Aggregated GET results are kept for 1–2 s, keyed by URL and authorization subject. Results are never shared across users, because permissions differ.
3. Conditional upstream requests (§7).
4. Health short-circuit, so dead members cost nothing (§6.1).

11. Licensing and usage

Route	Strategy	Notes
GET license?refresh&clientName	X	Fields from the lead member; status/license_status is the worst across members; instance_name is the proxy display name
GET license/details	L	Per-installation endpoint metadata. A merged view would need decisions about classification conflicts; not in version 1.
GET licensing/endpoints	U	Concatenate, merging rows with equal name_hash + scope_hash: max_daily_throughput is the sum of daily maxima. That is an over-estimate, but members do not expose daily series. monthly_throughput is summed per month.
GET licensing/settings/info, .../settings/test, .../masks	L	settings/test should be shown as healthy only if every member is healthy. Its result fields can be ANDed per connection type.
GET licensing/report/available	X	report_can_be_generated = AND across members; reason from the first member that cannot
GET licensing/report/file?spVersion=	X	Reports are signed per installation (ThroughputCollector → SignedReport) and cannot be merged. The proxy downloads each member's report zip and returns one outer zip, {customer}.throughput-reports-{timestamp}.zip, holding one entry per member ({memberKey}-…zip), with Content-Disposition. Whoever consumes the report must accept a bundle; to confirm with the licensing team.

12. Implementation outline

This section describes structure only; no code yet.

src/ServiceControl.Federation/                    (or a separate repository)
  Program.cs                                       host, auth, CORS, forwarded headers
  Configuration/FederationSettings.cs              members, lead, timeouts, limits
  Members/MemberClient.cs                          HttpClient per member, classification, ETag cache
  Members/MemberHealthMonitor.cs                   background probe, temporarily-unavailable state
  FanOut/FanOutEngine.cs                           parallel calls, timeouts, incomplete-results
  Aggregation/{Union,Sum,KeyMerge,SortedPageMerge,TopNWindow,FirstHit}.cs
  Aggregation/GroupOperationMerger.cs              §6.3
  Aggregation/MonitoredEndpointMerger.cs           §10
  Routing/OwnerResolver.cs, OwnerCache.cs          §5.2
  Routing/InstanceIdCodec.cs                       §5.3
  Http/ResponseComposer.cs                         Total-Count, Link, ETag, X-Particular-*
  Endpoints/PrimaryApi.cs                          minimal-API route table, one line per SC route + strategy
  Endpoints/MonitoringApi.cs
  Endpoints/Synthesised/{Root,MyRoutes,License,Report}.cs

• Route table as data. Each SC route is declared once with its strategy and parameters (merge key, comparator, owner extractor). That keeps the mapping in §§6–11 reviewable in one file.

• Forwarding. Streamed pass-through cases (bodies, routed commands, lead-member reads) use YARP IHttpForwarder, as the primary does for its remotes.

• Code reuse from ServiceControl.

  • DataVersion (ServiceControl.Infrastructure)
  • ServiceControl.Hosting (forwarded headers, auth, request id)
  • OpenIdConnectSettings

  The proxy does not reference persistence or the NServiceBus packages.

Tests

1. Route coverage approval test. Load the primary and Monitoring HttpApiRoutes.approved.txt files plus the licensing routes, and assert that every route has a strategy in the proxy route table. Adding a route to ServiceControl then fails the proxy build until it is classified.
2. Aggregator unit tests. Paging exactness: property-based tests comparing P against a global sort over random member datasets. Group operation merge tables. Monitored-series merge.
3. Multi-member acceptance tests. Extend the existing ServiceControl.MultiInstance.AcceptanceTests style to run two primaries and the proxy in process, and drive the ServicePulse-visible flows: list, retry, archive group, edit and retry, body fetch.
4. ServicePulse contract check. Run ServicePulse's frontend test mocks (src/Frontend/test/preconditions, test/mocks/service-control-instance-template.ts) against proxy-produced responses to catch shape drift.

13. Risks and open questions

1. ServicePulse is ahead of this ServiceControl checkout. ServicePulse reads platform_health_status and platform_health_version from the root documents of {SC} and {MU}, and calls POST license/amendDetails. Neither exists in this branch of ServiceControl. The proxy has to support them once the members do. Platform health is handled as described in §6.6 and §10; license/amendDetails goes to the lead member.
2. Failed-message id collisions (resolved: not expected). The same message id failing in the same endpoint name in two installations would give the same unique id. With pass-through ids (§5.1), the UI would show the row once (the lead member's) while Total-Count counts both, and commands would reach both owners.
3. Group comments and operation state are per member. The merged view can show "retrying" while one member is idle. That is by design (§6.3); the UI copy says nothing about members.
4. mass_transit_connector hides UI features. One MassTransit-enabled member would hide the flow and saga tabs for everyone. A product decision is needed.
5. Version skew. The reported version is the minimum, so one old member turns off newer features for all. This is intended, but should be visible: log it, and surface it in configuration/remotes.
6. Deep paging cost. P costs O(page·per_page) per member; it is bounded by DeepPagingMaxRows and then approximate. An alternative is keyset paging, which would need member API changes and is out of scope.
7. Load amplification (§10.1). Each poll costs N member calls, and the timeout budget has to fit inside the 5 s poll.
8. Owner cache misses after a proxy restart cost one probe per id, which matters for large batch commands. If that proves slow, add a member-side batch existence endpoint; that is the only member change this design might want.
9. Usage report bundle (§11). Needs confirmation from the licensing team that a bundle of signed reports is acceptable.
10. Where should the component live? (Resolved: a separate project in this repository for now; possibly an optional feature of a primary instance later.) The other options were a separate repository or the ServicePulse .NET host, which already proxies with YARP.
11. Where the decision is recorded. This change spans ServiceControl and ServicePulse and introduces constraints that are hard to discover from code: shared OIDC audience, ids that must not be rewritten, and owner routing. Per AGENTS.md, it is a candidate for an ADR under docs/decisions/ once the design is agreed. This draft is a working document, not that record.

14. Initial implementation

The code is in the src/ServiceControl.Federation library, with tests in src/ServiceControl.Federation.UnitTests. Both projects are in ServiceControl.slnx under /Instances/Federation/.

The primary instance references the library, so ServiceControl.Federation.dll ships in the primary's deploy folder and therefore in the servicecontrol container image. No Dockerfile changes are needed.

Hosting in the primary

Where	What
Settings.cs	Reads EnableFederation and FederatedInstances, and validates them at startup when federation is enabled
RunCommand.cs	Calls AddServiceControlFederation and MapServiceControlFederation when enabled, next to the integrated ServicePulse
Integrated ServicePulse	When federation is also enabled, ServicePulse is configured with /federation/api/, and with /federation/monitoring/ if any member has a Monitoring URL. /api/ stays the instance's own API.
Startup log	A Federation: line with the number of members

Layout

Path	What it holds
FederationHostingExtensions.cs	AddServiceControlFederation(FederationSettings) and MapServiceControlFederation(), the two calls the host makes
Configuration/FederationSettings.cs	Settings, parsing of the FederatedInstances JSON (§3.1), and validation
Members/	Member registry (lead first), MemberGateway (HTTP calls and failure classification), MemberHealthMonitor (background probe)
Aggregation/	Pure merge logic: SortedPageMerger (§6.4), RowOrder (comparators that mirror each persister's sort), FailureGroupMerger (§6.3), MonitoredEndpointMerger (§10), CommandOutcome (§9.1), Versions
Routing/	InstanceIdCodec (§5.3) and OwnerCache (§5.2)
Http/	Reading the incoming request target unchanged, writing responses, and composing ETags (§7)
Api/Federator.cs	The reusable strategies (L, B, F, S, U, P, W, R)
Api/PrimaryRoutes.cs, Api/MonitoringRoutes.cs	The route tables: one line per route with its strategy
Api/*Handlers.cs	Route-specific merges

How it differs from the design

• The member health probe treats any HTTP answer as reachable. Only connection failures and timeouts take a member out of rotation; each request then classifies its own response.
• A member that answers with an error status (other than 404) is reported in X-Particular-Incomplete-Results as error, as well as members that time out or are unreachable.
• Prefixed ids use ~ as the separator. Event log ids become {memberKey}~{id}, and instance_id becomes {memberKey}~{original}. . is used by ServiceControl's base64url padding, so it cannot be the separator.
• DELETE monitored-instance/... is broadcast rather than routed. Removal only drops the instance from a member's in-memory registry, which is harmless on members that do not know it.
• Members' own incomplete-results entries (for their audit instances) are passed through unchanged, so ServicePulse can still decode them to URLs. Failed members are named by their API URL.

Not implemented yet

• Request coalescing, the micro-cache, and conditional requests to members (§7, §10.1)
• Validating tokens in the proxy (§8). Tokens are forwarded and members enforce authorization.
• A strategy to answer /connection from several members (currently the lead member answers)
• Per-member weighting of failure-group comments (currently the lead member's comment wins)

Tests

• Route coverage: every route in the primary, licensing and Monitoring approval files must appear in a route table.
• Paging exactness: randomised comparison of the merged page against a global sort.
• Merge rules: failure groups, monitored endpoints, instance_id encoding, version selection.
• End to end: the proxy runs against two fake members over HTTP. Covered: paged merge, owner-routed batch commands, owner probing, root document rewriting, body routing by instance_id, group merge, and an unreachable member.
• Settings parsing: key derivation, empty lists, duplicate keys, invalid URLs.
• Primary settings approval: APIApprovals.PlatformSampleSettings includes the two new settings.

Running it

On any primary, for example in the servicecontrol container:

environment:
  SERVICECONTROL_ENABLEFEDERATION: "true"
  SERVICECONTROL_FEDERATEDINSTANCES: '[{"api_uri":"http://localhost:33333/api/","key":"local"},{"api_uri":"http://other-host:33333/api/","monitoring_uri":"http://other-host:33633/","key":"other"}]'
  SERVICECONTROL_ENABLEINTEGRATEDSERVICEPULSE: "true"

Then open the primary's root URL. The embedded ServicePulse shows the federated view.

• A standalone ServicePulse can use ?scu=http://<primary>:33333/federation/api/&mu=http://<primary>:33333/federation/monitoring/.
• Inside a container, localhost is the container itself. That is correct for the hosting instance; other members need addresses the container can reach.
• Verified locally: a Release build of the primary running on PostgreSQL persistence, with the learning transport and only environment variables, federating itself and a second installed instance. Its own /api was unchanged. The embedded ServicePulse's configuration pointed at /federation/..., and the federated root, failed-message list and remotes answered. A member's stopped Monitoring instance was reported in X-Particular-Incomplete-Results.