Add store-versions: store releases against GitHub releases

README.md

@@ -17,6 +17,7 @@ share for translations. One package, `session_ops`, deployed to one self-hosted
 | `release-stats` | On demand: download counts of the latest releases | [release-stats](docs/jobs/release-stats.md) |
 | `session-ops-silence` | Discord alerts for a job that failed, or stopped running | [session-ops-silence](docs/jobs/session-ops-silence.md) |
 | `token-expiry` | Discord alerts 14 days, 7 days and 24 hours before a token expires | [token-expiry](docs/jobs/token-expiry.md) |
+| `store-versions` | Every 5 minutes: each store's Session release against GitHub's, posted as they diverge | [store-versions](docs/jobs/store-versions.md) |
 
 Run by hand, not scheduled: [`sogs-ban`](docs/tools/sogs-ban.md) for community bans, and
 [`sogs-perms`](src/session_ops/sogs/perms.py) for a room's per-account permissions.

deploy/README.md

@@ -58,6 +58,8 @@ Each `/etc/session-ops/<name>.env` has a commented `<name>.env.example` beside i
 installed from [`env/`](env/), saying what goes in it.
 `/etc/session-ops/expiry.toml` is not a secret: the expiry dates
 [token-expiry](../docs/jobs/token-expiry.md) cannot read from an API, from `expiry.toml.example`.
+`/etc/session-ops/store-versions.toml` is not a secret either, and has to be mode 644: who
+[store-versions](../docs/jobs/store-versions.md) tags, from `store-versions.toml.example`.
 
 ## Checking
 

deploy/env/store-versions.env.example

@@ -0,0 +1,2 @@
+# /etc/session-ops/store-versions.env: where store-versions posts, and its own failures.
+STORE_VERSIONS_DISCORD_WEBHOOK_URL=

deploy/env/store-versions.toml.example

@@ -0,0 +1,11 @@
+# /etc/session-ops/store-versions.toml: who each platform's warnings tag. Not a
+# secret: mode 644. Without it, warnings tag nobody.
+
+[tag]
+#android = [111111111111111111]
+#android-fdroid-merged = [111111111111111111]
+#android-fdroid-live = [111111111111111111]
+#ios = [222222222222222222]
+#desktop = [333333333333333333]
+#desktop-flathub-merged = [333333333333333333]
+#desktop-flathub-live = [333333333333333333]

deploy/install.sh

@@ -71,6 +71,7 @@ move "$ETC/env" "$ETC/alerts.env" 600 root
 for name in zendesk github-prs crowdin publish alerts; do
     [ -e "$ETC/$name.env" ] || install -m 600 /dev/null "$ETC/$name.env"
 done
+[ -e "$ETC/store-versions.env" ] || install -m 600 /dev/null "$ETC/store-versions.env"
 # What each file takes, commented; the env files stay empty until filled, since a job
 # is enabled once its env files have content.
 install -m 644 "$ROOT"/deploy/env/*.example "$ETC/"

docs/jobs/store-versions.md

@@ -0,0 +1,52 @@
+# Store Versions Against GitHub Releases
+
+Every 5 minutes, compares each store's latest Session release with the platform's
+highest GitHub release, and posts as they diverge and come back.
+
+| | |
+| --- | --- |
+| Runs | `session-ops@store-versions.timer`, every 5 minutes |
+| Secrets | `/etc/session-ops/store-versions.env`: `STORE_VERSIONS_DISCORD_WEBHOOK_URL` |
+| Tags | `/etc/session-ops/store-versions.toml`, from [`store-versions.toml.example`](../../deploy/env/store-versions.toml.example); without it nobody is tagged |
+| Dry run | `session-ops run store-versions --dry-run` prints every version and what it would post |
+| Logs | `journalctl -u session-ops@store-versions -n 50 --no-pager` |
+
+| Platform | GitHub | Store | Warns when the store is |
+| --- | --- | --- | --- |
+| `android` | `session-android` | Play listing, scraped: Play has no version API | ahead |
+| `android-fdroid-merged` | `session-android` | `index-v1.json` on session-fdroid's `master` | ahead or behind |
+| `android-fdroid-live` | `session-android` | `fdroid.getsession.org`'s `index-v1.json` | ahead |
+| `ios` | `session-ios` | iTunes lookup API | ahead |
+| `desktop` | `session-desktop` | `deb.session.foundation`, `sid`, package `session-desktop` | ahead or behind |
+| `desktop-flathub-merged` | `session-desktop` | `network.loki.Session.metainfo.xml` on flathub/network.loki.Session's `master` | ahead or behind |
+| `desktop-flathub-live` | `session-desktop` | Flathub's appstream API for `network.loki.Session` | ahead |
+
+A store ahead of GitHub is a public build whose source is not published. The channels
+warned on either way are ours: each GitHub release opens a pull request on
+session-fdroid and on flathub/network.loki.Session that has to be merged, and the APT
+repo is published with the release. A store shipping a published prerelease above
+that release is in sync with it.
+
+| Post | When |
+| --- | --- |
+| ⚠️ warning, tagging the platform's people | it diverges; on a platform's first check, if it already has |
+| ⚠️ again | every 12 hours while unresolved, and at once if the gap grows |
+| ✅ back in sync, or no longer ahead | after a warning only |
+| ℹ️ new GitHub release, store updated | either moves without a warning |
+| ⚠️ source failing, untagged | a source has failed for an hour, then every 12 hours, and ✅ once it reads again |
+
+A store is held at the highest version it has served: iTunes and the CDNs re-serve the
+previous one for hours after a release, so a store returning to it changes nothing.
+Once a store has served a lower version for 6 hours, that version counts, with nothing
+to edit by hand: a bad read clears itself, a warning out is updated to the new gap, and
+a rollback warns on the channels warned on either way.
+
+State is `/var/lib/session-ops/store-versions/state.json`. Losing it makes the next run
+a first check. A message Discord refuses is posted again on the next run, as is one
+still unposted 100 seconds into a run. Every request has a wall-clock limit, 60 s for a
+source and 20 s for a post, so a run and its failure alert end inside the 4-minute
+timeout. A version that does not start with a digit, or runs past 64 characters, fails
+its source. A failed run is reported once, then every 12 hours until one succeeds.
+
+GitHub is read unauthenticated: one request per repository a run, 36 an hour of the 60
+allowed. The organisation refuses fine-grained tokens that live longer than 366 days.

pyproject.toml

@@ -34,6 +34,7 @@ session-ops = "session_ops.ops.runner:main"
 session-ops-alert = "session_ops.monitor.alert:main"
 session-ops-silence = "session_ops.monitor.silence:main"
 session-ops-token-expiry = "session_ops.monitor.token_expiry:main"
+store-versions = "session_ops.store_versions.job:main"
 
 [dependency-groups]
 dev = [

src/session_ops/jobs.toml

@@ -9,6 +9,8 @@
 #   env            variables the run refuses to start without
 #   channel_env    where it posts, and where its own failures are reported; a job
 #                  without one reports to ALERT_DISCORD_WEBHOOK_URL
+#   repeat_alert_hours  report the same failure again only this often until a run
+#                  succeeds, for a schedule short enough to repeat it every few minutes
 #   unit           further [Service] lines, over the template's hardening
 #
 #   max_age_hours  for a queued job, its queue's: the weekend's 72 h (73 h across
@@ -100,6 +102,21 @@ schedule = "*-*-* 09:00 Australia/Melbourne"
 max_age_hours = 27
 timeout = "5min"
 
+[[job]]
+name = "store-versions"
+description = "Store releases against GitHub releases, posted as they diverge"
+entry = "session_ops.store_versions.job:main"
+args = ["--state", "{state}/state.json"]
+user = "sessionops"
+env_files = ["/etc/session-ops/store-versions.env"]
+env = ["STORE_VERSIONS_DISCORD_WEBHOOK_URL"]
+channel_env = "STORE_VERSIONS_DISCORD_WEBHOOK_URL"
+# Unauthenticated GitHub allows 60 requests an hour; this makes 36, one per repo a run.
+schedule = "*:0/5"
+max_age_hours = 1
+timeout = "4min"
+repeat_alert_hours = 12
+
 [[job]]
 name = "crowdin-sync"
 description = "Crowdin translations into the platform repos"

src/session_ops/ops/registry.py

@@ -27,6 +27,7 @@ class Job:
     max_age_hours: float = None
     channel_env: str = "ALERT_DISCORD_WEBHOOK_URL"
     timeout: str = "30min"
+    repeat_alert_hours: float = None
     unit: tuple = field(default=())
     queued: bool = False
     after: tuple = ()

src/session_ops/ops/runner.py

@@ -14,23 +14,33 @@
 (session-ops-alert) knows this failure was already reported and stays quiet. The
 backstop is for the failures a run cannot report itself: killed, timed out, or
 Discord unreachable.
+
+A job with `repeat_alert_hours` reports the same failure again only that often,
+until a run succeeds.
 """
 import argparse
 import importlib
+import json
 import os
 import re
 import shutil
 import socket
 import sys
 import tempfile
+import time
 import traceback
 from dataclasses import dataclass, field
 
 from session_ops.ops import registry
 from session_ops.shared import discord, http
+from session_ops.shared.state import write_json
 from session_ops.zendesk import claude_cli
 
 ALERTED_MARKER = "alerted"
+LAST_ALERT = "last-alert.json"
+# Discord answers in a second; past this the OnFailure backstop reports the run instead,
+# and a job with a short timeout still has room for its alert.
+ALERT_SECONDS = 45
 # Present on a host rehearsing production: jobs publish beside it (env.rehearsing).
 REHEARSAL_MARKER = "/etc/session-ops/rehearsal"
 
@@ -130,7 +140,8 @@ def post_alert(job, message):
               file=sys.stderr)
         return False
     payload = {"content": message, "allowed_mentions": {"parse": []}}
-    return discord.post_to_discord(http.Session(), webhook, [payload]) == 1
+    session = http.Session(deadline=ALERT_SECONDS)
+    return discord.post_to_discord(session, webhook, [payload]) == 1
 
 
 def mark_alerted(state_dir):
@@ -147,6 +158,35 @@ def mark_alerted(state_dir):
               file=sys.stderr)
 
 
+def repeated_alert(state_dir, message, hours, now):
+    """Whether `message` was reported less than `hours` ago; records it when not.
+
+    Recorded whether or not the post lands: the backstop reports what it cannot."""
+    path = os.path.join(state_dir, LAST_ALERT)
+    try:
+        with open(path, encoding="utf-8") as handle:
+            last = json.load(handle)
+        if last["message"] == message and now - last["at"] < hours * 3600:
+            return True
+    except (OSError, ValueError, KeyError, TypeError):
+        pass
+    try:
+        write_json(path, {"message": message, "at": now})
+    except OSError as exc:
+        print(f"Could not record the alert ({exc}); the next failure alerts again.",
+              file=sys.stderr)
+    return False
+
+
+def forget_alert(state_dir):
+    try:
+        os.remove(os.path.join(state_dir, LAST_ALERT))
+    except FileNotFoundError:
+        pass
+    except OSError as exc:
+        print(f"Could not clear {LAST_ALERT} ({exc}).", file=sys.stderr)
+
+
 def call_target(function, argv):
     """Run one entry point, returning None, or its failure as one sentence.
 
@@ -203,13 +243,20 @@ def run(job, dry_run=False, extra=()):
     if outcome.summary:
         print(outcome.summary)
     if not error and not outcome.failures():
+        if job.repeat_alert_hours and not dry_run:
+            forget_alert(state_dir)
         return 0
 
     message = scrub(alert_message(job, socket.gethostname(), _step, error, outcome))
     code = 1 if error or outcome.fatal() else 0
     if dry_run:
         print(message)
         return code
+    if (code and job.repeat_alert_hours
+            and repeated_alert(state_dir, message, job.repeat_alert_hours, time.time())):
+        print(f"{message}\nAlready reported within {job.repeat_alert_hours:g}h; not posted.")
+        mark_alerted(state_dir)
+        return code
     if not post_alert(job, message):
         # Even an optional target's failure then fails the unit: OnFailure= is all
         # that is left to report it.

src/session_ops/shared/github.py

@@ -1,5 +1,5 @@
-"""GitHub's REST API for the jobs that publish to the platform repos: who they
-authenticate as, and the pull requests they keep open.
+"""GitHub's REST API: the org's releases, and for the jobs that publish to the platform
+repos, who they authenticate as and the pull requests they keep open.
 
 Publishing authenticates as a GitHub App, with an installation token minted per run
 and scoped to the repositories the job names: it expires in an hour, and pull
@@ -23,6 +23,7 @@
 APP_KEY_PATH = "/etc/session-ops/github-app.pem"
 
 ORG = "session-foundation"
+RELEASES_PAGE = 20
 REHEARSAL_PREFIX = "rehearsal/"
 BOT, DIRECT = "bot", "direct"
 # Every branch the App may write, and how: a BOT branch is the job's own, rebuilt and
@@ -116,6 +117,16 @@ def check(resp, what):
     return resp.json() if resp.content else None
 
 
+def releases(api, repo, count=RELEASES_PAGE):
+    """[(tag, page URL, prerelease)] of `repo`'s `count` most recently created
+    releases, drafts left out."""
+    found = check(api.request("GET", f"{API}/repos/{repo}/releases",
+                              params={"per_page": count}),
+                  f"reading {repo}'s releases")
+    return [(release["tag_name"], release.get("html_url", ""), bool(release.get("prerelease")))
+            for release in found if not release.get("draft")]
+
+
 def app_jwt(app_id, private_key, now=None):
     now = int(now or time.time())
     # Backdated a minute for clock drift; GitHub refuses one living past ten minutes.

src/session_ops/shared/http.py

@@ -11,6 +11,7 @@
     session = http.Session(attempts=10, timeout=60, rate=20)
     session.get(url)                       # the session's budget
     session.request("GET", url, attempts=2)  # a shorter one for optional data
+    session.request("GET", url, deadline=60)  # and a wall-clock limit on all of it
 """
 import contextvars
 import email.utils
@@ -34,6 +35,10 @@
 _attempts = contextvars.ContextVar("attempts", default=None)
 
 
+class DeadlineExceeded(requests.Timeout):
+    """No complete answer within the call's deadline."""
+
+
 def retry_after_seconds(resp, default):
     """Seconds to wait per the response's rate-limit headers, else `default`.
 
@@ -171,25 +176,59 @@ class Session(requests.Session):
 
     `limiter` may be shared between sessions, so that per-thread sessions against one
     API still draw on one budget. It paces each call, not each retry inside it.
+
+    `deadline` bounds a whole call, retries, waits and the body included, in seconds.
+    `timeout` cannot: it applies to each socket read, so a server trickling a byte at
+    a time never trips it.
     """
 
     def __init__(self, attempts=DEFAULT_ATTEMPTS, timeout=DEFAULT_TIMEOUT, rate=None,
-                 limiter=None):
+                 limiter=None, deadline=None):
         super().__init__()
         self.timeout = timeout
+        self.deadline = deadline
         self.limiter = limiter or (TokenBucket(rate) if rate else None)
         adapter = _Adapter(max_retries=_retry(attempts))
         self.mount("https://", adapter)
         self.mount("http://", adapter)
 
-    def request(self, method, url, *args, attempts=None, **kwargs):
+    def request(self, method, url, *args, attempts=None, deadline=None, **kwargs):
         if attempts is not None and attempts <= 0:
             raise ValueError("attempts must be at least 1")
         kwargs.setdefault("timeout", self.timeout)
         if self.limiter:
             self.limiter.acquire()
+        deadline = self.deadline if deadline is None else deadline
+        if deadline is None:
+            return self._send(method, url, attempts, args, kwargs)
+        return within(deadline, lambda: self._send(method, url, attempts, args, kwargs))
+
+    def _send(self, method, url, attempts, args, kwargs):
         token = _attempts.set(attempts)
         try:
             return super().request(method, url, *args, **kwargs)
         finally:
             _attempts.reset(token)
+
+
+def within(seconds, call):
+    """`call()`'s result, or DeadlineExceeded once `seconds` pass.
+
+    A daemon thread runs it, since no socket option bounds a whole exchange; one left
+    behind dies with the process instead of holding its exit."""
+    outcome = {}
+
+    def run():
+        try:
+            outcome["result"] = call()
+        except BaseException as exc:  # re-raised in the caller's thread
+            outcome["error"] = exc
+
+    worker = threading.Thread(target=run, daemon=True)
+    worker.start()
+    worker.join(seconds)
+    if worker.is_alive():
+        raise DeadlineExceeded(f"no complete answer within {seconds:g} s")
+    if "error" in outcome:
+        raise outcome["error"]
+    return outcome["result"]

src/session_ops/shared/state.py

@@ -69,12 +69,16 @@ def save_state(path, state, records, retention_days, version, extra=None):
         if last >= cutoff:
             kept[key] = record
 
+    write_json(path, {"version": version, "updated_at": stamp, **(extra or {}), "seen": kept})
+    return len(kept), len(seen) - len(kept)
+
+
+def write_json(path, data):
+    """Write `data` to `path` atomically: a crash mid-write cannot corrupt the state."""
     directory = os.path.dirname(path)
     if directory:
         os.makedirs(directory, exist_ok=True)
     temporary = f"{path}.tmp"
     with open(temporary, "w", encoding="utf-8") as handle:
-        json.dump({"version": version, "updated_at": stamp, **(extra or {}), "seen": kept},
-                  handle, indent=2)
-    os.replace(temporary, path)  # atomic: a crash mid-write cannot corrupt the state
-    return len(kept), len(seen) - len(kept)
+        json.dump(data, handle, indent=2)
+    os.replace(temporary, path)