From b4fe8678c720a541b8ee679a29f6f64bad1d4519 Mon Sep 17 00:00:00 2001 From: francesco Date: Fri, 16 Oct 2026 10:15:16 +0200 Subject: [PATCH 1/4] docs(distributing): record progress on the 100-star plan Co-Authored-By: Claude Opus 5.5 --- DISTRIBUTING.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DISTRIBUTING.md b/DISTRIBUTING.md index 0b78ef7..c74d740 100644 --- a/DISTRIBUTING.md +++ b/DISTRIBUTING.md @@ -468,6 +468,8 @@ Il messaggio per questo pubblico è **offline, senza API a pagamento, su un port | 13 | Attivare Discussions, annunciarvi le release, chiedere ai primi utenti di seguire il repo (Watch) | noi | XS | 0–2 | trattiene i visitatori, non ne porta | | — | Homepage del repo verso la pagina delle release o di installazione | noi | XS | — | evita un link fuorviante nelle anteprime | +**Stato al 16 ottobre 2026.** Fatti: 11 (topic: tolti api-marketplace, certified-api, openapi, openclaw, ai, claude-skills; aggiunti gnome, libadwaita, karpathy, ollama, mcp-server, obsidian; limite di 20) e homepage (ora `openapi.github.io/gknowledge`). Il 7 è in [PR #41](https://github.com/openapi/gknowledge/pull/41): pacchetto separato `gknowledge-mcp` (solo core e server, senza GTK, con `--init`), `server.json`, `glama.json` e workflow `mcp.yml`. Dopo il merge servono il trusted publisher su PyPI e l'environment `pypi`; poi la PR ad awesome-mcp-servers. Da fare: 9, 10, 13. + **Sequenza.** Settimane 1–2: preparazione (7, 9, 10, 11, 13 e homepage da parte nostra; pin, outreach pronta, karma Reddit, 3–5 commenti HN veri e il video da parte dell'utente). Settimana 3: tutto il lancio in 48 ore (Show HN, social, LinkedIn e blog il primo giorno; r/LocalLLaMA e r/ObsidianMD il secondo; r/selfhosted il venerdì), obiettivo 30+ stelle in 24 ore. Settimane 4–5: v0.3 con una funzione dal nome riconoscibile, TWIG, Phoronix, r/gnome, r/linux. Settimane 6–8: risposte nei thread, Medium, thread Showcase di r/developersIndia dell'8 novembre, controllo delle PR aperte. **Stima.** Con tutto il piano, la somma dei valori centrali è circa **+80–110 stelle**. Senza Show HN scende a **+45–70**: le 100 in 8 settimane diventano improbabili senza HN o un giorno in trending. L'alternativa a puntare tutto su HN è la combinazione Reddit + 48 ore concentrate. From 3c6c802273e872f78cc2e9c2ded1de91393bb11d Mon Sep 17 00:00:00 2001 From: francesco Date: Fri, 16 Oct 2026 10:28:09 +0200 Subject: [PATCH 2/4] docs(distributing): gknowledge-mcp published on PyPI and the MCP registry Co-Authored-By: Claude Opus 5.5 --- DISTRIBUTING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/DISTRIBUTING.md b/DISTRIBUTING.md index c74d740..a8a888d 100644 --- a/DISTRIBUTING.md +++ b/DISTRIBUTING.md @@ -321,6 +321,7 @@ Il percorso seguente "ripercorre i passi" dei progetti studiati nell'ordine in c | [awesome-hacktoberfest-2026 #22](https://github.com/Piyushhbhutoria/awesome-hacktoberfest-2026/pull/22) | lista Hacktoberfest (62★), sezione Coding for Beginners | 15 ott 2026 | PR aperta; ultimo merge del maintainer a nov 2025 | attendere merge | | [ollama/ollama #19001](https://github.com/ollama/ollama/pull/19001) | README Community Integrations (183k★), sezione RAG & Knowledge Bases | 14 ott 2026 | PR aperta; 80+ PR README in coda, merge rari | attendere | | [awesome-context-engineering #88](https://github.com/yzfly/awesome-context-engineering/pull/88) | lista awesome (211★, molto attiva, merge quasi quotidiani), sezione Memory & Compression accanto a LWC e Dory | 16 ott 2026 | PR aperta (EN + CN). La #69 per openapi-llm-wiki era stata chiusa come "vendor-specific catalog": il maintainer vuole metodi o strumenti riutilizzabili, descritti per meccanismo; ringraziato sulla #69 con link alla #88 | attendere merge | +| [PyPI `gknowledge-mcp`](https://pypi.org/project/gknowledge-mcp/) + [registro MCP ufficiale](https://registry.modelcontextprotocol.io/v0/servers?search=gknowledge) | pacchetto e registro MCP (`io.github.openapi/gknowledge`) | 16 ott 2026 | **pubblicati** 0.2.2 dal workflow `mcp.yml` (trusted publishing + OIDC); `uvx gknowledge-mcp --init` verificato | PR ad awesome-mcp-servers; controllare Glama e PulseMCP | | PyCoder's Weekly | newsletter Python (form Google) | 14 ott 2026 | modulo compilato, **invio bloccato** dal classificatore auto mode | inviare a mano con i testi di `emerging.md` | | TabNews, r/developersIndia, Hashnode, Viblo, Techpoint Africa, organizzatori Hack Day | comunità e stampa in Africa e paesi emergenti | 14 ott 2026 | testi pronti in `emerging.md` | pubblicare dal proprio account; r/developersIndia solo nel thread Showcase dell'8 nov | @@ -468,7 +469,7 @@ Il messaggio per questo pubblico è **offline, senza API a pagamento, su un port | 13 | Attivare Discussions, annunciarvi le release, chiedere ai primi utenti di seguire il repo (Watch) | noi | XS | 0–2 | trattiene i visitatori, non ne porta | | — | Homepage del repo verso la pagina delle release o di installazione | noi | XS | — | evita un link fuorviante nelle anteprime | -**Stato al 16 ottobre 2026.** Fatti: 11 (topic: tolti api-marketplace, certified-api, openapi, openclaw, ai, claude-skills; aggiunti gnome, libadwaita, karpathy, ollama, mcp-server, obsidian; limite di 20) e homepage (ora `openapi.github.io/gknowledge`). Il 7 è in [PR #41](https://github.com/openapi/gknowledge/pull/41): pacchetto separato `gknowledge-mcp` (solo core e server, senza GTK, con `--init`), `server.json`, `glama.json` e workflow `mcp.yml`. Dopo il merge servono il trusted publisher su PyPI e l'environment `pypi`; poi la PR ad awesome-mcp-servers. Da fare: 9, 10, 13. +**Stato al 16 ottobre 2026.** Fatti: 11 (topic: tolti api-marketplace, certified-api, openapi, openclaw, ai, claude-skills; aggiunti gnome, libadwaita, karpathy, ollama, mcp-server, obsidian; limite di 20) e homepage (ora `openapi.github.io/gknowledge`). Il 7 è in [PR #41](https://github.com/openapi/gknowledge/pull/41) (merged; pacchetto e registro pubblicati il 16 ott): pacchetto separato `gknowledge-mcp` (solo core e server, senza GTK, con `--init`), `server.json`, `glama.json` e workflow `mcp.yml`. Dopo il merge servono il trusted publisher su PyPI e l'environment `pypi`; poi la PR ad awesome-mcp-servers. Da fare: 9, 10, 13. **Sequenza.** Settimane 1–2: preparazione (7, 9, 10, 11, 13 e homepage da parte nostra; pin, outreach pronta, karma Reddit, 3–5 commenti HN veri e il video da parte dell'utente). Settimana 3: tutto il lancio in 48 ore (Show HN, social, LinkedIn e blog il primo giorno; r/LocalLLaMA e r/ObsidianMD il secondo; r/selfhosted il venerdì), obiettivo 30+ stelle in 24 ore. Settimane 4–5: v0.3 con una funzione dal nome riconoscibile, TWIG, Phoronix, r/gnome, r/linux. Settimane 6–8: risposte nei thread, Medium, thread Showcase di r/developersIndia dell'8 novembre, controllo delle PR aperte. From 001a2ee3106f6918429885027afa109f68bdd6e4 Mon Sep 17 00:00:00 2001 From: francesco Date: Mon, 19 Oct 2026 12:35:21 +0200 Subject: [PATCH 3/4] feat: flag sources that changed since ingest; Flatpak dev loop and PDF support Generated pages rarely go wrong because the model invents something; they go wrong because the source moved on. Every source already stores the hash it was ingested from, so drift is a hash comparison: - core/lint.py: ChangeDetector reports ingested sources whose original file no longer matches, with the pages that cite them (hashes memoised by size and mtime). - IngestionManager.refresh_source(): take the new version, update hash and size, re-ingest (builds on reingest_source from #30). - UI: changed sources are marked "changed since ingest"; Re-ingest becomes "Update"; a toast on open and when the window regains focus. - MCP: stale_sources tool, the lint step of Karpathy's pattern. Flatpak: - make start-flatpak runs the dev loop inside org.gnome.Platform, whose GTK/libadwaita differ from the host's; documented in CONTRIBUTING. - Vendor pypdf 6.19.0: the 0.2.2 bundle cannot import PDFs. - README no longer says "Flatpak planned": the Pages remote installs. Suggested by a reader on dev.to (FromTheArchitect, comment 3i4d5). Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 8 ++ Makefile | 22 +++++- README.md | 4 +- TODO.md | 39 +++++++++- .../flatpak/io.gknowledge.Gknowledge.yaml | 11 +++ gknowledge/core/ingestion.py | 32 ++++++++ gknowledge/core/lint.py | 74 +++++++++++++++++++ gknowledge/mcp.py | 19 +++++ gknowledge/ui/window.py | 45 ++++++++++- tests/test_core.py | 38 +++++++++- tests/test_mcp.py | 19 +++++ 11 files changed, 300 insertions(+), 11 deletions(-) create mode 100644 gknowledge/core/lint.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5578fee..0f96130 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,6 +8,14 @@ Thanks for helping. The project is young, so small, focused contributions are th sudo apt install python3-gi gir1.2-gtk-4.0 gir1.2-adw-1 # Debian/Ubuntu make test # headless core tests, no GTK or network needed make run # run from the source tree +make start # dev loop on a demo project, restarts on every save +``` + +Flatpak users get the GTK and libadwaita of the GNOME runtime, not yours. Before submitting UI work, run the same dev loop inside that runtime (the version is read from the Flatpak manifest): + +```bash +flatpak install --user flathub org.gnome.Platform//51 # once +make start-flatpak ``` ## Ground rules diff --git a/Makefile b/Makefile index 788884c..5cc5fe2 100644 --- a/Makefile +++ b/Makefile @@ -18,13 +18,14 @@ METAINFODIR := $(DATADIR)/metainfo INSTALL := install -.PHONY: all help check test run dev start demo-svg install uninstall clean dist pypi flatpak +.PHONY: all help check test run dev start start-flatpak demo-svg install uninstall clean dist pypi flatpak all: help help: @echo "gknowledge — targets:" @echo " make start Dev loop: run on a demo project and auto-restart on source changes" + @echo " make start-flatpak Same loop, but inside the GNOME runtime Flatpak users get" @echo " make run Run the app from the source tree (opens your last project)" @echo " make test Run the headless core test suite" @echo " make check Byte-compile all sources (syntax check)" @@ -63,6 +64,25 @@ start: fi XDG_CONFIG_HOME=$(CURDIR)/$(SANDBOX)/config $(PYTHON) build-aux/dev/watch.py $(DEMO) +# The same loop inside the Flatpak runtime: the host GTK and libadwaita can +# differ from what Flatpak users run, so check UI work here before a release. +# The runtime version is read from the manifest to keep the two in sync. +RUNTIME := org.gnome.Platform//$(shell sed -n "s/^runtime-version: *'\(.*\)'/\1/p" build-aux/flatpak/$(APP_ID).yaml) + +start-flatpak: + @flatpak info --user $(RUNTIME) >/dev/null 2>&1 || flatpak info $(RUNTIME) >/dev/null 2>&1 || { \ + echo ">> Missing runtime. Install it with:"; \ + echo " flatpak install --user flathub $(RUNTIME)"; exit 1; } + @mkdir -p $(SANDBOX)/config + @if [ ! -f $(DEMO)/gknowledge.json ]; then \ + echo ">> Creating and ingesting demo project in $(DEMO)"; \ + $(PYTHON) build-aux/dev/seed_demo.py $(DEMO); \ + fi + flatpak run --share=network --share=ipc --socket=wayland --socket=fallback-x11 \ + --device=dri --own-name=$(APP_ID) --filesystem=$(CURDIR) --cwd=$(CURDIR) \ + --env=XDG_CONFIG_HOME=$(CURDIR)/$(SANDBOX)/config \ + --command=python3 $(RUNTIME) build-aux/dev/watch.py $(DEMO) + install: @echo ">> Installing gknowledge into $(DESTDIR)$(PREFIX)" $(INSTALL) -d $(DESTDIR)$(LIBDIR) diff --git a/README.md b/README.md index 23f16f9..2a41bc9 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Karpathy LLM Wiki pattern Sign in with ChatGPT Ollama - Flatpak planned + Flatpak PRs welcome

@@ -179,7 +179,7 @@ Nothing is sent anywhere unless you pick a provider that is not offline. | Native GNOME / GTK 4 | **yes** | Tauri | Tauri | Electron | Electron | web + Electron | | Works offline, no key | **yes** | needs a model | AI optional | agent-driven | local model | self-host | | Plain Markdown vault | **yes** | yes | yes | yes | yes | reads yours | -| Linux packaging | `.deb`, Flatpak planned | `.deb`, AppImage | AppImage, `.deb` | `.deb`, `.rpm` | AppImage | pip, Docker | +| Linux packaging | `.deb`, Flatpak | `.deb`, AppImage | AppImage, `.deb` | `.deb`, `.rpm` | AppImage | pip, Docker | | License | MIT | GPL-3.0 | AGPL-3.0 | GPL-3.0 | AGPL-3.0 | AGPL-3.0 | Details and history of each project are in [`DISTRIBUTING.md`](DISTRIBUTING.md). diff --git a/TODO.md b/TODO.md index 6cf6621..2bfdd18 100644 --- a/TODO.md +++ b/TODO.md @@ -2,6 +2,37 @@ Actionable backlog derived from [`ROADMAP.md`](ROADMAP.md). Ordered by what unblocks the first 100 stars. `[ ]` open, `[x]` done. +## Feedback from dev.to: Flatpak first, drift over hallucination (2026-10-17) + +Source: [comment by Justin (@fromthearchitect)](https://dev.to/fromthearchitect/comment/3i4d5) on the launch article. Data/AI architect on Fedora, writing a GNOME photo app in Rust; starred the repo and will try gknowledge "once a Flatpak is available". Follow-up is tracked in `DISTRIBUTING.md` §8.1 (dev.to row). + +Flatpak (the `.deb` and `install.sh` do nothing for Fedora, Arch or openSUSE users) +- [x] Flatpak build is green in CI (`appstreamcli compose` succeeds since the GNOME 51 bump); every release attaches a bundle and `repos.yml` publishes the remote at `openapi.github.io/gknowledge` (closes #24). +- [x] `flatpak install --user https://openapi.github.io/gknowledge/io.gknowledge.Gknowledge.flatpakref` works on Ubuntu (2026-10-19): pulls GNOME 51 from Flathub, installs 0.2.2, the app launches. +- [ ] Inside the Flatpak: create a project, import through the portal, ingest, reopen after a restart. +- [ ] Verify on a non-Ubuntu distro (Fedora VM or toolbox) before replying to Justin. +- [x] README: badge and comparison table no longer say "Flatpak planned" (install section left as is). +- [x] Vendor `pypdf` 6.19.0 in the manifest (the 0.2.2 bundle has `No module named 'pypdf'`, so PDF import fails). Needs a green CI build. +- [ ] Push a release so the published bundle includes the stale-source check, then tag the remote as tested. +- [ ] Make the CI and release Flatpak jobs blocking (drop `continue-on-error`) now that they pass. +- [ ] Flathub itself stays paused (AI-assisted code policy, see P0 below); say so plainly if asked, and point to the Pages remote. + +Develop against the runtime users get +- [x] `make start-flatpak`: the `make start` loop run inside `org.gnome.Platform` (version read from the manifest), so host GTK/libadwaita differences show up before release. +- [x] CONTRIBUTING documents `make start-flatpak`. +- [ ] With Justin's permission, link his GNOME SDK setup write-up from CONTRIBUTING. + +Drift, not hallucination +- [x] `core/lint.py`: `ChangeDetector` flags ingested sources whose original file no longer matches the ingested hash, with the pages that cite them. +- [x] `IngestionManager.refresh_source()`: copy the new version, update hash and size, re-ingest (builds on #30's `reingest_source`). +- [x] UI: changed sources show "changed since ingest" with a warning icon; the Re-ingest button becomes "Update"; a toast on project open and when the window regains focus. +- [x] MCP tool `stale_sources` so agents can run the lint step of Karpathy's pattern. +- [ ] Open a GitHub issue describing the design, credit the suggestion, invite Justin's review. +- [ ] Mark affected pages as possibly outdated in the Knowledge pane (badge on the page, note in the page header). +- [ ] Trigger the check from `Gio.FileMonitor` on origin files (#3) instead of only on open and focus. +- [ ] Sources imported through the document portal: confirm `origin_path` stays readable across sessions inside the Flatpak. +- [ ] Emit drift as review items once the Review queue is split (#17). + ## UX review with a real ChatGPT plan (2026-10-13) Test: 6 short AI-paper notes (about 300 words each), provider "ChatGPT plan", model picked automatically (gpt-6.1-sol). @@ -58,7 +89,7 @@ Speed and feedback - [ ] Decide the app ID (`io.github..Gknowledge` or a verified domain). `io.gknowledge.*` needs a domain we control. - [x] Bump the runtime to GNOME 51 (current stable on Flathub). Not yet built in CI. - [x] Replace `--filesystem=home` with FileChooser / Documents portals; keep only `--share=network`. Done in the manifest, untested in a real Flatpak build. -- [ ] Build from `git`/`archive` sources instead of `type: dir`; vendor `pypdf`. +- [ ] Build from `git`/`archive` sources instead of `type: dir` (`pypdf` vendored). - [ ] Complete MetaInfo: screenshots (URLs still missing; developer, urls, branding added), `developer`, `url`, branding colors, release notes, summary of 35 characters or fewer. - [ ] Make `flatpak-builder-lint` pass with no exceptions. - [ ] Add a `.doap` file. (CI workflows for tests, metadata validation and Flatpak build exist; Flatpak job is non-blocking until proven.) @@ -92,7 +123,7 @@ Speed and feedback - [ ] "Insights" panel: weakly connected pages, hubs, suggested missing topics. ### Lint -- [ ] `core/lint.py`: orphan pages, duplicate titles, missing sources, stale pages, broken `[[links]]`. +- [ ] `core/lint.py`: orphan pages, duplicate titles, missing sources, broken `[[links]]` (stale sources done, see the dev.to feedback section). - [ ] Optional LLM contradiction check. - [ ] Emit lint findings as review items. - [ ] Append every lint run to `knowledge/log.md` using the parseable `## [date] op | title` prefix. @@ -165,10 +196,10 @@ Tasks: - [x] `build-aux/arch/PKGBUILD` ready for AUR (needs a tag and `updpkgsums`). - [x] `build-aux/snap/snapcraft.yaml` draft (untested). - [x] First release `v0.1.0` (2026-10-13) with `.deb`, version-independent `gknowledge_all.deb` and `install.sh` (`curl … | bash`). -- [ ] Fix the Flatpak build: `appstreamcli compose` fails with `file-read-error` on the MetaInfo (the CI job hides it with `continue-on-error`). +- [x] Fix the Flatpak build: `appstreamcli compose` succeeds in CI since 2026-10-19 (run 39223882741). - [ ] Test `sudo apt install ./gknowledge_all.deb` on a clean Ubuntu 24.04 VM. - [ ] Publish a signed APT repository on GitHub Pages (reprepro or aptly) and document `curl | gpg --dearmor` setup. -- [ ] Publish a Flatpak OSTree repo on GitHub Pages with a `.flatpakrepo` file; document `flatpak remote-add`. +- [x] Publish a Flatpak OSTree repo on GitHub Pages with a `.flatpakrepo` file; document `flatpak remote-add` (README documents the `.flatpakref` install). - [ ] Register `gknowledge` on the Snap Store and test the snap on Ubuntu 24.04; decide on `home` plug vs portals. - [ ] Create a Launchpad account and PPA; upload a source package (`debian/` with debhelper). - [ ] Submit AUR package. diff --git a/build-aux/flatpak/io.gknowledge.Gknowledge.yaml b/build-aux/flatpak/io.gknowledge.Gknowledge.yaml index 22e8378..a63d789 100644 --- a/build-aux/flatpak/io.gknowledge.Gknowledge.yaml +++ b/build-aux/flatpak/io.gknowledge.Gknowledge.yaml @@ -15,6 +15,17 @@ finish-args: # which goes through the document portal inside the sandbox. modules: + # PDF import; pure Python, installed from the wheel without network access. + - name: python3-pypdf + buildsystem: simple + build-commands: + - pip3 install --no-index --no-deps --no-build-isolation --find-links="file://${PWD}" + --prefix=${FLATPAK_DEST} pypdf + sources: + - type: file + url: https://files.pythonhosted.org/packages/3c/2c/c43c03eaf630435f023f1dc61ec4a4a78951ad5530a62c71cc89bde307b7/pypdf-6.19.0-py3-none-any.whl + sha256: 7e5d6e730e7dae87d560a2cee218b852f6498c8be61966f3cd02ead971e48d14 + - name: gknowledge buildsystem: simple build-commands: diff --git a/gknowledge/core/ingestion.py b/gknowledge/core/ingestion.py index 5443ce2..3ea3df4 100644 --- a/gknowledge/core/ingestion.py +++ b/gknowledge/core/ingestion.py @@ -23,6 +23,7 @@ from .extraction import ExtractionPipeline from .hashing import hash_file from .knowledge import KnowledgeWriter +from .lint import origin_of from .models import Job, JobState, PageKind, Source, SourceStatus, new_id, now from .project import Project @@ -126,6 +127,37 @@ def reingest_source(self, source_id: str) -> Source | None: writer.append_log("re-ingest", source.title) return source + def refresh_source(self, source_id: str) -> Source | None: + """Take the current content of a source's original file and re-ingest it. + + Used when lint reports the original changed after ingestion: the stored + copy is replaced, the hash updated, and the old knowledge detached. + """ + if self.is_running(): + raise RuntimeError("Cannot refresh a source while ingestion is running.") + source = self.store.get_source(source_id) + if source is None: + return None + origin = origin_of(source) + if origin is None: + raise FileNotFoundError(f"Original file is missing: {source.origin_path}") + digest = hash_file(origin) + other = self.store.source_by_hash(digest) + if other and other.id != source.id: + raise ValueError(f"Same content is already imported as “{other.title}”.") + + old_path = self._managed_path(self.project.sources_dir, source.rel_path) + dest = self.project.sources_dir / f"{origin.stem}-{digest[:8]}{origin.suffix}" + shutil.copy2(origin, dest) + if old_path is not None and old_path.is_file() and old_path != dest.resolve(): + old_path.unlink() + source.rel_path = dest.name + source.sha256 = digest + source.size = dest.stat().st_size + self.store.upsert_source(source) + self.store.log("source.refresh", source.title) + return self.reingest_source(source.id) + def remove_source(self, source_id: str) -> Source | None: """Remove the source and clean up knowledge no longer backed by it.""" if self.is_running(): diff --git a/gknowledge/core/lint.py b/gknowledge/core/lint.py new file mode 100644 index 0000000..f12b47a --- /dev/null +++ b/gknowledge/core/lint.py @@ -0,0 +1,74 @@ +"""Wiki lint: find knowledge that no longer matches its sources. + +A generated wiki rarely goes wrong because the model invented something; it +goes wrong because the source moved on and the page did not. Every source +keeps the SHA-256 of the file it was ingested from and every page keeps the +ids of the sources behind it, so drift is a hash comparison, not a guess. + +Only the stale-source check lives here for now; orphan pages, broken +``[[links]]`` and duplicate titles belong in this module too. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass, field +from pathlib import Path + +from .hashing import hash_file +from .models import Page, Source, SourceStatus +from .project import Project + + +@dataclass +class StaleSource: + source: Source + pages: list[Page] = field(default_factory=list) # pages that cite it + + +class ChangeDetector: + """Compare each source's original file with the content that was ingested. + + Hashes are memoised by (path, size, mtime), so calling :meth:`stale_sources` + on every refresh only re-reads files that were touched since the last call. + """ + + def __init__(self, project: Project): + self.project = project + self._memo: dict[tuple[str, int, int], str] = {} + + def current_hash(self, source: Source) -> str | None: + """Hash of the original file now, or None if it is gone or unreadable.""" + if not source.origin_path: + return None + try: + st = os.stat(source.origin_path) + except OSError: + return None + key = (source.origin_path, st.st_size, st.st_mtime_ns) + if key not in self._memo: + try: + self._memo[key] = hash_file(source.origin_path) + except OSError: + return None + return self._memo[key] + + def is_stale(self, source: Source) -> bool: + if source.status != SourceStatus.INGESTED: + return False # nothing generated from it yet + digest = self.current_hash(source) + return digest is not None and digest != source.sha256 + + def stale_sources(self) -> list[StaleSource]: + store = self.project.store + return [StaleSource(s, store.pages_for_source(s.id)) + for s in store.list_sources() if self.is_stale(s)] + + +def stale_sources(project: Project) -> list[StaleSource]: + return ChangeDetector(project).stale_sources() + + +def origin_of(source: Source) -> Path | None: + path = Path(source.origin_path) if source.origin_path else None + return path if path and path.is_file() else None diff --git a/gknowledge/mcp.py b/gknowledge/mcp.py index fbfaccd..a5ba946 100644 --- a/gknowledge/mcp.py +++ b/gknowledge/mcp.py @@ -21,6 +21,7 @@ from pathlib import Path from . import __version__ +from .core import lint from .core.ingestion import IngestionManager from .core.knowledge import KnowledgeWriter from .core.project import Project @@ -73,6 +74,13 @@ "ingest": {"type": "boolean", "default": True}}, "required": ["path"]}, }, + { + "name": "stale_sources", + "description": "Lint the wiki for drift: sources whose original file changed " + "after ingestion, with the pages that still reflect the old " + "version. Re-import or refresh them in the app.", + "inputSchema": {"type": "object", "properties": {}}, + }, { "name": "recent_activity", "description": "Recent operations from the wiki log (ingests, saved answers).", @@ -143,6 +151,17 @@ def add_source(self, path: str, ingest: bool = True) -> str: if ev in ("done", "warning", "error") else None) return "\n".join(notes) or "Imported." + def stale_sources(self) -> str: + stale = lint.stale_sources(self.project) + if not stale: + return "No drift: every ingested source matches its original file." + out = [] + for s in stale: + out.append(f"- {s.source.title} ({s.source.origin_path}) changed since ingest; " + f"{len(s.pages)} page(s) may be outdated:") + out += [f" - {p.slug}" for p in s.pages] + return "\n".join(out) + def recent_activity(self, limit: int = 20) -> str: rows = self.project.store.recent_log(int(limit)) return "\n".join(f"- {r['op']}: {r['detail']}" for r in rows) or "No activity yet." diff --git a/gknowledge/ui/window.py b/gknowledge/ui/window.py index 8460d6b..56c84e9 100644 --- a/gknowledge/ui/window.py +++ b/gknowledge/ui/window.py @@ -13,6 +13,7 @@ from .. import APP_NAME, __version__ from ..core.ingestion import IngestionManager from ..core.knowledge import KnowledgeWriter +from ..core.lint import ChangeDetector from ..core.models import JobState, PageKind, Source from ..core.project import Project from ..core.query import QueryEngine @@ -78,6 +79,10 @@ def __init__(self, app: Adw.Application): self.app = app self.project: Project | None = None self.ingestion: IngestionManager | None = None + self.changes: ChangeDetector | None = None + self._stale_ids: set[str] = set() + # Sources are often edited in another app: look for drift on return. + self.connect("notify::is-active", self._on_active_changed) self._selected_source_id: str | None = None self.provider = None @@ -283,6 +288,7 @@ def set_project(self, project: Project): self.project = project self.provider = build_provider(project.settings.as_dict()) self.ingestion = IngestionManager(project, self.provider) + self.changes = ChangeDetector(project) for stack in (self.side_stack, self.view_stack): while (child := stack.get_first_child()) is not None: @@ -308,6 +314,7 @@ def set_project(self, project: Project): self.refresh_all() self.view_stack.set_visible_child_name("sources") self._sync_sidebar() + self._announce_stale() def on_settings_saved(self, settings): if self.project: @@ -377,13 +384,17 @@ def _refresh_sources(self): self._update_source_action_buttons() return sources = self.project.store.list_sources() + self._stale_ids = self._find_stale() selected_row = None if not sources: self.sources_list.append(_placeholder_row("No sources yet. Import some.")) for s in sources: + stale = s.id in self._stale_ids row = Adw.ActionRow(title=GLib.markup_escape_text(s.title), - subtitle=s.status.value) - row.add_prefix(Gtk.Image.new_from_icon_name("text-x-generic-symbolic")) + subtitle="changed since ingest" if stale + else s.status.value) + row.add_prefix(Gtk.Image.new_from_icon_name( + "dialog-warning-symbolic" if stale else "text-x-generic-symbolic")) row.source = s self.sources_list.append(row) if s.id == self._selected_source_id: @@ -423,6 +434,11 @@ def _update_source_action_buttons(self): ) if hasattr(self, "reingest_btn"): self.reingest_btn.set_sensitive(available) + stale = self._selected_source_id in self._stale_ids + self.reingest_btn.set_label("Update" if stale else "Re-ingest") + self.reingest_btn.set_tooltip_text( + "The original file changed: ingest the new version" if stale + else "Extract this source again") if hasattr(self, "remove_source_btn"): self.remove_source_btn.set_sensitive(available) @@ -677,8 +693,12 @@ def _do_import_folder(self, folder): def _on_reingest_source(self, *_a): if not (self.project and self.ingestion and self._selected_source_id): return + source_id = self._selected_source_id try: - source = self.ingestion.reingest_source(self._selected_source_id) + if source_id in self._stale_ids: + source = self.ingestion.refresh_source(source_id) + else: + source = self.ingestion.reingest_source(source_id) except (OSError, RuntimeError, ValueError) as exc: self.toast(f"Could not re-ingest source: {exc}") return @@ -765,6 +785,25 @@ def done(): self.ingestion.start(progress=progress, on_finish=finished) self._update_source_action_buttons() + # -- drift --------------------------------------------------------------- + def _find_stale(self) -> set[str]: + if not self.changes: + return set() + return {s.source.id for s in self.changes.stale_sources()} + + def _announce_stale(self): + n = len(self._stale_ids) + if n: + self.toast(f"{n} source(s) changed since they were ingested") + + def _on_active_changed(self, *_a): + if not (self.is_active() and self.project and self.ingestion + and not self.ingestion.is_running()): + return + if self._find_stale() != self._stale_ids: + self._refresh_sources() + self._announce_stale() + # -- helpers ------------------------------------------------------------- def _warn(self, text: str): self.banner.set_title(GLib.markup_escape_text(text[:240])) diff --git a/tests/test_core.py b/tests/test_core.py index 3c7c8f2..d6bc2c4 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -10,6 +10,7 @@ from gknowledge.core.ingestion import IngestionManager from gknowledge.core.knowledge import KnowledgeWriter +from gknowledge.core.lint import ChangeDetector, stale_sources from gknowledge.core.models import Page, PageKind, Source, SourceStatus from gknowledge.core.project import Project, Settings from gknowledge.core.query import QueryEngine @@ -262,6 +263,40 @@ def test_import_and_ingest_epub(tmp_path): proj.close() +def test_lint_flags_and_refreshes_sources_whose_original_changed(tmp_path): + proj = make_project(tmp_path) + src_file = tmp_path / "solar.md" + src_file.write_text(SAMPLE, "utf-8") + mgr = IngestionManager(proj, build_provider(proj.settings.as_dict())) + source = mgr.import_file(src_file) + assert source is not None + assert stale_sources(proj) == [] # not ingested yet: nothing to drift + assert mgr.process_pending() == 1 + assert stale_sources(proj) == [] + + src_file.write_text(SAMPLE + "\nGeothermal plants run around the clock.\n", "utf-8") + stale = stale_sources(proj) + assert [s.source.id for s in stale] == [source.id] + assert any(p.kind == PageKind.SOURCE for p in stale[0].pages) + + old_rel = proj.store.get_source(source.id).rel_path + refreshed = mgr.refresh_source(source.id) + assert refreshed is not None and refreshed.status == SourceStatus.PENDING + assert refreshed.rel_path != old_rel + assert not (proj.sources_dir / old_rel).exists() + assert mgr.process_pending() == 1 + assert stale_sources(proj) == [] + current = proj.store.get_source(source.id) + assert "Geothermal" in (proj.sources_dir / current.rel_path).read_text("utf-8") + assert current.status == SourceStatus.INGESTED + + # A deleted original is not drift: the ingested copy is still the truth. + detector = ChangeDetector(proj) + src_file.unlink() + assert detector.stale_sources() == [] + proj.close() + + if __name__ == "__main__": # allow running without pytest import tempfile @@ -271,7 +306,8 @@ def test_import_and_ingest_epub(tmp_path): ("c", test_query_returns_citations), ("d", test_import_and_ingest_epub), ("e", test_reingest_clears_stale_pages_and_rebuilds_source), - ("f", test_remove_source_detaches_shared_pages_and_deletes_orphans)): + ("f", test_remove_source_detaches_shared_pages_and_deletes_orphans), + ("g", test_lint_flags_and_refreshes_sources_whose_original_changed)): d = tmp / name d.mkdir(parents=True, exist_ok=True) fn(d) diff --git a/tests/test_mcp.py b/tests/test_mcp.py index 59b1cfa..9b9d7dc 100644 --- a/tests/test_mcp.py +++ b/tests/test_mcp.py @@ -83,3 +83,22 @@ def test_mcp_without_init_refuses_missing_project(tmp_path): input="", capture_output=True, text=True, cwd=ROOT, timeout=60) assert proc.returncode == 2 assert not (tmp_path / "nope").exists() + + +def test_mcp_stale_sources_reports_drift(tmp_path): + proj = Project.create(tmp_path / "wiki", name="Drift") + proj.save_settings(Settings(provider="offline")) + note = tmp_path / "note.md" + note.write_text("# Mixture of Experts\n\nMixture of Experts models route each " + "token to a few expert networks.\n") + mgr = IngestionManager(proj) + mgr.import_file(note) + mgr.process_pending() + proj.close() + call = {"jsonrpc": "2.0", "id": 1, "method": "tools/call", + "params": {"name": "stale_sources", "arguments": {}}} + assert "No drift" in _session(tmp_path / "wiki", [call])[0]["result"]["content"][0]["text"] + + note.write_text(note.read_text() + "\nSwitch Transformers scaled them in 2021.\n") + text = _session(tmp_path / "wiki", [call])[0]["result"]["content"][0]["text"] + assert "Mixture of Experts" in text and "changed since ingest" in text From 4d3b1901266c0821e6cdb5180671e7d1284b8f33 Mon Sep 17 00:00:00 2001 From: francesco Date: Mon, 19 Oct 2026 12:37:55 +0200 Subject: [PATCH 4/4] docs(todo): record CI result for the vendored pypdf Co-Authored-By: Claude Opus 5.5 --- TODO.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index 2bfdd18..158bb77 100644 --- a/TODO.md +++ b/TODO.md @@ -12,7 +12,7 @@ Flatpak (the `.deb` and `install.sh` do nothing for Fedora, Arch or openSUSE use - [ ] Inside the Flatpak: create a project, import through the portal, ingest, reopen after a restart. - [ ] Verify on a non-Ubuntu distro (Fedora VM or toolbox) before replying to Justin. - [x] README: badge and comparison table no longer say "Flatpak planned" (install section left as is). -- [x] Vendor `pypdf` 6.19.0 in the manifest (the 0.2.2 bundle has `No module named 'pypdf'`, so PDF import fails). Needs a green CI build. +- [x] Vendor `pypdf` 6.19.0 in the manifest (the 0.2.2 bundle has `No module named 'pypdf'`, so PDF import fails). Builds green on PR #42. - [ ] Push a release so the published bundle includes the stale-source check, then tag the remote as tested. - [ ] Make the CI and release Flatpak jobs blocking (drop `continue-on-error`) now that they pass. - [ ] Flathub itself stays paused (AI-assisted code policy, see P0 below); say so plainly if asked, and point to the Pages remote. @@ -27,7 +27,7 @@ Drift, not hallucination - [x] `IngestionManager.refresh_source()`: copy the new version, update hash and size, re-ingest (builds on #30's `reingest_source`). - [x] UI: changed sources show "changed since ingest" with a warning icon; the Re-ingest button becomes "Update"; a toast on project open and when the window regains focus. - [x] MCP tool `stale_sources` so agents can run the lint step of Karpathy's pattern. -- [ ] Open a GitHub issue describing the design, credit the suggestion, invite Justin's review. +- [x] Design written up in PR #42, crediting the suggestion; invite Justin to review it when replying. - [ ] Mark affected pages as possibly outdated in the Knowledge pane (badge on the page, note in the page header). - [ ] Trigger the check from `Gio.FileMonitor` on origin files (#3) instead of only on open and focus. - [ ] Sources imported through the document portal: confirm `origin_path` stays readable across sessions inside the Flatpak.