Skip to content

HDDS-14451. Recon does not work with SQLite - #11649

Draft
smengcl wants to merge 2 commits into
apache:masterfrom
smengcl:HDDS-14451
Draft

smengcl wants to merge 2 commits into
apache:masterfrom
smengcl:HDDS-14451

Conversation

@smengcl

@smengcl smengcl commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 5.5)

What changes were proposed in this pull request?

Recon is documented to support SQLite through ozone.recon.sql.db.driver and ozone.recon.sql.db.jdbc.url, but several Derby specific paths break it. This PR makes SQLite work out of the box. Derby stays the default; switching the default is left to HDDS-16668.

  • ReconControllerModule: when the JDBC URL contains ${ozone.recon.db.dir}, the whole URL was replaced with a hard coded Derby URL. Now only the placeholder is resolved (with the same ozone.metadata.dirs fallback as the other Recon DBs), so a SQLite URL is kept. The default URL resolves to the same path as before. A custom URL containing the placeholder (a different Derby file name, or a SQLite URL that silently ran on Derby) now opens the configured DB, which starts empty and is repopulated.
  • DefaultDataSourceProvider: pick the provider by the URL scheme (jdbc:derby: / jdbc:sqlite:) instead of a substring match, so a DB path containing "derby" does not select the wrong driver.
  • SqliteDataSourceProvider: enable WAL and a 60s busy timeout (same as Derby's default lock wait timeout). SQLite allows one writer at a time; with the default rollback journal a long reader, such as the unhealthy container CSV export, blocks all writers until it finishes. WAL lets readers and the writer proceed concurrently, and the busy timeout makes writers wait instead of failing with SQLITE_BUSY. A journal_mode or busy_timeout set in the URL is kept. SQLiteDataSource ignores a URL busy_timeout (its config always sets one), so that value is applied explicitly.
  • ReconTaskStatusUpdaterManager: the column check queried Derby's SYS.SYSCOLUMNS with an upper case column name, which never matched (Derby stores the quoted lower case name), so last_task_run_status and is_current_task_running were reset to 0 on every restart, on Derby too. It now uses SqlDbUtils.columnExists (JDBC metadata), which works on both DBs.
  • AbstractReconSqlDBTest: a custom provider passed to the constructor was overwritten by the Derby provider in init(), so TestReconWithDifferentSqlDBs never actually ran against SQLite. It now keeps the custom provider.

With these changes the full Recon unit test suite passes on SQLite except for test side expectations (JDBC type codes, timestamp precision), which are handled in HDDS-16668.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14451

How was this patch tested?

New tests:

  • TestReconWithDifferentSqlDBs#testTaskStatusPreservedOnReload (Derby and SQLite): task status columns survive a reload. Fails without the fix on both DBs.
  • TestReconWithDifferentSqlDBs#testSqliteUsesWalAndBusyTimeout: WAL and busy timeout are set, and journal_mode and busy_timeout in the URL are kept.
  • TestReconWithDifferentSqlDBs#testSqliteSelectedByUrlScheme: a DB dir containing "derby" still gets the SQLite data source.
  • TestReconUtils#testSqlDbJdbcUrlResolvesReconDbDirPlaceholder: the placeholder resolves to the ozone.metadata.dirs fallback.

Recon unit tests pass locally.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 20, 2026 23:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants