Repository navigation
Conversation
Track datanode startup in phases (Initializing, Waiting for SCM, Preparing volumes, Loading containers, Checking volumes, Starting container services) with the elapsed time of each phase. Beginning or ending a phase ends all earlier phases. If container services fail to start, running phases are marked as failed. Container loading is tracked per volume: the container directories are counted up front to get the total, which is then used to estimate the remaining time. Volumes that failed to load are shown as failed. The remaining time is reported as unknown when none of the steps of a phase are running. The progress is exposed through the Hadoop:service=HddsDatanode,name=StartupProgress MXBean and shown on a new "Startup Progress" page of the datanode web UI, which refreshes until startup is complete. The tracker (StartupProgress) lives in hdds-server-framework so that it can be reused for OM and SCM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@spacemonkd please check it out and share your thoughts |
ptlrs
left a comment
ptlrs
left a comment
There was a problem hiding this comment.
Thanks for the PR @smengcl.
It would be helpful if we can get the status of all DNs in one place.
For example, the SCM page shows the status of all DNs today. Perhaps it could be updated to also show the new phase and ETA data?
| public void beginPhase(P phase) { | ||
| for (Map.Entry<P, Phase> entry : phases.entrySet()) { | ||
| if (entry.getKey() == phase) { | ||
| entry.getValue().begin(); | ||
| return; | ||
| } | ||
| entry.getValue().end(); | ||
| } | ||
| } |
There was a problem hiding this comment.
Perhaps we should disallow beginning a phase which has already ended
There was a problem hiding this comment.
Throwing here would break SCM HA. VersionEndpointTask runs once per SCM, so the 2nd and 3rd SCM call beginPhase(PREPARING_VOLUMES) after the phase has ended, and an exception there shuts down the datanode.
I'd keep it as a no-op. Updated javadoc and added a test.
|
Thanks @ptlrs for the review.
That would be work for a separate PR. This one is scoped for individual datanodes and is a big enough diff of itself. |
… test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-by: Claude Code (Opus 5.5)
What changes were proposed in this pull request?
A datanode with many containers can take a long time to start. Until then, operators can only follow the logs to tell which phase it is in and how much longer it will take.
This PR adds a "Startup Progress" page to the Datanode web UI.
StartupProgress(hdds-server-framework)A small tracker that can be reused for OM and SCM later.
PENDING,RUNNING,COMPLETEorFAILED.StartupProgressMXBean, so the data is also available via/jmx.Datanode
Phases (
DatanodeStartupPhase):VersionEndpointTask#checkVolumeSet: volume format and DB load)Other changes:
ContainerReadercounts the container directories of the volume withFile#list()up front to get the total, then loads containers with the existing loop. A volume that fails to load is shown as aFAILEDstep.FAILED.Hadoop:service=HddsDatanode,name=StartupProgress.extraNavLinksof the navmenu. The page shows each phase with its per volume steps (status, elapsed time, done / total, estimated remaining time), and refreshes every 2 seconds until startup completes or fails.The Datanode HTTP server already starts before the slow phases, so their progress is visible while the datanode starts. OM and SCM start their HTTP server near the end of startup, so they are left for a follow-up.
Screenshots
Rendered from this branch's UI code with mocked
/jmxdata.Loading containers, with one failed volume:
Startup complete:
Starting container services failed:
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17107
How was this patch tested?
TestStartupProgress: phase transitions, remaining time estimate for concurrent steps, failure marking, and a JMX round trip that checks the attribute names the UI reads.TestContainerReader#testContainerReaderProgress.TestRunningDatanodeStatenow checks that a failed container start marks startup as failed, and a successful one does not./jmxresponses (screenshots above).🤖 Generated with Claude Code