Repository navigation
Conversation
WolfLSZoom stops after maxLSIts evaluations, failed ones included; a search that cannot succeed used to go on until the bracket collapsed to adjacent doubles. A finite cubic trial step within 1% of a bracket end, or outside the bracket, is moved to 1% inside instead of replaced by bisection, which shrinks the bracket 100-fold rather than 2-fold when the minimiser is close to that end; bisection remains on every fifth zoom iteration, for a non-finite trial, and for a trial at an end of a bracket under about 50 ulps wide, where the margin rounds away. WolfeLineSearch refuses an ascent or NaN direction at once, so that the caller resets to the gradient without first spending evaluations.
Jenkins Console Log Machine informationDistributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 52 bits physical, 57 bits virtual CPU(s): 192 On-line CPU(s) list: 0-191 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 25 Model: 17 Model name: AMD EPYC 9474F 48-Core Processor Stepping: 1 Frequency boost: enabled CPU MHz: 1496.248 CPU max MHz: 4114.4229 CPU min MHz: 1500.0000 BogoMIPS: 7189.39 Virtualization: AMD-V L1d cache: 3 MiB L1i cache: 3 MiB L2 cache: 96 MiB L3 cache: 512 MiB NUMA node0 CPU(s): 0-47,96-143 NUMA node1 CPU(s): 48-95,144-191 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin |
|
Looks good. Claude assisted improvement The clamp (change 2) and the ascent refusal (change 3) help, but the zoom limit (change 1) turns successful searches into failures. I ran I tried limits of 40, 60 and 100 evaluations. Each limit that brings your jump case below develop's 55 evaluations also turns successful kink searches into failures, so I found no limit on all evaluations that works. The clamp also has a cost: with the minimizer near the other end of the bracket, a trial kept 1 % from one end removes only 1 % of the bracket, and the kink searches took more evaluations than on develop. I pushed one commit to your branch with two changes. The zoom limit counts only failed evaluations (the new parameter is
Quadratics keep your gain (8.4 to 4.1 evaluations per search, as with this PR), as does your stiff first step (23 to 6 evaluations). The stiff exponential and quartic families and a family whose evaluations fail beyond a point are unchanged in all three versions. The cost is your jump case: it takes 55 evaluations again, as on develop. That search fails in either case, and I think saving its 32 evaluations is worth less than the kink searches that the limit makes fail. The commit adds three tests to |
Jenkins Console Log Machine informationDistributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 43 bits physical, 48 bits virtual CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 23 Model: 49 Model name: AMD EPYC 7742 64-Core Processor Stepping: 0 Frequency boost: enabled CPU MHz: 1495.818 CPU max MHz: 3416.0681 CPU min MHz: 1500.0000 BogoMIPS: 4491.56 Virtualization: AMD-V L1d cache: 4 MiB L1i cache: 4 MiB L2 cache: 64 MiB L3 cache: 512 MiB NUMA node0 CPU(s): 0-63,128-191 NUMA node1 CPU(s): 64-127,192-255 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin |
This touches same code as #3448 and should preferably go after it.
AI use disclosure: This was developed with the help of Claude Code (Opus 5.5).
Submission Checklist
./runTests.py src/test/unitmake cpplintSummary
Three changes to
src/stan/optimization/bfgs_linesearch.hppthat cut the number of log-density gradient evaluations a line search spends when it cannot succeed or starts badly:WolfLSZoomstops aftermaxLSItsevaluations, failed ones included (new trailing parametermax_evals, defaultWolfeLineSearchpassesmaxLSIts, 20 in the services). Before, a zoom that could not succeed ran until the bracket collapsed to adjacent doubles;wolfeLineSearch_jump_terminates(Infinite loop inWolfLSZoomwhen using stan for optimization #3229) spends 55 evaluations.WolfeLineSearchreturns 1 before evaluating anything, andBFGSMinimizer::stepresets toThis PR and the PR of branch
feature/optimizer-first-step-capboth change the last argument of the twoWolfLSZoomcalls inWolfeLineSearch(1e-16, maxLSItshere,minRangethere); whichever is merged second resolves it tominRange, maxLSIts.Intended Effect
Fewer evaluations in failing or badly started line searches: on the three cases of the appendix, with the same return codes, 55 → 23, 56 → 1 and 23 → 6 evaluations. In whole runs, against develop on 20 model configurations (GLMs, a hierarchical logistic regression, embedded-Laplace GLMMs and GPs; 30–100 seeds each): at CmdStan's defaults, where few searches fail, the counts change little: 78 of 1650 L-BFGS and 57 of 900 dense BFGS runs differ, every configuration within 5% of develop (geometric mean 1.00 for both). With
tol_rel_obj = tol_rel_grad = 1, where about 30% of the L-BFGS runs end in a failing search at the precision limit, L-BFGS needs 0.84 times develop's gradient evaluations (0.66–1.00 per configuration) and dense BFGS 0.87, with as many runs at the best log density as develop (L-BFGS 796, dense BFGS 795 of 900).How to Verify
OptimizationBfgsLinesearch.WolfLSZoom_max_evals_counts_failures(src/test/unit/optimization/bfgs_linesearch_test.cpp): every evaluation fails, and the zoom gives up aftermax_evals= 5 of them. It tests the new parameter, so it does not compile on develop.bfgs_linesearch_test.cpp(includingwolfeLineSearch_jump_terminates,WolfLSZoom_failing_evaluations,WolfLSZoom_adjacent_bracket),bfgs_minimizer_test.cpp,bfgs_test.cpp,bfgs_exponential_boundary_test.cpp,services/optimize/{bfgs,lbfgs,bfgs_jacobian,lbfgs_jacobian}_test.cpp, and the Pathfinder testsservices/pathfinder/{eight_schools,normal_glm}_test.cpp.Side Effects
Measured with an in-process A/B harness against develop (gradient evaluation counts, platform independent); I can share the harness. Iterates of L-BFGS, BFGS and Pathfinder change where a zoom trial was near a bracket end or a zoom reached
maxLSItsevaluations: 88 of 1650 L-BFGS runs at CmdStan's defaults. Withtol_rel_obj = tol_rel_grad = 1, 396 instead of 371 of 900 L-BFGS runs (dense BFGS: 313 instead of 296) end with a line-search failure; the 35 (dense BFGS: 43) runs whose outcome changed, in either direction, all stay at the best log density. Pathfinder (960 runs): 19 change, the ELBO higher by more than 0.1 in 9 and lower in 3, the same failures, at most 1.05 times develop's evaluations.Documentation
Doc comments of
WolfLSZoomand of themaxLSItsparameter ofWolfeLineSearch.Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):
Jáchym Barvínek jachymb@gmail.com
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:
Appendix: reproduction
Evaluation counts of three line searches. Saved as
src/test/unit/optimization/bfgs_linesearch_evals_test.cpp(not part of this PR) and run withOn develop (all three fail):
On this branch (all three pass):