Optimizer: cap the first L-BFGS/BFGS step so that large initial gradients do not fail the first line search by jachymb · Pull Request #3448 · stan-dev/stan

jachymb · 2026-10-18T20:11:47Z

This should go after #3444 and probably also #3447. Creating just as a draft for now.

AI use disclosure: This was developed with the help of Claude Code (Opus 5.5).

Submission Checklist

Run unit tests: ./runTests.py src/test/unit
Run cpplint: make cpplint
Declare copyright holder and open-source license: see below

Summary

On the first iteration and after a Hessian reset, BFGSMinimizer::step tries the step $\alpha_0 \cdot (-g)$ with $\alpha_0$ = init_alpha (0.001). With a large initial gradient (Poisson regression with large exposures, a scale parameter initialised far out) that step moves the parameters by orders of magnitude too far: the evaluations overflow or overshoot, the line search halves the step 10 times or brackets nothing, and the run ends at iteration 1 with error 70 (line-search failure). Changes:

Cap. New LSOptions::maxStep0 $= s_{\max}$; the first trial step is $\alpha = \min(\alpha_0,\ s_{\max} / \lVert g \rVert_\infty)$. The default is $+\infty$, which keeps the old behaviour (Pathfinder does not set it). The L-BFGS and BFGS services set it to 1, so the first trial step moves no unconstrained parameter by more than 1.
No relative convergence right after a capped step. After a step that the cap shortened, the relative convergence tests (tol_rel_obj, tol_rel_grad) are skipped once: the inverse Hessian estimate rests on that one short step, and the relative gradient test judged convergence from it. The absolute tests still apply.
Zoom width. WolfeLineSearch gives up on a bracket narrower than $\min(10^{-16},\ 10^{-13} \alpha)$ instead of $10^{-16}$, where $\alpha$ is the first trial step. With the cap, a gradient above $10^{16}$ gives a first step below $10^{-16}$, whose bracket the zoom would refuse at once. For $\alpha \ge 10^{-3}$ nothing changes; a shorter first trial (a capped one, or a short interpolated one in a later iteration) gets a proportionally narrower limit. It is part of this PR because only the cap makes first steps below $10^{-16}$ occur in the services.
No step floor in a capped search. WolfeLineSearch takes its minAlpha argument (LSOptions::minAlpha, $10^{-12}$) as the smallest step it may return: its bracket starts at minAlpha, with the value and slope at 0 attached to that end. After the cap that floor forbids moving a parameter by less than $10^{-12} \lVert g \rVert_\infty$, 0.8 for a gradient of $8 \cdot 10^{11}$, and the search fails when the minimiser lies closer. BFGSMinimizer::step therefore passes 0 instead of minAlpha when the cap shortened the first trial step, so that its bracket starts at 0, where the value and slope are known. Uncapped searches keep the floor (Pathfinder never caps), and minAlpha keeps its second use, the lower bound of the interpolated first trial step.

This PR and the PR of branch feature/line-search-fewer-evaluations both change the last argument of the two WolfLSZoom calls in WolfeLineSearch (minRange here, 1e-16, maxLSIts there); whichever is merged second resolves it to minRange, maxLSIts. With both, that PR's bound of maxLSIts evaluations also ends a failing zoom that the narrower limit of change 3 would let run longer.

Intended Effect

With L-BFGS, runs that develop ends at iteration 1 with error 70 because of a large initial gradient reach the optimum (dense BFGS: 99 of 100 with tol_rel_obj = tol_rel_grad = 1, 2 of 100 at the defaults; see Side Effects). Against develop on 20 model configurations (GLMs, a hierarchical logistic regression, embedded-Laplace GLMMs and GPs; 30–100 seeds each), L-BFGS at CmdStan's defaults: none of develop's 220 line-search failures in 1650 runs remain (every run of two Poisson regressions with exposures up to 400, a tenth of two with exposures up to 20), and those runs end at the best log density. L-BFGS needs 0.66–0.91 times develop's gradient evaluations on five configurations (the two with a tenth failing among them) and stays within 5% on the rest, apart from the large-exposure ones, which develop stopped after 13 evaluations. With tol_rel_obj = tol_rel_grad = 1 all 900 L-BFGS runs end at the best log density (develop 794). Dense BFGS: none of develop's 106 failures in 900 runs remain, and with tol_rel_obj = tol_rel_grad = 1 898 instead of 793 runs end at the best log density.

How to Verify

New tests, all failing on develop:

OptimizationBfgsLinesearch.wolfeLineSearch_tiny_first_step (bfgs_linesearch_test.cpp): $\phi(x) = -x + 10^{17} x^2/2$ from the first step $2.35 \cdot 10^{-17}$. Develop gives up on the bracket at once (return code 1).
ServicesOptimize.lbfgs_large_initial_gradient and bfgs_large_initial_gradient (services/optimize/{lbfgs,bfgs}_jacobian_test.cpp): the services from $\sigma = e^{20}$, where the gradient is $2.4 \cdot 10^{17}$. Develop returns error 70 with $\sigma$ unchanged.
OptimizationBfgsMinimizerCosh.large_initial_gradient (bfgs_minimizer_test.cpp): $\sum_i \cosh x_i$ from $(30, 30)$ with maxStep0 = 1. It sets the new option, so it does not compile on develop; without that line (develop's behaviour) the first iteration fails (return code −1).
OptimizationBfgsMinimizerExp.no_relative_convergence_after_reset (bfgs_minimizer_test.cpp): $e^{30x} - 30x$ from $x = 2$ with maxStep0 = 1 and CmdStan's default tol_rel_grad. It guards change 2: without it the relative gradient test stops the run at $x = 1$. On develop (without its maxStep0 line) the first line search already fails.
OptimizationBfgsMinimizerStiffQuadratic.capped_step_without_floor (bfgs_minimizer_test.cpp): $10^{13} x^2/2$ from $x = 0.08$ with maxStep0 = 1. The capped first step $1.25 \cdot 10^{-12}$ overshoots, and the minimiser lies at a step of $10^{-13}$, below minAlpha. It guards change 4: without it the step fails (return code −1). It sets the new option, so it does not compile on develop.

Existing tests that reach the changed code pass unchanged: bfgs_linesearch_test.cpp, bfgs_minimizer_test.cpp, bfgs_test.cpp, bfgs_exponential_boundary_test.cpp, services/optimize/{bfgs,lbfgs,bfgs_jacobian,lbfgs_jacobian}_test.cpp, and the Pathfinder tests services/pathfinder/{eight_schools,normal_glm}_test.cpp.

Side Effects

In-process A/B harness against develop; I can share it. Pathfinder never caps; none of its 960 measured runs changes.

Dense BFGS keeps a too small inverse Hessian scale after a capped first step (addressed by feature/bfgs-update-sizing). At the defaults it then stops short of the best log density in 98 of 100 large-exposure Poisson runs (develop: error 70), 6 other former failures, 23 runs that develop ended there (7 others now reach it) and, net, 4 of 380 test-function starts.
With tol_rel_obj = tol_rel_grad = 1, line-search failures rise on some configurations (by at most 4 runs per configuration) but fall overall (L-BFGS 371 → 323, dense BFGS 296 → 205).
init_alpha of the services now bounds the first step, and a capped search may return a step below minAlpha.

More than 5% more evaluations:

	branch / develop
dense BFGS, both settings: ordered logistic, two linear regressions	1.13–1.25
dense BFGS, `tol_rel_obj = tol_rel_grad = 1`: Poisson regression $N = 500$, hierarchical logistic	1.08, 1.05
dense BFGS test functions, defaults: Rosenbrock, extended Rosenbrock, Powell singular	1.21, 1.31 (40 instead of 32 at the optimum), 1.06
the same with only a gradient tolerance: Rosenbrock, extended Rosenbrock	1.19, 1.26
L-BFGS, `tol_rel_obj = tol_rel_grad = 1`: Laplace Poisson GLMM, logistic $N = 20000$	1.11, 1.07

Documentation

Comment on LSOptions::maxStep0 and doc comments of the services' init_alpha parameter. The CmdStan documentation of init_alpha could say that the first step moves no unconstrained parameter by more than 1.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):

Jáchym Barvínek jachymb@gmail.com

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

LSOptions::maxStep0 limits the first trial step of the first iteration and after a Hessian reset to maxStep0 / |g|_inf (default +inf; the L-BFGS and BFGS services use 1). With a large initial gradient the first trial step overflowed or overshot by orders of magnitude, and the first line search failed. After a step that the cap shortened, the relative convergence tests are skipped once. A first trial step that the cap shortened is searched without the floor minAlpha, the smallest step WolfeLineSearch may return, so that its bracket starts at 0: the floor would forbid moving a parameter by less than minAlpha |g|_inf. The zoom gives up on a bracket narrower than min(1e-16, 1e-13 alpha) instead of 1e-16, so that a bracket from a first trial step below 1e-16 (a gradient above 1e16 with the cap) is still searched.

stan-buildbot · 2026-10-19T06:25:03Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.04	0.04	0.99	-0.97% slower
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.16	0.17	0.95	-5.73% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.86	0.88	0.98	-2.49% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	0.97	-3.06% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.11	0.11	1.02	1.85% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	8.14	8.04	1.01	1.17% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	3.72	3.68	1.01	1.1% faster
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	32.85	32.92	1.0	-0.22% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.39	0.39	1.0	0.33% faster
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	6.78	6.83	0.99	-0.64% slower
stat_comp_benchmarks/benchmarks/sir/sir.stan	167.47	169.94	0.99	-1.48% slower
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.21	3.21	1.0	0.15% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.61	0.63	0.96	-3.81% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	25.46	25.82	0.99	-1.4% slower
performance.compilation	390.99	388.59	1.01	0.61% faster
Mean result: 0.9907525285563511

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1498.179
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

avehtari · 2026-10-19T07:17:11Z

I tried (with Claude) the cap on a two-scale problem: f = K/2 x1^2 + 1/2 (x2 - 5)^2, from x1 in {1e-2, 1e-3, 1e-6, 1e-9} and x2 = 0, with K from 1e8 to 1e24 (36 cases) and CmdStan's default tolerances. The stiff x1 sets |g|_inf, so the capped first step moves x2 by about 1/K. With this PR, the runs that report convergence at a wrong point (TERM_ABSX or TERM_ABSF) go from 8 to 22, while the line-search failures go from 24 to 1. For the user, a failure is better than a wrong optimum.

The cause is the inverse Hessian scale. The first pair (s, y) comes from the stiff x1, so H is about 1/K in every direction, and the next steps along x2 are tiny as well. Skipping the absolute tests after the capped step is not enough on its own (22 to 16 wrong results): the runs then stop at iteration 2. #3444 does not help here either.

I pushed one commit to your branch that does two things after a capped first step: it skips tolAbsF and tolAbsX as well as the relative tests, and it starts the second iteration from -g again, so that the approximation is built from a pair that does not come from the capped step. It restarts only once. Restarting after every capped step made large-gradient problems into steepest descent (Rosenbrock scaled by 1e4: 950 evaluations instead of 47). Outcomes on the 36 cases (L-BFGS and dense BFGS give the same counts):

outcome	develop	original PR	with the commit
reaches `x2 = 5`	4	13	18
line-search failure	24	1	14
reports convergence with `x2` near 0	8	22	4

All 14 failures with the commit are at K >= 1e16, where the condition number exceeds 1/eps and the gradient in x1 is rounding noise, so a failure is the correct result there. Of the 4 wrong results, one is not capped (develop's behavior) and three are at K >= 1e16. For K <= 1e14 the commit reaches the optimum in every case except that uncapped one.

The cost is close to this PR's. Gradient evaluations (L-BFGS / dense BFGS):

problem	develop	original PR	with the commit
Rosenbrock scaled by 1e4	36 / 33	47 / 43	53 / 49
extended Rosenbrock, n = 10, scaled by 1e3	70 / 43	87 / 97	84 / 73
Poisson, n = 20, exposures to 400, from 3	109 / 78	87 / 119	125 / 89
exp sum, n = 5, from 5	190 / 94	48 / 114	41 / 161
quadratic, n = 20, condition 1e4	687 / 129	707 / 130	765 / 117

Unscaled Rosenbrock, the Poisson problem from 0 and the quadratic with condition 1e2 are within 10 % of this PR. Dense BFGS on the exp sum from 5 still stops early (f - f* = 1.7, original PR 393, develop 7613); that is the inverse Hessian scale that #3444 addresses.

With #3444 as well, the two changes do not interfere (#3444 changes bfgs_update.hpp, the commit bfgs.hpp). The two-scale counts stay as in the table above, and dense BFGS reaches the optimum on every problem in the cost table: the exp sum from 5 in 40 evaluations (f - f* = 1e-12), the Poisson problem from 3 in 44. A large sigma (log(sigma) from 15 to 30) next to mu ~ normal(5, 1) gives mu near 0 with dense BFGS on this PR, with or without the commit; with #3444 it reaches (3, 5) from every start. On top of #3444 and this PR, the commit costs extra evaluations only on scaled Rosenbrock with dense BFGS (59 instead of 47 at 1e2, 60 instead of 48 at 1e4). On develop + #3444 + this PR + the commit, the same 14 suites pass (97 tests, including the changed expectations of #3444), and the two new tests still fail without the commit. The only conflict between #3444 and this PR is in bfgs_minimizer_test.cpp, where both append tests after the minimize test, whose expected return code #3444 changes from 31 to 20.

The commit adds a services test (services/optimize/two_scale_test.cpp, model two_scale.stan: x1 ~ normal(0, 1e-6), x2 ~ normal(5, 1), from x1 = 3e-9, x2 = 0) for L-BFGS and dense BFGS. Both fail with the original PR's bfgs.hpp (they end at x2 = 5e-12) and pass with the added commit. The optimization, services/optimize and Pathfinder suites pass (14 suites, 95 tests, including your new tests). Pathfinder does not set maxStep0, so it is not affected.

jachymb · 2026-10-20T18:13:17Z

@avehtari You didn't actually push any commit yet, you are welcome to do so though.

avehtari · 2026-10-20T18:43:51Z

@jachymb Hah, we two have too many PRs, and I forgot to push that commit!

stan-buildbot · 2026-10-20T21:45:51Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.22	0.22	1.0	0.25% faster
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.06	0.06	1.01	1.15% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	6.3	6.31	1.0	-0.19% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.02	0.02	0.95	-5.62% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	8.22	8.37	0.98	-1.75% slower
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	4.44	4.44	1.0	-0.03% slower
stat_comp_benchmarks/benchmarks/sir/sir.stan	216.93	219.58	0.99	-1.22% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.93	0.88	1.05	5.14% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.61	0.61	0.99	-0.68% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	42.65	42.64	1.0	0.03% faster
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.58	0.59	0.99	-0.69% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	20.9	20.96	1.0	-0.29% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.11	0.11	1.0	-0.13% slower
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.17	3.2	0.99	-0.74% slower
performance.compilation	366.13	368.65	0.99	-0.69% slower
Mean result: 0.9968073012175874

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           43 bits physical, 48 bits virtual
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              23
Model:                                   49
Model name:                              AMD EPYC 7742 64-Core Processor
Stepping:                                0
Frequency boost:                         enabled
CPU MHz:                                 1497.193
CPU max MHz:                             3416.0681
CPU min MHz:                             1500.0000
BogoMIPS:                                4491.85
Virtualization:                          AMD-V
L1d cache:                               4 MiB
L1i cache:                               4 MiB
L2 cache:                                64 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

jachymb · 2026-10-20T22:56:57Z

@avehtari Thanks, I reproduced your numbers and your change seems like an overall improvement.

I pushed a one-liner that writes "capped first step, Hessian reset" to the iteration log for the restart

stan-buildbot · 2026-10-21T03:03:10Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.04	0.04	1.03	3.18% faster
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.16	0.16	1.0	0.27% faster
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.86	0.88	0.97	-2.66% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	0.94	-6.49% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.11	0.11	1.0	0.29% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	8.23	8.04	1.02	2.34% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	3.67	3.69	0.99	-0.53% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	32.79	33.23	0.99	-1.35% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.38	0.39	0.97	-3.42% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	6.83	6.83	1.0	0.07% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan	172.0	165.14	1.04	3.99% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.22	3.26	0.99	-1.24% slower
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.6	0.61	1.0	-0.22% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	25.7	25.8	1.0	-0.38% slower
performance.compilation	396.53	385.51	1.03	2.78% faster
Mean result: 0.998443945202529

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1497.751
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

jachymb marked this pull request as draft October 18, 2026 20:11

jachymb mentioned this pull request Oct 18, 2026

Optimizer: spend fewer evaluations in failing Wolfe line searches #3449

Draft

3 tasks done

avehtari added the algorithm label Oct 19, 2026

Restart once and skip the step tests after a capped first step

c67e001

avehtari mentioned this pull request Oct 19, 2026

Modify the opitimization line search to not return a point with non-finite gradient #3306

Closed

Note the Hessian reset after a capped first step in the iteration log

2dfee5e

Navigation Menu

Repository navigation

Optimizer: cap the first L-BFGS/BFGS step so that large initial gradients do not fail the first line search - #3448

Optimizer: cap the first L-BFGS/BFGS step so that large initial gradients do not fail the first line search#3448
jachymb wants to merge 3 commits into
stan-dev:developfrom
jachymb:feature/optimizer-first-step-cap

jachymb commented Oct 18, 2026

Uh oh!

stan-buildbot commented Oct 19, 2026

Uh oh!

avehtari commented Oct 19, 2026

Uh oh!

jachymb commented Oct 20, 2026

Uh oh!

avehtari commented Oct 20, 2026

Uh oh!

stan-buildbot commented Oct 20, 2026

Uh oh!

jachymb commented Oct 20, 2026

Uh oh!

stan-buildbot commented Oct 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jachymb commented Oct 18, 2026

Submission Checklist

Summary

Intended Effect

How to Verify

Side Effects

Documentation

Copyright and Licensing

Uh oh!

stan-buildbot commented Oct 19, 2026

Uh oh!

avehtari commented Oct 19, 2026

Uh oh!

jachymb commented Oct 20, 2026

Uh oh!

avehtari commented Oct 20, 2026

Uh oh!

stan-buildbot commented Oct 20, 2026

Uh oh!

jachymb commented Oct 20, 2026

Uh oh!

stan-buildbot commented Oct 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants