[jachymb]

A one-liner bugfix for a relatively subtle issue.

AI use disclosure: This was developed with the help of Claude Code (Opus 5.5).

Submission Checklist

• Run unit tests: ./runTests.py src/test/unit
• Run cpplint: make cpplint
• Declare copyright holder and open-source license: see below

Summary

WolfeLineSearch (src/stan/optimization/bfgs_linesearch.hpp) first tries the step $\alpha$ and multiplies it by 10 until it brackets an acceptable step, which WolfLSZoom then finds. In that first phase, sufficient decrease was tested as $f(\alpha_1) > f_0 + c_1 \alpha f'_0$ with the initial $\alpha$ instead of the current trial $\alpha_1 = 10^k \alpha$. After $k$ expansions the test was $10^k$ times too weak, and a step without sufficient decrease could be returned, although the function promises the strong Wolfe conditions. It now uses $\alpha_1$, as WolfLSZoom already does. The change is one token.

Intended Effect

Every step WolfeLineSearch accepts meets the strong Wolfe conditions it documents, so L-BFGS, BFGS and Pathfinder no longer go on from a step that did not decrease the objective enough. The case is rare: all 4350 L-BFGS and dense BFGS runs on 20 model configurations are unchanged, and it occurs in 3 of 960 Pathfinder runs (12 models, 40 seeds, two settings), all ordered logistic with the settings of Stan's Pathfinder tests (init_alpha = 1, inits in (-3, 3)). Two of them went on to ELBOs of $-2 \cdot 10^{71}$ and $-8 \cdot 10^{84}$ and now report an error instead.

How to Verify

New test OptimizationBfgsLinesearch.wolfeLineSearch_armijo_after_expansion (src/test/unit/optimization/bfgs_linesearch_test.cpp): a cubic Hermite interpolant of three points where the step is expanded once, to $\alpha_1 = 10$, which meets the curvature condition but not sufficient decrease for $\alpha_1$. It fails on develop, which accepts that step.

Existing tests that reach the changed code pass unchanged: bfgs_linesearch_test.cpp, bfgs_minimizer_test.cpp, bfgs_test.cpp, bfgs_exponential_boundary_test.cpp, services/optimize/{bfgs,lbfgs,bfgs_jacobian,lbfgs_jacobian}_test.cpp, and the Pathfinder tests services/pathfinder/{eight_schools,normal_glm}_test.cpp.

Side Effects

None.

Documentation

None.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):

Jáchym Barvínek jachymb@gmail.com

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[stan-buildbot]

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.04	0.04	1.03	3.02% faster
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.16	0.16	0.99	-0.79% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.86	0.87	0.99	-1.13% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.01	0.85% faster
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.1	0.11	0.99	-0.66% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	8.35	8.24	1.01	1.3% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	3.69	3.69	1.0	0.02% faster
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	33.08	35.35	0.94	-6.87% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.38	0.37	1.03	2.46% faster
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	6.78	6.81	1.0	-0.39% slower
stat_comp_benchmarks/benchmarks/sir/sir.stan	165.48	162.69	1.02	1.69% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.25	3.22	1.01	0.75% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.6	0.6	1.0	0.37% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	25.95	25.8	1.01	0.6% faster
performance.compilation	393.28	377.43	1.04	4.03% faster
Mean result: 1.0040818147805415

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1497.505
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[WardBrian]

I think this is correct and seems to match the code we have internal to the embedded laplace functions.

@SteveBronder has told me that he's interested in combining the wolfe implementations between the two repos, and I think this would definitely be worth thinking about...