Skip to content

Expanded numerical stability and performance for std_normal_lcdf, reused across distributions & functions - #3406

Open
andrjohns wants to merge 34 commits into
developfrom
normal-lcdf-opt
Open

Expanded numerical stability and performance for std_normal_lcdf, reused across distributions & functions#3406
andrjohns wants to merge 34 commits into
developfrom
normal-lcdf-opt

Conversation

@andrjohns

@andrjohns andrjohns commented Sep 18, 2026 •
edited by avehtari
Loading

Copy link
Copy Markdown
Collaborator

Fixes #2881. Implements most of #3364: the normal, lognormal and exp_mod_normal lcdf/lccdf through the shared std_normal_lcdf implementation, and the OpenCL part.

Summary

This PR extracts the key value and gradient calculations from the std_normal_lcdf header into reusable functions that other normal-cdf-related functions can delegate to, removing a lot of duplicated/inconsistent implementations.

The value and gradient calculations are also updated to replace the taylor expansions with the Cody rational approximations (previously only used for the tail).

Two other bugs were found & fixed for var_value<Eigen::VectorXd> handling:

  • as_array_or_scalar() now returns an array unchanged instead of re-wrapping it
  • ref_type_if keeps an rvalue arena_matrix by value rather than referencing a dead temporary

The valid ranges of inputs (before over/underflow) across all functions are expanded:

function develop value branch value develop gradient branch gradient
normal_lcdf, std_normal_lcdf [-5e30, 38.5] [-1e154, 38.5] (-1e308, 38.6] (-1e308, 38.6]
normal_lccdf, std_normal_lccdf [-38.5, 5e30] [-38.5, 1e154] [-38.6, 1e308) [-38.6, 1e308)
normal_cdf, std_normal_cdf [-37.5, 8.24] [-38.5, 8.28] [-37.5, 38.6] [-38.5, 38.6]
Phi [-37.5, 8.24] [-38.5, 8.28] [-38.6, 38.6] [-38.6, 38.6]
lognormal_lcdf [-38.5, 8.28] [< -500, 38.5] [-38.5, 37.6] [< -500, 37.6]
lognormal_lccdf [-8.28, 38.5] [-38.5, > 500] [-38.6, 38.5] [-38.6, > 500]
lognormal_cdf [-38.5, 8.28] [-38.5, 8.28] [-27.2, 37.6] [-38.5, 37.6]
exp_mod_normal_lpdf [-37.5, 1e308) [-1e154, 1e308) [-37.5, 1e308) (-1e308, 1e308)
exp_mod_normal_lcdf [-7.2, 37.9] [-2e5, ~745] [-8.2, ~745] [-2e5, ~745]
exp_mod_normal_lccdf [-7.2, ~745] [-38.4, 1e308) [-8.2, ~745] [-38.6, 1e308)
exp_mod_normal_cdf [-7.2, 37.9] [-38.4, 37.9] [-8.2, ~745] [-38.4, ~745]
skew_normal_lpdf [-38.5, 1e154] [-1e154, 1e154] [-38.5, 1e308) [-5e307, 1e308)

While performance is either unchanged or significantly improved across inputs:

function min slowest case max fastest case
normal_lcdf 1.66x double, N = 100000 3.56x var_valueEigen::Matrix, N = 1000
normal_lccdf 1.75x fvar, N = 100000 3.71x var_value, N = 1000
normal_cdf 0.92x fvar, N = 100000 2.13x double, N = 1000
std_normal_lcdf 1.78x fvar, N = 100000 5.08x var_valueEigen::Matrix, N = 1000
std_normal_lccdf 2.41x double, N = 100000 5.02x var_valueEigen::Matrix, N = 1000
std_normal_cdf 1.35x double, N = 100000 2.68x var_valueEigen::Matrix, N = 1000
Phi 1.17x fvar, N = 100000 1.65x double, N = 1000
lognormal_lcdf 1.37x var_value, N = 10000 2.60x double, N = 1000
lognormal_lccdf 1.32x fvar, N = 10000 2.43x double, N = 1000
lognormal_cdf 1.29x var_value, N = 10000 2.24x double, N = 1000
exp_mod_normal_lpdf 1.78x var_value, N = 10000 2.36x double, N = 1000
exp_mod_normal_lcdf 1.04x fvar, N = 100000 1.52x double, N = 1000
exp_mod_normal_lccdf 1.03x var_valueEigen::Matrix, N = 100000 1.51x double, N = 1000
exp_mod_normal_cdf 1.00x var_value, N = 100000 1.48x double, N = 1000
skew_normal_lpdf 1.58x var_value, N = 100000 2.49x double, N = 1000

Full timing comparisons

double (values only)

function N = 100 N = 1000 N = 10000 N = 100000
normal_lcdf 34.3 / 13.6 (2.52x) 34.5 / 13.4 (2.57x) 39.0 / 18.7 (2.08x) 40.0 / 24.2 (1.66x)
normal_lccdf 35.0 / 13.9 (2.52x) 34.8 / 13.6 (2.57x) 39.0 / 17.5 (2.24x) 40.2 / 19.4 (2.07x)
normal_cdf 28.3 / 13.7 (2.06x) 28.7 / 13.4 (2.13x) 28.4 / 16.9 (1.68x) 28.4 / 18.9 (1.50x)
std_normal_lcdf 36.3 / 14.3 (2.54x) 38.0 / 14.0 (2.71x) 39.8 / 17.8 (2.24x) 40.7 / 19.7 (2.06x)
std_normal_lccdf 36.8 / 13.4 (2.75x) 38.4 / 13.1 (2.93x) 39.9 / 15.4 (2.60x) 41.0 / 17.0 (2.41x)
std_normal_cdf 26.2 / 14.4 (1.82x) 26.8 / 14.1 (1.91x) 26.6 / 17.8 (1.50x) 26.6 / 19.7 (1.35x)
Phi 26.9 / 16.5 (1.64x) 27.2 / 16.5 (1.65x) 27.3 / 19.9 (1.37x) 27.2 / 20.8 (1.31x)
lognormal_lcdf 37.1 / 14.7 (2.53x) 36.8 / 14.1 (2.60x) 41.0 / 18.9 (2.17x) 42.8 / 20.8 (2.05x)
lognormal_lccdf 36.9 / 15.7 (2.36x) 36.7 / 15.1 (2.43x) 40.2 / 19.6 (2.05x) 42.3 / 21.6 (1.95x)
lognormal_cdf 32.0 / 15.3 (2.09x) 33.0 / 14.7 (2.24x) 36.1 / 19.2 (1.88x) 38.5 / 21.5 (1.79x)
exp_mod_normal_lpdf 29.3 / 13.4 (2.20x) 29.8 / 12.6 (2.36x) 31.1 / 14.6 (2.13x) 32.8 / 16.6 (1.97x)
exp_mod_normal_lcdf 57.5 / 38.5 (1.49x) 57.2 / 37.6 (1.52x) 57.7 / 49.9 (1.16x) 57.8 / 51.4 (1.12x)
exp_mod_normal_lccdf 57.6 / 38.8 (1.48x) 57.3 / 37.9 (1.51x) 57.6 / 50.6 (1.14x) 57.9 / 52.0 (1.11x)
exp_mod_normal_cdf 55.8 / 38.2 (1.46x) 55.6 / 37.4 (1.48x) 55.8 / 50.3 (1.11x) 56.0 / 51.8 (1.08x)
skew_normal_lpdf 30.9 / 12.9 (2.39x) 30.7 / 12.4 (2.49x) 33.2 / 17.0 (1.95x) 34.3 / 19.4 (1.77x)

var_value (Eigen::Matrix<var_value> y, var_value parameters, gradient)

function N = 100 N = 1000 N = 10000 N = 100000
normal_lcdf 62.2 / 18.6 (3.34x) 62.4 / 17.8 (3.50x) 70.1 / 29.5 (2.38x) 70.6 / 34.0 (2.08x)
normal_lccdf 67.2 / 18.7 (3.59x) 67.1 / 18.1 (3.71x) 73.6 / 21.4 (3.44x) 74.5 / 34.4 (2.16x)
normal_cdf 35.9 / 18.7 (1.92x) 36.1 / 17.9 (2.01x) 36.0 / 21.4 (1.69x) 36.1 / 22.9 (1.57x)
std_normal_lcdf 59.9 / 17.3 (3.47x) 61.5 / 17.2 (3.59x) 67.5 / 20.2 (3.34x) 68.2 / 21.6 (3.15x)
std_normal_lccdf 64.7 / 16.6 (3.89x) 66.1 / 16.4 (4.03x) 71.5 / 18.7 (3.83x) 72.0 / 20.2 (3.57x)
std_normal_cdf 32.2 / 18.3 (1.76x) 32.7 / 17.1 (1.91x) 32.7 / 20.1 (1.63x) 32.6 / 21.6 (1.51x)
Phi 37.8 / 26.3 (1.43x) 38.3 / 26.6 (1.44x) 38.5 / 31.4 (1.23x) 38.7 / 32.2 (1.20x)
lognormal_lcdf 32.5 / 20.7 (1.57x) 31.8 / 20.1 (1.58x) 34.6 / 25.2 (1.37x) 36.5 / 26.5 (1.38x)
lognormal_lccdf 32.5 / 21.2 (1.53x) 31.6 / 20.5 (1.54x) 33.9 / 25.6 (1.32x) 35.8 / 26.9 (1.33x)
lognormal_cdf 30.6 / 20.8 (1.47x) 29.9 / 20.2 (1.48x) 32.7 / 25.4 (1.29x) 34.6 / 26.7 (1.29x)
exp_mod_normal_lpdf 32.6 / 18.1 (1.80x) 32.5 / 17.2 (1.89x) 33.8 / 19.0 (1.78x) 35.6 / 20.0 (1.78x)
exp_mod_normal_lcdf 61.9 / 47.9 (1.29x) 61.4 / 46.1 (1.33x) 61.2 / 58.0 (1.06x) 61.8 / 59.5 (1.04x)
exp_mod_normal_lccdf 62.0 / 48.2 (1.29x) 61.3 / 47.3 (1.30x) 61.0 / 58.7 (1.04x) 62.1 / 60.0 (1.04x)
exp_mod_normal_cdf 60.0 / 48.1 (1.25x) 59.7 / 45.9 (1.30x) 59.4 / 57.9 (1.03x) 59.6 / 59.6 (1.00x)
skew_normal_lpdf 34.7 / 18.1 (1.92x) 34.3 / 17.4 (1.97x) 36.9 / 22.8 (1.62x) 38.5 / 24.4 (1.58x)

var_valueEigen::Matrix (var_valueEigen::VectorXd y, var_value parameters, gradient)

function N = 100 N = 1000 N = 10000 N = 100000
normal_lcdf 60.0 / 17.7 (3.39x) 60.1 / 16.9 (3.56x) 67.1 / 21.7 (3.10x) 67.9 / 33.0 (2.06x)
normal_lccdf 61.9 / 17.8 (3.48x) 61.3 / 17.1 (3.59x) 68.1 / 20.4 (3.35x) 68.9 / 33.3 (2.07x)
normal_cdf 35.0 / 17.8 (1.97x) 35.2 / 16.9 (2.08x) 35.2 / 20.3 (1.73x) 35.2 / 21.8 (1.61x)
std_normal_lcdf 59.0 / 12.1 (4.87x) 60.4 / 11.9 (5.08x) 65.9 / 15.3 (4.30x) 66.6 / 16.9 (3.95x)
std_normal_lccdf 60.2 / 12.5 (4.82x) 60.9 / 12.1 (5.02x) 66.1 / 15.1 (4.38x) 66.9 / 17.0 (3.94x)
std_normal_cdf 31.4 / 12.1 (2.59x) 31.8 / 11.9 (2.68x) 31.6 / 15.3 (2.06x) 31.8 / 16.9 (1.88x)
Phi 29.0 / 18.3 (1.59x) 28.9 / 18.1 (1.59x) 28.9 / 21.4 (1.35x) 29.1 / 22.5 (1.29x)
lognormal_lcdf 32.1 / 19.9 (1.61x) 31.1 / 19.1 (1.63x) 33.6 / 24.1 (1.39x) 35.4 / 25.5 (1.39x)
lognormal_lccdf 31.5 / 20.4 (1.54x) 30.6 / 19.3 (1.58x) 32.7 / 24.6 (1.33x) 34.5 / 25.9 (1.33x)
lognormal_cdf 29.8 / 19.9 (1.50x) 28.9 / 19.1 (1.51x) 31.5 / 24.3 (1.29x) 33.4 / 25.6 (1.31x)
exp_mod_normal_lpdf 31.8 / 17.3 (1.84x) 31.5 / 16.2 (1.95x) 32.7 / 18.0 (1.82x) 34.6 / 19.1 (1.81x)
exp_mod_normal_lcdf 60.8 / 46.9 (1.30x) 60.2 / 45.0 (1.34x) 60.1 / 56.9 (1.06x) 61.1 / 58.7 (1.04x)
exp_mod_normal_lccdf 60.8 / 47.1 (1.29x) 60.0 / 46.3 (1.29x) 60.0 / 57.7 (1.04x) 60.9 / 59.2 (1.03x)
exp_mod_normal_cdf 59.1 / 47.4 (1.25x) 58.4 / 45.1 (1.30x) 58.4 / 56.9 (1.03x) 59.2 / 58.5 (1.01x)
skew_normal_lpdf 34.1 / 17.2 (1.99x) 33.6 / 16.5 (2.03x) 36.0 / 21.8 (1.65x) 37.4 / 23.5 (1.59x)

fvar (tangent)

function N = 100 N = 1000 N = 10000 N = 100000
normal_lcdf 60.4 / 18.1 (3.33x) 60.7 / 17.5 (3.47x) 75.4 / 36.9 (2.04x) 79.0 / 45.4 (1.74x)
normal_lccdf 62.4 / 18.9 (3.30x) 61.7 / 17.9 (3.44x) 68.4 / 21.4 (3.20x) 80.5 / 46.0 (1.75x)
normal_cdf 36.3 / 18.7 (1.94x) 35.8 / 17.7 (2.02x) 36.0 / 21.1 (1.70x) 36.5 / 39.6 (0.92x)
std_normal_lcdf 60.4 / 17.4 (3.47x) 61.4 / 17.1 (3.59x) 67.3 / 20.0 (3.36x) 68.1 / 38.3 (1.78x)
std_normal_lccdf 61.7 / 17.0 (3.64x) 61.7 / 16.3 (3.78x) 67.0 / 18.6 (3.60x) 68.1 / 20.2 (3.37x)
std_normal_cdf 32.5 / 17.6 (1.85x) 32.5 / 17.1 (1.90x) 32.6 / 20.0 (1.62x) 33.0 / 21.6 (1.53x)
Phi 32.6 / 24.1 (1.35x) 32.8 / 24.5 (1.34x) 33.0 / 27.6 (1.20x) 33.1 / 28.3 (1.17x)
lognormal_lcdf 32.8 / 20.8 (1.58x) 31.6 / 19.8 (1.59x) 34.5 / 25.0 (1.38x) 36.3 / 26.2 (1.39x)
lognormal_lccdf 32.4 / 21.3 (1.52x) 31.3 / 20.1 (1.56x) 33.7 / 25.5 (1.32x) 35.8 / 26.7 (1.34x)
lognormal_cdf 30.8 / 20.6 (1.49x) 29.6 / 19.9 (1.48x) 32.5 / 25.2 (1.29x) 34.5 / 26.5 (1.30x)
exp_mod_normal_lpdf 33.2 / 18.0 (1.85x) 32.7 / 16.8 (1.94x) 34.2 / 18.9 (1.80x) 35.9 / 19.9 (1.80x)
exp_mod_normal_lcdf 61.8 / 47.6 (1.30x) 60.4 / 45.8 (1.32x) 61.1 / 57.8 (1.06x) 62.2 / 60.0 (1.04x)
exp_mod_normal_lccdf 61.9 / 47.8 (1.29x) 60.3 / 46.9 (1.28x) 61.2 / 58.7 (1.04x) 61.9 / 59.9 (1.03x)
exp_mod_normal_cdf 60.1 / 47.8 (1.26x) 58.7 / 45.7 (1.29x) 59.4 / 57.8 (1.03x) 60.6 / 60.1 (1.01x)
skew_normal_lpdf 35.1 / 18.0 (1.95x) 34.1 / 17.2 (1.98x) 37.0 / 23.1 (1.60x) 38.6 / 24.0 (1.61x)

Tests

Additional tests exercising the increased numerical stability are added, as well new mix tests for the distributions

Side Effects

N/A

Release notes

Increased numerical stability and performance of the normal, std_normal, lognormal and exp_mod_normal (LC)CDF functions

Checklist

  • Copyright holder: (fill in copyright holder information)

    The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
    - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
    - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

  • the basic tests are passing

    • unit tests pass (to run, use: ./runTests.py test/unit)
    • header checks pass, (make test-headers)
    • dependencies checks pass, (make test-math-dependencies)
    • docs build, (make doxygen)
    • code passes the built in C++ standards checks (make cpplint)

  • the code is written in idiomatic C++ and changes are documented in the doxygen

  • the new changes are tested

@andrjohns

andrjohns commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author

This duplicates a bunch of the implementation over in #3405, so will make some suggestions/comments on that one and then rebase this once that PR lands

@avehtari

avehtari commented Sep 18, 2026

Copy link
Copy Markdown
Member

I guess we should coordinate more as I also was preparing PR for std_normal and normal

@stan-buildbot

stan-buildbot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.04 0.04 0.98 -2.01% slower
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.16 0.17 1.0 -0.23% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan 0.86 0.88 0.98 -2.07% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 1.01 1.25% faster
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.11 0.1 1.05 4.43% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 8.32 8.31 1.0 0.12% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 3.69 3.71 1.0 -0.34% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 33.64 33.02 1.02 1.83% faster
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.38 0.38 0.99 -0.85% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 6.86 6.86 1.0 0.12% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan 133.47 135.53 0.98 -1.54% slower
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.21 3.2 1.0 0.45% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.6 0.59 1.01 1.07% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 25.27 25.0 1.01 1.07% faster
performance.compilation 393.17 392.02 1.0 0.29% faster
Mean result: 1.0026527691698786

Jenkins Console Log
Jenkins Build Stages
Commit hash: 98c64f32efad1c4821e30c9f76801a0d0c89d086

Machine information 
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1497.515
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@avehtari

avehtari commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

I reviewed this PR (with help from Claude). Looks great

Accuracy against mpmath

Worst ulp over 601 points per range:

range value gradient
deep tail [-300, -40] 1.69 0.85
[-40, -10] 1.51 0.87
[-10, -5.66] 1.60 0.85
seam [-5.8, -5.5] 1.54 3.12
[-5.66, -1] 1.71 5.92
seam [-0.7, -0.6] 2.37 2.75
near zero [-0.6, 0.6] 1.86 2.36
[0.6, 4] 7.25 4.81
[4, 10] 30.37 31.53

For comparison, develop's worst in-range relative error is 7.6e-06 to 5.6e-05, which is 3.4e+10 to 2.5e+11 ulp, across every branch except the far Mills tail. So this PR improves the gradient by about ten orders of magnitude.

erfcx looks slightly faster on the lower tail

Beyond removing the duplicated Cody coefficients, #3405 may improve speed. Comparing your internal std_normal_erfcx path against calling erfcx directly, value and gradient computed together:

range (scaled) CPU GPU
far tail [-40, -10] 1.16 1.35
Cody [-10, -4] 1.10 2.21
Cody edge [-4, -2.5] 1.06 1.03
interior [-2.5, -0.1] 1.11 1.93

Ratios above 1 mean the direct erfcx call is faster.

@spinkney

spinkney commented Sep 18, 2026

Copy link
Copy Markdown
Member

awesome! thanks @andrjohns for putting this together, it's much needed

@avehtari

avehtari commented Sep 18, 2026

Copy link
Copy Markdown
Member

Two more functions get the benefit with no extra work

Two functions that benefit from this std_normal_lcdf PR that are missing from the PR text are

function calls OpenCL version
ordered_probit_lpmf std_normal_lcdf x4 none
von_mises_cdf normal_cdf none

One function benefits from erfcx directly and is not covered here

wiener4_lcdf_unnorm line 128 is:

return std_normal_lcdf(-x) - std_normal_lpdf(x);

It calls std_normal_lcdf, so it looks like it inherits the repair., but doesn't. That expression is the log Mills ratio, and both terms approach -x^2/2 while their difference stays O(log x), so the subtraction cancels no matter how accurate the inputs are. Assuming both inputs are already correctly rounded, which is the best case after this PR:

x log Phi(-x) log phi(x) result as a difference via erfcx
10 -53.2 -50.9 -2.3123 4.6 ulp 0.4 ulp
30 -454.3 -450.9 -3.4023 74.0 ulp 0.0 ulp
100 -5005.5 -5000.9 -4.6053 195.7 ulp 0.3 ulp
300 -45006.6 -45000.9 -5.7038 6132.2 ulp 0.2 ulp

The loss grows without bound. log Phi(-x) - log phi(x) equals log(sqrt(pi/2) * erfcx(x * INV_SQRT_TWO)) exactly, so one erfcx call removes the subtraction. Out of scope for this PR, but worth recording as the one which calls std_normal_lcdf but not benefiting from this PR.

@SteveBronder

SteveBronder commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@andrjohns this is very cool! Because of how tied in this and #3405 are do you want to handle the #3405 review and then I'll review this once that is in?

@andrjohns

andrjohns commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

@andrjohns this is very cool! Because of how tied in this and #3405 are do you want to handle the #3405 review and then I'll review this once that is in?

Yep sounds like a plan to me!

@SteveBronder

SteveBronder commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

@andrjohns ping me when this is ready for review

@SteveBronder

SteveBronder commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

@andrjohns is this ready for me to look at?

@stan-buildbot

stan-buildbot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.23 0.23 1.01 1.31% faster
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.06 0.06 0.96 -4.44% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 6.32 6.31 1.0 0.13% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 0.98 -2.0% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 8.45 8.43 1.0 0.23% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 4.49 4.46 1.01 0.69% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan 165.47 165.13 1.0 0.2% faster
stat_comp_benchmarks/benchmarks/garch/garch.stan 0.89 0.89 1.0 -0.08% slower
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.7 0.71 0.99 -0.92% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 42.53 42.72 1.0 -0.45% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.6 0.6 0.99 -0.74% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 21.08 20.78 1.01 1.43% faster
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.11 0.11 1.0 -0.4% slower
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.19 3.19 1.0 0.22% faster
performance.compilation 386.86 387.1 1.0 -0.06% slower
Mean result: 0.9969507030598858

Jenkins Console Log
Jenkins Build Stages
Commit hash: 1a419c76226745cd916ab56e54b60394f2ebbdff

Machine information 
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           43 bits physical, 48 bits virtual
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              23
Model:                                   49
Model name:                              AMD EPYC 7742 64-Core Processor
Stepping:                                0
Frequency boost:                         enabled
CPU MHz:                                 1496.992
CPU max MHz:                             3416.0681
CPU min MHz:                             1500.0000
BogoMIPS:                                4491.85
Virtualization:                          AMD-V
L1d cache:                               4 MiB
L1i cache:                               4 MiB
L2 cache:                                64 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@nsiccha nsiccha mentioned this pull request Oct 3, 2026
Open
@SteveBronder
SteveBronder self-requested a review October 5, 2026 18:59
@avehtari avehtari mentioned this pull request Oct 7, 2026
Open
@avehtari

avehtari commented Oct 7, 2026

Copy link
Copy Markdown
Member

This PR fixes #2881

This was referenced Oct 8, 2026
Closed
Open
@avehtari

avehtari commented Oct 8, 2026

Copy link
Copy Markdown
Member

This PR also implements most of #3364: the normal, lognormal and exp_mod_normal lcdf/lccdf through the shared std_normal_lcdf implementation, and the OpenCL part.

SteveBronder and others added 4 commits October 14, 2026 12:47
@SteveBronder
Remove unnecessary function template values from calls to functions
7e99d35
@stan-buildbot
[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1
26256d4
@SteveBronder
Restore to_ref on lcdf inputs to cut expression test memory
201dcab 
Evaluating the *_val inputs in normal_lcdf_impl, lognormal_lcdf_impl and
exp_mod_normal_lcdf_impl lets expression and plain arguments share their
downstream instantiations. Peak g++ -O0 memory compiling the generated
expression tests drops from 33 GB to 9 GB for each exp_mod_normal
lcdf/lccdf/cdf and by about 35% for the normal and lognormal ones, which
pushed the expr CI pod past its 64 GiB limit.
@SteveBronder
Evaluate std_normal lcdf input once
f37ea7c 
std_normal_lcdf_impl read y_val without to_ref, so a double expression
argument was evaluated by check_not_nan and again by the kernel. This
failed the Prim expression tests for std_normal_lcdf, std_normal_lccdf
and std_normal_cdf.
@stan-buildbot

stan-buildbot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.04 0.04 1.0 0.24% faster
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.16 0.16 1.01 0.84% faster
stat_comp_benchmarks/benchmarks/garch/garch.stan 0.87 0.89 0.98 -2.37% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.0 -0.35% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.1 0.11 0.99 -1.52% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 7.98 8.38 0.95 -5.03% slower
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 3.78 3.7 1.02 2.05% faster
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 32.58 33.52 0.97 -2.89% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.38 0.39 0.98 -2.3% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 6.77 5.5 1.23 18.74% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan 135.06 165.44 0.82 -22.5% slower
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.23 3.2 1.01 0.84% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.61 0.62 0.98 -1.88% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 25.09 25.11 1.0 -0.06% slower
performance.compilation 406.92 390.45 1.04 4.05% faster
Mean result: 0.9980139932406499

Jenkins Console Log
Jenkins Build Stages
Commit hash: 42fcfa980d4483c56e7579401017b97de434ff5a

Machine information 
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1497.587
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

SteveBronder
SteveBronder previously approved these changes Oct 16, 2026
View reviewed changes

@SteveBronder SteveBronder left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good!

@stan-buildbot

stan-buildbot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.04 0.04 1.01 0.89% faster
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.16 0.16 0.99 -1.07% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan 0.86 0.85 1.0 0.33% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.98 -2.36% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.11 0.11 1.01 0.64% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 8.14 8.31 0.98 -2.14% slower
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 3.67 3.68 1.0 -0.31% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 33.52 33.13 1.01 1.15% faster
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.38 0.38 1.0 0.37% faster
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 6.79 5.51 1.23 18.82% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan 165.36 165.09 1.0 0.17% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.22 3.22 1.0 -0.1% slower
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.6 0.92 0.65 -52.99% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 25.75 27.25 0.94 -5.83% slower
performance.compilation 400.45 385.73 1.04 3.68% faster
Mean result: 0.9897284925109069

Jenkins Console Log
Jenkins Build Stages
Commit hash: 6a335dd02ea35aac291d2cb571d03092a8b16ad7

Machine information 
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1496.584
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@avehtari avehtari linked an issue Oct 17, 2026 that may be closed by this pull request
normal_lcdf and normal_lccdf give infinite gradients for infinite inputs. #2881
Open
@avehtari

avehtari commented Oct 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

@SteveBronder @andrjohns

The std_normal_lcdf helpers now use the merged erfcx code

On 2026-09-18 @andrjohns wrote that this PR duplicates much of #3405 and that he would rebase after #3405 is merged. #3405 was merged on 2026-09-20, and develop was merged into this branch on 2026-09-29, but the duplicated code stayed. The CPU helpers read the coefficient arrays from erfcx.hpp and evaluate them with their own code, and the OpenCL device function keeps full copies of the coefficient tables. @andrjohns has not commented here since 2026-09-19, so I pushed commit 5f33a10 with the cleanup. It changes only the internal helpers and leaves the distribution files alone.

What the commit changes

  • prim/fun/erfcx.hpp: internal::erfcx_cody_tail, erfcx_cody_middle and erfcx_small are now templates in the scalar type. I did not change their double code, so erfcx returns the same bits as before.
  • prim/fun/std_normal_lcdf_impl.hpp: I deleted std_normal_erfcx_tail and std_normal_erfcx_middle; the kernel calls erfcx_cody_tail and erfcx_cody_middle instead. std_normal_tail_correction calls erfcx_tail_polynomials with reverse iterators, which performs the same operations as the written-out Horner.
  • Central range |z| < 0.46875 sqrt(2): the kernel computes log Phi(z) = log(1/2) - z^2/2 + log(erfcx(-z/sqrt(2))) and the gradient sqrt(2/pi) / erfcx(-z/sqrt(2)) with erfcx_small. This replaces the Cody erf rational and its coefficient table, and the gradient needs no exp.
  • OpenCL: the std_normal_lcdf device function calls erfcx_tail_correction, erfcx_cody_tail, erfcx_cody_middle and erfcx_small from the erfcx device function, and I deleted its own coefficient tables. In elt_function_cl.hpp, the registrations of Phi, inv_Phi, std_normal_lcdf_impl and std_normal_lcdf_derivative now include erfcx_device_function first.
  • prim/prob/std_normal_lcdf_impl_test.cpp: new reference rows at z = -3, -0.6, -0.25, 0.25, 0.6, 0.66, 0.67 and 3 (mpmath at 50 digits, checked against mpmath.ncdf). They give the central range and both sides of its boundary absolute references.

The deep lower tail (z <= -4 sqrt(2)) keeps its log1p(tail_term) value and its neg_z / (1 + tail_term) gradient; only tail_term now comes from the shared code. A form through 1 / erfcx would give 0 / 0 in nested autodiff once erfcx^2 underflows, and std_normal_extreme_tail_derivatives checks third derivatives down to z = -1e50.

The diff touches 5 files: 59 lines added, 153 deleted.

Accuracy

Worst error in ulp (relative error / 2^-53) against mpmath at 50 digits, 2001 points per range, value / gradient:

range before after
[-300, -40] 2.73 / 1.33 2.73 / 1.33
[-40, -5.66] 2.93 / 1.38 2.93 / 1.38
[-5.66, -0.66] 2.46 / 7.01 2.46 / 6.50
[-0.66, 0.66] 2.93 / 4.01 6.43 / 2.74
[0.62, 0.70] (boundary) 4.58 / 3.66 8.77 / 3.14
[0.66, 5.66] 11.43 / 10.56 11.98 / 10.56
[5.66, 37] 512.84 / 512.48 509.62 / 512.48

Both lower tails give bitwise identical results. In the Cody middle and tail ranges the erfcx evaluation order rounds differently, and the worst errors move by at most 1.3 ulp in either direction.

In the central range the gradient improves from 4.0 to 2.7 ulp, but the value gets worse for 0 <= z < 0.66: up to 8.8 ulp on this grid and 9.6 ulp on a denser grid of 20001 points, against 4.6 before. There log(1/2) - z^2/2 + log(erfcx) adds terms of size 0.9 and 0.6 to reach about -0.3, and all three summation orders give the same worst case. 9.6 ulp is a relative error of 1e-15. I accept it for the speed gain below: many distributions call std_normal_lcdf, and most calls fall in the central and middle ranges.

Speed

Setup: Xeon E5-2680 v3, one core, performance governor, g++ 14.3.0, Stan default flags. I timed internal::std_normal_lcdf_value_grad<true> (value and gradient) and <false> (value only) on double, this PR's head against the commit, 3 runs × 3 rounds each with the two binaries alternated. Time relative to before (below 1 is faster):

range gradient and value value only
[-40, -5.66] 1.02 1.02
[-5.66, -0.66] 0.88 0.86
[-0.66, 0.66] 0.66 0.83
[0.66, 5.66] 0.90 0.91
[5.66, 37] 0.77 0.77
[-6, 6] uniform 0.87 0.89

The round-to-round spread was at most 3.2 %, and 6.6 % in the deep tail, whose code did not change. The middle ranges gain from the paired Horner evaluation in #3405. Above 5.66 the x^2 form of the Cody tail saves one division compared with (1 + u C(u)) / (sqrt(pi) x), u = 1/x^2. In the central range the gradient drops its exp call. I did not time the OpenCL code; the device function evaluates the same rationals as before, and its central branch also drops the exp.

Tests

I tested this PR's head 686fc67 plus the commit:

  • CPU, 20 suites, all pass: prim and mix std_normal_lcdf_impl_test, prim/rev/fwd/mix erfcx_test, prim and mix Phi_test, and the normal, std_normal, lognormal, exp_mod_normal and skew_normal suites that call the kernel.
  • OpenCL on a Tesla V100, 11 suites, all pass: opencl/rev std_normal_lcdf, std_normal_lccdf, normal_lcdf, normal_lccdf, lognormal_lcdf, Phi, inv_Phi, skew_normal_lpdf, exp_mod_normal_lcdf, exp_mod_normal_lpdf, and elt_function_cl_test.
  • clang-format 10 and cpplint (with the filters of make/cpplint) report nothing on the changed files.

@avehtari

avehtari commented Oct 19, 2026

Copy link
Copy Markdown
Member

Phi is slower in the original PR than on develop

The original PR computes Phi(x) as exp(log Phi(x)), on the CPU and in the OpenCL device function. That adds a log and an exp to every call. I timed Phi(double) against develop on a Xeon E5-2680 v3 (one core, g++ 14.3.0, Stan default flags): it takes 4.4 times as long for |x| < 0.66, 1.9 times on [-3, 3] and on [-37, -5], and 1.4 times on [5, 8]. Phi(var) with its gradient takes 1.2 to 1.5 times as long. Probit models call Phi once per observation, so many models would see this.

The change was made for accuracy, and develop does have a defect: its 0.5 * (1 + erf(x / sqrt(2))) branch on [-5, -0.67] loses up to 7.7e5 ulp. Only that branch is at fault. 0.5 * erfc(-x / sqrt(2)) has no cancellation for either sign of x.

I pushed commit bf5861a, which computes Phi that way on the CPU and in the OpenCL device function. Phi then no longer needs the std_normal_lcdf device function, so the Phi and inv_Phi registrations in elt_function_cl.hpp are back to their develop form. The gradient code of Phi does not change.

Accuracy

Worst relative error in ulp against mpmath, 47001 points on [-38.5, 8.5]:

x develop this PR before after bf5861a
[-38.5, -30) (cut off at -37.5) 1822 1704
[-30, -20) 1224 922 1224
[-20, -10) 470 440 470
[-10, -5) 118 93 118
[-5, -1) 766133 19.1 30.5
[-1, 0) 3.2 4.6 3.0
[0, 1) 1.3 2.6 1.3
[1, 8.5) 1.2 1.3 1.2

In the lower tail every version loses about x^2 / 2 ulp, because the argument x / sqrt(2) is rounded before erfc sees it, and the exp(log Phi) form has the same limit. The new form stays within a factor 1.6 of the exp(log Phi) form there and is better above -1.

Speed

Same setup as above, Phi(double), 3 runs × 3 rounds with the variants interleaved in one binary. Time relative to develop (below 1 is faster):

range this PR before after bf5861a
[-0.66, 0.66] 4.39 1.05
[-3, 3] 1.88 0.99
[-5, 5] 1.58 0.97
[-37, -5] 1.91 0.99
[5, 8] 1.42 0.95

The round-to-round spread was at most 2.4 %. I also tried erf above -0.67 and erfc below; it ran at the same speed, so I kept the single formula.

Tests

On this PR's head 5f33a10 plus bf5861a:

  • CPU, 22 suites, all pass: the 20 listed in my previous comment, plus prim and mix inv_Phi_test. That includes this PR's Phi_test, with its 1e-14 relative checks at -5 and -3, Phi(-40) == 0 and the tail values from R.
  • OpenCL on a Tesla V100, the same 11 suites as before, all pass, including opencl/rev Phi_test and inv_Phi_test.
  • clang-format 10 and cpplint report nothing on the changed files.

@avehtari

avehtari commented Oct 19, 2026

Copy link
Copy Markdown
Member

std_normal_cdf, normal_cdf and lognormal_cdf from erfc

The original PR also computes these three as exp(lcdf), and with double arguments they take 1.4 to 1.9 times as long as on develop.

I pushed commit 0d4ac36, which computes them from erfc in the same way as Phi in my previous comment. Each factor is 0.5 * erfc(-z / sqrt(2)), the cdf is their product, and each partial is the cdf times phi(z) / Phi(z), divided by sigma (and by y for lognormal_cdf).

A new helper, internal::std_normal_cdf_value_ratio in std_normal_lcdf_impl.hpp, returns Phi(z) and phi(z) / Phi(z) for a scalar or an Eigen array, generic in the scalar type so that nested autodiff works. Where Phi underflows it returns a ratio of 0: the product is 0 there anyway, and exp(-z^2/2) / Phi would be 0 / 0. The three functions keep the argument checks and the broadcasting of their lcdf implementations. I did not change the OpenCL versions, which still use exp(lcdf).

Accuracy

Scalar calls with mu = 0.3, sigma = 1.7, all arguments var, z = (y - mu) / sigma from -38 to 8 in steps of 0.25. Worst relative error in ulp against mpmath (100 digits; partials by mpmath.diff), develop / original PR / after 0d4ac36:

function z value d/dy
std_normal_cdf [-38, -30) 1500 / 950 / 1500 1000 / 950 / 1.7
std_normal_cdf [-10, -3) 3.5e5 / 51 / 89 65 / 51 / 1.5
std_normal_cdf [-3, 3) 60 / 6.3 / 9.8 6.3 / 3.3 / 3.2
normal_cdf [-30, -10) 1700 / 1200 / 1100 1700 / 1200 / 990
normal_cdf [-10, -3) 3.5e5 / 120 / 160 63 / 120 / 87
normal_cdf [-3, 3) 59 / 5 / 11 10 / 4.6 / 5.9
lognormal_cdf [-30, -10) 1700 / 1200 / 1100 returns 0 / 1200 / 990
lognormal_cdf [-10, -3) 62 / 120 / 150 64 / 110 / 85
lognormal_cdf [-3, 3) 12 / 4.7 / 10 9.2 / 3.2 / 4.9

The d/dmu and d/dsigma errors have the same size as d/dy, except near z = 0, where the lognormal_cdf d/dsigma crosses zero; its absolute error there is 2e-18 in all three versions. The commit keeps the fixes of the original PR: develop's 3.5e5 ulp values in std_normal_cdf and normal_cdf, and the zero lognormal_cdf gradients below z = -10. No result is NaN or infinite. Compared with exp(lcdf), the values are up to 2 times less accurate, from the rounding of the erfc argument that Phi has too, and the gradients are equal or better.

Speed

Same machine and flags as for Phi, three trees alternated, 3 runs × 3 rounds. K is the number of elements per call; with var, all arguments are var and one reverse pass covers 4096 elements. Time relative to develop, original PR / after 0d4ac36:

function range K double var
std_normal_cdf [-3, 3] 1 1.92 / 0.98 1.11 / 1.02
std_normal_cdf [-3, 3] 64 1.27 / 0.97 0.89 / 0.95
std_normal_cdf [-8, -3] 1 1.48 / 0.89 1.02 / 1.02
normal_cdf [-3, 3] 1 1.72 / 1.02 1.06 / 1.01
normal_cdf [-3, 3] 64 1.10 / 0.91 1.19 / 1.10
normal_cdf [-8, -3] 1 1.45 / 0.90 0.98 / 0.96
lognormal_cdf [-3, 3] 1 1.50 / 1.02 1.05 / 1.06
lognormal_cdf [-3, 3] 64 0.85 / 0.70 1.50 / 1.44
lognormal_cdf [-8, -3] 1 1.45 / 1.01 1.02 / 1.03

The round-to-round spread reached 14 % in some rows, so differences below about 0.1 are not significant. With double arguments all three are back at develop speed. One slowdown is left: lognormal_cdf of a vector with var arguments takes about 1.5 times as long as on develop, with either form, so the extra time is not in the erfc or exp(lcdf) code. I have not found where it is.

Tests

On this PR's head bf5861a plus 0d4ac36:

  • CPU, 19 unit suites, all pass: prim and mix std_normal_cdf_test and normal_cdf_test, mix lognormal_cdf_test (with this PR's tail and y = 0 checks), and the std_normal_lcdf_impl, Phi, inv_Phi, lcdf, lccdf, exp_mod_normal_cdf and skew_normal suites from before.
  • The generated test/prob tests of the three cdfs (runTests.py test/prob/normal test/prob/lognormal test/prob/std_normal -f _cdf_0): 17 test programs, 12000 cases, all pass.
  • OpenCL on a Tesla V100: opencl/rev std_normal_cdf_test, normal_cdf_test, lognormal_cdf_test and Phi_test pass. They compare the OpenCL results with the changed CPU code.
  • A compile check instantiates the three functions with fvar<fvar<var>>, std::vector<var>, Eigen vectors of fvar<double>, var_value column and row vectors, and mixed scalar and vector arguments.
  • clang-format 10 and cpplint report nothing on the changed files.

@stan-buildbot

stan-buildbot commented Oct 20, 2026

Copy link
Copy Markdown
Contributor
Name Old Result New Result Ratio Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan 0.22 0.23 0.98 -2.14% slower
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan 0.06 0.06 1.0 -0.29% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan 6.3 6.25 1.01 0.74% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 0.96 -3.85% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan 8.23 7.04 1.17 14.39% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan 4.45 4.47 0.99 -0.5% slower
stat_comp_benchmarks/benchmarks/sir/sir.stan 217.21 218.34 0.99 -0.52% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan 0.89 0.89 1.0 0.16% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan 0.61 0.62 1.0 -0.2% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan 42.45 42.64 1.0 -0.43% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan 0.59 0.59 1.01 0.98% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 20.93 20.82 1.01 0.54% faster
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan 0.11 0.11 1.01 0.65% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan 3.17 3.17 1.0 -0.02% slower
performance.compilation 365.45 365.67 1.0 -0.06% slower
Mean result: 1.0080687934934953

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information 
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           43 bits physical, 48 bits virtual
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              23
Model:                                   49
Model name:                              AMD EPYC 7742 64-Core Processor
Stepping:                                0
Frequency boost:                         enabled
CPU MHz:                                 1496.647
CPU max MHz:                             3416.0681
CPU min MHz:                             1500.0000
BogoMIPS:                                4491.85
Virtualization:                          AMD-V
L1d cache:                               4 MiB
L1i cache:                               4 MiB
L2 cache:                                64 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@avehtari

avehtari commented Oct 20, 2026

Copy link
Copy Markdown
Member

Temporaries that are read more than once are now evaluated once

The vector lognormal_cdf slowdown from my previous comment has a simple cause, and the same cause made three lcdfs slower after this PR was opened. Commits f4cd10e and c30ee6f fix it. They change only when Eigen evaluates an expression, so all results are bitwise identical.

In lognormal_cdf, z = (log(y) - mu) / sigma was a lazy Eigen expression. Eigen evaluated it when the kernel copied it into an array, and again for the sigma partial, so each element paid for log twice. lognormal_lcdf_impl, which lognormal_lccdf also uses, had the same z. f4cd10e wraps both in eval(...).

To find the lcdf slowdown, I timed every commit of this PR with vectors of 1000 elements and var arguments. One commit is responsible: 9a8deff ("Refactor normal log-CDF dispatch and density expressions") replaced evaluated temporaries with lazy expressions. Among them are z and scaled_slope in normal_lcdf_impl and lognormal_lcdf_impl, and log_b, lp, s_b and q in exp_mod_normal_lcdf_impl. Each of these is read two to four times, and Eigen repeats the work at every read: in exp_mod_normal_lcdf, lp = log_diff_exp(log_a, log_b) ran three times per element. c30ee6f evaluates these temporaries once, as the code did before 9a8deff. With double arguments each temporary is read once, so those paths did not change.

ns per element, Eigen::Matrix<var> of 1000 elements with var parameters, value and gradient, z uniform on [-3, 3] (Xeon E5-2680 v3, g++ 14.3.0, Stan default flags, 2 runs × 3 rounds, spread at most 3.7 %):

function develop this PR at a28c1c1 before f4cd10e after c30ee6f
std_normal_lcdf 97.5 42.7 37.5 37.1
normal_lcdf 104.8 48.9 55.9 43.0
lognormal_lcdf 59.3 59.8 65.8 54.7
exp_mod_normal_lcdf 91.2 129.0 218.5 115.3

std_normal_lcdf did not have the defect; its row shows the effect of the shared kernel alone. For lognormal_cdf of 64-element vectors with var arguments, the time relative to develop went from 1.29 to 1.06 with f4cd10e; when only y is var, so that no sigma partial is computed, it was already 1.02.

exp_mod_normal_lcdf is still slower than develop (0.79 of develop's speed with var, 0.72 with double). It evaluates two log cdfs and a log_diff_exp per element. Develop's form is faster but returns NaN or infinite values in large parts of the parameter space.

9a8deff also made exp_mod_normal_lpdf and skew_normal_lpdf 13 to 17 % slower with var, but there the faster kernel from 5f33a10 already makes up the difference, so I did not change them. neg_binomial_2_lpmf has the same pattern with log(mu + phi). Develop has it too, and #3434 already changes the phi partial of that function, so I will fix it there.

Speed of this PR compared with develop and with the table in the description

In short, the PR as opened made the lcdfs faster than develop but Phi and the cdfs slower. Commit 9a8deff then slowed down the var paths. With the commits from my comments, the PR is now faster than develop for every function in the table except exp_mod_normal_lcdf, apart from exp_mod_normal_lpdf and skew_normal_lpdf with var, which are at develop's time. The larger gains in the description most likely come from a slow erf/erfc in the C library of the machine used for it. Commit 9b38ecb removes the dependence of Phi and the cdfs on those two functions.

All tables give the time relative to develop: develop is 1.00, and a value below 1.00 means faster. The setup follows the description: vectors of N = 1000, Eigen::VectorXd with double parameters, and Eigen::Matrix<var> with var parameters and the gradient. The description does not give its inputs; I used z uniform on [-3, 3]. I ran all four versions in one job on an AMD EPYC 9965 (Zen 5, one whole node), with g++ 11.5, glibc 2.34 and the Stan default flags, 2 runs × 2 rounds, alternated; the spread between runs was at most 5.5 %. The columns are:

  • description: 1 / the speedup in the description (its N = 1000 column);
  • as opened: a28c1c1;
  • before my commits: 686fc67, which includes 9a8deff;
  • now: 9b38ecb, which includes all my commits (5f33a10 to 9b38ecb).

double:

function description as opened before my commits now
Phi 0.61 2.27 2.18 0.73
std_normal_cdf 0.53 1.44 1.43 0.82
normal_cdf 0.47 1.35 1.33 0.77
lognormal_cdf 0.45 1.03 1.07 0.77
std_normal_lcdf 0.37 0.93 0.91 0.80
normal_lcdf 0.39 0.89 0.88 0.78
lognormal_lcdf 0.38 0.82 0.85 0.74
exp_mod_normal_lcdf 0.66 1.62 1.67 1.62
exp_mod_normal_lpdf 0.42 0.98 0.93 0.83
skew_normal_lpdf 0.40 1.04 1.00 0.88

var (value and gradient):

function description as opened before my commits now
Phi 0.69 1.57 1.56 0.80
std_normal_cdf 0.52 1.00 1.02 0.63
normal_cdf 0.50 0.95 1.19 0.73
lognormal_cdf 0.68 1.26 1.72 0.93
std_normal_lcdf 0.28 0.43 0.44 0.39
normal_lcdf 0.29 0.44 0.56 0.40
lognormal_lcdf 0.63 1.06 1.44 0.97
exp_mod_normal_lcdf 0.75 1.62 2.77 1.38
exp_mod_normal_lpdf 0.53 0.92 1.03 0.98
skew_normal_lpdf 0.51 0.91 1.10 1.04

What the columns show:

  • As opened, the lcdfs were faster, because develop's std_normal_lcdf is the slow Taylor/A&S code. Phi and the cdfs were slower, because exp(log Phi) adds a log and an exp to every call where develop calls only erf or erfc. exp_mod_normal_lcdf was slower too: it evaluates two log cdfs and a log_diff_exp, but develop returns NaN or infinite values in large parts of the parameter space.
  • Before my commits, the double times are the same as at opening, but 9a8deff made the var paths up to 1.7 times slower (exp_mod_normal_lcdf 1.62 → 2.77, lognormal_cdf 1.26 → 1.72). My previous comment explains the cause and the fix (f4cd10e, c30ee6f).
  • Now every function is faster than develop with double, except exp_mod_normal_lcdf. With var the same holds, except that exp_mod_normal_lpdf and skew_normal_lpdf are at develop's time (0.98 and 1.04), not at the 0.92 and 0.91 they had when the PR was opened.

For N = 100 and 10000 the "now" values are within about 0.1 of these; at N = 100000 some var values move toward develop's (std_normal_cdf 0.91, normal_lcdf 0.55). With z on [-30, 30], develop's Taylor code is slower, and std_normal_lcdf and normal_lcdf now take 0.33 of develop's time with double and 0.26 to 0.28 with var, as in the description. On a Xeon E5-2680 v3 with g++ 14.3, the PR as opened showed the same pattern (for example Phi 2.2 and std_normal_cdf 1.3 times develop's time with double).

Why the description shows larger gains

The absolute times show where the difference is. ns per element, double, N = 1000:

function develop, description develop, EPYC PR as opened, description PR as opened, EPYC now, EPYC
Phi 27.2 8.9 16.5 20.3 6.5
std_normal_cdf 26.8 9.3 14.1 13.4 7.6
normal_cdf 28.7 10.1 13.4 13.7 7.8
lognormal_cdf 33.0 17.2 14.7 17.7 13.2
std_normal_lcdf 38.0 14.4 14.0 13.5 11.5
normal_lcdf 34.5 15.4 13.4 13.7 11.9
lognormal_lcdf 36.8 21.6 14.1 17.7 15.9
exp_mod_normal_lcdf 57.2 24.3 37.6 39.3 39.3
exp_mod_normal_lpdf 29.8 13.2 12.6 12.9 10.9
skew_normal_lpdf 30.7 13.3 12.4 13.9 11.7

The PR as opened ran at about the same speed on both machines (0.8 to 1.1 times the EPYC time), but develop took 1.7 to 3.1 times as long on the machine of the description. The largest factors (2.8 to 3.1) are for Phi, std_normal_cdf and normal_cdf, whose develop code is little more than an erf or erfc call per element. That fits a C library whose erf and erfc are much slower than glibc's. So the description measured a slow develop, not a faster branch. @andrjohns, which OS, compiler and CPU did you measure on?

On such a platform the erfc forms of Phi and the cdfs from bf5861a and 0d4ac36 would also be slow. Commit 9b38ecb therefore computes them without the C library's erf and erfc. With u = x / sqrt(2):

  • for |u| < 0.46875, Phi is 0.5 + 0.5 * erf(u) with the Cody (1969) rational for erf that this PR used before 5f33a10 (no exp; Phi is near 1/2, so nothing cancels);
  • otherwise it is 0.5 * exp(-(0.5 * x) * x) * erfcx(|u|) for x < 0, and one minus that for x > 0, with the erfcx rationals from erfcx.hpp.

Only exp comes from the C library. internal::std_normal_cdf_value_ratio, which std_normal_cdf, normal_cdf and lognormal_cdf use, computes the same values (bitwise equal to Phi) and the ratio phi / Phi as sqrt(2 / pi) / erfcx(-u) for x < 0 and in the central range; that needs no exp and has no 0 / 0 where Phi underflows. The OpenCL Phi device function uses the same formula, so the Phi and inv_Phi registrations include erfcx_device_function again. On glibc this form is also faster than develop's erf/erfc code, as the "now" column shows; on two Xeons (E5-2680 v3 and Gold 6248) Phi(double) takes 0.66 to 0.86 of develop's time over the input ranges I tried.

The new form is more accurate in the lower tail. Worst relative error of Phi in ulp against mpmath, 47001 points on [-38.5, 8.5]:

x develop original PR (exp(lcdf)) erfc (bf5861a) erfcx (9b38ecb)
[-38.5, -30) (cut off at -37.5) 1822 1704 515
[-30, -20) 1224 922 1224 258
[-20, -10) 470 440 470 130
[-10, -5) 118 93 118 34
[-5, -1) 766133 19.1 30.5 12.5
[-1, 0) 3.2 4.6 3.0 3.9
[0, 8.5) 1.3 2.6 1.3 1.4

In the tail the only error that grows with |x| is now one rounding of x * x; the erfc forms also round the argument x / sqrt(2), and erfc magnifies that. The ratio of the cdf helper improves more: on [-30, -1) from up to 1420 ulp to 4 to 7 ulp, and below -37.5, where Phi is subnormal and the erfc helper returned a wrong ratio, it is now within 4 ulp. std_normal_cdf follows Phi (worst value error 1500 ulp before, 2.9 now); normal_cdf and lognormal_cdf improve less, because the rounding of (y - mu) / sigma dominates there.

@avehtari

avehtari commented Oct 20, 2026

Copy link
Copy Markdown
Member

9a8deff also made skew_normal_lpdf slower with var arguments, for the same reason as the lcdfs in my previous comment: z and score became lazy expressions, and the partials and the sum read z up to five times and score up to three times. New commit 927099e evaluates both once in the autodiff branch. The results are bitwise identical.

A first version also evaluated z in the double path. That made double calls 7 % slower, because there z is read only twice and the allocation costs more than the second read. So the commit keeps z lazy in that path.

Time relative to develop (develop = 1.00, below 1.00 is faster), Xeon Gold 6248, g++ 14.3, vectors of N = 1000, z uniform on [-3, 3], 2 runs × 3 rounds:

skew_normal_lpdf as opened (a28c1c1) before (9b38ecb) after (927099e)
double 0.97 0.87 0.86
var 0.84 0.96 0.85

I also timed the OpenCL device functions on a Tesla V100, with each work item walking through the input range in registers and a baseline kernel subtracted. The device Phi from 9b38ecb is 2 to 4 times faster than the exp(lcdf) form of the original PR, and the device std_normal_lcdf_impl and its derivative are as fast as or faster than then. In [-5, 8] the device Phi takes up to 1.8 times as long as develop's, which uses erf there and loses up to 9e5 ulp on [-5, -1); the built-in erfc was slower than the current form on the V100. The difference is about 6 ps per evaluation, about 1 % of the cost per element of a Stan OpenCL call, so I left the device code as it is.

exp_mod_normal_lpdf has the same lazy temporaries, but evaluating them made it 3 % slower with var on the same machine, and it is already at its time from when the PR was opened (0.87), so I did not change it.

exp_mod_normal_lcdf, _lccdf and _cdf

New commit a17bba2 replaces the computation of these three functions on the CPU. The previous form, log F = log_diff_exp(log Phi(z), b) with b = v^2/2 - v z + log Phi(z - v) and v = lambda * sigma, has two failures at this PR's head (grid of z in [-1e4, 100] and v in [1e-8, 100], against mpmath):

function region value dy dsigma dlambda
lcdf left tail (z <= -30) 2.9e5 ulp 3.4e7 3.4e7 1.5e13
lcdf small v (<= 1e-4) non-finite (10 points)
lccdf small v 1.4e12 1.3e12 1.3e12 6.0e2

In the left tail both terms are of size z^2/2; for small v they are nearly equal. At (z, v) = (-1e4, 0.01) the lcdf value is about -5e7 with an absolute error of 1.6e-3.

With u = -z / sqrt(2), t = u + v / sqrt(2), E = erfcx and g = exp(-z^2/2) / 2, Phi(z) = g E(u) and exp(v^2/2 - v z) Phi(z - v) = g E(t), so F = g (E(u) - E(t)) and the z^2/2 terms cancel exactly. All partials are ratios of E(u), E(t), E'(t) and E(u) - E(t). The difference E(u) - E(t) is computed directly when E(t) < 0.99 E(u), and otherwise from a series in the derivatives of erfcx at the midpoint, or for arguments of at least 4 from the difference of the Cody tail rational; the rounded t - u is never used. For z > 0 the code uses F = 1 - (Phi(-z) + b) where F >= 1/4, and a form with two positive terms where F < 1/4. The three functions share one scalar loop.

Worst error over the value and the four partials after the change, no non-finite result: 510 ulp (lcdf) and 480 ulp (lccdf) on the grid above, and 640 and 900 ulp on a denser grid of 2500 points with |z| in [1e-4, 100] and v in [1e-10, 100]. One exception: dsigma near z = 1/v at large v (3.9e3 ulp at (0.0316, 31.6)), where this partial has a true zero.

The change also makes the functions faster. Time relative to develop, AMD EPYC 9965, vectors of N = 1000, bulk: z on [-3, 3], wide: z on [-30, 30]:

function inputs before double after double before var after var
lcdf bulk 1.43 0.80 1.37 0.94
lcdf wide 2.06 0.94 1.76 1.09
lccdf bulk 1.47 0.88 1.38 0.99
lccdf wide 2.17 0.79 1.91 0.89
cdf bulk 1.68 0.93 1.53 1.00
cdf wide 2.52 1.15 2.03 1.15

A Xeon Gold 6230 gives the same pattern. The cdf is exp(lcdf); develop computes it from erf directly, which is not accurate in the tails.

The OpenCL code still uses the previous form. A port of the new algorithm to the device is in testing, and I will add it as a separate commit.

@avehtari

avehtari commented Oct 20, 2026

Copy link
Copy Markdown
Member

OpenCL port of the exp_mod_normal cdfs

As announced in my previous comment, new commit 39886e0 moves the OpenCL versions of exp_mod_normal_lcdf, _lccdf and _cdf to the algorithm of a17bba2. The new device function exp_mod_normal_lcdf_part(z, v, upper, which) follows the CPU code branch for branewnch. The kernel generator only has functions with one output, so eight binary wrappers (exp_mod_normal_{lcdf,lccdf}_{value,d_z,d_v,d_s}) select the value or one of the three partials, and opencl/prim/exp_mod_normal_lcdf.hpp uses them. Compiled on the host, the device code agrees with the CPU code to 5e-14 relative on the 2500-point grid (1.2e-12 at the dsigma zero); the difference comes from the device erfcx, which evaluates its polynomials in a different order.

Time relative to develop, Tesla V100, matrix_cl (double, value only) and var_value<matrix_cl> with var parameters (value and gradient), N = 1e6; the run-to-run spread was 10 to 60 %:

function inputs before double after double before var after var
lcdf bulk 0.86 0.79 0.80 0.89
lcdf wide 0.58 0.52 0.64 0.85
lccdf bulk 1.13 0.99 1.09 1.21
lccdf wide 1.23 1.14 1.07 1.39
cdf bulk 1.19 1.09 1.05 1.19
cdf wide 1.28 1.20 1.10 1.51

The var path is slower than before because each of the four wrappers runs the whole device function, and the device compiler does not merge the four copies. The extra cost is 1 to 2 ns per element. A dedicated kernel that computes all four outputs in one pass would remove it, but it would not follow the pattern of the other distributions, so I did not write one.

@avehtari

avehtari commented Oct 20, 2026

Copy link
Copy Markdown
Member

@SteveBronder I'm now done improving this PR

@SteveBronder

SteveBronder commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Thanks! I'm running some benchmarks locally to see what is going on in more detail

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SteveBronder SteveBronder SteveBronder left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@andrjohns andrjohns

Labels

code cleanup numerics Numerical issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

normal_lcdf and normal_lccdf give infinite gradients for infinite inputs.

6 participants

@andrjohns @avehtari @stan-buildbot @spinkney @SteveBronder @yashikno