ssl.SSLSocket.read() / write() missing ERR_clear_error() before SSL_read_ex() / SSL_write_ex() causes spurious errors with cooperative threading

Bug report

Bug description:

Summary

_ssl__SSLSocket_read_impl and _ssl__SSLSocket_write_impl in Modules/_ssl.c do not call ERR_clear_error() before SSL_read_ex() / SSL_write_ex(). This allows stale entries on the per-thread OpenSSL error queue to corrupt the result of SSL_get_error(), causing spurious BrokenPipeError or OSError on healthy SSL connections.

Affected versions

All current CPython versions. Confirmed in 3.12 branch (line 2544) and main / 3.15-dev (line 2941).

Root cause

The do { ... } while() retry loop in _ssl__SSLSocket_read_impl (Modules/_ssl.c L2939-2942 on main):

do {
    Py_BEGIN_ALLOW_THREADS;
    retval = SSL_read_ex(self->ssl, mem, (size_t)len, &count);
    err = _PySSL_errno(retval == 0, self->ssl, retval);
    Py_END_ALLOW_THREADS;
    // ...
} while (err.ssl == SSL_ERROR_WANT_READ || err.ssl == SSL_ERROR_WANT_WRITE);

_PySSL_errno() calls SSL_get_error(ssl, retcode), which internally calls ERR_peek_last_error(). Per the OpenSSL documentation:

In addition to ssl and ret, SSL_get_error() inspects the current thread's OpenSSL error queue. Thus, SSL_get_error() must be called in the same thread that performed the TLS/SSL I/O operation, and no other OpenSSL function calls should appear in between. The current thread's error queue must be empty before the TLS/SSL I/O operation is attempted, or SSL_get_error() will not work reliably.

If stale error entries are present on the queue from a prior SSL operation (on the same thread but a different SSL object), SSL_get_error() misattributes them and returns SSL_ERROR_SYSCALL instead of the correct SSL_ERROR_WANT_READ.

The same issue exists in _ssl__SSLSocket_write_impl.

When this manifests

This bug is invisible in multi-threaded programs because each OS thread has its own OpenSSL error queue. It becomes critical in cooperative multitasking frameworks (gevent, eventlet, asyncio with SSL) where multiple coroutines/greenlets share a single OS thread and thus a single OpenSSL error queue.

Concrete scenario (gevent):

Greenlet A performs an SSL write on an HTTPS connection. The remote client has disconnected, so SSL_write_ex() → send() fails with EPIPE. OpenSSL pushes an error entry onto the (per-thread) error queue. The greenlet handles the exception, but the error queue is not cleared.
The gevent hub switches to Greenlet B, which is an AMQP consumer doing SSL_read_ex() on a healthy RabbitMQ connection.
SSL_read_ex() → recv() returns EAGAIN (no data available — normal for a non-blocking socket).
SSL_get_error() finds the stale error from step 1 via ERR_peek_last_error() and returns SSL_ERROR_SYSCALL instead of SSL_ERROR_WANT_READ.
_PySSL_errno() captures errno = 32 (stale EPIPE from step 1).
CPython exits the retry loop, enters PySSL_SetError(), and raises BrokenPipeError(errno=32, "Broken pipe") on a perfectly healthy connection.

Evidence

Disassembly: The compiled _ssl.cpython-312-x86_64-linux-gnu.so confirms no ERR_clear_error (PLT 0x9050) before SSL_read_ex (PLT 0x93b0) at the call site.
Production telemetry: At the moment of every BrokenPipeError, getsockopt(SO_ERROR) returns 0 (no kernel-level error), and tcpdump shows no FIN/RST from the remote side — the TCP connection is healthy.
Workaround validation: Calling ERR_clear_error() (via ctypes) before every _sslobj.read() in a monkey-patched ssl.SSLSocket.read() completely eliminates the spurious errors. Tested for 15+ minutes under production load with zero errors, after months of constant failures every ~90 seconds.

Proposed fix

Add ERR_clear_error() before SSL_read_ex() and SSL_write_ex() in their respective retry loops:

do {
    Py_BEGIN_ALLOW_THREADS;
    ERR_clear_error();  /* Prevent stale errors from affecting SSL_get_error() */
    retval = SSL_read_ex(self->ssl, mem, (size_t)len, &count);
    err = _PySSL_errno(retval == 0, self->ssl, retval);
    Py_END_ALLOW_THREADS;
    // ...

This matches OpenSSL's documented requirement and is consistent with how CPython already calls ERR_clear_error() in other SSL functions (e.g., _ssl__SSLSocket_do_handshake_impl, _ssl_ctx_new).

PySSL_SetError needs to be updated #115627 (commit ea9a296) — Improved PySSL_SetError handling for SSL_ERROR_SYSCALL, but only in main; does not add ERR_clear_error() before read/write calls.
[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) #127257 (commit 7f707fa) — ERR_LIB_SYS handling improvement, backported to 3.12; does not address this issue.

Reproducer

A minimal reproducer requires two SSL connections on the same OS thread. In pseudocode:

import ssl, socket, gevent

def writer_greenlet():
    """SSL connection that will fail, leaving stale error on queue"""
    ctx = ssl.create_default_context()
    sock = ctx.wrap_socket(socket.socket(), server_hostname="...")
    sock.connect(...)
    # Remote side disconnects
    sock.write(b"data")  # raises BrokenPipeError — leaves stale OpenSSL error

def reader_greenlet():
    """Healthy SSL connection that reads — gets spurious BrokenPipeError"""
    ctx = ssl.create_default_context()
    sock = ctx.wrap_socket(socket.socket(), server_hostname="...")
    sock.connect(...)
    # This should block waiting for data, but instead raises BrokenPipeError
    sock.read(4096)  # BrokenPipeError on a HEALTHY connection

gevent.joinall([
    gevent.spawn(writer_greenlet),
    gevent.spawn(reader_greenlet),
])

Versions

CPython: 3.12.12, also present in main (3.15-dev, commit d14e31e)
OpenSSL: 3.5.1 (also reproducible with 3.0.x, 3.2.x)
OS: RHEL 10.1
gevent: 25.4.1 / 25.8.2

The pseudocode reproducer is schematic — in practice, the trigger requires precise greenlet switching timing. The production scenario (AMQP consumers + HTTPS server in gevent) triggers it reliably every ~90 seconds.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

@kswia

Looks reasonably. Do you want to provide a PR, @kswia?

pythongh-148594: Clear OpenSSL error queue before SSL read/write

If there is a bug, way more _ssl.c code should be affected, since ERR_clear_error() is currently only called on OpenSSL failure. Only newPySSLSocket() and _ssl._SSLContext.load_cert_chain() call ERR_clear_error() before calling OpenSSL functions.

For example, ERR_clear_error() is not called before SSL_write_ex(). Is it also affected?

Concrete scenario (gevent):

Greenlet A performs an SSL write on an HTTPS connection. The remote client has disconnected, so SSL_write_ex() → send() fails with EPIPE. OpenSSL pushes an error entry onto the (per-thread) error queue. The greenlet handles the exception, but the error queue is not cleared.

Would it be possible to clear the error queue at this step?

Hi,

I’d be interested in investigating this further.

Before proposing another patch, I’d like to review the previous PR and determine whether the stale OpenSSL error queue can be reproduced deterministically without depending on gevent timing.

I’ll also check whether the problem is limited to SSL_read_ex() / SSL_write_ex(), or whether the appropriate fix belongs in a shared error-handling path, as suggested in the earlier discussion.

I’ll report back with findings and a test strategy before opening a PR.

@vstinner

Hi @vstinner @serhiy-storchaka,

I spent some time investigating this on current main.

From what I found, OpenSSL's documented contract is indeed that the thread-local error queue should be empty before calling SSL_read_ex(), SSL_write_ex(), and related SSL I/O functions, and that SSL_get_error() must immediately follow the I/O call.

While auditing _ssl.c, I also noticed that the same pattern exists for other SSL I/O operations (such as handshake/shutdown paths), so the problem appears broader than only SSL_read_ex() / SSL_write_ex().

However, I wasn't able to build a deterministic reproducer for the reported spurious BrokenPipeError locally. I experimented with injecting stale OpenSSL error-queue entries and with MemoryBIO/local socket scenarios, but those continued to produce the expected SSLWantReadError.

At this point my impression is that the remaining question is less about the OpenSSL contract itself and more about agreeing on the correct implementation scope and an acceptable deterministic regression test.

I'm happy to continue investigating or prototype a solution once there is agreement on the preferred direction.

I have a deterministic reproducer for this, and an explanation for why the attempt described above kept producing the expected SSLWantReadError.

There are two independent reasons for why injecting a stale entry didn't fire, either of which alone gives a false clean:

The entry must be injected after a completed handshake. Injecting it beforehand does nothing, because OpenSSL's own state_machine() calls ERR_clear_error().
The victim read must not be the first read on a freshly handshaken socket. That first read still runs the post-handshake state machine, which also clears the queue. Measured with a ctypes binding to ERR_get_error(): with a fresh pair per attempt the queue reads 0x4000065 immediately before read(), and read() still raises SSLWantReadError.

Reproducer. Self-contained: it generates its own certificate with the openssl CLI, completes an in-memory TLS 1.3 handshake between two wrap_bio endpoints, and reads with nothing pending — once with a clean queue, then after each of three ordinary public calls. No network, no timing, no gevent.

repro.py

"""
A leaked OpenSSL error-queue entry turns SSLWantReadError into a fatal SSLError.

SSL_get_error(3): "The current thread's error queue must be empty before the
TLS/SSL I/O operation is attempted, or SSL_get_error() will not work reliably."
_ssl.c never clears the queue before SSL_read_ex, and SSL_get_error() consults
ERR_peek_error() BEFORE it looks at the SSL object's rwstate -- so one stale
entry left by an unrelated call turns "feed me more bytes" into "the peer broke
the protocol".

Self-contained: generates its own certificate with the `openssl` CLI, completes
an in-memory TLS 1.3 handshake between two wrap_bio endpoints, then reads with
nothing pending -- once with a clean queue, and once after each of three
ordinary public calls that leave an entry behind.

The FIRST row is the control and MUST print SSLWantReadError.  If it does not,
this build differs from the one under review and the rest means nothing.

TWO THINGS THE PROBE'S SHAPE ENCODES, both measured, both of which produce a
false clean if you get them wrong:

  1. The stale entry must be injected AFTER a completed handshake.  Injecting
     it before does nothing, because OpenSSL's own state machine calls
     ERR_clear_error().
  2. All four rows must run on the SAME socket, and the clean-queue control
     must go first.  On a freshly handshaken socket the first read still runs
     the post-handshake state machine, which also clears the queue -- measured:
     with a fresh pair per row the queue reads 0x4000065 immediately before
     read() and read() still raises SSLWantReadError.  The control read is what
     settles the socket, so it is load-bearing rather than decorative.

Run:  python repro.py
"""

EXPECT = "clean"

import os
import shutil
import ssl
import subprocess
import sys
import tempfile

print("interp:", sys.implementation.name, sys.version.split()[0], "| openssl:", ssl.OPENSSL_VERSION)
if sys.implementation.name != "cpython":
    print("REFUSING TO RUN: not CPython")
    raise SystemExit(0)
if shutil.which("openssl") is None:
    print("REFUSING TO RUN: no `openssl` CLI to build a test certificate")
    raise SystemExit(0)

import _ssl

tmp = tempfile.mkdtemp(prefix="cpy0423-")
CERT = os.path.join(tmp, "keycert.pem")
subprocess.run(
    ["openssl", "req", "-x509", "-newkey", "rsa:2048", "-keyout", CERT, "-out", CERT,
     "-days", "1", "-nodes", "-subj", "/CN=localhost"],
    check=True, capture_output=True,
)


def handshaken_pair():
    sctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    sctx.load_cert_chain(CERT)
    cctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
    cctx.check_hostname = False
    cctx.verify_mode = ssl.CERT_NONE
    cin, cout = ssl.MemoryBIO(), ssl.MemoryBIO()
    sin, sout = ssl.MemoryBIO(), ssl.MemoryBIO()
    c = cctx.wrap_bio(cin, cout, server_hostname="localhost")
    s = sctx.wrap_bio(sin, sout, server_side=True)
    for _ in range(12):
        for a, out, inb in ((c, cout, sin), (s, sout, cin)):
            try:
                a.do_handshake()
            except (ssl.SSLWantReadError, ssl.SSLWantWriteError):
                pass
            data = out.read()
            if data:
                inb.write(data)
        if c.cipher() and s.cipher():
            return c, s
    raise SystemExit("PROBE BROKEN: handshake never completed")


def dirty_nothing():
    pass


def dirty_nid2obj():
    try:
        _ssl.nid2obj(9999999)
    except ValueError:
        pass


def dirty_txt2obj():
    try:
        _ssl.txt2obj("bogus")
    except ValueError:
        pass


def dirty_decode_cert():
    try:
        _ssl._test_decode_cert(os.path.join(tmp, "no-such-file.pem"))
    except Exception:
        pass


c, _s = handshaken_pair()
print("  handshake OK, cipher:", c.cipher()[0])

bad = 0
for label, dirty in (
    ("clean queue (CONTROL)", dirty_nothing),
    ("after _ssl.nid2obj(9999999)", dirty_nid2obj),
    ("after _ssl.txt2obj('bogus')", dirty_txt2obj),
    ("after _test_decode_cert(missing)", dirty_decode_cert),
):
    dirty()
    try:
        c.read(16)
        got = "read returned without raising"
    except ssl.SSLWantReadError as exc:
        got = f"SSLWantReadError: {exc}"
    except ssl.SSLError as exc:
        got = f"SSLError: {exc}"
    ok = got.startswith("SSLWantReadError")
    if not ok and label != "clean queue (CONTROL)":
        bad += 1
    print(f"  {'OK ' if ok else 'BAD'} {label:<34s} -> {got}")

print(f"VERDICT: {bad} of 3 ordinary calls turned a recoverable read into a fatal SSLError")
shutil.rmtree(tmp, ignore_errors=True)

Output on 3.14.3:

interp: cpython 3.14.3 | openssl: OpenSSL 3.5.5 27 Jan 2026
  handshake OK, cipher: TLS_AES_256_GCM_SHA384
  OK  clean queue (CONTROL)              -> SSLWantReadError: The operation did not complete (read)
  BAD after _ssl.nid2obj(9999999)        -> SSLError: [OBJ: UNKNOWN_NID] unknown nid
  BAD after _ssl.txt2obj('bogus')        -> SSLError: [ASN1: FIRST_NUM_TOO_LARGE] first num too large
  BAD after _test_decode_cert(missing)   -> SSLError: [BIO] system lib
VERDICT: 3 of 3 ordinary calls turned a recoverable read into a fatal SSLError

VERDICT: 3 of 3 on 11 interpreters: 3.10.20, 3.11.15, 3.12.13, 3.14.3, 3.14.4, 3.14.6+freethreaded, 3.15.0a6, and release-gil-nojit / debug-gil-nojit / release-ft-nojit / debug-ft-nojit at 3.16.0a0. Not build-gated or OpenSSL-version-gated within that range (host OpenSSL 3.5.5).

The three leakers are ordinary public calls — ssl._ASN1Object.fromname() reaches txt2obj, and ssl._ASN1Object.fromnid() reaches nid2obj. Any library on the same thread can leave the entry: _ssl doesn't have to be the producer, which is why a consumer-side fix is the general one.

The mechanism is that SSL_get_error() consults ERR_peek_error() before it looks at the SSL object's rwstate, so a pre-existing entry wins over SSL_ERROR_WANT_READ. SSL_get_error(3) states the precondition directly: "The current thread's error queue must be empty before the TLS/SSL I/O operation is attempted, or SSL_get_error() will not work reliably."

Regarding the question above about whether SSL_write_ex() is affected too, what is reproduced here is the read victim. The same missing pre-condition clear structurally applies to five further call sites, verified present at these coordinates but not individually reproduced: _ssl.c:2832 (SSL_write_ex), :1099 (SSL_do_handshake), :3100 (SSL_shutdown), :2901 (SSL_pending), :2701 (SSL_sendfile). (The reproduced read victim is :2989, SSL_read_ex.)

One negative that may help bound the scope: sweeping seven context-level entry points (load_verify_locations with cafile and cadata, load_cert_chain with a junk file and with a bad password, load_dh_params, set_groups, set_ciphers) with a clean queue versus a dirty one gave 0 of 7 with a changed message. So this looks like it belongs to the SSL I/O path rather than to the loaders.

Written with AI assistance; every result above was run on a real interpreter with the control arm shown, and the source coordinates were read rather than inferred.

@devdanzin

Hi @devdanzin,

Thanks for sharing the deterministic reproducer and the detailed explanation. I reviewed it, and it clearly explains why my earlier attempts kept producing SSLWantReadError instead of exposing the stale error queue behavior.

I'll reproduce this locally and review the additional SSL I/O call sites you identified. If the implementation direction is still open, I'd be happy to work on a patch and corresponding regression tests for review.

Fix abandoned TLS connections causing unrelated TLS connection teardo…

@zainnadeem786

I encountered what appears to be the same issue in a Slack Socket Mode application on macOS, using CPython 3.14.7 and OpenSSL 3.5.9.

The application uses ordinary OS. threads, without gevent or greenlets. The existing receiver thread retained OpenSSL error-queue entries and repeatedly raised BrokenPipeError immediately after receiving the WebSocket hello message. A fresh Socket Mode client in the same process had an empty error queue and successfully exchanged ping/pong messages during a 12-second diagnostic run.

The runtime inspection used ctypes; I would keep that diagnostic code out of any proposed regression test.

I have read #81891, the discussion on #148597, and the deterministic reproducer posted here on August 4. I have not yet validated that non-ctypes reproducer on current main.

@zainnadeem786, are you still working on a follow-up patch? I would be happy to help validate the reproducer and prepare regression tests using the existing test_ssl infrastructure. Before opening a PR., I would coordinate with anyone already working on this and discuss the implementation scope raised earlier in this issue.

@mahhie418

Hi @mahhie418 I'm not currently working on this you can work on it if you want 🙂

@zainnadeem786

Thanks for letting me know, @zainnadeem786!

I’ll start by validating the non-ctypes reproducer posted above on current main, then work on a regression test using the existing test_ssl infrastructure.

I’ll also investigate the affected SSL. I/O. paths and share the results and proposed patch scope here before opening a PR.

I have now validated the non-ctypes reproducer on main at 21a36aa79ca, using a CPython 3.16 debug build with OpenSSL 3.5.9 on macOS arm64.

Regression tests for read(), write(), and unwrap() fail before the patch and pass with it. They reuse the existing test_ssl infrastructure and certificate fixtures, without requiring ctypes, gevent, external network access, or certificate generation.

For the SSL. I/O. paths, the proposed patch calls ERR_clear_error() immediately before each SSL_read_ex(), SSL_write_ex(), SSL_do_handshake(), SSL_shutdown(), and SSL_sendfile() invocation, including retries. Error classification still immediately follows each operation.

The handshake and sendfile changes follow the same documented precondition. I have not reproduced a stale-error-queue failure in the handshake path, and Kernel TLS sendfile has not been tested on this macOS environment.

With the patch, test_ssl, the reference-leak check (-R 3:3), and the default test suite (-j0) passed, with platform and resource skips in the default suite. No leaks were reported after warm-up, and make patchcheck completed successfully.

I would welcome feedback on whether this is the appropriate scope before opening a PR.

Proposed patch: mahhie418@78574ef

I’ve opened #159641 with the proposed fix and regression tests.

Navigation Menu

Repository navigation

ssl.SSLSocket.read() / write() missing ERR_clear_error() before SSL_read_ex() / SSL_write_ex() causes spurious errors with cooperative threading #148594

Bug report

Bug description:

Summary

Affected versions

Root cause

When this manifests

Evidence

Proposed fix

Related

Reproducer

Versions

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ssl.SSLSocket.read() / write() missing ERR_clear_error() before SSL_read_ex() / SSL_write_ex() causes spurious errors with cooperative threading #148594

Description

Bug report

Bug description:

Summary

Affected versions

Root cause

When this manifests

Evidence

Proposed fix

Related

Reproducer

Versions

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

serhiy-storchaka commented on Apr 15, 2026

vstinner commented on Apr 15, 2026

vstinner commented on Apr 15, 2026

zainnadeem786 commented on Jul 10, 2026

zainnadeem786 commented on Jul 10, 2026

devdanzin commented on Aug 4, 2026

zainnadeem786 commented on Aug 4, 2026

mahhie418 commented on Oct 16, 2026

zainnadeem786 commented on Oct 16, 2026

mahhie418 commented on Oct 16, 2026

mahhie418 commented on Oct 16, 2026

mahhie418 commented on Oct 17, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions