Free-threading: codecs.encode() crashes when a codec search function is unregistered concurrently #159399

Crash report

In the free-threaded build, codecs.encode() and codecs.decode() can crash when another thread calls codecs.unregister(). codec_getitem() in Python/codecs.c, which backs PyCodec_Encoder(), PyCodec_Decoder(), PyCodec_Encode() and PyCodec_Decode(), gets the CodecInfo tuple from _PyCodec_Lookup() and reads the item with PyTuple_GET_ITEM(), which returns a borrowed reference. It then releases the tuple before taking its own reference to the item:

    v = PyTuple_GET_ITEM(codecs, index);
    Py_DECREF(codecs);
    return Py_NewRef(v);

The lookup cache normally keeps the tuple alive, but codecs.unregister() clears it. If that happens between the lookup and the Py_DECREF(), the tuple can hold the last reference to the encoder, which is freed before Py_NewRef(). The caller then calls a freed object.

import codecs
import threading
from functools import partial

def identity(input, errors="strict"):
    return input, len(input)

def search(name):
    if name != "racy":
        return None
    # The returned CodecInfo holds the only references to these callables.
    return codecs.CodecInfo(partial(identity), partial(identity), name=name)

done = threading.Event()

def writer():
    for _ in range(100_000):
        codecs.register(search)
        codecs.lookup("racy")
        codecs.unregister(search)  # clears the lookup cache
    done.set()

def reader():
    while not done.is_set():
        try:
            codecs.encode("x", "racy")
        except LookupError:
            pass

threads = [threading.Thread(target=writer)]
threads += [threading.Thread(target=reader) for _ in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()

On main (d6f2129) built with --disable-gil --with-pydebug --with-address-sanitizer, this crashed in 5 out of 5 runs, each within a few seconds:

==273017==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x593bcce88b94 bp 0x777ae06df6c0 sp 0x777ae06df6b0 T4)
    #0 0x593bcce88b94 in PyType_HasFeature Include/object.h:755
    #1 0x593bcce88b94 in _PyVectorcall_FunctionInline Include/internal/pycore_call.h:95
    #2 0x593bcce894fb in PyVectorcall_Function Objects/call.c:259
    #3 0x593bcce8d88c in _PyObject_Call Objects/call.c:346
    #4 0x593bcce8da06 in PyObject_Call Objects/call.c:373
    #5 0x593bcd158e5b in _PyCodec_EncodeInternal Python/codecs.c:415
    #6 0x593bcd15cffe in PyCodec_Encode Python/codecs.c:497
    #7 0x593bcd349728 in _codecs_encode_impl Modules/_codecsmodule.c:135
SUMMARY: AddressSanitizer: SEGV Include/object.h:755 in PyType_HasFeature

codec_getitem_checked(), a few lines below in the same file, takes the reference before releasing the tuple. The same code is in 3.13, 3.14 and 3.15. The default build is not affected, since nothing can clear the cache between the lookup and the Py_DECREF().

This is the bug described in gh-157077, which was closed together with a batch of other reports (see gh-157198).

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-159399: Fix crash in codecs.encode() on a concurrently unregistered codec #159400