Skip to content

tarfile: a pax size record followed by a GNU long-name header makes tarfile duplicate, invent or skip members #159435

Open

Description

@gpshead

Bug report

impacted versions: since gh-83869 landed (shipped in 3.13.16, 3.14.8, patch releases, still unreleased in the 3.15 branch to become 3.15.1)

Bug report

Bug description

The gh-83869 change (PR GH-18562) rewrote how TarInfo._proc_pax() recomputes the offset of the next header when a pax extended header carries a size record. 3.13.16 Lib/tarfile.py,

lines 1616-1629 vs the prior code (expand to see)
            if "size" in pax_headers:
                # If the extended header replaces the size field,
                # we need to recalculate the offset where the next
                # header starts.
                offset = next.offset + BLOCKSIZE
                if next.isreg() or next.type not in SUPPORTED_TYPES:
                    try:
                        size = PAX_NUMBER_FIELDS["size"](pax_headers["size"])
                    except ValueError:
                        size = 0
                    offset += next._block(size)
                tarfile.offset = offset

            next.offset = self.offset

(3.13.15: offset = next.offset_data, then offset += next._block(next.size).)

next is whatever self._fromtarfile(tarfile) returned for the header that follows the pax header, and the new formula assumes next.offset is the position of the member's own header block, so that its data starts at next.offset + BLOCKSIZE. That is not true when another special header sits between the pax header and the member, or when the member's header is followed by blocks of its own.

verbose claude explanation (expand to see)
  • A GNU L (long name) or K (long link) header: _proc_gnulong() reads the long-name body, fetches the real member, then sets next.offset = self.offset (line 1466), i.e. the position of the L/K header. The computed offset falls short by BLOCKSIZE + _block(len(long-name body)), 1024 bytes for names up to 511 bytes. For a non-regular member no size is added at all, so the offset lands inside the long-name body.
  • A second x header: the inner _proc_pax() ends with next.offset = self.offset (line 1629), so the outer one computes from the inner header's position.
  • An old GNU sparse member (type S) with extension header blocks: _proc_sparse() reads them and puts offset_data after them (lines 1487-1502); the new formula ignores them.

3.13.15 started from next.offset_data, which _proc_builtin() / _proc_sparse() set from the real file position, so none of this mattered; it was wrong in the GNU sparse 1.0 case that gh-83869 fixed, where _proc_gnusparse_10() has already moved offset_data past the sparse map (and next.size may be the file's apparent size). The opposite order (L, then x, then the member) is unaffected, and so is an x header without a size record, unless a size is inherited from an earlier global g header.

What happens with the sequence x(size=…), L, member, in the random-access modes ("r", "r:…"; not "r|…"), for iteration, getmembers() and extractall() alike (extractall() walks the same member list, with or without filter="data"):

  • With a pax size of 1 to 512 bytes (for a name up to 511 bytes) the next header is read from the member's own header block again, so the member is listed and extracted twice: once under its long name and once under the name in its own ustar header (member-short in case A below; in a real archive, the truncated long name).
  • Whatever bytes sit at the computed offset are parsed as a header. A valid header block placed in the long-name body after the NUL-terminated name, or inside the member's data, becomes a member that GNU tar, bsdtar and Go's archive/tar do not see, and is extracted (case C).
  • With a larger pax size the offset lands inside the member's data; if those bytes are not a valid header, TarFile.next() treats that as end of archive (invalid header at a non-zero offset, with the default ignore_zeros=False) and every following member is silently missing from the listing and the extraction (case B). A pax size of 0 does the same, landing in the long-name text.
  • With the pair x(size=0), L nested n times, each step puts the offset back inside the enclosing long-name body and the rest of the archive is read again as the next long-name body, so listing reads about n/2 times the archive size in total and yields n members where 3.13.15 yields one: a 6 MB archive takes seconds, and the same archive gzipped is about 40 KB and several times slower (case D).

In stream mode ("r|…") the computed offset is usually behind the stream position, so next() raises StreamError("seeking backwards is not allowed") (cases A and C). Once the pax size, in whole blocks, is at least 512 bytes more than the long-name body (for a name up to 511 bytes: any pax size above 512) the offset is not behind, and a stream listing goes wrong silently instead (case B); a stream extractall() reads the member's data first and so still ends in StreamError.

GNU tar 1.35, bsdtar (libarchive 3.7.2) and Go's archive/tar (1.27) list archives A, B and C from the code below as 3.13.15 does: the long-named member, then next.txt. So apart from the duplicated and missing members, 3.13.16's tarfile can be shown a different set of members than other readers for the same bytes, which matters wherever one component inspects or lists an archive with tarfile and another extracts it with a different reader, or the reverse. As far as we know no tar writer emits x followed by L for one member (GNU tar and bsdtar write either pax records or GNU long-name headers, not both), so real archives of this shape should be rare; a crafted one is a few lines, below.

Reproducer

Standard library only; it builds the archives in memory from raw 512-byte blocks (expand to see)
"""tarfile: a pax header carrying 'size', followed by a GNU long-name header.
Builds tiny archives in memory from raw 512-byte blocks (stdlib only) and
prints what tarfile lists and extracts."""
import gzip, io, os, sys, tarfile, tempfile, time

B = 512
LONG = "long-name-" * 12 + "file.txt"         # 128 chars: needs a GNU 'L' header

def hdr(name, size, type=b"0", mode=0o644):    # one ustar header block, valid checksum
    b = bytearray(B)
    b[0:len(name)] = name.encode()
    b[100:108] = b"%07o\0" % mode
    b[108:116] = b[116:124] = b"%07o\0" % 0
    b[124:136] = b"%011o\0" % size
    b[136:148] = b"%011o\0" % 0
    b[148:156] = b" " * 8
    b[156:157] = type
    b[257:265] = b"ustar\x0000"
    b[148:156] = b"%06o\0 " % sum(b)
    return bytes(b)

def blk(data):                                 # pad to whole blocks
    return data + b"\0" * (-len(data) % B)

def pax(**kv):                                 # 'x' extended header + records
    body = b""
    for k, v in kv.items():
        rec = f" {k}={v}\n".encode()
        n = len(rec) + 1
        while n != len(rec) + len(str(n)):
            n = len(rec) + len(str(n))
        body += str(n).encode() + rec
    return hdr("././@PaxHeader", len(body), b"x") + blk(body)

def longname(name, extra=b""):                 # GNU 'L' header + body (+ bytes planted after the name)
    body = blk(name.encode() + b"\0") + extra
    return hdr("././@LongLink", len(body), b"L") + body

def reg(name, data):                           # regular file member
    return hdr(name, len(data)) + blk(data)

EOF = b"\0" * (2 * B)
cases = {
    "A: x(size=5) L member next":
        pax(size=5) + longname(LONG) + reg("member-short", b"hello") + reg("next.txt", b"bye") + EOF,
    "B: same with size=1536":
        pax(size=1536) + longname(LONG) + reg("member-short", b"z" * 1536) + reg("next.txt", b"bye") + EOF,
    "C: A with a header block planted in the long-name body":
        pax(size=5) + longname(LONG, extra=hdr("PLANTED", 0, mode=0o4755))
        + reg("member-short", b"hello") + reg("next.txt", b"bye") + EOF,
}

def show(names):
    return ["LONG" if n == LONG else n for n in names]

print("Python", sys.version.split()[0])
for label, buf in cases.items():
    print(label)
    for mode in ("r:", "r|"):
        try:
            with tarfile.open(fileobj=io.BytesIO(buf), mode=mode) as tf:
                names = show(ti.name for ti in tf)
            print(f"   list {mode}  {names}")
        except tarfile.TarError as e:
            print(f"   list {mode}  {type(e).__name__}: {e}")
    with tempfile.TemporaryDirectory() as d:
        with tarfile.open(fileobj=io.BytesIO(buf), mode="r:") as tf:
            tf.extractall(d, filter="data")
        print(f"   extractall  {show(sorted(os.listdir(d)))}")

# D: listing time. n nested levels, each = x(size=0) + an 'L' header whose
# body claims everything up to the final member's header.
def nested(n):
    out = b""
    for i in range(n):
        out += pax(size=0) + hdr("././@LongLink", (n - 1 - i) * 1536 + B, b"L")
    return out + blk(b"name\0") + reg("last", b"") + EOF

def timed(label, buf):
    t = time.perf_counter()
    with tarfile.open(fileobj=io.BytesIO(buf), mode="r:*") as tf:
        count = len(tf.getmembers())
    print(f"D: {label}, {len(buf) / 1e6:.2f} MB: {count} member(s) in {time.perf_counter() - t:.2f} s")

for n in (1000, 2000, 4000):
    timed(f"{n} levels", nested(n))
timed("4000 levels gzipped", gzip.compress(nested(4000)))

Output on 3.13.15 (3.14.7 and 3.15.0rc2 print the same member lists and counts):

Python 3.13.15
A: x(size=5) L member next
   list r:  ['LONG', 'next.txt']
   list r|  ['LONG', 'next.txt']
   extractall  ['LONG', 'next.txt']
B: same with size=1536
   list r:  ['LONG', 'next.txt']
   list r|  ['LONG', 'next.txt']
   extractall  ['LONG', 'next.txt']
C: A with a header block planted in the long-name body
   list r:  ['LONG', 'next.txt']
   list r|  ['LONG', 'next.txt']
   extractall  ['LONG', 'next.txt']
D: 1000 levels, 1.54 MB: 1 member(s) in 0.00 s
D: 2000 levels, 3.07 MB: 1 member(s) in 0.00 s
D: 4000 levels, 6.15 MB: 1 member(s) in 0.00 s
D: 4000 levels gzipped, 0.04 MB: 1 member(s) in 0.01 s

Output on 3.13.16 (one Linux x86-64 machine; for D, read the growth rather than the decimals):

Python 3.13.16
A: x(size=5) L member next
   list r:  ['LONG', 'member-short', 'next.txt']
   list r|  StreamError: seeking backwards is not allowed
   extractall  ['LONG', 'member-short', 'next.txt']
B: same with size=1536
   list r:  ['LONG']
   list r|  ['LONG']
   extractall  ['LONG']
C: A with a header block planted in the long-name body
   list r:  ['LONG', 'PLANTED', 'member-short', 'next.txt']
   list r|  StreamError: seeking backwards is not allowed
   extractall  ['PLANTED', 'LONG', 'member-short', 'next.txt']
D: 1000 levels, 1.54 MB: 1000 member(s) in 0.18 s
D: 2000 levels, 3.07 MB: 2000 member(s) in 0.84 s
D: 4000 levels, 6.15 MB: 4000 member(s) in 3.59 s
D: 4000 levels gzipped, 0.04 MB: 4000 member(s) in 15.56 s

LONG stands for the 128-character name. In C, PLANTED is the header block placed in the long-name body after the name (mode 04755 in the archive; the data filter strips the setuid bit on extraction). In B, next.txt is never listed or extracted and no error is raised.

A possible direction

Not a patch, just what we tried (a plain revert to 3.13.15's formula would bring gh-83869 back): keep the gh-83869 idea of adding the pax size to the start of the member's data, but take that start from the member's offset_data as its own _proc_*() method recorded it rather than from next.offset + BLOCKSIZE, i.e. save data_offset = next.offset_data in _proc_pax() right after the self._fromtarfile(tarfile) call (before _proc_gnusparse_10() moves offset_data past the sparse map) and use offset = data_offset at line 1620. With that two-line change to 3.13.16's tarfile.py the reproducer prints the 3.13.15 output, the old-sparse and global-header variants above also read as in 3.13.15, test_sparse_file_10_pax_size (the gh-83869 test) still passes, and the whole of 3.13.16's Lib/test/test_tarfile.py passes. We have not thought about it beyond these cases.

CPython versions tested on:

3.13, 3.14, 3.15

Operating systems tested on:

Linux

Activity

  1. added
    stdlibStandard Library Python modules in the Lib/ directory
    on Oct 14, 2026
  2. commit-msuyog commented on Oct 15, 2026

    @commit-msuyog

    I can reproduce this with the script in the report. I’d like to work on a fix and add tests.

    @gpshead, is anyone already working on this? If not, I’ll open a PR soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions