Add GC benchmarks and Python implementation filtering

doc/benchmarks.rst

@@ -117,6 +117,13 @@ skips the operations after creation.
 Note that this benchmark does not create any reference cycles that the garbage
 collector will need to break to free memory.
 
+The ``btree_gc_only_frozen`` variant freezes the startup heap before creating
+the tree. An ordinary holder references a frozen anchor so the measured full
+collection runs with references into a nonempty permanent generation. Tree
+construction, freezing, and unfreezing are outside timing. The permanent
+generation must be empty when this benchmark starts. This variant runs on
+CPython, which provides the required GC freeze API.
+
 
 chameleon
 ---------
@@ -250,6 +257,56 @@ Create 100,000 point objects which compute ``math.cos()``, ``math.sin()`` and
    than testing performance of class attributes.
 
 
+gc_collect_large, gc_collect_self_lists
+---------------------------------------
+
+``gc_collect_large`` uses the same cyclic graph as ``gc_collect`` with 10,000
+independent bidirectional rings of 21 nodes each, for 210,000 garbage nodes.
+``gc_collect_self_lists`` uses 1,000,000 independent one-element lists that
+each contain themselves. Its small per-object traversal emphasizes the cost
+of processing a large garbage list.
+
+Both benchmarks keep the graph rooted until ``gc.disable()`` is called, then
+time only the explicit full collection. PyPy may still run minor collections
+while automatic major collections and finalizers are disabled. Construction
+and checks are outside timing. ``--cycles`` controls the number of rings or
+self-referential lists.
+
+
+gc_weakref_recheck, gc_resurrection_partial, gc_resurrection_full
+-----------------------------------------------------------------
+
+Collect 10,000 independent rings of 21 nodes each with one callback event:
+
+* ``gc_weakref_recheck`` runs one weakref callback that observes a cleared
+  reference and retains no fixture objects.
+* ``gc_resurrection_partial`` runs one finalizer that retains half the rings.
+* ``gc_resurrection_full`` runs one finalizer that retains every ring.
+
+The resurrection variants add one finalizing node and a list of roots. Timing
+includes the collections needed to reclaim unrescued garbage. If an older
+collector defers all garbage after resurrection, the partial variant includes
+the second collection needed to finish that work. Setup, validation, and the
+final collection of rescued objects are outside timing.
+
+These variants run on CPython and use its integer reclamation counts to verify
+the measured work. ``--cycles`` controls the number of rings and must be even.
+
+
+gc_tuple_survivors
+------------------
+
+Retain 200,000 fresh tuples containing 16 ``None`` items each and time
+``gc.collect(0)``. Construction and cleanup are outside timing. ``--entries``
+and ``--width`` control the population and tuple size.
+
+On GIL-enabled CPython, the retained root is aged before the tuples are created,
+and the young-generation collection can untrack the tuples. Free-threaded
+CPython scans the full GC heap for this call. PyPy also performs a full
+collection and may reclaim construction temporaries, so those results have a
+broader scope than the GIL-enabled CPython result.
+
+
 genshi
 ------
 

doc/custom_benchmarks.rst

@@ -324,21 +324,26 @@ All other PEP 621 fields are optional (e.g. ``requires-python = ">=3.8"``,
 The ``[tool.pyperformance]`` Section
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-=============== ===== === === ===
-field           type  R   B   F 
-=============== ===== === === ===
-tool.name       str   X       X
-tool.tags       [str]     X    
-tool.extra_opts [str]     X    
-tool.inherits   file           
-tool.runscript  file      X    
-tool.datadir    file      X    
-=============== ===== === === ===
+=========================== ===== === === ===
+field                       type  R   B   F
+=========================== ===== === === ===
+tool.name                   str   X       X
+tool.tags                   [str]     X
+tool.python_implementations [str]     X
+tool.extra_opts             [str]     X
+tool.inherits               file
+tool.runscript              file      X
+tool.datadir                file      X
+=========================== ===== === === ===
 
 "R": required
 "B": inferred from the inherited metadata
 "F": inferred from filename
 
 * tags: optional list of names to group benchmarks
+* python_implementations: optional nonempty list of supported names from
+  ``sys.implementation.name``, such as ``["cpython"]``. Names are normalized
+  to lowercase. Omitting this field allows every implementation unless the
+  inherited metadata restricts it. A supplied list replaces the inherited list.
 * extra_opts: optional list of args to pass to ``tool.runscript``
 * runscript: the benchmark script to use instead of run_benchmark.py.

doc/usage.rst

@@ -559,9 +559,16 @@ significant.
 Omitting the ``-b`` option will result in the default group of benchmarks being
 run Omitting ``-b`` is the same as specifying `-b default`.
 
-To run every benchmark pyperformance knows about, use ``-b all``. To see a full
+To select every benchmark eligible for the target interpreter, use ``-b all``. To see a full
 list of all available benchmarks, use `--help`.
 
+The ``run`` and ``list`` commands check compatibility against the interpreter
+selected by ``--python``. A benchmark must support its Python language version
+and, when restricted by metadata, its implementation. Unsupported implementations
+are reported on stderr and omitted from groups and explicit selections. If no
+benchmarks remain, ``run`` reports that no benchmark was run.
+``list_groups`` continues to show declared group membership without filtering.
+
 Negative benchmarks specifications are also supported: `-b -2to3` will run every
 benchmark in the default group except for 2to3 (this is the same as
 `-b default,-2to3`). `-b all,-django` will run all benchmarks except the Django

pyperformance/_benchmark.py

@@ -167,6 +167,11 @@ def extra_opts(self):
     def python(self):
         return SpecifierSet(self._get_metadata_value("python", ""))
 
+    @property
+    def python_implementations(self):
+        """Return supported implementation names, or an empty sequence for all."""
+        return self._get_metadata_value("python_implementations", ())
+
     # Other metadata keys:
     # * base
     # * dependencies

pyperformance/_benchmark_metadata.py

@@ -26,6 +26,7 @@
     "metafile": None,
     "name": None,
     "tags": None,
+    "python_implementations": None,
     "datadir": None,
     "runscript": None,
     "extra_opts": None,
@@ -38,6 +39,7 @@
 #    metafile
 #    tags
 #    python
+#    python_implementations
 #    dependencies  # (from requirements.in)
 #    requirements  # (from lock file or requirements.txt)
 #    datadir
@@ -219,6 +221,18 @@ def _resolve_value(field, value, rootdir):
                 raise ValueError("Invalid tag 'all'")
             elif tag == "":
                 raise ValueError("Invalid empty tag")
+    elif field == "python_implementations":
+        if not isinstance(value, list):
+            raise TypeError("python_implementations should be a list of names")
+        if not value:
+            raise ValueError("python_implementations should not be empty")
+        implementations = []
+        for name in value:
+            if not isinstance(name, str):
+                raise TypeError("python_implementations should be a list of names")
+            _utils.check_name(name)
+            implementations.append(name.lower())
+        value = implementations
     elif field == "datadir":
         if not os.path.isabs(value):
             value = os.path.join(rootdir, value)

pyperformance/cli.py

@@ -313,14 +313,31 @@ def _manifest_from_options(options):
     return _manifest.load_manifest(options.manifest)
 
 
-def _benchmarks_from_options(options):
+def _benchmarks_from_options(options, python):
+    """Resolve selected benchmarks for the interpreter the command will use.
+
+    options: Parsed options containing the manifest and benchmark selection.
+    python: Target executable as a string, or information returned by _pythoninfo.get_info().
+    Return: Eligible benchmarks, or None when no benchmarks were requested.
+    """
     if not getattr(options, "benchmarks", None):
         return None
+
+    from pyperformance import _pythoninfo
+
+    info = _pythoninfo.get_info(python) if isinstance(python, str) else python
     manifest = _manifest_from_options(options)
-    return _select_benchmarks(options.benchmarks, manifest)
+    return _select_benchmarks(options.benchmarks, manifest, info)
+
 
+def _select_benchmarks(raw, manifest, info):
+    """Resolve a selection and omit benchmarks unsupported by the target.
 
-def _select_benchmarks(raw, manifest):
+    raw: Benchmark names and groups in command-line selection syntax.
+    manifest: Catalog used to expand the selection.
+    info: Target interpreter information returned by _pythoninfo.get_info().
+    Return: Eligible benchmarks after explicit exclusions are applied.
+    """
     from pyperformance import _benchmark_selections
 
     # Get the raw list of benchmarks.
@@ -342,14 +359,23 @@ def parse_entry(o, s):
 
     # Get the selections.
     selected = []
-    this_python_version = ".".join(map(str, sys.version_info[:3]))
+    python_version = ".".join(map(str, info.sys.version_info[:3]))
+    implementation = info.sys.implementation.name.lower()
     for bench in _benchmark_selections.iter_selections(manifest, parsed_infos):
         if isinstance(bench, str):
             logging.warning(f"no benchmark named {bench!r}")
             continue
-        # Filter out any benchmarks that can't be run on the Python version we're running
-        if this_python_version in bench.python:
-            selected.append(bench)
+        implementations = bench.python_implementations
+        if implementations and implementation not in implementations:
+            supported = ", ".join(implementations)
+            print(
+                f"Skipping {bench.name}: target implementation {implementation} is not supported (supported: {supported})",
+                file=sys.stderr,
+            )
+            continue
+        if python_version not in bench.python:
+            continue
+        selected.append(bench)
 
     return selected
 
@@ -376,10 +402,10 @@ def _main():
 
         action = options.venv_action
         if action == "create":
-            benchmarks = _benchmarks_from_options(options)
+            benchmarks = _benchmarks_from_options(options, info or sys.executable)
             cmd_venv_create(options, root, info, benchmarks)
         elif action == "recreate":
-            benchmarks = _benchmarks_from_options(options)
+            benchmarks = _benchmarks_from_options(options, info or sys.executable)
             cmd_venv_recreate(options, root, info, benchmarks)
         elif action == "remove":
             cmd_venv_remove(options, root)
@@ -402,12 +428,12 @@ def _main():
         cmd_show(options)
         sys.exit()
     elif options.action == "run":
-        benchmarks = _benchmarks_from_options(options)
+        benchmarks = _benchmarks_from_options(options, options.python)
         cmd_run(options, benchmarks)
     elif options.action == "compare":
         cmd_compare(options)
     elif options.action == "list":
-        benchmarks = _benchmarks_from_options(options)
+        benchmarks = _benchmarks_from_options(options, options.python)
         cmd_list(options, benchmarks)
     elif options.action == "list_groups":
         manifest = _manifest_from_options(options)

pyperformance/commands.py

@@ -178,7 +178,10 @@ def cmd_run(options, benchmarks):
         print('ERROR: "%s" is not an absolute path' % executable)
         sys.exit(1)
 
-    suite, errors = run_benchmarks(benchmarks, executable, options)
+    if benchmarks:
+        suite, errors = run_benchmarks(benchmarks, executable, options)
+    else:
+        suite, errors = None, []
 
     if not suite:
         print("ERROR: No benchmark was run")

pyperformance/data-files/benchmarks/MANIFEST

@@ -28,11 +28,18 @@ base64	<local>
 bpe_tokeniser	<local>
 btree	<local>
 btree_gc_only	<local:btree>
+btree_gc_only_frozen	<local:btree>
 concurrent_imap	<local>
 coroutines	<local>
 coverage	<local>
 gc_traversal	<local>
 gc_collect	<local>
+gc_collect_large	<local:gc_collect>
+gc_collect_self_lists	<local:gc_collect>
+gc_weakref_recheck	<local:gc_resurrection>
+gc_resurrection_partial	<local:gc_resurrection>
+gc_resurrection_full	<local:gc_resurrection>
+gc_tuple_survivors	<local>
 generators	<local>
 chameleon	<local>
 chaos	<local>

pyperformance/data-files/benchmarks/bm_btree/bm_btree_gc_only_frozen.toml

@@ -0,0 +1,4 @@
+[tool.pyperformance]
+name = "btree_gc_only_frozen"
+python_implementations = ["cpython"]
+runscript = "run_frozen_benchmark.py"