Skip to content

conformance: Add conformance tests for implementation and platform checks. #2375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Josverl wants to merge 10 commits into
base: main
Choose a base branch
from
+2,197 −27
Draft

conformance: Add conformance tests for implementation and platform checks. #2375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
dea1cd3
Update specification for directives for sys.implementation and sys.pl…
Josverl Aug 15, 2026
51e52c5
Update docs/spec/directives.rst
Josverl Aug 15, 2026
5128e74
Clarify language in directives regarding named attribute usage in ver…
Josverl Aug 15, 2026
f928655
spec/directives: Process review comments.
Josverl Aug 15, 2026
bc8affa
spec/directives: Process additional review comments.
Josverl Sep 8, 2026
41f0b65
spec/directives: Process additional review comments.
Josverl Sep 21, 2026
a81264b
Address review comments and suggestions.
Josverl Oct 19, 2026
9f941e3
Add additional version or platform conformance tests.
Josverl Oct 23, 2026
13933ad
Document version and platform checking conformance approach.
Josverl Oct 23, 2026
41941f2
Conformance: Add conformance results for implementation and platform …
Josverl Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions conformance/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,32 @@ To add a new test, create a new ".py" file in the `tests` directory. Its name mu

If a test is updated (augmented or fixed), the process is similar to when adding a new test. Run the conformance test tool to generate new results and manually examine the output of each supported type checker. Then update the conformance status accordingly. Once the conformance status has been updated, rerun the conformance test tool to regenerate the summary report.

## Version and Platform Checking Tests

The version and implementation checks use the suite's Python 3.12 CPython target,
with no implementation override. Tests cover both reachable and unreachable
branches: intentional errors in reachable branches must be reported, errors in
unreachable branches must be suppressed, and `assert_type` checks verify branch
selection. Existing three-element version comparisons and `os.name` checks remain
optional; tuple, set, and list membership checks are required.

Platform tests use nonexistent platform names for portable nonmatches and guard
positive Linux membership checks with `sys.platform == "linux"`. Tagged errors in
the outer `else` branch keep the expectations valid on non-Linux targets. Run on a
Linux target to exercise the positive membership cases. Prefix checks include the
empty prefix (always a match) and a nonexistent prefix (never a match). A guarded
`freebsd13` scenario tests a matching `freebsd` prefix and a nonmatching `linux`
prefix; run with a `freebsd13` target to exercise these branches.

The ordinary suite does not validate all target configuration mechanisms. Separate
targeted runs are needed for Windows/Linux selection, the versioned FreeBSD
scenario, and non-CPython implementation versions distinct from the language version.
The current runner has no target-scenario matrix, and the checker CLIs examined
for this change expose language-version/platform options but no implementation-name
or implementation-version options. Non-CPython configuration coverage is therefore
deferred until suitable checker support is available; missing implementation-check
support is still recorded in the ordinary conformance results.

## Updating a Type Checker

Type checker versions are locked in `uv.lock`.
Expand Down
74 changes: 74 additions & 0 deletions conformance/results/mypy/directives_implementation_name.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
conformant = "Unsupported"
notes = """
Does not prune branches for sys.implementation.name equality or membership checks.
"""
conformance_automated = "Fail"
errors_diff = """
Line 21: Unexpected errors ['directives_implementation_name.py:21: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 23: Unexpected errors ['directives_implementation_name.py:23: error: Expression is of type "int | str", not "int" [assert-type]']
Line 27: Unexpected errors ['directives_implementation_name.py:27: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 32: Unexpected errors ['directives_implementation_name.py:32: error: Expression is of type "int | str", not "int" [assert-type]']
Line 36: Unexpected errors ['directives_implementation_name.py:36: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 42: Unexpected errors ['directives_implementation_name.py:42: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 44: Unexpected errors ['directives_implementation_name.py:44: error: Expression is of type "int | str", not "int" [assert-type]']
Line 54: Unexpected errors ['directives_implementation_name.py:54: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 56: Unexpected errors ['directives_implementation_name.py:56: error: Expression is of type "int | str", not "int" [assert-type]']
Line 60: Unexpected errors ['directives_implementation_name.py:60: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 65: Unexpected errors ['directives_implementation_name.py:65: error: Expression is of type "int | str", not "int" [assert-type]']
Line 68: Unexpected errors ['directives_implementation_name.py:68: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 74: Unexpected errors ['directives_implementation_name.py:74: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 84: Unexpected errors ['directives_implementation_name.py:84: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 86: Unexpected errors ['directives_implementation_name.py:86: error: Expression is of type "int | str", not "int" [assert-type]']
Line 90: Unexpected errors ['directives_implementation_name.py:90: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 95: Unexpected errors ['directives_implementation_name.py:95: error: Expression is of type "int | str", not "int" [assert-type]']
Line 98: Unexpected errors ['directives_implementation_name.py:98: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 104: Unexpected errors ['directives_implementation_name.py:104: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 114: Unexpected errors ['directives_implementation_name.py:114: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 116: Unexpected errors ['directives_implementation_name.py:116: error: Expression is of type "int | str", not "int" [assert-type]']
Line 120: Unexpected errors ['directives_implementation_name.py:120: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 125: Unexpected errors ['directives_implementation_name.py:125: error: Expression is of type "int | str", not "int" [assert-type]']
Line 128: Unexpected errors ['directives_implementation_name.py:128: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 134: Unexpected errors ['directives_implementation_name.py:134: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
"""
output = """
directives_implementation_name.py:18: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:21: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:23: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:27: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:30: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:32: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:36: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:39: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:42: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:44: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:51: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:54: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:56: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:60: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:63: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:65: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:68: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:70: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:72: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:74: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:81: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:84: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:86: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:90: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:93: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:95: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:98: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:100: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:102: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:104: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:111: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:114: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:116: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:120: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:123: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:125: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_name.py:128: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:130: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:132: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_name.py:134: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
"""
25 changes: 25 additions & 0 deletions conformance/results/mypy/directives_implementation_version.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
conformant = "Unsupported"
notes = """
Does not prune branches for sys.implementation.version comparisons using the CPython target version.
"""
conformance_automated = "Fail"
errors_diff = """
Line 21: Unexpected errors ['directives_implementation_version.py:21: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 23: Unexpected errors ['directives_implementation_version.py:23: error: Expression is of type "int | str", not "int" [assert-type]']
Line 27: Unexpected errors ['directives_implementation_version.py:27: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 32: Unexpected errors ['directives_implementation_version.py:32: error: Expression is of type "int | str", not "int" [assert-type]']
Line 37: Unexpected errors ['directives_implementation_version.py:37: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
Line 39: Unexpected errors ['directives_implementation_version.py:39: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]']
"""
output = """
directives_implementation_version.py:18: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:21: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:23: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_version.py:27: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:30: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:32: error: Expression is of type "int | str", not "int" [assert-type]
directives_implementation_version.py:35: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:37: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:39: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
directives_implementation_version.py:41: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment]
"""
Loading