Regression segfault under CPython 3.12a5 in the SymPy test suite

[oscarbenjamin]

Crash report

This comes from sympy/sympy#24776 which adds CPython 3.12 prerelease testing in SymPy's CI.

This is seen with CPython 3.12a5 but not with 3.11 or earlier versions.

The reproducer is to run the SymPy test suite:

pip install sympy==1.11.1

Then

>>> import sympy
>>> sympy.test(subprocess=False)
...
(it takes about 5 minutes)
...
sympy/functions/special/tests/test_delta_functions.py[3] ...                                                                  [OK]
sympy/functions/special/tests/test_elliptic_integrals.py[4] ....                                                              [OK]
sympy/functions/special/tests/test_error_functions.py[24] ................Segmentation fault: 11

I don't yet have a simpler reproducer for this because it seems to be non-deterministic but the SymPy test suite reliably invokes a segfault under 3.12 alpha 5 after about 5 minutes. The tests that are running at the time of the segfault will pass if run in isolation. Running the whole test suite though will cause it to fail randomly at one of a few specific places. I don't have a simpler reproducer because running a smaller part of the test suite does not reproduce the problem.

Error messages

Usually:

Segmentation fault: 11

In one case I have also seen (on OSX):

Bus error: 10

Your environment

• CPython versions tested on: 3.12 alpha 5 (the problem is not seen with 3.11 or earlier).
• Operating system and architecture: Ubuntu and OSX, both on x84-64 CPU.

The problem was seen initially in GitHub Actions CI on an Ubuntu 20.04 runner but I have also reproduced it locally in OSX (an Intel-CPU Macbook).

I don't immediately have a setup that I can use to bisect this but I will get one set up soon to narrow this down.

Linked PRs

• gh-102250: Fix double-decref in COMPARE_AND_BRANCH error case #102287

[sobolevn]

@oscarbenjamin do you have time to debug it to help us? Which test exacty fails? Or is it a combination / sequence of tests?

[corona10]

I got another crash from sympy but not sure it's CPython issue or Sympy issue at this moment.
(Buf AFAIK, sympy is pure-python code..)

sympy/core/tests/test_relational.py[64] ...............f.....................f..
...Modules/gcmodule.c:115: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x1064b25e0
object refcount : 6
object type     : 0x145c63630
object type name: floor
object repr     : floor(x)

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x000000010197c580 (most recent call first):
  Garbage-collecting
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/facts.py", line 533 in <genexpr>
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/facts.py", line 533 in deduce_all_facts
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 546 in _ask
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 483 in getit
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/add.py", line 703 in _eval_is_zero
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 539 in _ask
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 483 in getit
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 1561 in is_eq
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 1555 in is_eq
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 629 in __new__
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/tests/test_relational.py", line 967 in test_issue_10927
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 1281 in test_file
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 1146 in test
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 568 in _test
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 498 in test
  File "<stdin>", line 1 in <module>
[1]    21140 abort      python

[OTheDev]

main (sympy.__version__ == '1.11.1'):

Python 3.12.0a5+ (heads/main:a35fd38b57, Feb 25 2023, 23:27:09) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
>>> sympy.test(subprocess=False, seed=96737803)
============================= test process starts ==============================
executable:         /...omitted.../python.exe  (3.12.0-alpha-5) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
numpy:              None
random seed:        96737803
hash randomization: off
...
sympy/functions/combinatorial/tests/test_comb_numbers.py[24] ....zsh: segmentation fault  ./python.exe

Second run

sympy/functions/special/tests/test_bsplines.py[11] .........wzsh: segmentation fault  ./python.exe

Not on 3.11.2 (Feb 8 Gzipped source tarball, sympy.__version__ == '1.11.1') though:

Python 3.11.2 (main, Feb 26 2023, 15:08:59) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
>>> sympy.test(subprocess=False, seed=96737803)
============================= test process starts ==============================
executable:         /...omitted.../python.exe  (3.11.2-final-0) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
numpy:              None
random seed:        96737803
hash randomization: off
...
True

Moreover

Python 3.11.2+ (heads/3.11:7a0dc8a802, Feb 26 2023, 16:22:23) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
...
True

It looks like if CPython is the culprit, it probably should be bisectable in 3.12. Unfortunately, the test suite is quite long! :)

[oscarbenjamin]

I got another crash from sympy but not sure it's CPython issue or Sympy issue at this moment. (Buf AFAIK, sympy is pure-python code..)

The crash you showed suggests this is a segfault arising from a refcounting bug. Yes, SymPy is pure Python. It can be used with other optional dependencies that are not pure Python like numpy, gmpy2 etc but this bug is seen without any of those. A refcounting bug here is definitely a CPython bug.

[oscarbenjamin]

do you have time to debug it to help us? Which test exacty fails? Or is it a combination / sequence of tests?

I am trying to bisect it currently but it's a slow process because of the time taken to build cpython and then run the sympy test suite. Also it is difficult because the failure itself is not completely deterministic. As I said in the OP running individual tests does not reproduce the failure so it is not possible to narrow it down to less than running the whole test suite.

Actually I have narrowed it down a little bit. It seems that the first 1/6 of the test suite is enough to reproduce this:

import sympy
sympy.test(subprocess=False, split="1/6")

That does not reduce the time taken for the crash to show up but it does reduce the time taken to reproduce the crash not happening which is useful for bisecting.

I haven't quite finished bisecting but I'm currently down to this range:

commit df10571a130c46d67756239e76ad69a3d730a990 (refs/bisect/bad)
Author: Steve Dower <steve.dower@python.org>
Date:   Mon Jan 16 16:05:39 2023 +0000

    gh-100320: Fix path calculations on Windows when python.exe is moved outside of the normal location (GH-100947)

commit 7b14c2ef194b6eed79670aa9d7e29ab8e2256a56
Author: Mark Shannon <mark@hotpy.org>
Date:   Mon Jan 16 12:35:21 2023 +0000

    GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983)

commit b1a74a182d8762bda51838401ac92b6ebad9632a
Author: Nikita Sobolev <mail@sobolevn.me>
Date:   Mon Jan 16 13:46:07 2023 +0300

    gh-101056: Fix memory leak in `formatfloat()` in `bytesobject.c` (#101057)

commit b82049993f74185da71adf2eb8d6c8f15db063e1
Author: scrazzz <scruzism@mail.com>
Date:   Mon Jan 16 13:44:50 2023 +0300

    Docs: Remove duplicate `TraversableResources` reference (#101068)

commit 74c20b6ecee125e76c187c3f78c55f39501702ae (HEAD)
Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Date:   Mon Jan 16 16:11:57 2023 +0530

    GH-100997: fix refleak in `_testinternalcapi.c` (#101058)

commit 8e9d08b062bbabfe439bc73f82e3d7bb3800189e
Author: Oleg Iarygin <oleg@arhadthedev.net>
Date:   Mon Jan 16 00:36:31 2023 +0400

    gh-101041: Fix a misspelled name of `utctimetuple` in a doc warning (GH-101042)

commit 5e9df888dd5ab0e59d1cebc30c998a17aa65a3e2
Author: Cristián Maureira-Fredes <cmaureir@users.noreply.github.com>
Date:   Sun Jan 15 16:12:37 2023 +0100

    Fix `ctypes` typo in `set_exception` (#101040)

commit f6307d44167e9c110a1db442f7a33deb365e89a8 (refs/bisect/good-f6307d44167e9c110a1db442f7a33deb365e89a8)
Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Date:   Sun Jan 15 20:39:26 2023 +0530

    GH-100892: consolidate `HEAD_LOCK/HEAD_UNLOCK` macros (#100953)

[OTheDev]

Tests pass in 3.12.0a4 (Head 3d5d3f7, Jan 10)

[oscarbenjamin]

Bisect complete:

sympy/functions/elementary/tests/test_miscellaneous.py[18] ......s...........                                                                             [OK]
sympy/functions/elementary/tests/test_piecewise.py[63] ....ww...w..............................................w......                                    [OK]
sympy/functions/elementary/tests/test_trigonometric.py[86] ./bug.sh: line 12: 30073 Bus error: 10           ./python.exe -c 'import sympy; sympy.test(subprocess=False, split="1/6")'
$ git bisect bad
7b14c2ef194b6eed79670aa9d7e29ab8e2256a56 is the first bad commit
commit 7b14c2ef194b6eed79670aa9d7e29ab8e2256a56
Author: Mark Shannon <mark@hotpy.org>
Date:   Mon Jan 16 12:35:21 2023 +0000

    GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983)

:040000 040000 a01bf959d999386154fb4dfef5d568c0288cb316 dbd2b3a4a61424abfa5f9e726973b9d03bd24bfd M	Doc
:040000 040000 0fac1d9d6fc2e2066a40e527653ecdae70011c23 d3479ece49974dc513a50d3bd1d5684834e95970 M	Include
:040000 040000 58458255181b3044deeb7c7f852c89e37d398508 f24643aa3a7f1afbdd58639b6e4496f59f1da4f9 M	Lib
:040000 040000 3a605b820cd18bd683911c0a274405ab875a6c79 11052d23b2fb46e6cb0964518d8c215dbc5b3bcd M	Misc
:040000 040000 9364b4ad219faab45e789b146bd93c5694865d79 9a9702c99b36e043768cfde5b3dbdb97ba8972f3 M	Objects
:040000 040000 40b1d7e8c2b8c5af2fd1186256c49bebd3e9c954 83b3b65a9a27d8cb683313f6d280e9bfd20f16ca M	Python
:040000 040000 0cae5606a7c50631954bb014dd67b0384f201210 12b0ea6eb5eafad436ccf7564bacd0a72f40ab59 M	Tools

That's 7b14c2e from gh-100983.

I will do a bit more testing with the parent commit (b1a74a1) to confirm that the bug is not seen there.

CC @markshannon

[OTheDev]

Full test suite green for me in b1a74a1.

[markshannon]

Does running the sympy/functions/special/tests/test_error_functions.py test repeatedly by itself cause the segfault?

[OTheDev]

Attempting to run the full test suite at 7b14c2e, I encounter a segfault at

...
sympy/diffgeom/tests/test_diffgeom.py[16] ................                                               [OK]
sympy/diffgeom/tests/test_function_diffgeom_book.py[4] ..zsh: segmentation fault  ./python.exe

However, running sympy/diffgeom/tests/test_function_diffgeom_book.py repeatedly (100 times) doesn't seem to reproduce the segfault - likewise for sympy/functions/special/tests/test_error_functions.py.

[oscarbenjamin]

I will do a bit more testing with the parent commit (b1a74a1) to confirm that the bug is not seen there.

I confirm the full test suite passes with the parent commit so I'm confident that the problem begines with 7b14c2e.

Does running the sympy/functions/special/tests/test_error_functions.py test repeatedly by itself cause the segfault?

I tried something similar to this. Exactly where it crashes is different on different runs but for example it sometimes crashes in the diffgeom tests so I made a script to run just the diffgeom tests 100 times and that didn't give any crash. I'll try again with:

import sympy

for n in range(1000):
    print()
    print('-------------------------------------- n =', n)
    print()
    sympy.test('test_error_functions', subprocess=False)

It's possible that it would be different by setting the environment variable SYMPY_USE_CACHE=no although that makes everything much slower.

How many repetitions should I expect to need?

[dtrodrigues]

The crash @corona10 mentioned is reproducible with --with-pydebug enabled. A simpler reproducer is

from sympy.core.symbol import symbols
x = symbols('x')
assert x > 0

The bytecode for both is below

b1a74a1 (good):

  4           0 RESUME                   0

  5           2 LOAD_GLOBAL              1 (NULL + symbols)
             14 LOAD_CONST               1 ('x')
             16 CALL                     1
             26 STORE_FAST               0 (x)

  6          28 LOAD_FAST                0 (x)
             30 LOAD_CONST               2 (0)
             32 COMPARE_OP              68 (>)
             36 POP_JUMP_IF_TRUE         2 (to 42)
             38 LOAD_ASSERTION_ERROR
             40 RAISE_VARARGS            1
        >>   42 LOAD_CONST               0 (None)
             44 RETURN_VALUE

7b14c2e (bad):

  4           0 RESUME                   0

  5           2 LOAD_GLOBAL              1 (NULL + symbols)
             14 LOAD_CONST               1 ('x')
             16 CALL                     1
             26 STORE_FAST               0 (x)

  6          28 LOAD_FAST                0 (x)
             30 LOAD_CONST               2 (0)
             32 COMPARE_AND_BRANCH      68 (>)
             36 POP_JUMP_IF_TRUE         2 (to 42)
             38 LOAD_ASSERTION_ERROR
             40 RAISE_VARARGS            1
        >>   42 LOAD_CONST               0 (None)
             44 RETURN_VALUE

with the difference between the two being COMPARE_OP vs COMPARE_AND_BRANCH.

I can't cause this alone to cause a segfault when --with-pydebug is not enabled.

Edit: Removing https://github.com/sympy/sympy/blob/sympy-1.11.1/sympy/core/tests/test_relational.py#L860-L869 from the sympy test suite allow sthe subset of the test suite which I was running on (split='1/12') to run without segfaulting anymore. Adding the test back in reintroduces the segfault that appears later on.