Remove public low-level API functions for Py_complex?

[skirpichev]

Historically, in CPython we have functions like _Py_c_sum and _Py_c_prod to do arithmetic with Py_complex (a custom type for complex numbers). Recent pr python/cpython#124829 added support for mixed-mode arithmetic and this API was extended with functions like _Py_cr_prod (i.e. when one argument is a real number).

It was suggested by @vstinner in python/cpython#124829 (comment) either to remove this from the public API or rename properly (e.g. like Py_complex_add() and Py_complex_add_real(), in the GNU GSL style).

With python/cpython#124829 - complex arithmetic in CPython follows to most implementations of the C99+ Annex G. So, users can use native double _Complex type instead of the Py_complex. Removing low-level arithmetic API possibly open a door to eventually also switch CPython to use native complex type internally (like for floats).

I see several arguments to keep these functions.

First, not all Tier 1 PEP 11's platforms have Annex G support: it's Windows with MSVC. I don't know if here are some plans to add such support. On another hand, this platform has a different native complex type, _Dcomplex, with few arithmetic primitives (only multiplication) to do math with.

Second, special arithmetic functions will make sense if we go beyond most Annex G implementations, i.e. add support for pure-imaginary numbers. This was recently suggested: https://discuss.python.org/t/77073/. Probably, we can conclude that this proposal has no enough support.

Third, current Annex G has no special case for real/complex division. So, current CPython complex arithmetic has a small incompatibility with the standard. This is something, going to be fixed by N3460.

CPython issue: python/cpython#128813

[vstinner]

If I understand correctly, this Py_complex API duplicates the Python C API using complex objects. Examples:

• _Py_c_sum() on Py_complex: PyNumber_Add() on complex objects
• _Py_c_pow() on Py_complex: PyNumber_Power() on complex objects
• etc.

I'm not sure why we would have to provide a low-level Py_complex API in addition to Python complex API. The Python C API is already big, and it's mostly based on objects.

The low-level API is private but it's documented which is uncommon: https://docs.python.org/dev/c-api/complex.html#complex-numbers-as-c-structures

I would suggest to deprecate this API, and then move it to the internal C API.

---

I ran a code search on PyPI top 8,000 projects using the regex _Py_c_sum|_Py_c_diff|_Py_c_neg|_Py_c_prod|_Py_c_quot|_Py_c_pow|_Py_c_abs, it only found a single line in a single project:

numba-0.61.0.tar.gz: numba-0.61.0/numba/_helperlib.c: *out = _Py_c_pow(*a, *b);

/* provide complex power */
NUMBA_EXPORT_FUNC(void)
numba_cpow(Py_complex *a, Py_complex *b, Py_complex *out) {
    errno = 0;
    *out = _Py_c_pow(*a, *b); // <========== HERE ============
    if (errno == EDOM) {
        /* _Py_c_pow() doesn't bother returning the right value
           in this case, as Python raises ZeroDivisionError */
        out->real = out->imag = Py_NAN;
    }
}

Link: https://github.com/numba/numba/blob/main/numba/_helperlib.c

_Py_c_pow() is about 30 lines of C code, it can easily be copied in numba if needed.

[davidhewitt]

For what it's worth we moved from _Py_c functions to the PyNumber_x methods in PyO3 about a year ago in PyO3/pyo3#4201

[skirpichev]

API duplicates the Python C API using complex objects. Examples:

_Py_c_sum() on Py_complex: PyNumber_Add() on complex objects
[...]

Generally, they aren't equal performance-wise. But typical scenario could be something like extension of the cmath module, i.e. adding some mathematical functions. I suspect, that here performance impact might be low. I'll do tests.

I ran a code search on PyPI top 8,000 projects using the regex _Py_c_sum|_Py_c_diff|_Py_c_neg|_Py_c_prod|_Py_c_quot|_Py_c_pow|_Py_c_abs, it only found a single line in a single project

This is a good argument for removing, anyway.

[skirpichev]

Here some quick tests on toy sin() implementation (taylor series with fixed number of terms). With high-level API and with _Py_c_* functions, respectively.

$ python -m timeit -s 'from demo import sin_hh as s;z=0.1j' 's(z)'
50000 loops, best of 5: 9.25 usec per loop
$ python -m timeit -s 'from demo import sin_ll as s;z=0.1j' 's(z)'
100000 loops, best of 5: 3.14 usec per loop

This is on 3.13.

Details

/* mod.c */
#include <Python.h>

static PyObject *
sin_ll(PyObject *self, PyObject *arg)
{
    Py_complex z = PyComplex_AsCComplex(arg);

    if (z.real == -1 && PyErr_Occurred()) {
        return NULL;
    }

    Py_complex s = z;
    unsigned long fact = 1;
    Py_complex num = z;
    Py_complex den;
    Py_complex z2 = _Py_c_prod(z, z);
    int sign = 1;

    for (unsigned long i = 1; i < 50;) {
        i += 2;
        fact *= i * (i-1);
        num = _Py_c_prod(num, z2);
        sign *= -1;
        den = (Py_complex) {fact * sign, 0};
        s = _Py_c_sum(s, _Py_c_quot(num, den));
    }
    return PyComplex_FromCComplex(s);
}

static PyObject *
sin_hh(PyObject *self, PyObject *arg)
{
    PyObject *s = PyNumber_Positive(arg);
    unsigned long fact = 1;
    PyObject *num = PyNumber_Positive(s);
    PyObject *den;
    PyObject *z2 = PyNumber_Multiply(s, s);
    PyObject *tmp;
    int sign = 1;

    for (unsigned long i = 1; i < 50;) {
        i += 2;
        fact *= i * (i-1);
        tmp = PyNumber_Multiply(num, z2);
        Py_SETREF(num, tmp);
        sign *= -1;
        den = PyComplex_FromDoubles(fact * sign, 0);
        tmp = PyNumber_TrueDivide(num, den);
        Py_SETREF(den, tmp);
        tmp = PyNumber_Add(s, den);
        Py_DECREF(den);
        Py_SETREF(s, tmp);
    }
    Py_DECREF(num);
    Py_DECREF(z2);
    return s;
}

static PyMethodDef module_methods[] = {
    {"sin_ll", (PyCFunction)sin_ll, METH_O, NULL},
    {"sin_hh", (PyCFunction)sin_hh, METH_O, NULL},
    {NULL}  // sentinel
};


static struct PyModuleDef demo_spec = {
    PyModuleDef_HEAD_INIT,
    "demo",
    NULL,
    -1,
    module_methods
};


PyMODINIT_FUNC PyInit_demo(void) {
    return PyModule_Create(&demo_spec);
}

# pyproject.toml
[build-system]
requires = ["setuptools>=74.1"]
build-backend = "setuptools.build_meta"

[project]
name = "demo"
version = "1.0"

[tool.setuptools]
ext-modules = [{name = "demo", sources = ["mod.c"]}]

[malemburg]

The performance difference is mostly due to the PyNumber_* APIs doing lots of type slot checks to figure out what to run, implementing coercion and requiring allocating new temporary objects. My guess is that the latter costs most.

And that's expected, since the PyNumber_* API is part of the high level abstract C API.

I'd opt for removing their "private" status of using a leading underscore. Since these are documented APIs of the C API, they are in fact public APIs (and have been for ages). The leading underscore is the real bug that needs to be fixed here.

[encukou]

While it'll take time for <complex.h> Annex G to be available & reliable in all contexts, I don't think we need to promote CPython's functions, especially for new code.

One thing I didn't check: aren't Python's floor-division & modulo different from C? If they are, it would make sense to expose the functions, and at that point it would make sense to expose the entire suite.

Anyway, whether we add non-underscored names or not, I think the pre-3.14 should be soft-deprecated. There's no reason to force users to change working code that uses documented API.

[skirpichev]

One thing I didn't check: aren't Python's floor-division & modulo different from C?

Hmm, we don't have floor-division & modulo for CPython complex type. Ditto for C _Complex types (% not defined).

Though, current CPython implementation for division differ from C: the C standard has specialized version for complex/real, but not for real/complex. This is something, going to be fixed by N3460.

So, unless we are going to add arithmetic rules for pure-imaginary numbers - there are no real needs for this low-level C-API. Users who care about performance might use native double _Complex type, if available.

While it'll take time for <complex.h> Annex G to be available & reliable in all contexts, I don't think we need to promote CPython's functions, especially for new code.

Ok, so far two members of the C-API WG are for this decision. @encukou, should I propose a formal vote to soft-deprecate this API?

[vstinner]

If it's useless, why not deprecating it, instead of soft deprecating it?

[encukou]

It's only useless if your compiler supports <complex.h>, which is a C11 extension that we officially don't require. (And MSVC doesn't support it, AFAIK.)

[skirpichev]

It's only useless if your compiler supports <complex.h>, which is a C11 extension that we officially don't require.

I think it's fine to assume additional requirements for extra functionality (i.e. user want to extend the cmath bag of functions). On MVCS we have _Dcomplex (or use non-MS compiler), people also could use external libraries like GSL for complex arithmetic. Unless we aren't going for something beyond the Annex G standard - we don't offer our users too much with low-level C-API.

Probably then we should also deprecate Py_complex struct, PyComplex_FromCComplex() and PyComplex_AsCComplex(), providing instead (conditionally, if <complex.h> support is available)

PyObject * PyComplex_FromNativeComplex(double _Complex v);
double _Complex PyComplex_AsNativeComplex(PyObject *op);

[malemburg]

Shouldn't this discussion be happening on Discourse ?

…

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> Python Database Interfaces ... http://products.egenix.com/

________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

[serhiy-storchaka]

Note that there are PyOS_mystricmp, PyOS_snprintf, PyOS_strtol, etc in the C API. There were also functions like Py_UNICODE_strcmp. They provide platform independent implementations.

Even when they do not work with Python objects and are not necessary for using the C API, they are convenient. Since they are needed for internal use, providing them to user has low cost.

_Py_c_sum and others were added in the same commit (python/cpython@ae110af) as _PyObject_NewVar and _PyTuple_Resize. Should we consider them a public API or a private API?

[vstinner]

I don't understand well the status of this issue :-( I see different options:

• Do nothing
• Soft deprecate _Py_c_xxx() functions
• Deprecate _Py_c_xxx() functions
• Promote _Py_c_xxx() functions to public Py_complex_xxx() functions (and maybe deprecate _Py_c_xxx() functions)

What's your preference?

[skirpichev]

Maybe I should open a discourse thread, as suggested?

I see different options:

Lets simplify a bit:

1. do nothing (this actually just delay decision)
2. deprecate _Py_c_*() (maybe soft-deprecate)
3. promote them to public API (with some naming scheme we can decide later) & do pt.2

For me the question is: do we want to do something beyond the C standard in complex arithmetic? If not - _Py_c_*() seems redundant, as people could use native complex types to do math.