invalid value encountered in divide when running pp.normalize_pearson_residuals

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of scanpy.
(optional) I have confirmed this bug exists on the master branch of scanpy.

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Hi, I'm trying to follow Analytic Pearson residuals
but after getting first 2 nomalizations, I'm stucked with Pearson residuals normalization. It seems like a devided by 0 issue? I don't know. What else I can do to check if the matrix is the problem. already did gene (min_cells=3) and cell filtering(min_genes=200).

Minimal code sample (that we can copy&paste without having any data)

analytic_pearson = sc.experimental.pp.normalize_pearson_residuals(adata, inplace=False)

/home/sxykdx/miniconda3/envs/scanpy/lib/python3.10/site-packages/scanpy/experimental/pp/_normalization.py:59: RuntimeWarning: invalid value encountered in divide
  residuals = diff / np.sqrt(mu + mu**2 / theta)

analytic_pearson["X"]
Output:

array([[-0.08038502, -0.10195383, -0.24513291, ...,  1.47699586,
        -0.08709449, -0.16926342],
       [-0.08623174, -0.109369  ,  3.53739781, ..., -0.54014355,
        -0.09342913, -0.18157157],
       [-0.07625086, -0.09671059, -0.2325321 , ..., -0.47777291,
        12.02085262,  6.06603257],
       ...,
       [-0.02799957, -0.03551299, -0.08540438, ..., -0.17560885,
        -0.03033674, -0.05896328],
       [-0.02840246, -0.03602399, -0.08663319, ..., -0.17813493,
        -0.03077326, -0.05981169],
       [-0.02914286, -0.03696307, -0.08889143, ..., -0.18277714,
        -0.03157547, -0.06137084]])

adata.layers["analytic_pearson_residuals"] = analytic_pearson["X"]
adata.layers["analytic_pearson_residuals"].sum(1)

Output:
array([nan, nan, nan, ..., nan, nan, nan])

Versions

scanpy==1.9.3 anndata==0.9.1 umap==0.5.3 numpy==1.23.5 scipy==1.10.1 pandas==2.0.1 scikit-learn==1.2.2 statsmodels==0.13.5 python-igraph==0.10.3 pynndescent==0.5.10

[Paste the output of scanpy.logging.print_versions() leaving a blank line after the details tag]

I've run into the same issue. In my case adata.layers["analytic_pearson_residuals"].sum(1) gives an array of nans because there are nans in analytic_pearson["X"], as indicated by RuntimeWarning.

I am still only investigating this, but if treating nans as 0 is OK there is numpy.nansum function that could be used instead of sum.

@LuckyMD

@LuckyMD any idea why that could be?

It seems that some genes did not express in all cells. I tried to used sc.pp.filter_genes(adata, min_cells=1) to filter out those genes, and addressed this issue.

in the recipe, after hvg, we should filter cells again, because with a subset of genes, the previous cell filter retaining cells might not have count:

   # After subsetting to HVG, before normalize_pearson_residuals
    cell_sums = adata_pca.X.sum(axis=1)
    if isinstance(cell_sums, DaskArray):
        cell_sums = cell_sums.compute()
    cell_sums = np.asarray(cell_sums).ravel()
    valid_cells = cell_sums > 1
    if not valid_cells.all():
        warnings.warn(f"Removing {(~valid_cells).sum()} cells with zero counts in HVG genes.", UserWarning)
        adata_pca = adata_pca[valid_cells].copy()

scverse#2496

Actually, I am not sure if hvg subset before normalization is a good idea.
and I see _pearson_residuals in _normalization.py and _calculate_res_sparse in _highly_variable_genes.py
intrinsically the same. in one way we might not need to write twice, in another, in the recipe we might need to calculate twice.

Imo we should check whether variance is zero in a gene and if that is the case just set the normalized value to 0 far all cells.

Navigation Menu

Repository navigation

`invalid value encountered in divide` when running `pp.normalize_pearson_residuals` #2496

Minimal code sample (that we can copy&paste without having any data)

Versions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

invalid value encountered in divide when running pp.normalize_pearson_residuals #2496

Description

Minimal code sample (that we can copy&paste without having any data)

Versions

Activity

mcjmigdal commented on Oct 25, 2023

flying-sheep commented on Nov 7, 2023

WeipengMO commented on Nov 24, 2023

brainfo commented on Jan 27, 2026

brainfo commented on Jan 29, 2026

MLubetzki commented on Sep 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`invalid value encountered in divide` when running `pp.normalize_pearson_residuals` #2496