Skip to content
This repository was archived by the owner on Apr 20, 2026. It is now read-only.
This repository was archived by the owner on Apr 20, 2026. It is now read-only.

I failed to reproduce SpliceAI training based on the source code. #155

Open

Description

@shiro-kur

Hello. Great work. I am trying to reproduce SpliceAI model from the source code you provided.
Following the README.txt instruction, I started training, but I found the training was only valid when the context of 80 bp.
I mean when I increase the context to 400~10,000bp, the validation metrics become almost zero.

The installed softwere version is here. And further below, I added the output log by $ python -u train_model.py 2000 1.

Could you tell me advices to successfully reproduce the training?

$ conda list
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_tflow_select             2.1.0                       gpu
absl-py                   0.15.0             pyhd3eb1b0_0
astor                     0.8.0                    py27_0
backports                 1.1                pyhd3eb1b0_1
backports.weakref         1.0.post1                  py_1
blas                      1.1                    openblas    conda-forge
c-ares                    1.19.1               h5eee18b_0
ca-certificates           2024.12.31           h06a4308_0
certifi                   2020.6.20          pyhd3eb1b0_3
cudatoolkit               10.0.130                      0
cudnn                     7.6.5                cuda10.0_0
cupti                     10.0.130                      0
enum34                    1.1.6                    py27_1
funcsigs                  1.0.2                    py27_0
future                    0.18.2                   py27_0
futures                   3.3.0                    py27_0
gast                      0.2.2                    py27_0
google-pasta              0.2.0              pyhd3eb1b0_0
grpcio                    1.12.1           py27hdbcaa40_0
h5py                      2.9.0            py27h7918eee_0
hdf5                      1.10.4               hb1b8bf9_0
intel-openmp              2023.1.0         hdb19cb5_46306
keras                     2.2.4                         0
keras-applications        1.0.8                      py_1
keras-base                2.2.4                    py27_0
keras-preprocessing       1.1.0                      py_1
libblas                   3.9.0           1_h86c2bf4_netlib    conda-forge
libcblas                  3.9.0           6_ha36c22a_netlib    conda-forge
libffi                    3.4.4                h6a678d5_1
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran-ng            7.5.0               ha8ba4b0_17
libgfortran4              7.5.0               ha8ba4b0_17
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
liblapack                 3.9.0           6_ha36c22a_netlib    conda-forge
libopenblas               0.2.20               h9ac9557_7
libprotobuf               3.11.2               hd408876_0
libstdcxx-ng              11.2.0               h1234567_1
linecache2                1.0.0                      py_1
markdown                  3.1.1                    py27_0
mkl                       2020.2                      256
mkl-service               2.3.0            py27he904b0f_0
mkl_fft                   1.0.10                   py27_0    conda-forge
mkl_random                1.0.2                    py27_0    conda-forge
mock                      2.0.0                    py27_0
ncurses                   6.4                  h6a678d5_0
numpy                     1.16.5           py27h95a1406_0    conda-forge
numpy-base                1.14.3           py27h2b20989_0
openblas                  0.3.3             h9ac9557_1001    conda-forge
openssl                   3.4.0                h7b32b05_1    conda-forge
opt_einsum                2.3.2                      py_0
pbr                       5.5.1                      py_0
pip                       19.3.1                   py27_0
protobuf                  3.11.2           py27he6710b0_0
python                    2.7.18               h42bf7aa_3
pyyaml                    5.2              py27h7b6447c_0
readline                  8.2                  h5eee18b_0
scikit-learn              0.20.4          py27_blas_openblashebff5e3_0    conda-forge
scipy                     1.2.1            py27h921218d_2    conda-forge
setuptools                44.0.0                   py27_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.45.3               h5eee18b_0
tensorboard               1.14.0                   py27_0    conda-forge
tensorflow                1.14.0          gpu_py27he9627f8_0
tensorflow-base           1.14.0          gpu_py27h8d69cac_0
tensorflow-estimator      1.14.0           py27h5ca1d4c_0    conda-forge
tensorflow-gpu            1.14.0               h0d30ee6_0
termcolor                 1.1.0                    py27_1
tk                        8.6.14               h39e8969_0
traceback2                1.4.0                      py_0
unittest2                 1.1.0              pyhd3eb1b0_0
webencodings              0.5.1                    py27_1
werkzeug                  0.16.1                     py_0
wheel                     0.37.1             pyhd3eb1b0_0
wrapt                     1.11.2           py27h7b6447c_0
yaml                      0.1.7                had09818_2
zlib                      1.2.13               h5eee18b_1

$ python -u train_model.py 2000 1
Using TensorFlow backend.
Context nucleotides: 2000
Sequence length (output): 5000
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            (None, None, 4)      0
__________________________________________________________________________________________________
conv1d_1 (Conv1D)               (None, None, 32)     160         input_1[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, None, 32)     128         conv1d_1[0][0]
__________________________________________________________________________________________________
activation_1 (Activation)       (None, None, 32)     0           batch_normalization_1[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D)               (None, None, 32)     11296       activation_1[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, None, 32)     128         conv1d_3[0][0]
__________________________________________________________________________________________________
activation_2 (Activation)       (None, None, 32)     0           batch_normalization_2[0][0]
__________________________________________________________________________________________________
conv1d_4 (Conv1D)               (None, None, 32)     11296       activation_2[0][0]
__________________________________________________________________________________________________
add_1 (Add)                     (None, None, 32)     0           conv1d_4[0][0]
                                                                 conv1d_1[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, None, 32)     128         add_1[0][0]
__________________________________________________________________________________________________
activation_3 (Activation)       (None, None, 32)     0           batch_normalization_3[0][0]
__________________________________________________________________________________________________
conv1d_5 (Conv1D)               (None, None, 32)     11296       activation_3[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, None, 32)     128         conv1d_5[0][0]
__________________________________________________________________________________________________
activation_4 (Activation)       (None, None, 32)     0           batch_normalization_4[0][0]
__________________________________________________________________________________________________
conv1d_6 (Conv1D)               (None, None, 32)     11296       activation_4[0][0]
__________________________________________________________________________________________________
add_2 (Add)                     (None, None, 32)     0           conv1d_6[0][0]
                                                                 add_1[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, None, 32)     128         add_2[0][0]
__________________________________________________________________________________________________
activation_5 (Activation)       (None, None, 32)     0           batch_normalization_5[0][0]
__________________________________________________________________________________________________
conv1d_7 (Conv1D)               (None, None, 32)     11296       activation_5[0][0]
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, None, 32)     128         conv1d_7[0][0]
__________________________________________________________________________________________________
activation_6 (Activation)       (None, None, 32)     0           batch_normalization_6[0][0]
__________________________________________________________________________________________________
conv1d_8 (Conv1D)               (None, None, 32)     11296       activation_6[0][0]
__________________________________________________________________________________________________
add_3 (Add)                     (None, None, 32)     0           conv1d_8[0][0]
                                                                 add_2[0][0]
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, None, 32)     128         add_3[0][0]
__________________________________________________________________________________________________
activation_7 (Activation)       (None, None, 32)     0           batch_normalization_7[0][0]
__________________________________________________________________________________________________
conv1d_9 (Conv1D)               (None, None, 32)     11296       activation_7[0][0]
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, None, 32)     128         conv1d_9[0][0]
__________________________________________________________________________________________________
activation_8 (Activation)       (None, None, 32)     0           batch_normalization_8[0][0]
__________________________________________________________________________________________________
conv1d_10 (Conv1D)              (None, None, 32)     11296       activation_8[0][0]
__________________________________________________________________________________________________
add_4 (Add)                     (None, None, 32)     0           conv1d_10[0][0]
                                                                 add_3[0][0]
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, None, 32)     128         add_4[0][0]
__________________________________________________________________________________________________
activation_9 (Activation)       (None, None, 32)     0           batch_normalization_9[0][0]
__________________________________________________________________________________________________
conv1d_12 (Conv1D)              (None, None, 32)     11296       activation_9[0][0]
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, None, 32)     128         conv1d_12[0][0]
__________________________________________________________________________________________________
activation_10 (Activation)      (None, None, 32)     0           batch_normalization_10[0][0]
__________________________________________________________________________________________________
conv1d_13 (Conv1D)              (None, None, 32)     11296       activation_10[0][0]
__________________________________________________________________________________________________
add_6 (Add)                     (None, None, 32)     0           conv1d_13[0][0]
                                                                 add_4[0][0]
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, None, 32)     128         add_6[0][0]
__________________________________________________________________________________________________
activation_11 (Activation)      (None, None, 32)     0           batch_normalization_11[0][0]
__________________________________________________________________________________________________
conv1d_14 (Conv1D)              (None, None, 32)     11296       activation_11[0][0]
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, None, 32)     128         conv1d_14[0][0]
__________________________________________________________________________________________________
activation_12 (Activation)      (None, None, 32)     0           batch_normalization_12[0][0]
__________________________________________________________________________________________________
conv1d_15 (Conv1D)              (None, None, 32)     11296       activation_12[0][0]
__________________________________________________________________________________________________
add_7 (Add)                     (None, None, 32)     0           conv1d_15[0][0]
                                                                 add_6[0][0]
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, None, 32)     128         add_7[0][0]
__________________________________________________________________________________________________
activation_13 (Activation)      (None, None, 32)     0           batch_normalization_13[0][0]
__________________________________________________________________________________________________
conv1d_16 (Conv1D)              (None, None, 32)     11296       activation_13[0][0]
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, None, 32)     128         conv1d_16[0][0]
__________________________________________________________________________________________________
activation_14 (Activation)      (None, None, 32)     0           batch_normalization_14[0][0]
__________________________________________________________________________________________________
conv1d_17 (Conv1D)              (None, None, 32)     11296       activation_14[0][0]
__________________________________________________________________________________________________
add_8 (Add)                     (None, None, 32)     0           conv1d_17[0][0]
                                                                 add_7[0][0]
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, None, 32)     128         add_8[0][0]
__________________________________________________________________________________________________
activation_15 (Activation)      (None, None, 32)     0           batch_normalization_15[0][0]
__________________________________________________________________________________________________
conv1d_18 (Conv1D)              (None, None, 32)     11296       activation_15[0][0]
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, None, 32)     128         conv1d_18[0][0]
__________________________________________________________________________________________________
activation_16 (Activation)      (None, None, 32)     0           batch_normalization_16[0][0]
__________________________________________________________________________________________________
conv1d_19 (Conv1D)              (None, None, 32)     11296       activation_16[0][0]
__________________________________________________________________________________________________
add_9 (Add)                     (None, None, 32)     0           conv1d_19[0][0]
                                                                 add_8[0][0]
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, None, 32)     128         add_9[0][0]
__________________________________________________________________________________________________
activation_17 (Activation)      (None, None, 32)     0           batch_normalization_17[0][0]
__________________________________________________________________________________________________
conv1d_21 (Conv1D)              (None, None, 32)     21536       activation_17[0][0]
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, None, 32)     128         conv1d_21[0][0]
__________________________________________________________________________________________________
activation_18 (Activation)      (None, None, 32)     0           batch_normalization_18[0][0]
__________________________________________________________________________________________________
conv1d_22 (Conv1D)              (None, None, 32)     21536       activation_18[0][0]
__________________________________________________________________________________________________
add_11 (Add)                    (None, None, 32)     0           conv1d_22[0][0]
                                                                 add_9[0][0]
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, None, 32)     128         add_11[0][0]
__________________________________________________________________________________________________
activation_19 (Activation)      (None, None, 32)     0           batch_normalization_19[0][0]
__________________________________________________________________________________________________
conv1d_23 (Conv1D)              (None, None, 32)     21536       activation_19[0][0]
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, None, 32)     128         conv1d_23[0][0]
__________________________________________________________________________________________________
activation_20 (Activation)      (None, None, 32)     0           batch_normalization_20[0][0]
__________________________________________________________________________________________________
conv1d_24 (Conv1D)              (None, None, 32)     21536       activation_20[0][0]
__________________________________________________________________________________________________
add_12 (Add)                    (None, None, 32)     0           conv1d_24[0][0]
                                                                 add_11[0][0]
__________________________________________________________________________________________________
batch_normalization_21 (BatchNo (None, None, 32)     128         add_12[0][0]
__________________________________________________________________________________________________
activation_21 (Activation)      (None, None, 32)     0           batch_normalization_21[0][0]
__________________________________________________________________________________________________
conv1d_25 (Conv1D)              (None, None, 32)     21536       activation_21[0][0]
__________________________________________________________________________________________________
batch_normalization_22 (BatchNo (None, None, 32)     128         conv1d_25[0][0]
__________________________________________________________________________________________________
activation_22 (Activation)      (None, None, 32)     0           batch_normalization_22[0][0]
__________________________________________________________________________________________________
conv1d_26 (Conv1D)              (None, None, 32)     21536       activation_22[0][0]
__________________________________________________________________________________________________
add_13 (Add)                    (None, None, 32)     0           conv1d_26[0][0]
                                                                 add_12[0][0]
__________________________________________________________________________________________________
batch_normalization_23 (BatchNo (None, None, 32)     128         add_13[0][0]
__________________________________________________________________________________________________
activation_23 (Activation)      (None, None, 32)     0           batch_normalization_23[0][0]
__________________________________________________________________________________________________
conv1d_27 (Conv1D)              (None, None, 32)     21536       activation_23[0][0]
__________________________________________________________________________________________________
batch_normalization_24 (BatchNo (None, None, 32)     128         conv1d_27[0][0]
__________________________________________________________________________________________________
activation_24 (Activation)      (None, None, 32)     0           batch_normalization_24[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D)               (None, None, 32)     1056        conv1d_1[0][0]
__________________________________________________________________________________________________
conv1d_11 (Conv1D)              (None, None, 32)     1056        add_4[0][0]
__________________________________________________________________________________________________
conv1d_28 (Conv1D)              (None, None, 32)     21536       activation_24[0][0]
__________________________________________________________________________________________________
add_5 (Add)                     (None, None, 32)     0           conv1d_2[0][0]
                                                                 conv1d_11[0][0]
__________________________________________________________________________________________________
conv1d_20 (Conv1D)              (None, None, 32)     1056        add_9[0][0]
__________________________________________________________________________________________________
add_14 (Add)                    (None, None, 32)     0           conv1d_28[0][0]
                                                                 add_13[0][0]
__________________________________________________________________________________________________
add_10 (Add)                    (None, None, 32)     0           add_5[0][0]
                                                                 conv1d_20[0][0]
__________________________________________________________________________________________________
conv1d_29 (Conv1D)              (None, None, 32)     1056        add_14[0][0]
__________________________________________________________________________________________________
add_15 (Add)                    (None, None, 32)     0           add_10[0][0]
                                                                 conv1d_29[0][0]
__________________________________________________________________________________________________
cropping1d_1 (Cropping1D)       (None, None, 32)     0           add_15[0][0]
__________________________________________________________________________________________________
conv1d_30 (Conv1D)              (None, None, 3)      99          cropping1d_1[0][0]
==================================================================================================
Total params: 360,579
Trainable params: 359,043
Non-trainable params: 1,536
__________________________________________________________________________________________________
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1521: The name tf.log is deprecated. Please use tf.math.log instead.

('BATCH_SIZE:', 24)
2025-01-29 14:47:09.947963: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2025-01-29 14:47:09.955182: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2304005000 Hz
2025-01-29 14:47:09.956888: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xed23240 executing computations on platform Host. Devices:
2025-01-29 14:47:09.956935: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2025-01-29 14:47:09.960022: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2025-01-29 14:47:10.349228: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.349398: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9dae4b0 executing computations on platform CUDA. Devices:
2025-01-29 14:47:10.349430: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA GeForce RTX 3070 Laptop GPU, Compute Capability 8.6
2025-01-29 14:47:10.349676: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.349745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA GeForce RTX 3070 Laptop GPU major: 8 minor: 6 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2025-01-29 14:47:10.352229: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2025-01-29 14:47:10.368687: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2025-01-29 14:47:10.377357: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2025-01-29 14:47:10.380307: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2025-01-29 14:47:10.400169: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2025-01-29 14:47:10.412138: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2025-01-29 14:47:10.444871: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2025-01-29 14:47:10.445058: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.445105: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.445126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2025-01-29 14:47:10.445220: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2025-01-29 14:47:10.445651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-01-29 14:47:10.445675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2025-01-29 14:47:10.445684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2025-01-29 14:47:10.445938: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.445959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1409] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2025-01-29 14:47:10.445984: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.446045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6762 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3070 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6)
2025-01-29 14:47:21.554931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
--------------------------------------------------------------

Validation set metrics:

Acceptor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000  0.0000  0.0001  0.0004  0.0002  0.0000  0.0000  0.0000  0.0000  12893

Donor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000  0.0000  0.0001  0.0004  0.0002  0.0000  0.0000  0.0000  0.0000  12927

Training set metrics:

Acceptor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000  0.0001  0.0005  0.0008  0.0001  0.0000  0.0000  0.0000  0.0000  13117

Donor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000  0.0000  0.0004  0.0008  0.0001  0.0000  0.0000  0.0000  0.0000  13141
Learning rate: 0.00100
--- 6627.18091917 seconds ---

Activity

  1. kishorejaganathan commented on Jan 29, 2025

    @kishorejaganathan
    Collaborator

    Could you try adding a batch normalization layer before the convolution layer? Some things have changed in Keras and this seems to be required unless you're using the exact version used for training (2.0.5).

  2. shiro-kur commented on Jan 30, 2025

    @shiro-kur
    Author

    Thank you very much for such an informative solution. Do I have to add the bath normalization both within the Resblocks and outside of them? I mean whether I have to add batchnorms before the yellow convolution layers.

    Image

  3. kishorejaganathan commented on Jan 30, 2025

    @kishorejaganathan
    Collaborator

    Just one batch normalization before the last convolution layer Conv(3, 1, 1) is enough. You can leave the residual blocks as is.

  4. shiro-kur commented on Jan 30, 2025

    @shiro-kur
    Author

    Thank you. I will try the script below. I just added a bachnorm layer after the cropping and before the last convolution layer.

    def SpliceAI(L, W, AR):
        # L: Number of convolution kernels
        # W: Convolution window size in each residual unit
        # AR: Atrous rate in each residual unit
    
        assert len(W) == len(AR)
    
        CL = 2 * np.sum(AR*(W-1))
    
        input0 = Input(shape=(None, 4))
        conv = Conv1D(L, 1)(input0)
        skip = Conv1D(L, 1)(conv)
    
        for i in range(len(W)):
            conv = ResidualUnit(L, W[i], AR[i])(conv)
            if (((i+1) % 4 == 0) or ((i+1) == len(W))):
                # Skip connections to the output after every 4 residual units
                dense = Conv1D(L, 1)(conv)
                skip = add([skip, dense])
    
        skip = Cropping1D(CL//2)(skip)
    
        output0 = [[] for t in range(1)]
    
        for t in range(1):
            skip = BatchNormalization()(skip)
            output0[t] = Conv1D(3, 1, activation='softmax')(skip)
    
        model = Model(inputs=input0, outputs=output0)
    
        return model
    
  5. shiro-kur commented on Jan 30, 2025

    @shiro-kur
    Author

    I am sorry, but I got almost same result even after I added a batch normalization. Are there something wrong with the model structure yet?

    $ python train_model.py 2000 1
    Using TensorFlow backend.
    Context nucleotides: 2000
    Sequence length (output): 5000
    WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
    
    WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
    
    WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
    
    WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
    
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            (None, None, 4)      0
    __________________________________________________________________________________________________
    conv1d_1 (Conv1D)               (None, None, 32)     160         input_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_1 (BatchNor (None, None, 32)     128         conv1d_1[0][0]
    __________________________________________________________________________________________________
    activation_1 (Activation)       (None, None, 32)     0           batch_normalization_1[0][0]
    __________________________________________________________________________________________________
    conv1d_3 (Conv1D)               (None, None, 32)     11296       activation_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_2 (BatchNor (None, None, 32)     128         conv1d_3[0][0]
    __________________________________________________________________________________________________
    activation_2 (Activation)       (None, None, 32)     0           batch_normalization_2[0][0]
    __________________________________________________________________________________________________
    conv1d_4 (Conv1D)               (None, None, 32)     11296       activation_2[0][0]
    __________________________________________________________________________________________________
    add_1 (Add)                     (None, None, 32)     0           conv1d_4[0][0]
                                                                     conv1d_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_3 (BatchNor (None, None, 32)     128         add_1[0][0]
    __________________________________________________________________________________________________
    activation_3 (Activation)       (None, None, 32)     0           batch_normalization_3[0][0]
    __________________________________________________________________________________________________
    conv1d_5 (Conv1D)               (None, None, 32)     11296       activation_3[0][0]
    __________________________________________________________________________________________________
    batch_normalization_4 (BatchNor (None, None, 32)     128         conv1d_5[0][0]
    __________________________________________________________________________________________________
    activation_4 (Activation)       (None, None, 32)     0           batch_normalization_4[0][0]
    __________________________________________________________________________________________________
    conv1d_6 (Conv1D)               (None, None, 32)     11296       activation_4[0][0]
    __________________________________________________________________________________________________
    add_2 (Add)                     (None, None, 32)     0           conv1d_6[0][0]
                                                                     add_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_5 (BatchNor (None, None, 32)     128         add_2[0][0]
    __________________________________________________________________________________________________
    activation_5 (Activation)       (None, None, 32)     0           batch_normalization_5[0][0]
    __________________________________________________________________________________________________
    conv1d_7 (Conv1D)               (None, None, 32)     11296       activation_5[0][0]
    __________________________________________________________________________________________________
    batch_normalization_6 (BatchNor (None, None, 32)     128         conv1d_7[0][0]
    __________________________________________________________________________________________________
    activation_6 (Activation)       (None, None, 32)     0           batch_normalization_6[0][0]
    __________________________________________________________________________________________________
    conv1d_8 (Conv1D)               (None, None, 32)     11296       activation_6[0][0]
    __________________________________________________________________________________________________
    add_3 (Add)                     (None, None, 32)     0           conv1d_8[0][0]
                                                                     add_2[0][0]
    __________________________________________________________________________________________________
    batch_normalization_7 (BatchNor (None, None, 32)     128         add_3[0][0]
    __________________________________________________________________________________________________
    activation_7 (Activation)       (None, None, 32)     0           batch_normalization_7[0][0]
    __________________________________________________________________________________________________
    conv1d_9 (Conv1D)               (None, None, 32)     11296       activation_7[0][0]
    __________________________________________________________________________________________________
    batch_normalization_8 (BatchNor (None, None, 32)     128         conv1d_9[0][0]
    __________________________________________________________________________________________________
    activation_8 (Activation)       (None, None, 32)     0           batch_normalization_8[0][0]
    __________________________________________________________________________________________________
    conv1d_10 (Conv1D)              (None, None, 32)     11296       activation_8[0][0]
    __________________________________________________________________________________________________
    add_4 (Add)                     (None, None, 32)     0           conv1d_10[0][0]
                                                                     add_3[0][0]
    __________________________________________________________________________________________________
    batch_normalization_9 (BatchNor (None, None, 32)     128         add_4[0][0]
    __________________________________________________________________________________________________
    activation_9 (Activation)       (None, None, 32)     0           batch_normalization_9[0][0]
    __________________________________________________________________________________________________
    conv1d_12 (Conv1D)              (None, None, 32)     11296       activation_9[0][0]
    __________________________________________________________________________________________________
    batch_normalization_10 (BatchNo (None, None, 32)     128         conv1d_12[0][0]
    __________________________________________________________________________________________________
    activation_10 (Activation)      (None, None, 32)     0           batch_normalization_10[0][0]
    __________________________________________________________________________________________________
    conv1d_13 (Conv1D)              (None, None, 32)     11296       activation_10[0][0]
    __________________________________________________________________________________________________
    add_6 (Add)                     (None, None, 32)     0           conv1d_13[0][0]
                                                                     add_4[0][0]
    __________________________________________________________________________________________________
    batch_normalization_11 (BatchNo (None, None, 32)     128         add_6[0][0]
    __________________________________________________________________________________________________
    activation_11 (Activation)      (None, None, 32)     0           batch_normalization_11[0][0]
    __________________________________________________________________________________________________
    conv1d_14 (Conv1D)              (None, None, 32)     11296       activation_11[0][0]
    __________________________________________________________________________________________________
    batch_normalization_12 (BatchNo (None, None, 32)     128         conv1d_14[0][0]
    __________________________________________________________________________________________________
    activation_12 (Activation)      (None, None, 32)     0           batch_normalization_12[0][0]
    __________________________________________________________________________________________________
    conv1d_15 (Conv1D)              (None, None, 32)     11296       activation_12[0][0]
    __________________________________________________________________________________________________
    add_7 (Add)                     (None, None, 32)     0           conv1d_15[0][0]
                                                                     add_6[0][0]
    __________________________________________________________________________________________________
    batch_normalization_13 (BatchNo (None, None, 32)     128         add_7[0][0]
    __________________________________________________________________________________________________
    activation_13 (Activation)      (None, None, 32)     0           batch_normalization_13[0][0]
    __________________________________________________________________________________________________
    conv1d_16 (Conv1D)              (None, None, 32)     11296       activation_13[0][0]
    __________________________________________________________________________________________________
    batch_normalization_14 (BatchNo (None, None, 32)     128         conv1d_16[0][0]
    __________________________________________________________________________________________________
    activation_14 (Activation)      (None, None, 32)     0           batch_normalization_14[0][0]
    __________________________________________________________________________________________________
    conv1d_17 (Conv1D)              (None, None, 32)     11296       activation_14[0][0]
    __________________________________________________________________________________________________
    add_8 (Add)                     (None, None, 32)     0           conv1d_17[0][0]
                                                                     add_7[0][0]
    __________________________________________________________________________________________________
    batch_normalization_15 (BatchNo (None, None, 32)     128         add_8[0][0]
    __________________________________________________________________________________________________
    activation_15 (Activation)      (None, None, 32)     0           batch_normalization_15[0][0]
    __________________________________________________________________________________________________
    conv1d_18 (Conv1D)              (None, None, 32)     11296       activation_15[0][0]
    __________________________________________________________________________________________________
    batch_normalization_16 (BatchNo (None, None, 32)     128         conv1d_18[0][0]
    __________________________________________________________________________________________________
    activation_16 (Activation)      (None, None, 32)     0           batch_normalization_16[0][0]
    __________________________________________________________________________________________________
    conv1d_19 (Conv1D)              (None, None, 32)     11296       activation_16[0][0]
    __________________________________________________________________________________________________
    add_9 (Add)                     (None, None, 32)     0           conv1d_19[0][0]
                                                                     add_8[0][0]
    __________________________________________________________________________________________________
    batch_normalization_17 (BatchNo (None, None, 32)     128         add_9[0][0]
    __________________________________________________________________________________________________
    activation_17 (Activation)      (None, None, 32)     0           batch_normalization_17[0][0]
    __________________________________________________________________________________________________
    conv1d_21 (Conv1D)              (None, None, 32)     21536       activation_17[0][0]
    __________________________________________________________________________________________________
    batch_normalization_18 (BatchNo (None, None, 32)     128         conv1d_21[0][0]
    __________________________________________________________________________________________________
    activation_18 (Activation)      (None, None, 32)     0           batch_normalization_18[0][0]
    __________________________________________________________________________________________________
    conv1d_22 (Conv1D)              (None, None, 32)     21536       activation_18[0][0]
    __________________________________________________________________________________________________
    add_11 (Add)                    (None, None, 32)     0           conv1d_22[0][0]
                                                                     add_9[0][0]
    __________________________________________________________________________________________________
    batch_normalization_19 (BatchNo (None, None, 32)     128         add_11[0][0]
    __________________________________________________________________________________________________
    activation_19 (Activation)      (None, None, 32)     0           batch_normalization_19[0][0]
    __________________________________________________________________________________________________
    conv1d_23 (Conv1D)              (None, None, 32)     21536       activation_19[0][0]
    __________________________________________________________________________________________________
    batch_normalization_20 (BatchNo (None, None, 32)     128         conv1d_23[0][0]
    __________________________________________________________________________________________________
    activation_20 (Activation)      (None, None, 32)     0           batch_normalization_20[0][0]
    __________________________________________________________________________________________________
    conv1d_24 (Conv1D)              (None, None, 32)     21536       activation_20[0][0]
    __________________________________________________________________________________________________
    add_12 (Add)                    (None, None, 32)     0           conv1d_24[0][0]
                                                                     add_11[0][0]
    __________________________________________________________________________________________________
    batch_normalization_21 (BatchNo (None, None, 32)     128         add_12[0][0]
    __________________________________________________________________________________________________
    activation_21 (Activation)      (None, None, 32)     0           batch_normalization_21[0][0]
    __________________________________________________________________________________________________
    conv1d_25 (Conv1D)              (None, None, 32)     21536       activation_21[0][0]
    __________________________________________________________________________________________________
    batch_normalization_22 (BatchNo (None, None, 32)     128         conv1d_25[0][0]
    __________________________________________________________________________________________________
    activation_22 (Activation)      (None, None, 32)     0           batch_normalization_22[0][0]
    __________________________________________________________________________________________________
    conv1d_26 (Conv1D)              (None, None, 32)     21536       activation_22[0][0]
    __________________________________________________________________________________________________
    add_13 (Add)                    (None, None, 32)     0           conv1d_26[0][0]
                                                                     add_12[0][0]
    __________________________________________________________________________________________________
    batch_normalization_23 (BatchNo (None, None, 32)     128         add_13[0][0]
    __________________________________________________________________________________________________
    activation_23 (Activation)      (None, None, 32)     0           batch_normalization_23[0][0]
    __________________________________________________________________________________________________
    conv1d_27 (Conv1D)              (None, None, 32)     21536       activation_23[0][0]
    __________________________________________________________________________________________________
    batch_normalization_24 (BatchNo (None, None, 32)     128         conv1d_27[0][0]
    __________________________________________________________________________________________________
    activation_24 (Activation)      (None, None, 32)     0           batch_normalization_24[0][0]
    __________________________________________________________________________________________________
    conv1d_2 (Conv1D)               (None, None, 32)     1056        conv1d_1[0][0]
    __________________________________________________________________________________________________
    conv1d_11 (Conv1D)              (None, None, 32)     1056        add_4[0][0]
    __________________________________________________________________________________________________
    conv1d_28 (Conv1D)              (None, None, 32)     21536       activation_24[0][0]
    __________________________________________________________________________________________________
    add_5 (Add)                     (None, None, 32)     0           conv1d_2[0][0]
                                                                     conv1d_11[0][0]
    __________________________________________________________________________________________________
    conv1d_20 (Conv1D)              (None, None, 32)     1056        add_9[0][0]
    __________________________________________________________________________________________________
    add_14 (Add)                    (None, None, 32)     0           conv1d_28[0][0]
                                                                     add_13[0][0]
    __________________________________________________________________________________________________
    add_10 (Add)                    (None, None, 32)     0           add_5[0][0]
                                                                     conv1d_20[0][0]
    __________________________________________________________________________________________________
    conv1d_29 (Conv1D)              (None, None, 32)     1056        add_14[0][0]
    __________________________________________________________________________________________________
    add_15 (Add)                    (None, None, 32)     0           add_10[0][0]
                                                                     conv1d_29[0][0]
    __________________________________________________________________________________________________
    cropping1d_1 (Cropping1D)       (None, None, 32)     0           add_15[0][0]
    __________________________________________________________________________________________________
    batch_normalization_25 (BatchNo (None, None, 32)     128         cropping1d_1[0][0]
    __________________________________________________________________________________________________
    conv1d_30 (Conv1D)              (None, None, 3)      99          batch_normalization_25[0][0]
    ==================================================================================================
    Total params: 360,707
    Trainable params: 359,107
    Non-trainable params: 1,600
    __________________________________________________________________________________________________
    WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
    
    WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1521: The name tf.log is deprecated. Please use tf.math.log instead.
    
    ('BATCH_SIZE:', 24)
    2025-01-30 13:58:28.639903: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
    2025-01-30 13:58:28.656287: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2303970000 Hz
    2025-01-30 13:58:28.659294: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x171745f0 executing computations on platform Host. Devices:
    2025-01-30 13:58:28.659767: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
    2025-01-30 13:58:28.675315: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
    2025-01-30 13:58:29.297746: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
    Your kernel may have been built without NUMA support.
    2025-01-30 13:58:29.298041: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x164470d0 executing computations on platform CUDA. Devices:
    2025-01-30 13:58:29.298106: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA GeForce RTX 3070 Laptop GPU, Compute Capability 8.6
    2025-01-30 13:58:29.298353: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
    Your kernel may have been built without NUMA support.
    2025-01-30 13:58:29.298860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
    name: NVIDIA GeForce RTX 3070 Laptop GPU major: 8 minor: 6 memoryClockRate(GHz): 1.62
    pciBusID: 0000:01:00.0
    2025-01-30 13:58:29.302254: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2025-01-30 13:58:29.349039: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
    2025-01-30 13:58:29.377121: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
    2025-01-30 13:58:29.383589: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
    2025-01-30 13:58:29.445837: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
    2025-01-30 13:58:29.486619: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
    2025-01-30 13:58:29.568987: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    2025-01-30 13:58:29.569295: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
    Your kernel may have been built without NUMA support.
    2025-01-30 13:58:29.569474: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
    Your kernel may have been built without NUMA support.
    2025-01-30 13:58:29.569546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
    2025-01-30 13:58:29.570152: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2025-01-30 13:58:29.571258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
    2025-01-30 13:58:29.571344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
    2025-01-30 13:58:29.571404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
    2025-01-30 13:58:29.571811: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
    Your kernel may have been built without NUMA support.
    2025-01-30 13:58:29.571892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1409] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
    2025-01-30 13:58:29.572031: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
    Your kernel may have been built without NUMA support.
    2025-01-30 13:58:29.572156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6762 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3070 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6)
    2025-01-30 13:58:45.374483: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    --------------------------------------------------------------
    
    Validation set metrics:
    
    Acceptor:
    Warning: y_pred contains NaN or infinity. Converting to 0.
    0.0000  0.0001  0.0004  0.0007  0.0002  0.0000  0.0000  0.0000  0.0000  13600
    
    Donor:
    Warning: y_pred contains NaN or infinity. Converting to 0.
    0.0000  0.0001  0.0004  0.0007  0.0002  0.0000  0.0000  0.0000  0.0000  13603
    
    Training set metrics:
    
    Acceptor:
    Warning: y_pred contains NaN or infinity. Converting to 0.
    0.0004  0.0005  0.0007  0.0010  0.0001  0.0000  0.0000  0.0000  0.0000  14763
    
    Donor:
    Warning: y_pred contains NaN or infinity. Converting to 0.
    0.0004  0.0005  0.0009  0.0010  0.0001  0.0000  0.0000  0.0000  0.0000  14766
    Learning rate: 0.00100
    --- 5948.97182894 seconds ---
    --------------------------------------------------------------
    
  6. kishorejaganathan commented on Feb 3, 2025

    @kishorejaganathan
    Collaborator

    Hmm, in that case I am not really sure. Can you try training on the CPU once (using Keras 2.0.5) and see if that works?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions