Skip to content
This repository was archived by the owner on Apr 20, 2026. It is now read-only.
This repository was archived by the owner on Apr 20, 2026. It is now read-only.

Way to many TEMP files #148

Open
Open

Description

@ttys42

People are starting to use spliceAI on our HPC infrastructure. While running we noticed that TEMP folders are created with multiple millions (~4M) files each only a few kb.
In a HPC system with a massive parallel filesystem this is creating massive overhead.
Is there a way to change this behaviour? So that these TEMP files are combined in any way or switch to another method?
For example storing this data in a local database would improve performance massively.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions