Skip to content

[Large Databases]: Make tree searches by name more efficient #7752

Description

@grantfitzsimmons

For collections with exceptionally large taxonomy trees (e.g., ~1 million rows), the taxon search box severely degrades server performance. Every keystroke sends a query using a leading wildcard (LIKE '%pattern%'), which prevents standard index usage and forces a full table scan. In environments hosting multiple collections, a single user typing a scientific name rapidly queues up multiple 5-second scans.

SELECT taxon.TaxonID, IFNULL(taxon.FullName, '') FROM taxon WHERE taxon.TaxonTreeDefID IN (1) AND taxon.FullName LIKE '%hydrolea%' ORDER BY IFNULL(taxon.FullName, '') LIMIT 0, 1000

This consumes disk I/O and starves other collections on the same server, potentially resulting in 502 Gateway Timeouts for other users.

Proposal
To improve performance and reduce server load, searches should fetch a limited number of results per query (for example, limiting the initial fetch to 25 or 50 records instead of 1000) and paginate the remaining results when scrolling. Fetching smaller chunks will reduce the I/O tax per keystroke while still allowing users to find the correct name.

Describe alternatives you've considered
Several other strategies could be employed either alongside or instead of pagination to mitigate this issue:

  1. Implement a strict debounce on the input field, waiting 500ms after the user stops typing before firing the query to the backend.
  2. Prevent the query from firing until the user has typed at least 3 or 4 characters.
  3. Migrate to using full text indexes and MATCH() AGAINST() for substring searches, which can utilize indexes for substring matches and resolve queries in milliseconds instead of seconds.

Reported By
Tomislav, UT Austin on BoldDesk in ticket #616

Additional context
This was identified via slow query logs where the taxon search in a botany collection (PRC) accounted for 72% of all slow queries on the server, causing cascading 502 errors for a vertebrate paleo collection (VPL) sharing the same MariaDB instance.

Activity

  1. added
    2 - QueriesIssues that are related to the query builder or queries in general
    4 - PerformanceIssues related to performance, concurrency, and optimization
    on Feb 23, 2026
  2. changed the title [-][Large Databases]: Improve tree search fetching and performance[/-] [+][Large Databases]: Make tree searches more efficient[/+] on Feb 23, 2026
  3. changed the title [-][Large Databases]: Make tree searches more efficient[/-] [+][Large Databases]: Make tree searches by name more efficient[/+] on Feb 23, 2026
  4. dshorthouse commented on Mar 18, 2026

    @dshorthouse

    Having had to deal with issues like this in a past life on MySQL, you might see some improvement with your proposed solution but it'd not be as marked as making a leap to using Elasticsearch where its edge n-grams are designed exactly for this use-case. You might however see far better performance improvement if you did LIKE 'pattern%' for a "starts-with" query albeit with an impact on the user experience. But, given this sounds like an auto-complete use-case, it might be acceptable.

  5. added 5 commits that reference this issue on Apr 4, 2026
    7b33f29
    93bde49
    9d95089
    d0e2aa3
    57ac51a
  6. modified the milestones: 7.x, 7.12.3 on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

2 - QueriesIssues that are related to the query builder or queries in general4 - PerformanceIssues related to performance, concurrency, and optimization

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions