Skip to content

check_file: an oversize .parquet file passes validation because the size rejection is overwritten #485

Description

@toolshedlabs-hash

Summary

check_file returns is_check_passed: True for a parquet file it has already rejected as over
the size limit, so files.upload(..., check=True) goes ahead and uploads it.

lib/utils/files.py, the size branch does not return:

if file_size > MAX_FILE_SIZE_GB * NUM_BYTES_IN_GB:
    report_dict["message"] = f"Maximum supported file size is ..."
    report_dict["is_check_passed"] = False     # falls through
elif file_size == 0:
    ...
    return report_dict                         # the empty case does return
else:
    report_dict["file_size"] = file_size

and further down it merges the format result over the top:

report_dict.update(data_report_dict)

_check_parquet ends its success path with an unconditional report_dict["is_check_passed"] = True
at line 364, so the merge puts the rejection back to True. _check_jsonl and _check_csv never
set the flag to True, so they are not affected. Parquet is the only one.

That flag is the upload guard, in three places: resources/files.py:161, resources/files.py:347,
and lib/cli/api/files/upload.py:34.

Reproduction

together 2.28.0, pyarrow 25.0.0, Python 3.14.0, fresh venv.

MAX_FILE_SIZE_GB is 50.1 and I did not want to generate a 50 GB file, so for the rows marked
"over the limit" I set together.lib.utils.files.MAX_FILE_SIZE_GB to 1e-9 for that one call and
restored it afterwards. Nothing else is patched. The under-the-limit rows run on the stock
constant. The other four rows are there so a reader can tell the patched threshold is not simply
forcing everything to True.

file over the limit is_check_passed file_size message
valid parquet yes True None Maximum supported file size is ...
valid parquet no True 1956 Checks passed
valid jsonl yes False None Maximum supported file size is ...
valid jsonl no True 50 Checks passed
malformed jsonl yes False None Error parsing json payload. Unexpected format on line 2.
csv, purpose fine-tune yes False None CSV files are not supported for fine-tune.
empty file yes False 0 File is empty

Full report dict for the first row:

{"is_check_passed": true,
 "message": "Maximum supported file size is 1e-09 GB. Found file with size of 0.0 GB.",
 "found": true, "file_size": null, "utf8": null, "line_type": null, "text_field": null,
 "key_value": null, "has_min_samples": null, "num_samples": 2, "load_json": null,
 "load_csv": null, "filetype": "parquet"}

file_size is null in that report as well, since the size branch never assigns it.

Suggested fix

Return from the oversize branch the way the empty branch already does:

if file_size > MAX_FILE_SIZE_GB * NUM_BYTES_IN_GB:
    report_dict["message"] = f"Maximum supported file size is ..."
    report_dict["file_size"] = file_size
    report_dict["is_check_passed"] = False
    return report_dict

Making _check_parquet set is_check_passed only when it is not already False would also work,
though the early return keeps the whole function honest for any check added later.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions