Summary
check_file returns is_check_passed: True for a parquet file it has already rejected as over
the size limit, so files.upload(..., check=True) goes ahead and uploads it.
lib/utils/files.py, the size branch does not return:
if file_size > MAX_FILE_SIZE_GB * NUM_BYTES_IN_GB:
report_dict["message"] = f"Maximum supported file size is ..."
report_dict["is_check_passed"] = False # falls through
elif file_size == 0:
...
return report_dict # the empty case does return
else:
report_dict["file_size"] = file_size
and further down it merges the format result over the top:
report_dict.update(data_report_dict)
_check_parquet ends its success path with an unconditional report_dict["is_check_passed"] = True
at line 364, so the merge puts the rejection back to True. _check_jsonl and _check_csv never
set the flag to True, so they are not affected. Parquet is the only one.
That flag is the upload guard, in three places: resources/files.py:161, resources/files.py:347,
and lib/cli/api/files/upload.py:34.
Reproduction
together 2.28.0, pyarrow 25.0.0, Python 3.14.0, fresh venv.
MAX_FILE_SIZE_GB is 50.1 and I did not want to generate a 50 GB file, so for the rows marked
"over the limit" I set together.lib.utils.files.MAX_FILE_SIZE_GB to 1e-9 for that one call and
restored it afterwards. Nothing else is patched. The under-the-limit rows run on the stock
constant. The other four rows are there so a reader can tell the patched threshold is not simply
forcing everything to True.
| file |
over the limit |
is_check_passed |
file_size |
message |
| valid parquet |
yes |
True |
None |
Maximum supported file size is ... |
| valid parquet |
no |
True |
1956 |
Checks passed |
| valid jsonl |
yes |
False |
None |
Maximum supported file size is ... |
| valid jsonl |
no |
True |
50 |
Checks passed |
| malformed jsonl |
yes |
False |
None |
Error parsing json payload. Unexpected format on line 2. |
| csv, purpose fine-tune |
yes |
False |
None |
CSV files are not supported for fine-tune. |
| empty file |
yes |
False |
0 |
File is empty |
Full report dict for the first row:
{"is_check_passed": true,
"message": "Maximum supported file size is 1e-09 GB. Found file with size of 0.0 GB.",
"found": true, "file_size": null, "utf8": null, "line_type": null, "text_field": null,
"key_value": null, "has_min_samples": null, "num_samples": 2, "load_json": null,
"load_csv": null, "filetype": "parquet"}
file_size is null in that report as well, since the size branch never assigns it.
Suggested fix
Return from the oversize branch the way the empty branch already does:
if file_size > MAX_FILE_SIZE_GB * NUM_BYTES_IN_GB:
report_dict["message"] = f"Maximum supported file size is ..."
report_dict["file_size"] = file_size
report_dict["is_check_passed"] = False
return report_dict
Making _check_parquet set is_check_passed only when it is not already False would also work,
though the early return keeps the whole function honest for any check added later.
Summary
check_filereturnsis_check_passed: Truefor a parquet file it has already rejected as overthe size limit, so
files.upload(..., check=True)goes ahead and uploads it.lib/utils/files.py, the size branch does not return:and further down it merges the format result over the top:
_check_parquetends its success path with an unconditionalreport_dict["is_check_passed"] = Trueat line 364, so the merge puts the rejection back to True.
_check_jsonland_check_csvneverset the flag to True, so they are not affected. Parquet is the only one.
That flag is the upload guard, in three places:
resources/files.py:161,resources/files.py:347,and
lib/cli/api/files/upload.py:34.Reproduction
together 2.28.0, pyarrow 25.0.0, Python 3.14.0, fresh venv.
MAX_FILE_SIZE_GBis 50.1 and I did not want to generate a 50 GB file, so for the rows marked"over the limit" I set
together.lib.utils.files.MAX_FILE_SIZE_GBto 1e-9 for that one call andrestored it afterwards. Nothing else is patched. The under-the-limit rows run on the stock
constant. The other four rows are there so a reader can tell the patched threshold is not simply
forcing everything to True.
Full report dict for the first row:
{"is_check_passed": true, "message": "Maximum supported file size is 1e-09 GB. Found file with size of 0.0 GB.", "found": true, "file_size": null, "utf8": null, "line_type": null, "text_field": null, "key_value": null, "has_min_samples": null, "num_samples": 2, "load_json": null, "load_csv": null, "filetype": "parquet"}file_sizeis null in that report as well, since the size branch never assigns it.Suggested fix
Return from the oversize branch the way the empty branch already does:
Making
_check_parquetsetis_check_passedonly when it is not already False would also work,though the early return keeps the whole function honest for any check added later.