md5[FILE]: document that the hash includes the file extension; consider an MD5 of the content #1913

Description

md5[FILE] from the System module (md5(FILE file) = FORMULA STRING 'md5($1)') hashes the stored FILE value, which consists of a byte with the extension length, the extension and the content. This is consistent with FILE being "extension + bytes": two FILE values with the same bytes but different extensions are different values, and the hash works well for detecting changes of a FILE property.

However, the documentation (System, File helpers) only says "md5[FILE] — MD5 hash", while the common meaning of "MD5 of a file" is the hash of its bytes only (md5sum, certutil -hashfile, HTTP Content-MD5). Code that uses md5[FILE] to compare with a checksum provided by a sender, or to detect that the same file was received again under another extension, gets a mismatch without any hint why.

Steps to reproduce

1. Create C:/tmp/x.txt containing exactly three bytes abc (no newline): python -c "open('C:/tmp/x.txt','wb').write(b'abc')"
2. Run via POST /eval:

   contentMd5 (RAWFILE f) = FORMULA STRING 'md5($1)';
   
   run() {
       LOCAL f = FILE ();
       READ 'file://C:/tmp/x.txt' TO f;
       EXPORT JSON FROM ext = extension(f()), md5File = md5(f()), md5Content = contentMd5(RAWFILE(f()));
   }
   

Actual behavior

HTTP 200:

{"ext":"txt","md5File":"4d9ddfa08fbafc7488a52eb7f4d49e6c","md5Content":"900150983cd24fb0d6963f7d28e17f72"}

• 900150983cd24fb0d6963f7d28e17f72 is the MD5 of the content abc — the same as md5sum or certutil -hashfile C:\tmp\x.txt MD5.
• 4d9ddfa08fbafc7488a52eb7f4d49e6c is the MD5 of the bytes 03 74 78 74 61 62 63 — extension length, txt, content. The same file named x.csv would give 6fe3582997ecdab9beedef7c534246e5 (MD5 of 03 63 73 76 61 62 63).

Expected behavior

Either the documentation states that md5[FILE] hashes the FILE value (extension length + extension + content), or there is a documented way to get the MD5 of the content only, matching md5sum/certutil (today it works via a custom formula over RAWFILE).

Where in the source

system/System.lsf:339 — md5(FILE file) = FORMULA STRING 'md5($1)';: PostgreSQL hashes the stored FILE value, which is built in FileData.getBytes() (api module, lsfusion/base/file/FileData.java, lines 54–58): extension length byte, extension, content.

Request

1. Document in System (File helpers) that md5[FILE] hashes the FILE value including the extension, and point to RAWFILE for a content hash.
2. Optionally add a ready-made content hash (e.g. md5[RAWFILE]), so that a custom FORMULA is not needed.

Impact

Content-based checks ("this file has already been imported", comparison with a sender's checksum) silently fail if they use md5[FILE]: the same bytes with another extension give another hash, and the hash never matches external tools.