Add Expanded Entropy ingest module - #36
david-gary wants to merge 1 commit into
Conversation
|
How does this handle compressed files? |
|
No, it doesn't tell them apart. An archive with no dominant byte value (gzip or zip output of data that was already dense) gets flagged the same as ciphertext, on every measure including Min. That's a limit of any byte-frequency test. I put a file like that in the sample corpus (limitation_compressed.tar.gz) and called it out in the README and the paper for that reason. The main thing Min entropy helps with is the other big source of false positives: files that mix compressed streams with structure, like PDFs, SQLite databases, and Office docs. Those have a dominant byte (padding, page structure, a common character), so they stay under the threshold with Min while Shannon at 6.0 flags them. On my test collection that took false positives from 46 down to 11 with no false negatives, and the 11 left were archives and PDFs. In practice the hits are LIKELY_NOTABLE leads with all four values in the comment, so identified archives can be filtered out of the Encryption Suspected results by MIME type. For archives specifically I'd lean on the Embedded File Extractor anyway, since it already reports archives it can't open as Encryption Detected. If it's useful, I can add an option to skip files whose type Autopsy has already identified, the way the built-in detector limits itself to application/octet-stream. |
|
When we looked at this, we found that cosine similarity worked really well. “Using purpose-built functions and block hashes to enable small block and sub-file forensics” — Simson Garfinkel, Alex Nelson, Douglas White, and Vassil Roussev. Digital Investigation 7 (2010), S13–S23. DOI. Why don't you try implementing that and see how it works? |
|
Will do right away. Thanks! |
Great! It's wonderful to have an enthusiastic set of eyes looking at this problem after 16 years. |
|
I've used your suggestions to help build out a second stage for this tool. It runs only on files the entropy test flags. Up to four blocks are sampled across the file, each one is rotated by one to four bytes and subtracted from itself modulo 256, and the byte histogram of the block is compared with the histograms of the differences by cosine. A block below the MCV counts as Huffman-coded, and a file is called compressed when two or more of its sampled blocks are. I departed from the paper in a few places. The four difference histograms are pooled and compared with a single cosine rather than taking the smallest of the four; the pooled histogram is less noisy, and at the same false positive rate it recognizes more deflate blocks (78% against 73% at 16 KB, 53% against 47% at 4 KB). The minimum cosine is still available as a setting. The MCV is a quantile over a large negative class rather than the minimum over a training sample. Since AES output is indistinguishable from random data, 3,200 MB of AES-CTR keystream is scored and the threshold is the score that one block in ten thousand falls below, so the false positive rate is chosen and comes with a confidence bound. Blocks come in 64, 16 and 4 KB sizes, with a file judged at the largest size that gives it two whole blocks, so a verdict is available from 8 KB up. The positive class is 273 files and 336 MB: deflate, bzip2 and xz encodings of synthetic text, the Python standard library, a DLL and the files of govdocs1 thread 001, with real jars from an Autopsy installation and images from Windows and govdocs1 compressed as found. AES-CBC copies of the synthetic and govdocs1 files form a second negative set. At 64 KB, 98% of deflate blocks fall below the threshold and none of 2,618 encrypted blocks do; at 16 KB it is 78% with 3 of 10,531 encrypted blocks miscalled, and at 4 KB 53% with 6 of 42,214. Per file, every deflate file judged at 64 or 16 KB is called compressed, 18 of the 26 small files judged at 4 KB are, and none of the 56 encrypted copies is. Among the files the first stage actually flags, which excludes most jars because Min entropy at 6 bits clears them on its own, the count is 39 of 43 deflate files and 29 of 29 JPEGs. Two things stood out in the data. The govdocs1 material is recognized in 45% to 47% of its 4 KB blocks, within the range in the paper, while deflate of prose and source code is nearly invisible at that size (under 3% of blocks) whether the text is synthetic or real, and only comes into reach at 64 KB. The cause appears to be the symbol stream: text with long matches deflates into short dynamic codes that leave little byte structure in a 4 KB block, and the same text encoded with literals only is recognized in every block. For the comparison I used FP/(FP+TN) from the confusion matrices in Tables 5 and 6, which gives 0.0074% and 0.39%, since the 0.0197% and 0.45% listed there are false positives as a share of positive calls. One limitation is worth stating. The test measures departure from randomness at byte lags of one to four, and ciphertext that carries plaintext structure produces the same signal. Of 23 such files I built (PDFs encrypted with the standard security handler, AES-ECB of the source material, and ZIP archives of stored 2 KB and 8 KB ciphertext members), 16 are called compressed, every encrypted PDF among them; the archives with 8 KB members are not. Ten of the 23 never reach the second stage, since their plaintext keeps whole-file Min entropy under 6 bits (eight of the ten PDFs among them); of the 13 that do, 6 are called compressed. The verdict text now says so and the evaluation reports the set separately. The xz blocks the test does catch also turned out to be already-compressed content inside the govdocs1 Office files, which LZMA re-emits nearly unchanged, rather than xz's own output. The module also records the type Autopsy identified on each result and can leave identified compressed formats unflagged, so the statistical test is for data without a header. The evaluation script and the output of this run are in the repository (tests/compression_eval.py, tests/compression_eval_results.txt), and the method and results are in the paper with the 2010 paper cited. Sincerely, thank you for the guidance. |
|
I'm glad that you found the paper interesting and that you were able to do fun things with the ideas. |
This branch has not been deployed
2 participants
Adds a listing for Expanded Entropy, a Python ingest module that flags possibly encrypted files by byte entropy.
It uses Min entropy in place of the Shannon entropy that the built-in Encryption Detection module uses. Min entropy depends only on the most frequent byte value, so structured files such as PDFs and SQLite databases, which a Shannon test at 6.0 bits/byte reports as false positives, stay clear while ciphertext is still flagged. The module computes Shannon, Min, Rényi and Collision entropy for every file and records all four values in the Encryption Suspected result, with the measure, threshold and other options configurable in the ingest settings panel.
The module is hosted at https://github.com/david-gary/expanded_entropy (MIT). It was tested with Autopsy 4.23.1 (Jython 2.7.4, Sleuth Kit 4.15.0): unit tests, tests under Autopsy's bundled Jython with the Sleuth Kit jars on the classpath, and a scripted command-line ingest checked against a labelled sample corpus. The README on the entry follows the template in DocsForDevelopers.