An Autopsy file ingest module that flags possibly
encrypted files by byte entropy, using Min entropy in place of the Shannon
entropy that Autopsy's own detector relies on, then tells compressed data
from encrypted data on the files it flags and notes the type Autopsy
identified for them. The paper in paper/
sets out the reasoning and the evidence in full. This file covers what the
module does, how to install it, and how to run its tests.
Autopsy's encryption detector flags a file when the Shannon entropy of its bytes reaches a threshold. The threshold is 7.5 bits per byte by default, and the settings panel allows it to be lowered to 6.0, the most permissive setting Autopsy allows and the one where the choice of measure matters most. Shannon entropy measures how spread out the byte values are, so a file whose bytes take many different values scores high even when one value dominates. PDFs and SQLite databases are built that way, with compressed streams sitting next to runs of structural bytes, and a Shannon test at 6.0 reports them as false positives. Autopsy's own detector avoids that only by never testing a file whose type it can identify (see below).
Min entropy looks only at the most frequent byte's share. It is -log₂ of that share, so a file reaches 6.0 under Min entropy only if no byte value makes up more than 1/64 of its content. Ciphertext meets that condition easily and sits near 8.0, while a PDF with a few percent of padding bytes does not. In the earlier experiment the unencrypted variants clustered around 4 bits under Min entropy, two bits below the lowest threshold Autopsy allows. The four measures the module computes are ordered Min ≤ Rényi(3) ≤ Collision ≤ Shannon for every file, so at a fixed threshold a switch from Shannon to Min can only remove flags and never adds one.
For a file made of one dominant byte value plus uniform noise, the share that value needs before each measure drops below 6.0:
| Measure | Bits per byte | Dominant share needed to clear 6.0 |
|---|---|---|
| Shannon | -Σ pᵢ log₂ pᵢ | 37% |
| Collision (Rényi, α = 2) | -log₂ Σ pᵢ² | 11% |
| Rényi (α = 3) | 1/(1-α) log₂ Σ pᵢᵅ | 6.1% |
| Min | -log₂ max pᵢ | 1.6% |
An earlier measurement over 27 files of mixed types, each also encoded, hashed and encrypted twelve different ways, compared the four measures at the 6.0 threshold: Shannon raised 46 false positives among 243 unencrypted variants, Min raised 11 with none on text or SQLite files, and no measure missed any of the 108 encrypted variants. The paper reports the details.
For each file it streams the bytes into a 256-bin histogram, computes all four
measures, and creates an Encryption Suspected result (TSK_ENCRYPTION_SUSPECTED,
Autopsy's own artifact type for this) when the chosen measure reaches the
threshold. The result's comment records all four values, so the analyst can
compare them whichever measure is driving the flag. Min entropy at 6.0 is the
default. The settings panel lets you pick any measure, change the threshold
and the Rényi order, cap how many bytes are read per file, skip known-good
files, and flag when any of the four measures trips. The second stage's
statistic and strictness are settings too, as are the options to leave files
that look compressed, or that Autopsy identified as a compressed format,
unflagged. Settings are kept in Autopsy's generic settings map, so they
survive between ingest runs the way a Java module's do.
The switch has one cost of its own. Because Min entropy never exceeds Shannon entropy, ciphertext that carries a large run of a single byte value, such as an encrypted container whose unused regions are zero-filled, can fall below the threshold under Min while Shannon at the same threshold would still flag it. Nothing of the kind was in the test collection; the flag-if-any option covers evidence where such containers are expected.
Autopsy's built-in detector applies its Shannon test only to files of unknown
MIME type (application/octet-stream) that are at least 5 MB and, by default,
whose size is a whole multiple of 512 bytes; files with a .db extension are
tested from 64 KB up without the 512-byte rule. This module measures every
regular file of 1 KB or more that no hash set has marked as known, so it sees
the files on which the choice of measure makes a difference.
tests/entropy_cli.py runs the same math as the module from the command line.
Over the eight labelled files in sample_data/ (on Windows, py -3 in place
of python3):
python3 tests/entropy_cli.py sample_data/*.bin sample_data/*.gz sample_data/*.sqlite sample_data/*.pdf sample_data/*.csv sample_data/*.txtfile Shannon Min Renyi Collisn flag 2nd stage
----------------------------------------------------------------------------------------------------------------------
flagged_encrypted_aes256.bin 7.995 7.710 7.985 7.990 ENCRYPTED? random 0/2 @16K
flagged_random_bytes.bin 7.999 7.864 7.997 7.998 ENCRYPTED? random 0/3 @64K
limitation_compressed.tar.gz 7.999 7.807 7.996 7.997 ENCRYPTED? random 0/2 @64K
limitation_deflate_prose.gz 7.996 7.623 7.988 7.992 ENCRYPTED? compressed 3/3 @64K
minclears_database.sqlite 6.883 4.578 5.554 5.939 ENCRYPTED? (Min would clear)
minclears_document.pdf 7.396 2.728 4.090 5.287 ENCRYPTED? (Min would clear)
trueneg_data_table.csv 4.869 2.949 4.062 4.408
trueneg_text_english.txt 4.183 2.653 3.539 3.789
----------------------------------------------------------------------------------------------------------------------
scanned 8 files | Shannon flagged 6 | Min flagged 4 | false positives Min avoids: 2 | Min hits that look compressed: 1
Shannon flags the PDF and the SQLite database, and Min clears both while still catching the AES ciphertext and the random file. The two compressed files are flagged by every measure. That is the known limit of entropy-based detection, since dense compression and encryption look alike at the byte level, and it is what the second stage is for.
Byte entropy of any kind cannot separate a gzip stream from a ciphertext, so on the files the first stage flags the module runs the Huffman-coded data discriminator of Garfinkel, Nelson, White and Roussev (DFRWS 2010): rotate a block by a few bytes, subtract the rotated copy from the original byte by byte, and compare the byte histogram of the original with that of the difference by vector cosine. Random or encrypted data stays similar to itself, with a cosine close to 1. Huffman-coded data, a stream of variable-length code words that line up at some shifts, does not. The module pools the four difference histograms (shifts of one to four bytes) into one cosine, which is less noisy than the paper's smallest cosine over the shifts and recognizes more compressed data at the same false positive rate; the paper's statistic is a setting. It judges up to four blocks spread across the file, at the largest of 64 KB, 16 KB and 4 KB that gives the file two whole blocks, and calls the file compressed when at least two of them fall below the threshold for that size. The verdict goes into the result comment, and an option leaves files that look compressed unflagged.
The thresholds are set from encrypted data, as in the paper, but from a lot
of it: tests/compression_eval.py scores 3,200 MB of AES keystream and takes
the score that 1 in 10,000 blocks fall below (a relaxed setting uses 1 in
1,000). Encrypted output is indistinguishable from random by design, so the
negative class can be made as large as needed, and the per-block false
positive rate is then known to within about a factor of two, rather than
being whatever the smallest score in a training sample of 0.1% of the data
happened to be. Under the two-block rule an encrypted file of four sampled
blocks is miscalled with probability under 1 in 10 million, at most 2.5 in
10 million.
Ciphertext interleaved with plaintext structure produces the same departure from randomness at short byte lags that Huffman coding does: an encrypted PDF keeps its syntax around the encrypted streams, an encrypted archive keeps its headers between small members, and a block cipher in ECB mode repeats ciphertext blocks wherever the plaintext repeats. Such files are called compressed, and the verdict says so. The evaluation reports them as a family of their own, and the option to leave compressed-looking files unflagged should stay off where password-protected documents or archives matter. Autopsy's own Encryption Detection module checks PDFs, Office documents and Access databases for password protection by their structure and can run alongside.
The same script scores compressed data of several kinds: deflate, bzip2 and xz encodings of synthetic prose and logs, the corpus's CSV table and SQLite database, the Python standard library's source, Python's documentation, a DLL, and the text and binary files of a govdocs1 thread, plus real jars from an Autopsy install and PNG and JPEG images from Windows and govdocs1: 273 files, 336 MB. Blocks below the default thresholds, with AES-encrypted copies of the synthetic and govdocs1 files and the structured ciphertext alongside:
| Blocks | Encrypted copies | Structured ciphertext | deflate | bzip2 | JPEG | PNG | xz |
|---|---|---|---|---|---|---|---|
| 64 KB | 0 of 2,618 | 55% of 2,817 | 98% of 3,869 | 74% of 407 | 100% of 172 | 90% of 84 | 19% of 339 |
| 16 KB | 3 of 10,531 | 20% of 11,269 | 78% of 15,578 | 60% of 1,640 | 99% of 765 | 86% of 340 | 12% of 1,370 |
| 4 KB | 6 of 42,214 | 12% of 44,942 | 53% of 62,279 | 40% of 6,547 | 82% of 3,125 | 55% of 1,369 | 6% of 5,500 |
Per file, with the module's own sampling and rule, every deflate file judged in 64 KB or 16 KB blocks is called compressed (139 of 139), 18 of the 26 small ones judged in 4 KB blocks are, and 157 of 165 in all. Nineteen jars have no high-entropy block among those sampled (their members appear to be stored rather than deflated), and two have a single one, which cannot meet the two-block rule. bzip2: 9 of 9. JPEG: 48 of 63, all 20 files of 128 KB and more, 13 of the 14 between 32 KB and 128 KB, and 15 of the 29 smaller ones. PNG: 4 of 5, the miss being the synthetic image. xz: none. None of the 56 encrypted copies is called compressed under any setting; 16 of the 23 structured ciphertext files are, among them every encrypted PDF and every archive of 2 KB ciphertext members, while the archives of 8 KB members are not.
Those counts cover every file. The first stage sees only files whose whole-file Min entropy reaches 6 bits, and most jars fall short of that: the typical jar sits near 5 bits, the NetBeans jars of stored members between 1.6 and 3.4. Most encrypted PDFs fall short too, since their plaintext syntax keeps a few byte values common. Among the files the first stage would flag, the second stage calls compressed 39 of 43 deflate files, 9 of 9 bzip2, 29 of 29 JPEG, 2 of 3 PNG and 0 of 9 xz, none of the encrypted copies, and 6 of the 13 structured ciphertext files.
The paper reported 21% of compressed 4 KB blocks and 48% of 16 KB blocks recognized on compressed text and disk images; from its confusion matrices, 0.0074% and 0.39% of its encrypted blocks were miscalled (the paper's own false positive figures, 0.0197% and 0.45%, count miscalls as a share of all positive calls). With the paper's statistic at the default strictness this corpus gives 47% and 73%, at 3 of 42,214 and 0 of 10,531 encrypted blocks; the pooled statistic gives the table above. The compressed govdocs1 files are recognized in about half of their 4 KB blocks (47% for the text, 45% for the binaries), while deflate of prose, source code and logs, synthetic or real (Python's own documentation), is nearly invisible at that size (under 3% of blocks) and needs 64 KB blocks, where every synthetic source is at 78% or more. Match-heavy text deflates into long matches with short dynamic codes, which leave little byte structure in a 4 KB block; literal-heavy data does not, and the same text encoded with literals only (zlib's Huffman-only strategy) is recognized in every block. Beyond deflate, bzip2 is Huffman coded and recognized, and JPEG's entropy-coded data is the clearest signal of all. xz is a range coder whose own output looks random; the xz blocks recognized here are already-compressed content embedded in the govdocs1 Office files (deflate streams and images), which LZMA cannot compress and re-emits with much of its byte structure. Deflate that stores incompressible input as it is, such as a gzip of random data, looks random because it is.
For the formats a header settles, the module also notes the type Autopsy
identified (Type: application/x-gzip, a compressed format.), asking
Autopsy's file type detector when the identification module did not run,
and a setting leaves such files unflagged. Both skip options are aids for
triage, not defences: the type comes from the file's header, and the second
stage samples blocks at offsets that follow from the file size, so a file
made to pass them will pass them. Bit-level rotations and more shifts were
tried as well and gained a point or two at two to three times the cost, so
they are not shipped. The evaluation script and the output of the run these
numbers come from, tests/compression_eval_results.txt, are in the
repository.
ExpandedEntropy/ the folder to copy into Autopsy's python_modules
expanded_entropy.py factory, file ingest module, settings and panel
entropy_measures.py the four measures, runs on Jython 2.7 and CPython 3
huffman_discriminator.py the second stage (Garfinkel et al., 2010)
README.md install steps and settings reference
tests/
test_entropy.py the formulas (CPython)
test_huffman_discriminator.py the second stage's mechanics and separation
test_entropy_cli.py the scanner samples what the module samples
test_ingest_module.py the ingest module's behaviour (CPython, fakes for Java)
jython_module_test.py the module under Autopsy's real Jython and jars
run_jython_tests.py launcher for the above
autopsy_e2e.py a real Autopsy ingest over the sample corpus
autopsy_env.py finds the local Autopsy install
compression_eval.py compressed-versus-encrypted evaluation of the second stage
compression_eval_results.txt output of the run the documented figures come from
entropy_cli.py command-line scanner for files or folders
sample_data/ labelled corpus, see its README
paper/ the paper: motivation, design, evidence
- In Autopsy, Tools > Python Plugins opens the
python_modulesfolder. - Copy the whole
ExpandedEntropy/folder into it (the__pycache__folder and$py.classfiles that the tests leave behind can be left out). - Restart Autopsy.
- When running ingest, tick Expanded Entropy and click it to choose the measure and threshold.
Results appear under Analysis Results > Encryption Suspected. The settings are described in ExpandedEntropy/README.md.
Add sample_data/ as a Logical Files data source and run ingest with only
this module enabled. With the default Min setting, Encryption Suspected lists
four files: the two flagged_ files and the two limitation_ files, and the
comment on the gzip of prose says it looks compressed.
Switch the measure to Shannon and run again, and the two minclears_ files
join them.
Those two are the false positives that the default setting avoids. Per-file
expectations and measured values are in
sample_data/README.md.
The tests are arranged in three layers, and each layer covers something the
previous one does not. The commands below say python3; on Windows use
py -3 in its place, since the python3 shim that Windows installs is not
Python.
1. The math and the module's behaviour, plain Python 3, no Autopsy needed. From the repository root:
python3 -m unittest discover -s tests -vtest_entropy.py checks the four formulas against known values, the Rényi
special cases and limits, the direct reading of the Min threshold, and the
ordering Shannon ≥ Collision ≥ Rényi(3) ≥ Min that makes Min the most
conservative measure. test_ingest_module.py runs the ingest module's own
code against small stand-ins for the Java and Autopsy classes it imports,
including the one Jython trait its error handling depends on (Java exceptions
are not Python exceptions), and covers: which files are skipped, the
sorted-buffer histogram, chunked, partial and capped reads, cancellation, the
flag decision for every measure, the artifact's type, score and comment, the
API fallback chain, the settings map and panel, one summary per job, the
second stage's sampling, tiers and verdicts, the type note and both skip
options, and the sample corpus. test_huffman_discriminator.py pins the
second stage's mechanics (both statistics, the tiers, the sampling and the
verdict rule) and the separation it relies on. test_entropy_cli.py checks
that the command-line scanner samples the blocks the module samples. 175
tests.
2. Under Autopsy's own interpreter and jars.
python3 tests/run_jython_tests.pyRuns jython_module_test.py with the Jython 2.7 that Autopsy bundles and the
Sleuth Kit and Autopsy jars on the classpath, headless. It confirms that the
module compiles under Jython; that its factory, settings, module and panel are
instances of Autopsy's interfaces; that the settings survive Java
serialization, which is how Autopsy stores them; that every Java class and
method the module calls exists in the installed version (a real jarray
buffer sorted with Arrays.sort, a real TSK_COMMENT attribute, the real
artifact type and score, a real Swing panel); and that Java exceptions from
the file and Blackboard APIs are logged rather than escaping; that the
second stage's sampled blocks come out of the real byte buffer and reach the
right verdicts on the corpus under every statistic and strictness; and that
Autopsy's real file type detector can be built and its failure on a file is
survived. 38 tests. Needs Autopsy installed; set AUTOPSY_HOME if it is not
under Program Files.
3. A real Autopsy ingest.
python3 tests/autopsy_e2e.pyInstalls the module into Autopsy's python_modules, runs Autopsy's command
line ingest over sample_data/ as a logical files data source with Autopsy's
own modules disabled and this one enabled, then opens the case database and
compares the Encryption Suspected results with the corpus labels. Autopsy
opens briefly; allow a few minutes for the whole run the first time. The
script replaces any ExpandedEntropy folder already in your python_modules
and rewrites Autopsy's command-line ingest profile for the duration of the
run, restoring it afterwards. Start Autopsy once before the first run so its
profile folder exists. Output from Autopsy 4.23.1, trimmed to the report:
Autopsy ran for 14 s; the ingest job itself took 0.6 s
Ingest modules that ran: Expanded Entropy 1.2.0
file bytes expected autopsy type second stage
-------------------------------------------------------------------------------------------------------------
README.md 4015 clear clear
flagged_encrypted_aes256.bin 36432 flag flagged 0 of 2 sampled 16 KB blocks look Huffman-coded rather than random; consistent with random or encrypted data.
flagged_random_bytes.bin 196608 flag flagged 0 of 3 sampled 64 KB blocks look Huffman-coded rather than random; consistent with random or encrypted data.
generate_corpus.py 7032 clear clear
limitation_compressed.tar.gz 150235 flag flagged application/x-gzip, a compressed format. 0 of 2 sampled 64 KB blocks look Huffman-coded rather than random; consistent with random or encrypted data.
limitation_deflate_prose.gz 206067 flag flagged application/x-gzip, a compressed format. 3 of 3 sampled 64 KB blocks look Huffman-coded rather than random; likely compressed; ciphertext interleaved with plaintext structure, as in an encrypted PDF or archive, reads the same way.
minclears_database.sqlite 974848 clear clear
minclears_document.pdf 47322 clear clear
trueneg_data_table.csv 57960 clear clear
trueneg_text_english.txt 49050 clear clear
-------------------------------------------------------------------------------------------------------------
10 files checked, 4 flagged, 0 mismatches
analysis result significance: LIKELY_NOTABLE
The command-line profile the script writes carries no saved settings for the module, so the run uses the module's defaults, Min entropy at 6.0 with the second stage on; the other measures are covered by the first two layers. The type column is the note from Autopsy's file type detector and the last column is the second stage's verdict; both come from the end of the same comment, which starts the same way for every flagged file. The generator script and the README that sit next to the corpus are ingested with it and stay clear.
The second layer found two real bugs in earlier versions of the module. Under
Jython, except Exception does not catch Java exceptions, and a settings
class written in Python can be saved by Autopsy but never loaded again. Both
are fixed, and both now have tests.
Developed and tested against Autopsy 4.23.1 (Jython 2.7.4, Sleuth Kit 4.15.0)
on Windows. Flagging uses AbstractFile.newAnalysisResult with a
LIKELY_NOTABLE score, the same path as Autopsy's own detector, with fallbacks
for older Blackboard APIs. The test scripts look for Autopsy in the usual
places (Program Files on Windows, the application bundle on macOS,
~/autopsy-*, /opt and /usr/share elsewhere); set AUTOPSY_HOME
otherwise.
MIT. See LICENSE.