Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,13 @@ The Google Apps Script Productivity Suite is a collection of automation scripts
**Key mechanisms:**

- **Thread deduplication:** Embeds the thread ID in the message separator line `------------------------------[THREAD:<id>]` and uses a separate thread separator line `==============================`. Before processing, checks for existing thread content identified by these markers and removes it (idempotent reprocessing).
- **Attachment deduplication:** MD5 hash-based content dedup. If a file with the same hash exists in the target folder, skips upload. If same name but different content, renames with numeric suffix.
- **Attachment deduplication:** SHA-256 hash-based content dedup. If a file with the same hash exists in the target folder, skips upload. If same name but different content, renames with numeric suffix.
- **Prepend ordering:** Processes threads oldest-first with `insertParagraph(0, ...)` so newest content appears at the top of the document.
- **Batch processing:** Configurable `batchSize` (default 250 threads) to handle GAS execution time limits.
- **Label management:** Removes trigger label, adds processed label after archiving.
- **Rebuild capability:** `rebuildDoc()` and `rebuildAllDocs()` can reconstruct documents from all processed threads.

**Google APIs used:** GmailApp, DriveApp, DocumentApp, Utilities (MD5)
**Google APIs used:** GmailApp, DriveApp, DocumentApp, Utilities (SHA-256)

### calendar-to-sheets

Expand Down
2 changes: 1 addition & 1 deletion docs/component-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
| `rebuildDoc(config, ...)` | src/index.js | Core | Rebuilds single document with state tracking |
| `sortThreadsByLastMessageDate(threads)` | src/index.js | Utility | Sorts threads by last message date ascending (oldest-first) for prepend-based processing |
| `getCleanBody(text)` | gas-utils.js | Shared Utility | Strips quoted replies, legal footers, normalizes whitespace |
| `getFileHash(blob)` | gas-utils.js | Shared Utility | MD5 hash for content-based attachment deduplication |
| `getFileHash(blob)` | gas-utils.js | Shared Utility | SHA-256 hash for content-based attachment deduplication |
| `getProcessConfig()` | config.gs | Config | Returns array of label/doc/folder configuration objects |

### calendar-to-sheets
Expand Down
4 changes: 2 additions & 2 deletions src/gas-utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ function getCleanBody(text) {
}

/**
* getFileHash - compute MD5 hex digest of a "blob" (Buffer or object with getBytes())
* getFileHash - compute SHA-256 hex digest of a "blob" (Buffer or object with getBytes())
*/
function getFileHash(blob) {
let bytes
Expand All @@ -56,7 +56,7 @@ function getFileHash(blob) {
throw new TypeError('Unsupported blob type')
}

const hash = crypto.createHash('md5').update(bytes).digest('hex')
const hash = crypto.createHash('sha256').update(bytes).digest('hex')
return hash
}

Expand Down
4 changes: 2 additions & 2 deletions src/gmail-to-drive-by-labels/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ A robust Google Apps Script designed to automate the archiving of Gmail threads.

**Content-Based De-duplication:**

- Uses MD5 hashing (digital fingerprinting) to detect if a file is an exact duplicate of one already in the folder, even if the filename is different.
- Uses SHA-256 hashing (digital fingerprinting) to detect if a file is an exact duplicate of one already in the folder, even if the filename is different.

**Safe Renaming:**

Expand Down Expand Up @@ -82,7 +82,7 @@ sequenceDiagram
Script->>Drive: Check if file exists (by name)

alt File exists
Script->>Drive: Compare MD5 hashes
Script->>Drive: Compare SHA-256 hashes

alt Hashes match (duplicate)
Script->>Doc: Insert "[DUPLICATE SKIPPED]"
Expand Down
6 changes: 3 additions & 3 deletions src/gmail-to-drive-by-labels/code.gs
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@ function processLabelGroup(config) {
if (existingFile.getSize() === newFileBlob.getBytes().length) {
console.log('[processLabelGroup] Size match, checking hash')

// 3. Deep Check: Compare MD5 Hashes (The "Fingerprint")
// 3. Deep Check: Compare SHA-256 Hashes (The "Fingerprint")
var existingHash = getFileHash(existingFile.getBlob())
var newHash = getFileHash(newFileBlob)

Expand Down Expand Up @@ -658,12 +658,12 @@ function getCleanBody(text) {
}

/**
* Generates an MD5 hash (fingerprint) for a file blob.
* Generates a SHA-256 hash (fingerprint) for a file blob.
* Returns a string representing the binary content.
*/
function getFileHash(blob) {
var digest = Utilities.computeDigest(
Utilities.DigestAlgorithm.MD5,
Utilities.DigestAlgorithm.SHA_256,
blob.getBytes()
)
return digest
Expand Down
4 changes: 2 additions & 2 deletions src/gmail-to-drive-by-labels/src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@

/**
* Determine whether an attachment already exists in the folder by comparing
* file size (fast fail) then MD5 hash against every existing file of the same name.
* file size (fast fail) then SHA-256 hash against every existing file of the same name.
*
* @param {Object} existingFiles - Drive file iterator (from getFilesByName)
* @param {Object} newFileBlob - Blob of the incoming attachment
Expand Down Expand Up @@ -238,7 +238,7 @@
if (DocumentApp) {
try {
headingPara.setHeading(DocumentApp.ParagraphHeading.HEADING_3)
} catch (e) {

Check warning on line 241 in src/gmail-to-drive-by-labels/src/index.js

View workflow job for this annotation

GitHub Actions / autofix

'e' is defined but never used

Check warning on line 241 in src/gmail-to-drive-by-labels/src/index.js

View workflow job for this annotation

GitHub Actions / autofix

'e' is defined but never used
const style = {}
style[DocumentApp.Attribute.BOLD] = true
headingPara.setAttributes(style)
Expand Down Expand Up @@ -541,7 +541,7 @@
// Try to set heading, fallback to bold if Doc is busy
try {
headingPara.setHeading(DocumentApp.ParagraphHeading.HEADING_3)
} catch (e) {

Check warning on line 544 in src/gmail-to-drive-by-labels/src/index.js

View workflow job for this annotation

GitHub Actions / autofix

'e' is defined but never used

Check warning on line 544 in src/gmail-to-drive-by-labels/src/index.js

View workflow job for this annotation

GitHub Actions / autofix

'e' is defined but never used
const style = {}
style[DocumentApp.Attribute.BOLD] = true
headingPara.setAttributes(style)
Expand Down Expand Up @@ -593,7 +593,7 @@
if (existingFile.getSize() === newFileBlob.getBytes().length) {
console.log('[processLabelGroup] Size match, checking hash')

// 3. Deep Check: Compare MD5 Hashes (The "Fingerprint")
// 3. Deep Check: Compare SHA-256 Hashes (The "Fingerprint")
const existingHash = getFileHash(existingFile.getBlob())
const newHash = getFileHash(newFileBlob)

Expand Down
18 changes: 12 additions & 6 deletions src/gmail-to-drive-by-labels/tests/gas-utils.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -122,19 +122,25 @@ describe('getCleanBody', () => {
})

describe('getFileHash', () => {
test('computes md5 for a Buffer', () => {
test('computes sha256 for a Buffer', () => {
const buf = Buffer.from('hello world')
expect(getFileHash(buf)).toBe('5eb63bbbe01eeed093cb22bb8f5acdc3')
expect(getFileHash(buf)).toBe(
'b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'
)
})

test('computes md5 for an object with getBytes()', () => {
test('computes sha256 for an object with getBytes()', () => {
const blob = { getBytes: () => Buffer.from('abc') }
expect(getFileHash(blob)).toBe('900150983cd24fb0d6963f7d28e17f72')
expect(getFileHash(blob)).toBe(
'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad'
)
})

test('computes md5 for an object with bytes property', () => {
test('computes sha256 for an object with bytes property', () => {
const blob = { bytes: Buffer.from('test') }
expect(getFileHash(blob)).toBe('098f6bcd4621d373cade4e832627b4f6')
expect(getFileHash(blob)).toBe(
'9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
)
})

test('throws error for unsupported blob type', () => {
Expand Down
6 changes: 3 additions & 3 deletions test-utils/setup.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,17 @@ global.Utilities = {
},
// Provide a simple sleep stub used in code
sleep: (ms) => {},
// Mock MD5 hash computation
// Mock SHA-256 hash computation
computeDigest: (algorithm, bytes) => {
// Simple deterministic hash for testing
// Convert bytes to a string and create a fake hash
const crypto = require('node:crypto')
const hash = crypto.createHash('md5').update(Buffer.from(bytes)).digest()
const hash = crypto.createHash('sha256').update(Buffer.from(bytes)).digest()
// Return as array of numbers (like GAS does)
return Array.from(hash)
},
DigestAlgorithm: {
MD5: 'MD5',
SHA_256: 'SHA_256',
},
Charset: {
US_ASCII: 'US-ASCII',
Expand Down
Loading