[SVLS-9189] feat(logs): add config param DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE by lym953 · Pull Request #1239 · DataDog/datadog-lambda-extension

lym953 · 2026-05-27T19:10:24Z

Background

For durable functions, the extension holds logs and waits to get the invocation context from the tracer so the extension can enrich the logs.

Problem

When tracing is off (which is not recommended for durable functions), logs are flushed when:

the buffer is full (buffer size is all logs for 50 invocations). This increases delay. Or
when extension shuts down. However, this is not guaranteed to happen due to race conditions, causing missing logs.

This PR

Adds lambda_durable_function_log_buffer_size config field (env var DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE, default: 0) controlling the max number of request ID keys held in held_logs.
When set to 0, logs bypass the hold mechanism and are sent immediately without durable execution context enrichment.
Setting the default to 0 for now because the tracer changes for durable functions are not released. Later, after tracer changes are released, we can change the default back to 50 and ask customers to install the tracer for durable functions.

Test plan

Steps

Build an extension layer
install it on a durable function
execute the function
wait for a few minutes until the environment shuts down

Result

Before:

No log appears in Datadog
and the invocation doesn't appear either

After:

The invocation appears

Logs also appear

Adds `durable_function_log_buffer_size` (env: `DD_DURABLE_FUNCTION_LOG_BUFFER_SIZE`, default: 50) to control the max number of request ID keys held in `held_logs` waiting for durable execution context. Setting it to 0 disables holding entirely, so logs are flushed immediately without durable context enrichment — useful when the tracer is not installed and the buffer would otherwise delay log delivery until full or until extension shutdown. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Default to 0 (disabled) until tracer-side durable execution support is released. Customers can set DD_DURABLE_FUNCTION_LOG_BUFFER_SIZE=50 to opt into context enrichment once the tracer is available. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds a new DD_DURABLE_FUNCTION_LOG_BUFFER_SIZE configuration option that controls the maximum number of request IDs whose logs are held in the durable-function log buffer. The previous hardcoded constant (HELD_LOGS_MAX_KEYS = 50) is replaced by this configurable value, defaulted to 0, which bypasses the hold mechanism entirely so logs are flushed immediately without durable execution context enrichment. This avoids missing-log issues when the tracer is not installed on durable functions.

Changes:

New durable_function_log_buffer_size config field (env + yaml + default), defaulting to 0.
LambdaProcessor reads this value, replacing the HELD_LOGS_MAX_KEYS constant, and short-circuits the hold path when set to 0.
Existing durable-arn tests explicitly set held_logs_max_keys = 50 to preserve prior behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
bottlecap/src/config/mod.rs	Adds `durable_function_log_buffer_size: usize` field with doc and default 0.
bottlecap/src/config/env.rs	Adds env var deserialization, merge call, and updates default-config test.
bottlecap/src/config/yaml.rs	Updates yaml test expected struct with new field.
bottlecap/src/logs/lambda/processor.rs	Replaces constant with configurable cap; adds zero-disables fast path; updates tests.

litianningdatadog · 2026-05-27T20:47:31Z


        let processing_rules = &datadog_config.logs_config_processing_rules;
        let logs_enabled = datadog_config.serverless_logs_enabled;
+        let held_logs_max_keys = datadog_config.durable_function_log_buffer_size;


nit: rename held_logs_max_keys to durable_function_log_buffer_size or sth. similar for better readability.

Renamed it as lambda_durable_function_log_buffer_size

duncanista · 2026-05-28T16:33:09Z

+    /// durable execution context enrichment. Useful when the tracer is not installed.
+    /// Default is `0`.
+    #[serde(deserialize_with = "deserialize_option_lossless")]
+    pub durable_function_log_buffer_size: Option<usize>,


Wondering if this should include the LAMBDA prefix, since we're gonna migrate to the config package from serverless-components eventually,

that way it makes it more descriptive that its for the lambda product

Renamed it as lambda_durable_function_log_buffer_size

I don't see the rename?

I see it only in the processor, not the config

Good catch. Fixed.

…log_buffer_size Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…lambda_durable_function_log_buffer_size Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

litianningdatadog

left some comments

…e loop Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…m 0 to 5 (#1324) ## Background `DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE` means how many logs (in terms of number of invocations) the extension buffers to wait for enrichment. If it's N (which is 5 right now), then the extension holds all logs for up to N invocations, waits for traces for these invocations to be sent from the tracer, then uses the metadata from the traces to enrich the logs. If the function is not a durable function, then the extension holds all logs (for up to N invocations) at cold start. Then, as soon as it receives the `platform.InitStart` event and learns that the function is not a durable function, it releases all held logs and no longer holds logs. Ideally, its default value should be a non-zero value, so the behavior is correct for both durable functions and non-durable functions, i.e.: - logs for durable functions are held, and - logs for non-durable functions are not held. Actually, the default was 5 at the beginning. However, when the log holding logic was released, tracer-side changes that adds the necessary metadata to traces had not been released. As a result, the extension never gets the metadata it's waiting for, and some logs are somehow dropped. Therefore, #1239 changes the default back to 0 as a temporary mitigation of this issue. Now, all the tracer-side changes (`datadog-lambda-js`, `datadog-lambda-python`) have been released, so we can change the default back to 5. The dropped-logs problem won't happen as long as the user uses the latest version for the tracer. ## Overview Changes the default of `lambda_durable_function_log_buffer_size` (`DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE`) from `0` to `5`, so instrumenting a durable function no longer requires setting this env var to get logs enriched with durable execution context. ## Testing - `cargo test --lib` — 541 passed, 0 failed - `cargo clippy --all-targets -- -D warnings` — clean - `cargo fmt --check` — clean No manual test on a live durable function yet. ### Existing integration tests **Non-durable functions — covered.** These suites ensures logs for non-durable functions are flushed correctly: - `integration-tests/tests/on-demand.test.ts` — node/python/java/dotnet, 2 invocations each; asserts the `Hello world!` log is retrievable by request ID for both the cold and warm invocation. - `integration-tests/tests/lmi.test.ts` — Managed Instance mode; asserts logs exist and that the `Hello world!` log is present. - `integration-tests/tests/auth.test.ts` — asserts `logs.length > 0` under delegated auth. **Durable functions — not covered**. However, we have done lots of manual tests using `DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE == 5` and ensured this works well for durable functions. ## Next steps: Update onboarding doc saying if extension v100+ is used, then the user doesn't need to set `DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE` to 5 since it's 5 by default. 🤖 Partially generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

This comment has been minimized.

Sign in to view

lym953 changed the title ~~feat(logs): add DD_DURABLE_FUNCTION_LOG_BUFFER_SIZE config param~~ [SVLS-9189] feat(logs): add config param DD_DURABLE_FUNCTION_LOG_BUFFER_SIZE May 27, 2026

lym953 marked this pull request as ready for review May 27, 2026 20:39

lym953 requested a review from a team as a code owner May 27, 2026 20:39

lym953 requested review from Copilot and litianningdatadog May 27, 2026 20:39

Copilot started reviewing on behalf of lym953 May 27, 2026 20:39 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

litianningdatadog reviewed May 28, 2026

View reviewed changes

duncanista reviewed May 28, 2026

View reviewed changes

lym953 and others added 3 commits June 1, 2026 11:57

refactor(logs): rename held_logs_max_keys to lambda_durable_function_…

192655b

…log_buffer_size Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

style: fix cargo fmt

947c6a4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor(logs): rename remaining durable_function_log_buffer_size to …

47442b3

…lambda_durable_function_log_buffer_size Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

litianningdatadog reviewed Jun 4, 2026

View reviewed changes

Comment thread bottlecap/src/config/env.rs

litianningdatadog reviewed Jun 4, 2026

View reviewed changes

Comment thread bottlecap/src/logs/lambda/processor.rs

litianningdatadog reviewed Jun 4, 2026

View reviewed changes

lym953 changed the title ~~[SVLS-9189] feat(logs): add config param DD_DURABLE_FUNCTION_LOG_BUFFER_SIZE~~ [SVLS-9189] feat(logs): add config param DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE Jun 8, 2026

fix(logs): guard hold_log against zero buffer size to prevent infinit…

fe0be91

…e loop Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

litianningdatadog approved these changes Jun 12, 2026

View reviewed changes

Merge branch 'main' into yiming.luo/durable-log-buffer-size

eb28aa7

lym953 merged commit f8b1e0b into main Jun 12, 2026
59 checks passed

lym953 deleted the yiming.luo/durable-log-buffer-size branch June 12, 2026 17:53

lym953 mentioned this pull request Aug 13, 2026

[SVLS-8582] feat(logs): change default of durable log buffer size from 0 to 5 #1324

Merged

Navigation Menu

Uh oh!

[SVLS-9189] feat(logs): add config param DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE - #1239

[SVLS-9189] feat(logs): add config param DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE#1239
lym953 merged 7 commits into
mainfrom
yiming.luo/durable-log-buffer-size

lym953 commented May 27, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Uh oh!

litianningdatadog May 27, 2026

Uh oh!

lym953 Jun 1, 2026

Uh oh!

duncanista May 28, 2026

Uh oh!

lym953 Jun 1, 2026

Uh oh!

duncanista Jun 4, 2026

Uh oh!

duncanista Jun 4, 2026

Uh oh!

lym953 Jun 4, 2026

Uh oh!

Uh oh!

Uh oh!

litianningdatadog left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

lym953 commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Problem

This PR

Test plan

Steps

Result

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

litianningdatadog May 27, 2026

Choose a reason for hiding this comment

Uh oh!

lym953 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

duncanista May 28, 2026

Choose a reason for hiding this comment

Uh oh!

lym953 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

duncanista Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

duncanista Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

lym953 Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

litianningdatadog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lym953 commented May 27, 2026 •

edited

Loading