[v25.3.x] [CORE-16880] kafka: autocreate internal topics with correct properties

[vbotbuildovich]

Backport of PR #31234

• Command: git cherry-pick -x 5f2855f f2d1854 932c5bc 2be356f f6ac88a
• Commits backported: 5
• Conflicts resolved: 3
• Commits skipped (already on target): 0
• Backport branch: ai-backport-pr-31234-v25.3.x-1784756347

Conflict details

• 5f2855f (src/v/security/audit/BUILD): the target branch uses the monolithic //src/v/cluster target while the source branch splits it into fine-grained sub-targets. Kept the umbrella //src/v/cluster and added only //src/v/cluster:topic_configuration (the dep the commit actually introduced, needed by the new audit_log_topic.cc/.h which use cluster::topic_properties); dropped the other cluster sub-deps that are only present on the source branch's split layout.
• 932c5bc (src/v/cluster/BUILD): huge conflict spanning the whole cluster library region because the source branch splits it into many fine-grained targets absent on the target branch. The commit's only change here was adding //src/v/config to the members_table target; on the target branch members_table.cc is compiled by the monolithic cluster target, which already depends on //src/v/config, so the intent is already satisfied. Resolved to the target-branch (HEAD) version.
• 932c5bc (src/v/kafka/server/BUILD): same split-vs-monolithic divergence. Kept HEAD and added only //src/v/cluster:topic_configuration (needed by group_initializer.cc, which now uses cluster::topic_configuration); the other incoming cluster sub-deps are provided by the umbrella //src/v/cluster on the target branch.
• 2be356f (src/v/kafka/server/handlers/metadata.cc): two include-block conflicts (target branch had extra base/likely.h and utils/to_string.h includes) resolved by keeping both sides in sorted order and adding the commit's new cluster/members_table.h and security/audit/audit_log_topic.h includes. One body conflict where the commit removed the inline default topic_configuration cfg block (now handled by the merged-in autocreate_topic_configuration()); applied that removal while preserving the target branch's tout = create_topic_timeout_ms() (the commit left the tout line as unchanged context; the timeout accessor is a pre-existing branch divergence the commit did not intend to change).

Fixes: #31240,

[vbotbuildovich]

Retry command for Build#87659

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/data_migrations_api_test.py::DataMigrationsApiTest.test_higher_level_migration_api

[vbotbuildovich]

CI test results

test results on build#87659

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(FAIL)	DataMigrationsApiTest	test_higher_level_migration_api	null	integration	https://buildkite.com/redpanda/redpanda/builds/87659#019f90e3-c6ff-4406-895e-70ebf33390c4	9/11	Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DataMigrationsApiTest&test_method=test_higher_level_migration_api