TEZ-4466: Measure exact stream read time while fetching by Aggarwal-Raghav · Pull Request #530 · apache/tez

Aggarwal-Raghav · 2026-08-05T17:52:57Z

new MeasuredDataInputStream( 
    new MeasuredInputStream(            //  STOPWATCH WRAPPER (the actual logic)
        new DataInputStream(          
            new BufferedInputStream(    
                connection.getInputStream()
            )
        )
    )
)

tez-yetus · 2026-08-05T19:14:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+1 💚	mvninstall	2m 58s		master passed
+1 💚	compile	2m 32s		master passed
+1 💚	checkstyle	0m 28s		master passed
+1 💚	javadoc	0m 25s		master passed
+0 🆗	spotbugs	0m 47s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 15s		the patch passed
+1 💚	codespell	0m 50s		No new issues.
+1 💚	compile	2m 32s		the patch passed
+1 💚	javac	2m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 18s	/results-checkstyle-tez-runtime-library.txt	tez-runtime-library: The patch generated 1 new + 90 unchanged - 0 fixed = 91 total (was 90)
+1 💚	javadoc	0m 24s		the patch passed
+1 💚	spotbugs	0m 54s		the patch passed
			_ Other Tests _
-1 ❌	unit	60m 21s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		76m 17s

Reason	Tests
Failed junit tests	tez.runtime.library.api.TestTezRuntimeConfiguration

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/2/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 95db0ba57cb8 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/2/testReport/
Max. process+thread count	2124 (vs. ulimit of 5500)
modules	C: tez-runtime-library U: tez-runtime-library
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2026-08-05T20:42:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+1 💚	mvninstall	3m 7s		master passed
+1 💚	compile	2m 34s		master passed
+1 💚	checkstyle	0m 25s		master passed
+1 💚	javadoc	0m 25s		master passed
+0 🆗	spotbugs	0m 48s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 21s		the patch passed
+1 💚	codespell	0m 49s		No new issues.
+1 💚	compile	2m 35s		the patch passed
+1 💚	javac	2m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed
+1 💚	spotbugs	0m 53s		the patch passed
			_ Other Tests _
+1 💚	unit	62m 27s		root in the patch passed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		78m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/3/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 86ed268fca1e 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/3/testReport/
Max. process+thread count	1444 (vs. ulimit of 5500)
modules	C: tez-runtime-library U: tez-runtime-library
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/3/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2026-08-06T07:45:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 53s		master passed
+1 💚	compile	2m 28s		master passed
+1 💚	checkstyle	0m 44s		master passed
+1 💚	javadoc	0m 51s		master passed
+0 🆗	spotbugs	0m 53s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 45s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 26s		the patch passed
+1 💚	codespell	0m 54s		No new issues.
+1 💚	compile	2m 34s		the patch passed
+1 💚	javac	2m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed
+1 💚	spotbugs	1m 51s		the patch passed
			_ Other Tests _
-1 ❌	unit	57m 52s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		78m 17s

Reason	Tests
Failed junit tests	tez.runtime.library.common.shuffle.TestFetcher

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/4/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux c6758e3de655 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/4/testReport/
Max. process+thread count	2123 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/4/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Aggarwal-Raghav · 2026-08-06T17:46:28Z

Ran basic groupby query, SET tez.runtime.optimize.local.fetch=false; ran with this on local cluster.

--
SELECT 
  group_key, 
  COUNT(id), 
  SUM(rand_val1), 
  MAX(padding)
FROM large_data
GROUP BY group_key;

tez-yetus · 2026-08-06T17:50:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 54s		master passed
+1 💚	compile	2m 33s		master passed
+1 💚	checkstyle	0m 43s		master passed
+1 💚	javadoc	0m 53s		master passed
+0 🆗	spotbugs	0m 50s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 45s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 12s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 32s		the patch passed
+1 💚	javac	2m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	javadoc	0m 48s		the patch passed
+1 💚	spotbugs	1m 50s		the patch passed
			_ Other Tests _
+1 💚	unit	59m 59s		root in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		80m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/6/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 13e32d590552 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/6/testReport/
Max. process+thread count	1380 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/6/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Aggarwal-Raghav · 2026-08-06T18:01:38Z

This is ready @abstractdog from my end, requesting your eagle eye review.

Aggarwal-Raghav · 2026-08-06T18:13:10Z

    this.badIdErrs = badIdErrsCounter;
    this.connectionErrs = connectionErrsCounter;
    this.wrongReduceErrs = wrongReduceErrsCounter;
+    this.ioTimeCounter = inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_IO_TIME_MILLISECONDS);


1 thing to call out is even tez.runtime.shuffle.measure.io.time=false the counter will have SHUFFLE_IO_TIME_MILLISECONDS: 0 to prevent this the following is required or we can have it but might be misleading looking at the counter stats..

this.ioTimeCounter = conf.getBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_MEASURE_IO_TIME, TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_MEASURE_IO_TIME_DEFAULT) ? inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_IO_TIME_MILLISECONDS) : null;

Addressed in b1dd37b

Aggarwal-Raghav · 2026-08-06T18:37:11Z

New test:
With set tez.runtime.shuffle.measure.io.time=false;

With set tez.runtime.shuffle.measure.io.time=true;

tez-yetus · 2026-08-06T19:57:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 52s		master passed
+1 💚	compile	2m 31s		master passed
+1 💚	checkstyle	0m 43s		master passed
+1 💚	javadoc	0m 52s		master passed
+0 🆗	spotbugs	0m 49s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 45s		tez-runtime-library in master has 143 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 11s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 30s		the patch passed
+1 💚	javac	2m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	javadoc	0m 49s		the patch passed
+1 💚	spotbugs	1m 46s		the patch passed
			_ Other Tests _
+1 💚	unit	61m 24s		root in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		81m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/7/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux cb72d4f2d834 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `d348bf1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/7/testReport/
Max. process+thread count	2124 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-530/7/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 0120635 to e459166 Compare August 5, 2026 17:57

Aggarwal-Raghav mentioned this pull request Aug 5, 2026

TEZ-4748: Expose SHUFFLE_IO_TIME_MILLISECONDS via TaskCounter #531

Closed

Aggarwal-Raghav force-pushed the TEZ-4466 branch from e459166 to 4ad0480 Compare August 5, 2026 19:23

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 4ad0480 to 3bd8eaa Compare August 6, 2026 06:26

Aggarwal-Raghav force-pushed the TEZ-4466 branch from 3bd8eaa to d99959a Compare August 6, 2026 10:34

TEZ-4466: Measure exact stream read time while fetching

65ef7ac

Aggarwal-Raghav force-pushed the TEZ-4466 branch from d99959a to 65ef7ac Compare August 6, 2026 16:29

Aggarwal-Raghav commented Aug 6, 2026

View reviewed changes

Fix SHUFFLE_IO_TIME off counter stat behaviour

b1dd37b

Navigation Menu

Uh oh!

TEZ-4466: Measure exact stream read time while fetching - #530

TEZ-4466: Measure exact stream read time while fetching#530
Aggarwal-Raghav wants to merge 2 commits into
apache:masterfrom
Aggarwal-Raghav:TEZ-4466

Aggarwal-Raghav commented Aug 5, 2026 •

edited

Loading

Uh oh!

tez-yetus commented Aug 5, 2026

Uh oh!

tez-yetus commented Aug 5, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav Aug 6, 2026

Uh oh!

Aggarwal-Raghav Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Aggarwal-Raghav commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tez-yetus commented Aug 5, 2026

Uh oh!

tez-yetus commented Aug 5, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

tez-yetus commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Aggarwal-Raghav commented Aug 5, 2026 •

edited

Loading