Repository navigation
Zip archiver v2.10.1 needs much more heap than v2.9.1 #5
Description
In the Eclipse Tycho project we discoverd that the zip plexus archiver v2.10.1 does need much more heap than in version 2.9.1 (https://bugs.eclipse.org/bugs/show_bug.cgi?id=470074).
The exception we see:
Caused by: org.apache.maven.plugin.MojoExecutionException: Error packing p2 repository
at org.eclipse.tycho.plugins.p2.repository.ArchiveRepositoryMojo.execute(ArchiveRepositoryMojo.java:55)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: org.codehaus.plexus.archiver.ArchiverException: Problem creating zip: Execution exception (and the archive is probably corrupt but I could not delete it)
at org.codehaus.plexus.archiver.AbstractArchiver.createArchive(AbstractArchiver.java:1007)
at org.eclipse.tycho.plugins.p2.repository.ArchiveRepositoryMojo.execute(ArchiveRepositoryMojo.java:53)
... 21 more
Caused by: java.io.IOException: Execution exception
at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.close(AbstractZipArchiver.java:764)
at org.codehaus.plexus.archiver.AbstractArchiver.createArchive(AbstractArchiver.java:994)
... 22 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:121)
at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.write(ByteArrayOutputStream.java:152)
at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:129)
at org.codehaus.plexus.archiver.zip.DeferredScatterOutputStream.writeOut(DeferredScatterOutputStream.java:36)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor$ScatterGatherBackingStoreCompressor.writeOut(StreamCompressor.java:274)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor.writeCounted(StreamCompressor.java:257)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:248)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded(StreamCompressor.java:241)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor.writeDeflated(StreamCompressor.java:222)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor.write(StreamCompressor.java:190)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:169)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.ScatterZipOutputStream.addArchiveEntry(ScatterZipOutputStream.java:97)
at org.codehaus.plexus.archiver.commonscompress.archivers.zip.ParallelScatterZipCreator$2.call(ParallelScatterZipCreator.java:175)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[ERROR]
[ERROR]
Activity
krosenvold commentedon Jun 16, 2015
Due to the parallel compression algorithm we use more memory now than earlier versions, this is expected; I can update the docs about this. But do you have any indications that the memory usage is excessive ?
krosenvold commentedon Jun 16, 2015
Issue #6 has just been fixed on 2.x branch and is probably the root cause of this issue. Issue #6 was causing memory usage way beyond the epxpected increase. 2.10.3 will be released soonish.
mpilone commentedon Aug 31, 2015
I'm seeing this same issue when using the Assembly plugin to create a zip file. The assembly plugin v2.5.5 uses the archiver version 2.10.2. I tried overriding the archiver version to 2.10.3 but I get the same problem.
I'm attempting to zip a directory containing about 650 MB of other jars. I have my Maven ops configured for -Xmx512m and the zip creation fails with the heap error. My naive interpretation is that depending on how many CPUs you have and how fast they are, you may need to ensure that the heap size is larger than the maximum zip file you might create which doesn't seem like a great design.
I created a simple test case that consistently reproduces the issue on my machine. Run it with the command
mvn clean compile exec:java package. Download it here: https://www.dropbox.com/s/7n87uklcijm18lp/archivermemory.zip?dl=0This will generate about 600 MB of random binary files and then attempt to create a zip of them using the assembly plugin. On my MacBook Pro with 8 processors/cores and -Xmx512m this always results in a heap error. My guess is that you'll have to tweak the values depending on the machine. Having to tweak the numbers is part of the problem because the failure isn't consistent on other machines including my CI server which makes it hard to debug across a team.
Relevant output:
mpilone commentedon Sep 1, 2015
A quick look at the code shows that DeferredScatterOutputStream creates an OffloadingOutputStream with a 100 MB buffer. The commons ParallelScatterZipCreator will create a thread pool with a thread per available CPU. This means, on my 8 core machine, it would be possible for 800 MB of data to be buffered before the stream is offloaded to disk. It seems like this memory requirement would go up by 100 MB per CPU core which means the build memory requirements would be variable by CPU count which seems odd. Maybe some maximum amount of memory should be used (50% of heap) and then divided across available CPUs so the threads offload to disk sooner.
eicki commentedon Feb 10, 2016
As we ran into the same problem fixing it is simple, see attached patch
plexus-archiver-patch.txt
Stephan202 commentedon May 16, 2016
Thanks for that patch @eicki. The issue described here matches MSOURCES-94. In the context of that ticket I tested your patch and it seems to work well.
Do you mind opening a PR?
Added proper bound on memory usage, patch by Björn Eickvonder
jglick commentedon Aug 13, 2025
3f0c84d may have helped but I still see
maven-war-pluginbeing the culprit for OOMEs pretty often. For example:apparently consuming ⅓ of the heap.
slawekjaranowski commentedon Aug 13, 2025
@jglick please create a new issue, as this one is closed
jglick commentedon Aug 13, 2025
It was more of a comment; perhaps this is by design, that up to 100Mb of heap may be consumed, and you must plan your
-Xmxaccordingly. This just came as a surprise to me, since I would have expected all ZIP entries to be streamed from their source on disk, never buffered more than a few Kb at a time.slawekjaranowski commentedon Aug 13, 2025
It looks strange for me ... should be investigated why such many resources are needed ...