This repository was archived by the owner on Mar 20, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 20, 2023. It is now read-only.
CLI option --poll-until-tasks-complete is too slow on many tasks #346
Description
Problem Description
I use cli option --poll-until-tasks-complete too wait until all jobs completed. This option is too slow when you have many tasks (e. g. 80.000). So this option is unusable for me.
Batch Shipyard Version
3.9.1
Activity
alfpark commentedon Mar 17, 2020
You can also try the counting API:
shipyard jobs tasks counts. Note that this is eventually consistent.If that is insufficient, is it possible to add task deletion into your workflow to reduce the count? Otherwise, there will need to be a feature to support OData filtering on that command.
maroony commentedon Mar 18, 2020
My use case: I have about 170 jobs with each 500 tasks. Together these are 85.000 tasks (total task number).
Workaround 1:
shipyard jobs tasks count:Here I have first compute the total task number with the command above. Than I have to write a while loop with this command and compare output line
* completed:. Are you seeing a better solution here?Workaround 2: Delete tasks when they are completed:
How could this be easily done? And if this could be done, the first call of
poll-until-tasks-completewould be unusable slow as before.Workaround 3: Do you have other ideas?
alfpark commentedon Apr 6, 2020
Apologies for the delay:
Workaround 1, that's correct except you should be able to programmatically parse the output with
--raw.Workaround 2, correct, a filter will need to be implemented.
For other workarounds, unfortunately you may need to perform a short term integration with Azure CLI and combine some features between the two.
maroony commentedon Apr 16, 2020
What about the
statscommand?Could this be fast enough? Does this only output jobs specified in the jobs configuration file or does this command lists all jobs in the batch account? The documentation is not clear to me:
"
statswill generate a statistics summary of a job or jobs"alfpark commentedon Apr 21, 2021
Apologies for the delay. Without a
--jobidit will list all jobs in the account. Internally it will call get task counts for each job which is the same as Workaround 1.