Repository navigation
Manage the amount requests to the Web API to avoid rate limiting #12
Description
I think you could use something like https://github.com/JMPerez/promise-throttle to limit how many requests per second are made. You have an example of how this is done on https://github.com/JMPerez/spotify-dedup/blob/master/app/scripts/main.js
If you like the idea I can work on a Pull Request to add this.
Activity
watsonbox commentedon Jul 1, 2015
Yes, I like this idea. I guess it would only ensure no throttling errors when using a personal application though, since according to the docs "Rate limiting is applied on an application basis (based on client id), regardless of how many users are using it."
I'm going to add a progress bar for the bulk export soon, too, which would be a pre-requisite for adding this kind of slowed down exporting I think.
JMPerez commentedon Jul 1, 2015
You can still have rate limiting issues if several users are using the site at the same time, but you would have more control on the amount of requests a given user makes per unit of time.
pavelkomarov commentedon Sep 19, 2019
My version implements rate limiting, 10 queries per second to the server. This is essential to make exporting really large playlists or all together work. Even for users with insanely large collections, it works in only a few minutes. Progress bar not super necessary.
I've hard-coded my client id, so everyone who uses it racks up traffic under the same limit. So if lots of users at once happen to all try to Export All at the same time, then yes they'll still run in to rate limiting problems. But this is unlikely until there are thousands of users every day, so I've chosen to just ignore it. If we ever get to that point, then I'll have to do something like give everyone their own client id.
exportify.net
watsonbox commentedon Nov 3, 2020
Terribly overdue 🙈 but I'll see what I can do in updating this repo to support this and using as much of the work that's been done as possible. Thank you @pavel-aicradle for advancing it. I think this issue will best serve as the main one for dealing with this, put there's a lot of useful discussion in these:
pavelkomarov commentedon Nov 3, 2020
#70 gives you my changes, which include the fix to this.
Implement rate limiting using Retry-After header [#12]
Implement rate limiting using Retry-After header [#12]
Implement rate limiting using Retry-After header [#12]
Add rate limiting and export progress tracking [#12]
watsonbox commentedon Nov 17, 2020
Should now be fixed by #75