arXiv.org PDFs denying access to User-Agent "getpapers/(TDM Crawler contact@contentmine.org)" #167

Activity

1. tarrow commented on Oct 19, 2017

   

   tarrow

   on Oct 19, 2017

   Contributor

   More actions

   We have clearly made a mistake here. I imagine that we're hammering them too hard/not following a delay between requests etc...

   Probably the answer is to

   1. fix it so we're playing by their rules
   2. release a new version with the new version number in the UserAgent
   3. let them know that we've fixed it in the new version (should they keep blocking the old one?)

   Also: did we get an email to contact@contentmine.org
2. sedimentation-fault commented on Dec 17, 2017

   

   sedimentation-fault

   on Dec 17, 2017

   More actions

   ...or simply spoof the UserAgent header with some innocent string and move on. :-)
   A list of most common UA strings can be found in:
   https://techblog.willshouse.com/2012/01/03/most-common-user-agents/
3. rossmounce commented on Aug 22, 2018

   

   rossmounce

   on Aug 22, 2018

   Member

   More actions

   I also bumped into this issue just now. A pity...
4. merkys commented on Sep 24, 2018

   

   merkys

   on Sep 24, 2018

   More actions

   Same problem here. It's important to maintain arXiv downloader working, thus I suggest

   1. fix it so we're playing by their rules

   For starters, crawl delays at least of 15 seconds must be introduced.

   3. let them know that we've fixed it in the new version (should they keep blocking the old one?)

   Yes and yes. It is a bit strange that arXiv discourages automated access to /api, but this is probably (?) a bug.
5. petermr commented on Sep 25, 2018

   

   petermr

   on Sep 25, 2018

   Member

   More actions

   OK I will write to Paul.
6. rossmounce commented on Sep 25, 2018

   

   rossmounce

   on Sep 25, 2018

   Member

   More actions

   @petermr actually, you might be better-off emailing the lead software architect at arxiv (Erick Peirson). I've found him to be quite helpful & communicative: brp53@cornell.edu https://erickpeirson.github.io/
7. petermr commented on Sep 25, 2018

   

   petermr

   on Sep 25, 2018

   Member

   More actions

   Thanks Ross.
8. sdruskat commented on Aug 9, 2019

   

   sdruskat

   on Aug 9, 2019

   More actions

   Just to re-iterate that arXiv will block your IP (or your employer's) if you use the getpapers API as is to try and download PDFs.

   May I suggest switching off the PDF download for now until a refactoring to conform with the current API guidelines is in place?
9. petermr commented on Aug 9, 2019

   

   petermr

   on Aug 9, 2019 via email

   Member

   More actions

   On Fri, Aug 9, 2019 at 10:39 AM Stephan Druskat ***@***.***> wrote: Just to re-iterate that arXiv will block your IP (or your employer's) if you use the getpapers API as is to try and download PDFs.

   Thank you. I'll add a comment

   …

   May I suggest switching off the PDF download for now until a refactoring to conform with the current API guidelines is in place? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#167?email_source=notifications&email_token=AAFTCS3WPOGO2VNC33QO3ZTQDU3OVA5CNFSM4D7XGVWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD36FS2A#issuecomment-519854440>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFTCSYJXCODE4VA3FT5J7TQDU3OVANCNFSM4D7XGVWA> .

   -- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK