New Extractor Idea: `scihub-dl` to auto-detect inline DOI numbers and download academic paper PDFs

[pirate]

New extractor idea: SCIHUB

e.g. take this academic paper for example: https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1

If a full paper PDF is available on scihub, e.g.: https://sci-hub.se/https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1 it could be downloaded to a ./archive/<timestmap>/scihub/ output folder.

# try downloading via verbatim URL first
$ scihub.py -d https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1'
DEBUG:Sci-Hub:Successfully downloaded file with identifier https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1

We could also look for a DOI number in the page URL or page html contents e.g.: 10.1016/j.cub.2019.11.030 using a regex and try downloading that.

# otherwise try downloading via any regex-extracted bare DOI numbers on the page or in the URL
$ scihub.py -d '10.1016/j.cub.2019.11.030'
DEBUG:Sci-Hub:Successfully downloaded file with identifier 10.1016/j.cub.2019.11.030

$ ls
c28dc1242df6f931c29b9cd445a55597-.cub.2019.11.030.pdf

New Dependencies:

• https://github.com/zaytoun/scihub.py

New Extractors:

• extractors/scihub.py

New Config Options:

• SAVE_SCIHUB=True

[danisztls]

Captchas would be a problem.

[pirate]

In my testing so far it's not a problem as long as you're not doing dozens of papers at a time.

[jezcope]

Two more avenues to explore:

1. Zotero has a big collection of translators to get full text from many different publishers and repositories https://github.com/zotero/translators
2. Unpaywall has a huge database of open access copies of articles https://unpaywall.org/

Would be great to capture full text for academic publications where its available.

[pirate]

Parsing DOIs out of PDF/HTML content:

• https://github.com/MicheleCotrufo/pdf2doi
• https://www.findingyourway.io/blog/2019/03/13/2019-03-13_extracting-doi-from-text/
• https://stackoverflow.com/questions/27910/finding-a-doi-in-a-document-or-page

Found good candidate extractor dependencies to try for the major free scientific paper DBs:

• https://github.com/Tishacy/SciDownl
• • https://pypi.org/project/arxiv-dl/
• https://github.com/viown/libgen-dl
• https://github.com/unpywall/unpywall
• https://openaccessbutton.org/?from=integration&id=
• https://github.com/bibcure/scihub2pdf
• https://github.com/dheison0/annas-archive-api

and more via...

• https://github.com/mrossinek/cobib ⭐️ (awesome bibliography manager that supports DOI search)
• https://gist.github.com/taskylizard/5ba73bf97dccf159316edcf4c6520856#-academic-papers
• https://github.com/cipher387/osint_stuff_tool_collection
• https://www.reddit.com/r/FREEMEDIAHECKYEAH/wiki/tools-index

something like this might be interesting for linking together citations too https://inciteful.xyz/

[benmuth]

I have a rough script for this working (just using scihub.py as a module and downloading a pdf with the url->doi fallback). I had to slightly modify scihub.py to get it to work, but I was thinking we can just add the modified version to the vendor directory or something. It looks like scihub.py isn't available on PyPI anyway.

Should I just wait for the plugin system? Or I can create a repo with the script and modified scihub.py so it can be looked at and improved in the meantime. Or I can try to minimally integrate it with ArchiveBox (just get it working as a oneshot?) and open a PR for feedback and testing, whatever makes sense.