Skip to content

Use conda to fetch Node.js for ReadTheDocs #1220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

pjbollinger
Copy link
Contributor

I was checking out this project when I noticed there was issues with ReadTheDocs, so I figured I'd try to provide some assistance. Feel free to use this PR or incorporate it into your existing #1207. Let me know if you have any questions!


ReadTheDocs builds are failing on the main branch due to the need for the latest Node.js. This is a temporary fix by installing a manually specified binary on the documentation building machine.

Locally, this works as expected. When running the Docker container without update-node-and-npm-without-sudo.sh results in the same errors found here: https://readthedocs.org/projects/remotestoragejs/builds/13065050/

After adding the script to conf.py the documentation is able to build normally.

Also, I added an auto-update for the copyright.

@raucao
Copy link
Member

raucao commented Mar 6, 2021

Welcome, and thanks for your first contribution here! 👏

It looks like the installation itself works. But then, unfortunately, the build is still run on the system's own node.js. There are some deprecation warnings just before the exception, showing current: {"node":"8.10.0","npm":"3.5.2"}.

(When you push more commits to this PR, the RTD build should be run again, so you can verify the results yourself.)

@pjbollinger
Copy link
Contributor Author

Thanks! I'm guessing RTD spins up a new shell at some point that causes the environment variables to get wiped. I'll have to think about this some more. 🤔 I find it strange that it's not reproducible in the Docker image but I've found mismatches in my day job between what the Docker image does and what RTD does in their pipeline. 🤷

@raucao
Copy link
Member

raucao commented Mar 6, 2021

Someone mentioned something about Conda/anaconda in an issue on the RTD Docker repo. Maybe that's helpful somehow:

readthedocs/readthedocs-docker-images#107 (comment)

@pjbollinger
Copy link
Contributor Author

Sounds good. I'll try to do some more testing without Conda to hopefully make it easier to remove this fix when RTD updates everything.

@pjbollinger
Copy link
Contributor Author

Oh yeah, it's finally working! 🎉

From the last commit message:

I noticed the biggest issue was the typedoc was being pulled from /usr/local/bin.
sphinx-js is built to NOT use npx typedoc, so we have to do a bit of hacks to get this to work.

That bit of hacks is chmod +x the typedoc executable we install with npm install and then adding the path of said executable to the PATH environment variable.

Sorry for all the commits as I worked through this; I can work on squashing them tomorrow.

Please checkout the docs from this PR to verify they are working as expected: https://remotestoragejs--1220.org.readthedocs.build/en/1220/

ReadTheDocs builds are failing on the main branch due to the need for the latest Node.js.
This is a temporary fix by using conda to install a more recent version of Node.js specified in the `environment.yml` file.

Even with the Node.js installed by conda, ReadTheDocs does not activate the conda environment, so it does not have immediate access to the new Node.js/npm.
Therefore, we have to manipulate the environment variable `PATH` in ``conf.py` to include the directory of the conda `bin` location.

Since `sphinx-js` does not use `npx` to run `typedoc`, we have to manually add its location to the `PATH` environment variable as well.
By default, the `typedoc` installed to the repository by `npm install` does not have executable permissions, so we have to add them in `conf.py`.

Added an auto-update for the copyright.
Added a helper script for replicating ReadTheDocs build process.
Added documentation on how to replicate ReadTheDocs build process on a local machine.
@pjbollinger pjbollinger force-pushed the docs/fetch-nodejs-for-readthedocs branch from 42e50a5 to 34a08d2 Compare March 7, 2021 11:10
@pjbollinger pjbollinger changed the title Fetch a more recent Node.js for ReadTheDocs Use conda to fetch Node.js for ReadTheDocs Mar 7, 2021
@raucao
Copy link
Member

raucao commented Mar 7, 2021

Fantastic!

I followed the new docs about running the RTD build locally in Docker, but getting a Sphinx exception after it having successfully downloaded and installed all the dependencies:

Traceback (most recent call last):
  File "/home/docs/.conda/envs/rtd_env/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
    mod = import_module(extname)
  File "/home/docs/.conda/envs/rtd_env/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'sphinx_js'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/.conda/envs/rtd_env/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/home/docs/.conda/envs/rtd_env/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
    self.setup_extension(extension)
  File "/home/docs/.conda/envs/rtd_env/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/docs/.conda/envs/rtd_env/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname,
sphinx.errors.ExtensionError: Could not import extension sphinx_js (exception: No module named 'sphinx_js')

Extension error:
Could not import extension sphinx_js (exception: No module named 'sphinx_js')

I'm not sure if it needs to be fixed within this PR, as RTD is finally green again, and the docs were built correctly there. Your call, @pjbollinger.

@pjbollinger
Copy link
Contributor Author

@raucao Do you mind creating a new issue with a full log of the execution? I'm wondering if the local script needs to be tweaked...

Since ReadTheDocs is a check for all PRs, I'd be inclined to merge this and I can address the local issue separately. However, I'm just a drive-by contributor wanting to help out with the CI stuff, so I feel it's more your call than mine. 😅

@raucao
Copy link
Member

raucao commented Mar 7, 2021

@raucao Do you mind creating a new issue with a full log of the execution? I'm wondering if the local script needs to be tweaked...

Sure, will do!

Since ReadTheDocs is a check for all PRs, I'd be inclined to merge this and I can address the local issue separately.

Alright, let's do it then. 👍

@raucao raucao merged commit 7924185 into remotestorage:master Mar 7, 2021
@raucao raucao mentioned this pull request Mar 7, 2021
@pjbollinger pjbollinger deleted the docs/fetch-nodejs-for-readthedocs branch March 7, 2021 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants