Failure to load OPeNDAP in Colab

[giovaniceotto]

Recently, reading OPeNDAP files when running notebooks on Google Colab resutss in an OSError: [Errno -68] NetCDF: I/O failure

Relevant info:

• netCDF4 version: 1.6.0
• Python version: Python 3.7.13
• Environment: Google Colab

Example code:

import netCDF4

ds = netCDF4.Dataset('http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc')

Error:

Colab Notebook link: https://colab.research.google.com/drive/17juZulxoz8sahoDJVuU4uOMSnBC_xXDA?usp=sharing

[giovaniceotto]

Additional information: When using "netCDF4<1.6.0", the code runs fine, without any errors.

[Zeitsperre]

I've run into this issue as well:

Additionally, it looks as though this issue only happens when in a pure python environment running netCDF4==1.6.0. My test ensemble based on Anaconda and netCDF4==1.6.0 seems to be opening this OPeNDAP URL perfectly fine!

[Zeitsperre]

Update: I think the wheel being supplied via PyPI is broken. After compiling netCDF4==1.6.0 from sources, everything seems to be working perfectly fine. I needed to run the following after other dependencies were installed in my environment.

$ python -m pip install --upgrade --force-reinstall --no-deps --no-cache-dir netcdf4 --no-binary netcdf4

You may want to consider rebuilding your wheels (for manylinux * x86_64, anyway).

For reference: CSHS-CWRA/RavenPy@5bab92e

[jswhit]

You may want to consider rebuilding your wheels (for manylinux * x86_64, anyway).

What specifically do you think needs to be done differently when building the manylinux wheels?

[ocefpaf]

BTW, the screenshots in #1179 (comment) make it hard to read about the real issue. It seems that the problem is with curl and some SSL certificates. I get this when I try it locally:

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success

@jswhit what version/build of curl are you using in the wheel?

[jswhit]

curl i s 7.75.0, openssl is 1.0.2u

[ocefpaf]

It may be a version problem or something with the wheel build. I guess the only way to figure this out is to add an OPeNDAP test during the wheel test phase. This is how I reproduced it locally in case you want to check it out:

conda create --name TEST python=3 pip
pip install netcdf4

python -c "import netCDF4; ds = netCDF4.Dataset('http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc')"

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc'

Then, if I do a conda install netcdf4, the test pass and I have these packages installed:

  Name       Version  Build                    Channel    
──────────────────────────────────────────────────────────
  curl       7.83.1   h2283fc2_0               conda-forge
  hdf5       1.12.2   nompi_h4df4325_100       conda-forge
  libcurl    7.83.1   h2283fc2_0               conda-forge
  libnetcdf  4.8.1    nompi_h21705cb_103       conda-forge
  netcdf4    1.6.0    nompi_py310h9fd08d4_101  conda-forge
  openssl    3.0.5    h166bdaf_1               conda-forge

[ngam]

@ocefpaf @jswhit any way we could help here? FWIW, this issue is not present on M1 Macs.

[ocefpaf]

I believe that the wheel should be rebuilt. One test folks could try is to conda install it on colab and see if that works.

[barronh]

I was having the same problem. I installed once with "%pip install netcdf4<1.6" and once with "%pip install netcdf4<1.6". I can confirm it works with one, but not the other.

For each, I used ldd to look at the linked libraries. Below are two lists. The first is the full list where I have sorted to align libraries. The second is just the places where version number or library has changed. I'm not sure if this helps, but I am hoping someone who knows more about this would have some thoughts.

--- v15.txt	2022-09-19 17:49:58.161994184 +0000
+++ v16.txt	2022-09-19 17:49:59.071994132 +0000
@@ -1,18 +1,20 @@
-	linux-vdso.so.1 (0x00007ffecfb0d000)
+	linux-vdso.so.1 (0x00007ffdd1397000)
-	/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 (0x00007f99aaa79000)
+	/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 (0x00007fddc9e71000)
-	libnetcdf-cb1b5117.so.18.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-cb1b5117.so.18.0.0 (0x00007f99aa8bf000)
+	libnetcdf-0ef85704.so.19 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-0ef85704.so.19 (0x00007fddc9b8c000)
-	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f99aa6a0000)
+	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fddc996d000)
-	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f99aa2af000)
+	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fddc957c000)
-	libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f99aa094000)
+	libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007fddc9361000)
-	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f99a9d0b000)
+	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fddc8fd8000)
-	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f99a996d000)
+	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fddc8c3a000)
-	libhdf5_hl-f3ea9bc7.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f3ea9bc7.so.200.0.0 (0x00007f99aad38000)
+	libhdf5_hl-f45ebdf9.so.200.1.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f45ebdf9.so.200.1.0 (0x00007fddca12c000)
-	libhdf5-3e8ef4e5.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-3e8ef4e5.so.200.0.0 (0x00007f99a9382000)
+	libhdf5-115e850b.so.200.2.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-115e850b.so.200.2.0 (0x00007fddc85cc000)
-	libsz-57467d8a.so.2.0.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libsz-57467d8a.so.2.0.1 (0x00007f99aad31000)
+	libsz-57467d8a.so.2.0.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libsz-57467d8a.so.2.0.1 (0x00007fddca125000)
-	libaec-e300f322.so.0.0.10 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.10 (0x00007f99aad25000)
+	libaec-e300f322.so.0.0.12 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.12 (0x00007fddca115000)
-	libcurl-8c767087.so.4.7.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libcurl-8c767087.so.4.7.0 (0x00007f99a9095000)
+	libcurl-8c767087.so.4.7.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libcurl-8c767087.so.4.7.0 (0x00007fddc7e33000)
-	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f99a8e78000)
+	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fddc83af000)
+	libzstd-c4cc7033.so.1.5.2 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libzstd-c4cc7033.so.1.5.2 (0x00007fddc8120000)
+	libblosc-a9e8a7c7.so.1.21.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libblosc-a9e8a7c7.so.1.21.1 (0x00007fddc8289000)
-	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f99a8c74000)
+	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fddc8a36000)
-	/lib64/ld-linux-x86-64.so.2 (0x00007f99aace9000)
+	/lib64/ld-linux-x86-64.so.2 (0x00007fddca0e1000)
-	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f99a8a4e000)
+	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fddc79f5000)
-	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f99a8836000)
+	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fddc7c1b000)

Below is a more narrow list of just where versions have changed.

-	libnetcdf-cb1b5117.so.18.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-cb1b5117.so.18.0.0 (0x00007f99aa8bf000)
+	libnetcdf-0ef85704.so.19 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-0ef85704.so.19 (0x00007fddc9b8c000)
-	libhdf5_hl-f3ea9bc7.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f3ea9bc7.so.200.0.0 (0x00007f99aad38000)
+	libhdf5_hl-f45ebdf9.so.200.1.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f45ebdf9.so.200.1.0 (0x00007fddca12c000)
-	libhdf5-3e8ef4e5.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-3e8ef4e5.so.200.0.0 (0x00007f99a9382000)
+	libhdf5-115e850b.so.200.2.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-115e850b.so.200.2.0 (0x00007fddc85cc000)
-	libaec-e300f322.so.0.0.10 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.10 (0x00007f99aad25000)
+	libaec-e300f322.so.0.0.12 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.12 (0x00007fddca115000)
+	libzstd-c4cc7033.so.1.5.2 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libzstd-c4cc7033.so.1.5.2 (0x00007fddc8120000)
+	libblosc-a9e8a7c7.so.1.21.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libblosc-a9e8a7c7.so.1.21.1 (0x00007fddc8289000)