netCDF failing roundtrip with unicode chars

[DeltaSigma130]

Problem

When directories contain unicode chars, ds.to_netcdf() works, but xr.open_dataset() fails with a FileNotFoundError:

Code

import numpy as np
import xarray as xr
from pathlib import Path

da = xr.DataArray(np.random.rand(2,3,3), coords=[('x', range(2)), ('y', range(3)), ('z', range(3))])
ds = xr.Dataset({'foo': da})

p = Path('test_λ850') / 'data.nc'
p.parent.mkdir()
ds.to_netcdf(p)  #  Successful file creation
# ds.to_netcdf(str(p))  # also works just fine.

dsl = xr.open_dataset(p)  # ERROR: as shown below
# with xr.open_dataset(p) as dsl:  # ERROR
#     pass
# dsl = xr.open_dataset(str(p))  # ERROR

Returning error:

>>> dsl = xr.open_dataset(p)
Traceback (most recent call last):
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('C:\\Users\\<USER>\\Documents\\Projects\\Jupyter\\test_λ850\\data.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\api.py", line 501, in open_dataset
    backend_ds = backend.open_dataset(
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\netCDF4_.py", line 547, in open_dataset
    store = NetCDF4DataStore.open(
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\netCDF4_.py", line 379, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\netCDF4_.py", line 327, in __init__
    self.format = self.ds.data_model
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\netCDF4_.py", line 388, in ds
    return self._acquire()
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\netCDF4_.py", line 382, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\xarray\backends\file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "src\netCDF4\_netCDF4.pyx", line 2307, in netCDF4._netCDF4.Dataset.__init__
  File "src\netCDF4\_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:\\Users\\<USER>\\Documents\\Projects\\Jupyter\\test_\xce\xbb850\\data.nc'

Environment

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.9.1 (default, Dec 11 2020, 09:29:25) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: English_United States.1252
libhdf5: 1.10.6
libnetcdf: 4.6.1

xarray: 0.18.0
pandas: 1.2.1
numpy: 1.20.2
scipy: 1.6.2
netCDF4: 1.5.7
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.5.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.3.2
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 52.0.0.post20210125
pip: 21.1.1
conda: None
pytest: None
IPython: 7.19.0
sphinx: None

(One note is that I am using miniconda. The conda: None line in the env must only check for the big version.)

[headtr1ck]

Thanks for the report and sorry for the late reply :)

This is not an issue of xarray but rather netCDF4.
The corresponding issue can be found here: Unidata/netcdf4-python#941
It seems that there is no solution yet.

Since it seems that it is a HDF5 issue I don't know if any other backend could fix this issue for you (I tried h5netcdf without success).

I will close this issue for now since we cannot fix it anyway.