Embedding metadata is still not read or displayed in TensorBoard

[ldorigo]

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:

Diagnostics

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='lenz-ubuntu', release='5.11.0-25-generic', version='#27~20.04.1-Ubuntu SMP Tue Jul 13 17:41:23 UTC 2021', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: '/home/luca/medrel/medrel_venv'

--- check: installed_packages
INFO: installed: tensorboard==2.6.0
INFO: installed: tensorflow==2.6.0
INFO: installed: tensorflow-estimator==2.6.0
INFO: installed: tensorboard-data-server==0.6.1

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.6.0'

--- check: tensorflow_python_version
2021-08-15 14:57:17.271237: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-08-15 14:57:17.271258: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
INFO: tensorflow.__version__: '2.6.0'
INFO: tensorflow.__git_version__: 'v2.6.0-rc2-32-g919f693420e'

--- check: tensorboard_data_server_version
INFO: data server binary: '/home/luca/medrel/medrel_venv/lib/python3.8/site-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.6.1'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/luca/medrel/medrel_venv/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'lenz-ubuntu'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=1458200, st_dev=2050, st_nlink=2, st_uid=1004, st_gid=1004, st_size=4096, st_atime=1629031205, st_mtime=1629032212, st_ctime=1629032212)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/luca/medrel/medrel_venv/lib/python3.8/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
abbreviations==0.2.5
absl-py==0.13.0
anyio==3.3.0
appdirs==1.4.4
argon2-cffi==20.1.0
astunparse==1.6.3
async-generator==1.10
attrs==21.2.0
Babel==2.9.1
backcall==0.2.0
biopython==1.79
black==21.6b0
bleach==3.3.1
blis==0.7.4
cachetools==4.2.2
catalogue==2.0.4
certifi==2021.5.30
cffi==1.14.6
chardet==4.0.0
charset-normalizer==2.0.4
clang==5.0
click==7.1.2
conllu==4.4
cupy-cuda101==9.2.0
cymem==2.0.5
data-science-types==0.2.23
debugpy==1.4.1
decorator==5.0.9
defusedxml==0.7.1
deprecation==2.1.0
en-core-sci-lg==0.4.0
en-core-sci-md==0.4.0
entrypoints==0.3
fastrlock==0.6
flatbuffers==1.12
gast==0.4.0
gensim==4.0.1
google-auth==1.34.0
google-auth-oauthlib==0.4.5
google-pasta==0.2.0
grpcio==1.39.0
gviz-api==1.9.0
h5py==3.1.0
idna==3.2
ipykernel==6.0.3
ipython==7.25.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==3.0.1
joblib==1.0.1
json5==0.9.6
jsonschema==3.2.0
jupyter-client==6.1.12
jupyter-core==4.7.1
jupyter-packaging==0.10.4
jupyter-server==1.10.1
jupyterlab==3.0.14
jupyterlab-pygments==0.1.2
jupyterlab-server==2.6.1
jupyterlab-widgets==1.0.0
keras==2.6.0
keras-nightly==2.5.0.dev2021032900
Keras-Preprocessing==1.1.2
Lasagne==0.1
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib-inline==0.1.2
mistune==0.8.4
more-itertools==8.8.0
murmurhash==1.0.5
mypy==0.902
mypy-extensions==0.4.3
nbclassic==0.3.1
nbclient==0.5.3
nbconvert==6.1.0
nbformat==5.1.3
nest-asyncio==1.5.1
networkx==2.6.2
nltk==3.6.2
nmslib==2.1.1
notebook==6.4.0
numpy==1.21.1
oauthlib==3.1.1
obonet==0.3.0
opt-einsum==3.3.0
packaging==21.0
pandocfilters==1.4.3
parso==0.8.2
pathspec==0.8.1
pathy==0.6.0
pexpect==4.8.0
pickleshare==0.7.5
pip==20.0.2
pkg-resources==0.0.0
preshed==3.0.5
prometheus-client==0.11.0
prompt-toolkit==3.0.19
protobuf==3.17.3
psutil==5.8.0
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.6.1
pycparser==2.20
pydantic==1.8.2
pydot==1.4.2
Pygments==2.9.0
pyparsing==2.4.7
pyrsistent==0.18.0
pysbd==0.3.4
python-dateutil==2.8.1
python-Levenshtein==0.12.2
pytz==2021.1
PyYAML==5.4.1
pyzmq==22.1.0
regex==2021.4.4
requests==2.26.0
requests-oauthlib==1.3.0
requests-unixsocket==0.2.0
rsa==4.7.2
scikit-learn==0.24.2
scipy==1.7.1
scispacy==0.4.0
Send2Trash==1.7.1
setuptools==57.4.0
six==1.15.0
smart-open==5.1.0
sniffio==1.2.0
spacy==3.0.7
spacy-legacy==3.0.8
sqlparse==0.4.1
srsly==2.4.1
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-profile==2.5.0
tensorboard-plugin-wit==1.8.0
tensorflow==2.6.0
tensorflow-estimator==2.6.0
termcolor==1.1.0
terminado==0.10.1
testpath==0.5.0
Theano==1.0.5
thinc==8.0.8
threadpoolctl==2.2.0
toml==0.10.2
tomlkit==0.7.2
tornado==6.1
tqdm==4.62.0
traitlets==5.0.5
typer==0.3.2
types-python-dateutil==0.1.4
typing-extensions==3.7.4.3
Unidecode==1.2.0
urllib3==1.26.6
vprof==0.38
wasabi==0.8.2
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.1.0
Werkzeug==2.0.1
wheel==0.36.2
widgetsnbextension==3.5.1
wrapt==1.12.1

Issue description

Specifying the metadata file for the embeddings projector does not work - it fails silently and never gets loaded.

Sample code:

    tensorboard_callback = tf.keras.callbacks.TensorBoard(
        log_dir="/hdd/logs",
        profile_batch=(2, 5),
        histogram_freq=1,
        embeddings_freq=1,
        embeddings_metadata="/hdd/logs/metadata.tsv",
    )

This is the exact same issue as #4246 (#4246). It was closed despite the "solution" being a manual workaround that only fixes part of the problem.

Reopening to get a proper fix.

[bileschi]

Thanks for reporting this issue, @ldorigo ! Sorry you are having trouble?

Would it be too much to ask for a minimal reproduction? The embedding projector was inherited by and unfamiliar to much the TB team, and you may be able to save us a great deal of time and get a much faster fix.

Thanks!
Sincerely,
Stan on behalf on the TensorBoard team.

[ldorigo]

Hi, unfortunately I'm away from my workstation for the next week, so won't be able to provide an example very soon. If still needed when I'm back I'll be happy to oblige.

Note that the issue I linked to describes how to reproduce it in some more detail, maybe that is enough for you.