[loaders-] guess types (hdf5), understand unsigned int type (npy)

[maxfl]

• hdf5: read datatype codes from 1d arrays and guess column types. Done similarly to npy for floats and (un)signed integers.
• npy: treat unsigned integer datatype as integer. Was anytype previously.
• pandas: fix hdf loader

Only existing loaders were modified, no other added.

[CLAassistant]

All committers have signed the CLA.

[saulpw]

Nitpick: ItemColumn is the preferred nomenclature; ColumnItem is deprecated. (All column types are named FooColumn.)

[maxfl]

fixed

[saulpw]

Thanks, @maxfl ! The pandas loader is a little rusty so I appreciate that it's getting some updates.

[maxfl]

@saulpw, you are welcome. There are a few other items that could be improved and I will definitely look more into the loaders as I work with vd more.

On a related note, it is possible to extend vd with root loader. I'm not proposing a PR now as I need to gain some more experience and walk through the check list.

By the way, visidata is wonderful. It is simplified my life so much. Thank you a lot!

[anjakefala]

Hey @maxfl, did you test the numpy loader?

When I created one that creates small arrays, and opened it with VisiData, the VisiData sheet was just blank. Did you notice something similar?

[anjakefala]

This was mine, for reference:

import numpy as np

dtype = [
    ('id', np.uint16),
    ('value', np.uint8),
    ('score', np.float32)
]

structured_data = np.zeros(20, dtype=dtype)
for i in range(20):
    structured_data[i] = (i, i % 255, i * 1.5)

np.save('visidata_test.npy', structured_data)

array_2d = np.arange(100, dtype=np.uint8).reshape(10, 10)
np.save('visidata_test_2d.npy', array_2d)

uint8_array = np.array([0, 127, 255], dtype=np.uint8)
uint16_array = np.array([0, 32767, 65535], dtype=np.uint16)
uint32_array = np.array([0, 2147483647, 4294967295], dtype=np.uint32)
uint64_array = np.array([0, 9223372036854775807, 18446744073709551615], dtype=np.uint64)

np.save('uint8_test.npy', uint8_array)
np.save('uint16_test.npy', uint16_array)
np.save('uint32_test.npy', uint32_array)
np.save('uint64_test.npy', uint64_array)

[maxfl]

dear @anjakefala, thank you for testing and providing an example.

In fact I did test the updates and I do use visidata on a daily basis to work with hdf5/npz/root and tsv files. The only change I did for npy/npz is adding support for unsigned int to already existing int by modifying a single line. The original implementation by design works only with record arrays and requires columns to have names. It is not working with regular 1d/2d arrays.

I think I might import the relevant code from hdf5 to numpy loader to make it work with regular arrays, but may be it does not belong to this particular PR.

[anjakefala]

Thank you for that context @maxfl! I will add a small test file for npy, and then will go ahead and merge. =)

[maxfl]

Thank you for that context @maxfl! I will add a small test file for npy, and then will go ahead and merge. =)

following PR on 2d numpy in !2724