[question] [feature request] support for Dict and Tuple spaces

[AloshkaD]

I want to train using two images from different cameras and an array of 1d data from a sensor. I'm passing these input as my env state. Obviously I need a cnn that can take those inputs, concatenate, and train on them. My question is how to pass these input to such a custom cnn in polocies.py. Also, I tried to pass two images and apparently dummy_vec_env.py had trouble with that.
obs = env.reset() File "d:\resources\stable-baselines\stable_baselines\common\vec_env\dummy_vec_env.py", line 57, in reset self._save_obs(env_idx, obs) File "d:\resources\stable-baselines\stable_baselines\common\vec_env\dummy_vec_env.py", line 75, in _save_obs self.buf_obs[key][env_idx] = obs ValueError: cannot copy sequence with size 2 to array axis with dimension 80

I appreciate any thoughts or examples.

[araffin]

Hello,

Could you please provide a minimal code to reproduce the error?

[AloshkaD]

@araffin So for simplicity, say I need to pass two rgb images (observations from two cameras onboard a robot) of the size (80,160,4) as states like this

`class MyCustomEnv(gym.Env):

def __init__(self):

    self.observation_space = spaces.Box(low=0, high=255, shape=(80,160,4), dtype=np.float32)

    self.state =(np.zeros((80, 160,4), dtype=np.uint8),np.zeros((80, 160,4), dtype=np.uint8))

    .
    .
    .
def step(self, action):
    .
    .
    .
    self.state = self.rgbimage_1,self.rgbimage_2 
    return self.state, reward, done, info
def reset(self):
    .
    .
    .
    self.state = self.rgbimage_1,self.rgbimage_2 
    return self.state

`
I hope this is good enough. I also suspect my definition of the observation_space might not be correct but I tried different methods to define an observation_space for two images and nothing worked. I saw that you are a contributor here and I hope you would be able to help with defining the ob_space too
for the record, I tried to build an observation space like this

`self.nested_observation_space = spaces.Dict({

    'sensors':  spaces.Dict({

        #'position': spaces.Box(low=-100, high=100, shape=(3,)),

        #'velocity': spaces.Box(low=-1, high=1, shape=(3,)),

        'front_cam': spaces.Tuple((

            spaces.Box(low=0, high=255, shape=(80, 160, 4)),

            spaces.Box(low=0, high=255, shape=(80, 160, 4))

        )),
        }) 
        })

`
but that didn't work either and returned the error

env = DummyVecEnv([lambda: env]) # The algorithms require a vectorized environment to run File "d:\stable-baselines\stable_baselines\common\vec_env\dummy_vec_env.py", line 31, in __init__ shapes[key] = box.shape
for simplicity I passed this and still got the same error
self.nested_observation_space =spaces.Tuple(( spaces.Box(low=0, high=255, shape=(80, 160, 4)), spaces.Box(low=0, high=255, shape=(80, 160, 4)) ))

I can send you the complete class if you like.
Thanks

[AloshkaD]

The problem appears to be with vectorizing the env.. I get
"d:stable-baselines\stable_baselines\common\vec_env\dummy_vec_env.py", line 35, in __init__ self.buf_obs = {k: np.zeros((self.num_envs,) + tuple(shapes[k]), dtype=dtypes[k]) for k in self.keys} File "d:\stable-baselines\stable_baselines\common\vec_env\dummy_vec_env.py", line 35, in <dictcomp> self.buf_obs = {k: np.zeros((self.num_envs,) + tuple(shapes[k]), dtype=dtypes[k]) for k in self.keys} TypeError: 'NoneType' object is not iterable

for defining the state like this
self.observation_space =spaces.Tuple(( spaces.Box(low=0, high=255, shape=(80, 160, 4), dtype=np.uint8), spaces.Box(low=0, high=255, shape=(80, 160, 4), dtype=np.uint8) ))

[araffin]

Hello,
Dict and Tuple spaces are not supported for observations spaces. Did you try concatenating the images along the channel axis?

[AloshkaD]

I could concatenate the images and then separate them when fed to the cnn. I could also pad the signal with zeros and concatenate it as a 2x2 channel. I'm worried about the scalability of this approach.

[pulver22]

Hello,
Dict and Tuple spaces are not supported for observations spaces. Did you try concatenating the images along the channel axis?

Why aren't they supported? I also would like to pass image + scalars an input to the policy at the current stage this is not possible. I don't know if it's more convenient to write a code for this or just add a vector of scalar at the end of the image and then separate it later.

[hill-a]

hey,

@pulver22 Well Tuple space could be supported with some effort (IIRC you can feed tuples into the feed_dict with a tf.concat of placeholders).

However Dict would require quite a bit of reworking for it to be compatible with all the models, as each placeholder for each tensor would be called by name, and not by sequential order.

EDIT: if anyone can see a quick hack that could work in https://github.com/hill-a/stable-baselines/blob/master/stable_baselines/common/input.py without changing anything else, it would be awsome to hear from you.

EDIT2: Just tried the Tuple with tf.concat and tf.stack, and it doesn't seem to want to play nice. Which makes sense when you think about concatenating an image of 90x128 integers with 4 floating point value, the code would need to flatten the input, and make it all floating point numbers; and that would only work with MLP policies.

[AloshkaD]

@hill-a there seem to be a hack proposed by @Atcold here but it does not seem to generalize to all envs
Atcold@dbc329f

[hill-a]

@AloshkaD That seems to be more of a alterations of the models, which is exactly what I would like to avoid doing; as it might generate more unforeseen issues and bugs when changing all the models in such a way.

I was hoping to be able to simply change the input parsing code (stable_baselines/common/input.py) (that almost all the models use).

However, if this unlikely to be possible, then a redesign of the return of the input parsing code might be a more viable solution to this problem.

[AloshkaD]

Agreed! thank you @hill-a and @araffin.