DQN for Atari - frame stacking

[gordicaleksa]

Why is there no frame stacking in the Atari wrapper but only max-pooling over the last 2 observations?
I can see that OpenAI's gym does the same thing with AtariPreprocessing class.

The original DM's DQN paper had max-pooling as well but it stacked 3-5 frames before feeding it into DQN.

I saw that the results of SB3's DQN match that of SB2 but does it match DM's implementation?
Could you point me to a paper/result that showed that this doesn't affect the performance on Atari?
And if so why did DM use it in their paper?

Thank you!

[araffin]

Hello,
please take a look at the documentation first next time (it is part of the issue template...): https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#id2

[gordicaleksa]

Hi! That did help but it still doesn't fully answer my question - how to practically run the original DQN setup on Atari. I did take a look at the docs before asking this question - I'm aware your time is sparse.

I've only seen DQN used on CartPole-v1 in the docs and I wonder why is that.

If I replace A2C in the link you've linked (and set n_envs=1 since DQN doesn't support VecEnvs) with DQN I get:
"Unable to allocate 26.3 GiB for an array with shape (1000000, 1, 4, 84, 84) and data type uint8"

I'd expect it to be smart enough to allocate (10^6, 1, 84, 84) and then just fetch the 4 frames.
Is there something else I'm missing out here? Or is it done like that by design? Some sort of tradeoff between less optimal memory footprint but maybe faster fetching time? If that's the case having an option of choosing smaller memory would be, I assume, a great option for those of us lacking that much memory.

If you feel like I should update the title of the question please advise - I'd be hesitant to reopen yet another question, but I'll do whatever you see fit.

[araffin]

Hello,
you should take a look at the RL zoo (as mentioned in the doc) and at the optimize_memory_usage parameter.
https://github.com/DLR-RM/rl-baselines3-zoo

the memory usage discussion (i think you are refering to LazyFrames) : #37 (comment)

currently, the replay buffer size is set to 10000 in the zoo but should be in fact set to a higher value if you want to be closer to original hyperparameters.

[gordicaleksa]

I've already seen the optimize_memory_usage parameter, thanks for flagging it in any case - but it's a slight (2x?) optimization as far as I could tell.

What I was suggesting was a k-x optimization (in the case where we have 4 frames - 4x). The issue thread you linked is precisely what I was thinking about!

The only PyTorch DQN implementation (that I'm aware of) that allocates (10^6, 1, 84, 84) bytes is this one (PT reimplementation of Berkley's TF implementation)

So just to confirm SB3 currently doesn't have a way to train DQN via this "lazy frame" method?
Looking at class VecFrameStack I don't see that option whereas gym's FrameStack uses LazyFrame by default.

[araffin]

but it's a slight (2x?) optimization as far as I could tell.

well, I wouldn't say 2x is "slight", especially when it works all the time (not only with frame-stacking).

What I was suggesting was a k-x optimization (in the case where we have 4 frames - 4x). The issue thread you linked is precisely what I was thinking about!
So just to confirm SB3 currently doesn't have a way to train DQN via this "lazy frame" method?

This is currently not implemented, we would welcome a PR implementing such feature (if you plan to do so, please open an issue first to discuss the details) if it does not add to much complexity.