[Question] Importing trained model in SB3 to Matlab

[mohanafathollahi]

❓ Question

Hi,

I trained SAC model in SB3 and I defined custom gym environment for that. After training the model, I wanted to import it to Matlab. I considered below approaches:

1. I used clip action in my custom gym environment. Since ONNX does not support post processing, like clipping, the output of onnx was not correct and applying clip on the generated onnx output gave nonsense result.
2. SB3 using pytorch and my project is not image classification or segmentation models to be supported by importNetworkFromPyTorchcan in Matlab.
3. I think Tensorflow does not have this limitation in Matlab. I tried to switch to SB2 to have a model in tensorflow but SB2 needs tensorflow==1.5 and apparently I need to downgrade python version from 3.8 to lower versions.
4. Manually building the network by optimum weights and biases, which is a bit time consuming because I have 4 models.

Do you have any recommendation to facilitate importing trained model from SB3 to Matlab.

Thank you

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

I used clip action in my custom gym environment. Since ONNX does not support post processing, like clipping, the output of onnx was not correct and applying clip on the generated onnx output gave nonsense result.

ONNX cannot trace torch.clamp()?
Also, no clipping is needed for SAC: https://araffin.github.io/post/sac-massive-sim/

SB3 using pytorch and my project is not image classification

SAC actor is just fully connected layers, I'm surprised it would not work, see doc for tracing: https://stable-baselines3.readthedocs.io/en/master/guide/export.html#trace-export-to-c

https://de.mathworks.com/help/deeplearning/ref/importnetworkfrompytorch.html

Manually building the network by optimum weights and biases, which is a bit time consuming because I have 4 models.

That's problably the easiest/fastest. Why 4 models? you only need the actor? (and doing it one time should be same for any SAC trained model).
print(model.policy) to have the architecture: https://stable-baselines3.readthedocs.io/en/master/guide/export.html#manual-export

SAC actor:

stable-baselines3/stable_baselines3/sac/policies.py

Line 25 in 656de97

class Actor(BasePolicy):

In Jax (more readable): https://github.com/araffin/sbx/blob/8238fccc19048340870e4869813835b8fb02e577/sbx/common/policies.py#L251-L262

[mohanafathollahi]

Thank you for your response.
I figure out that in the source code, specifically in the common/policies.py file and the predict function, there are additional steps, such as the unscale_action function.
This function rescale the actions produced by the model to fit within the range [low, high], which corresponds to the lower and upper limits of the action space. After applying this function to the output of the ONNX model, I was able to generate the same output that the model.predict function produces.