How can we optimise synthcity's memory and gpu usage?

[sharmuz]

Question

Are there any best practices for optimising synthcity's use of memory and gpu?

Further Information

• I'm attempting to fit a TimeGAN model on a dataset of 17M rows (split across 25 UIDs = 25 dataframes passed to temporal_dfs)
• I'm running on a fairly beefy cluster: 24 cores, 220GB RAM, Nvidia A100
• I get a memory allocation error during the execution of fit() as it's attempting to allocate many TB of memory
• I tried using just 1 of the 25 dfs (~70k rows) but it's still too much

WRT gpu..

• synthcity.utils.constants.DEVICE is correctly being set to "cuda" so I expect that this will be proliferated into pytorch
• However looking at gpu utilisation I see it's not being used at all...
• Same prob even with ctgan on a simple dataset: no gpu utilisation
• If I train a catboost model in the same session, passing in DEVICE, it works and I see utilisation spike

Anything you might suggest I try?
@robsdavis

System Information

• OS: Ubuntu 20.04
• Python: 3.10.15
• Synthcity: 0.2.10

Additional Context

BTW must say that synthcity rocks 👌