Handling data split in many files

[ksachdeva]

Hi,

Thanks for this excellent work.

Let's say that I have a data file per class e.g.

1-a.pickle - contains the pandas data frame for class 1
1-b.pickle - contains the pandas data frame for class 1
2.pickle - contains the pandas data frame for class 2

I would like to create a sliding window such that they operate on 1-a.pickle and 1-b.pickle independently.

During the training phase, in a mini-batch I would like to get the shuffled windows from my dataset i.e. 1-a.pickle, 1-b.pickle, and 2.pickle

Please guide

Regards
Kapil

[oguiza]

Thanks for your comments @ksachdeva!
You can apply SlidingWindow independently to each pandas data frame. You’ll get 3 X and 3 y numpy arrays. You can then concatenate all the Xs and all the ys and generate splits using the get_splits function.