TransVisDrone

Arxiv Preprint: https://arxiv.org/abs/2210.08423

Major Update: Paper Accepted to 2023 IEEE International Conference on Robotics and Automation (ICRA) 🎉

Project Page link

Pretrained models available at Drive

Codes and Visualizations coming soon! Cleaning up training & validation runs, will upload pretrained models soon

Processing NPS daataset

Download annotations from Dogfight Github understand the annotation format. Download the videos from Original NPS site Extract all the frames in folder called AllFrames starting from 0th index (without skip), plot annotations over videos to verify annotations are mapping to videos.
This applies to NPS only, My original frame extraction & annotations of NPS starts from index 0 but to follow tph-yolov5's visdrone style I converted 0 index to 1 index in Step 1. However this is not reuired for FL-drones & AOT dataset processing.

Step1 :

To account for that I created conversion script nps_to_visdrone.py to symlink data with index offseting.

Step2 :

Convert visdrone to yolov5 style Please change the root paths accordingly. Train, Val, Test split followed as in dogfight paper. However contrary to dogfight we don't only use every 4th frame but all the frames in training & testing.

Step3:

After following Step 1 & Step2 data folder should look like /home/tu666280/NPS-Data-Uncompressed/AllFrames/{train, test, val} for annotations /home/tu666280/NPSvisdroneStyle/{train/labels, val/labels, test/labels} you can keep the root folder same as long as frame & annotation indices are matching. Create a new folder like this /home/tu666280/NPS/Videos/{train, val, test} & symlink your original videos or create video_length_dict.pkl with python dictionary format as {"int(video_id)" such as 1,2 : int(num_frames)}

Finally make into the format of NPS.yaml where there are train & val folders containing frames in Clip_{id}{frame_id.zfill(5)}.png & annotations in the format of Clip{id}_{frame_id.zfill(5)}.txt in yolov5 format. Videos folder with either videos symlinks in it or video_length_dict.pkl where its a python dictionary stored in {"int(video_id)": "int(num_frames)"}.

Processing FL-drones dataset

Download fl-drones dataset annotations as described above. Fl-drones dataset is not publicly available & needs to be obtained from permission with authors. I obtained it as our research collaborater had obtained prior permission. For each video half the frames are in training & rest are in validation or testing. There's no test section of this dataset. Validation is testing & we don't do cross-validation style training.

Step1 :

Convert fl-drones annotations to Visdrone style.

Step2 :

If annotations are not in yolov5 format then run convert to yolov5 script

Step 3:

Videos folder will be generated by Step-1.

Finally make into the format of FLDrone.yaml where there are train & val folders containing frames in Clip_{id}{frame_id.zfill(5)}.png & annotations in the format of Clip{id}_{frame_id.zfill(5)}.txt in yolov5 format. Videos folder with either videos symlinks in it or video_length_dict.pkl where its a python dictionary stored in {"int(video_id)": "int(num_frames)"}.

Processing AOT

Coming soon. Download the AOT dataset there are three parts, 1st part should suffice for running my experiements.

1. Link to understand the AOT dataset: Challenge
2. Starter Notebook you need to clone http://gitlab.aicrowd.com/amazon-prime-air/airborne-detection-starter-kit.git you can download flight by flight, keep partial=False & download all flights in part1. Starter kit is already present in the folder aotcore, I renamed core folder in the starter kit as aotcore.
3. Or you can download all files on this s3 bucket
4. Suppose you save entire part1 in dataset_root = /home/c3-0/datasets/Amazon-AOT/data/part1
5. cp conversion_scripts/aot_to_visdrone.py ./ copy aot_to_visdrone.py outside in main root folder &
6. Set variables dataset_root & aot_yolo_root_dir in the file aot_to_visdrone.py. It will create symlinks from locally downloaded files at dataset_root to destination folder aot_yolo_root_dir this destination folder will be used in training & testing. Test spilts are created to speedup testing parallely, if not required you can consider clustering all test flights in same yaml file by changing the setting in aot_to_visdrone.py.
7. Run python aot_to_visdrone.py such that aot_to_visdrone.py, aotcore & aot_flight_ids are all in same directory. If some path issues occur consider exporting aotcore path.
8. For training create AOT.yaml like shown in data/AOT.yaml changing appropriate paths
9. To run training refer to comments in submit-train.slurm python train.py --img 1280 --adam --batch 4 --epochs 80 --data ./data/AOT.yaml --weights ./pretrained/yolo5l.pt \ --hy ./data/hyps/hyp.VisDrone_1.yaml --cfg ./models/yolov5l-xl.yaml --project ./runs/train/AOT \ --name image_size_1280_YOLOXL_1_frames_SwinTR_AOT_with_yolo_weights --exist-ok
10. Above command will run training for 1 frame model, to run for 3 frames change --hy ./data/hyps/hyp.Visdrone_3.yaml for 5 frames change it to --hy ./data/hyps/hyp.Visdrone.yaml, also note that above training requires 2 GPUs with 49 GB VRAM capacity
11. For testing refer to comments in sumbit-test.slurm python val.py --data ./data/AOTTestSplits/AOTTest_$SLURM_ARRAY_TASK_ID.yaml \ --weights ./runs/train/AOT/image_size_1280_temporal_SwinTR_1_frames_AOT_with_yolo_weights/weights/best.pt \ --batch-size 5 --img 1280 --num-frames 1 \ --project ./runs/val/AOT/image_size_1280_temporal_SwinTR_1_frames_AOT_with_yolo_weights --name best \ --task test --exist-ok --save-aot-predictions --save-json-gt this should be launched on 8 == No.of AOT test splits we created gpus simultaneously, each gpu should be of capacity 24 GB VRAM, test if it can work with smaller VRAM as well, reduce batch_size maybe.
12. Finally run python evaluate_aot.py --results_folder ./runs/val/AOTtest/epoch_17/aotpredictions --evaluation_folder /some/destination/folder --detection_threshold 0.2 to get AOT challenge metrics evaluated. This needs the path to the folder where your AOT predictions are stored, in step 11.

Training NPS, FL-drones & AOT

Please follow whatever parameters are set in submit-train.slurm & submit-test.slurm. In training ampere in SBATCH refers to 42 gb NVIDIA ampere gpu. AOT training requires 2 of those. All the dataset trainings are commented in submit-train.slurm file. Uncomment as per your need, only modify --data flag with updated {NPS, FLDrone, AOT}.yaml with your respective data paths.

Running pre-trained checkpoints

Please follow runs/train/NPS//weights, runs/FL//weights, runs/AOT//weights todownload the weights files. While best runs are saved in runs/val/

Evaluate AOT results

AOT results can't only be evaluated using COCO criteria, it has its own grading criteria. To run evaluation, save all the predictions & run evaluate_aot.py with given arguments. To fasten up the evaluation, I split up AOT test data in chunks, get predictions parallely using SBATCH Array & then fuse all the predictions in one.

Citation

If you find our work useful in your research, please consider citing:

@INPROCEEDINGS{10161433,
  author={Sangam, Tushar and Dave, Ishan Rajendrakumar and Sultani, Waqas and Shah, Mubarak},
  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)}, 
  title={TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone Detection in Aerial Videos}, 
  year={2023},
  volume={},
  number={},
  pages={6006-6013},
  keywords={Performance evaluation;Visualization;Image edge detection;Robot vision systems;Transformers;Throughput;Real-time systems},
  doi={10.1109/ICRA48891.2023.10161433}}

Contact

If you have any questions, please feel free to contact us:

Tushar Sangam: [email protected]

Ishan Dave: [email protected]

References

• yolov5-tph
• ultralytics/yolov5
• SwinTransformer