Action Recognition Models

C3D

Introduction

@ARTICLE{2014arXiv1412.0767T,
author = {Tran, Du and Bourdev, Lubomir and Fergus, Rob and Torresani, Lorenzo and Paluri, Manohar},
title = {Learning Spatiotemporal Features with 3D Convolutional Networks},
keywords = {Computer Science - Computer Vision and Pattern Recognition},
year = 2014,
month = dec,
eid = {arXiv:1412.0767}
}

Model Zoo

UCF-101

config resolution gpus backbone pretrain top1 acc top5 acc testing protocol inference_time(video/s) gpu_mem(M) ckpt log json
c3d_sports1m_16x1x1_45e_ucf101_rgb.py 128x171 8 c3d sports1m 83.27 95.90 10 clips x 1 crop x 6053 ckpt log json

Notes:

  1. The author of C3D normalized UCF-101 with volume mean and used SVM to classify videos, while we normalized the dataset with RGB mean value and used a linear classifier.

  2. The gpus indicates the number of gpu (32G V100) we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  3. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

For more details on data preparation, you can refer to UCF-101 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train C3D model on UCF-101 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/c3d/c3d_sports1m_16x1x1_45e_ucf101_rgb.py \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test C3D model on UCF-101 dataset and dump the result to a json file.

python tools/test.py configs/recognition/c3d/c3d_sports1m_16x1x1_45e_ucf101_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy

For more details, you can refer to Test a dataset part in getting_started.

CSN

Introduction

@inproceedings{inproceedings,
author = {Wang, Heng and Feiszli, Matt and Torresani, Lorenzo},
year = {2019},
month = {10},
pages = {5551-5560},
title = {Video Classification With Channel-Separated Convolutional Networks},
doi = {10.1109/ICCV.2019.00565}
}

@inproceedings{ghadiyaram2019large,
  title={Large-scale weakly-supervised pre-training for video action recognition},
  author={Ghadiyaram, Deepti and Tran, Du and Mahajan, Dhruv},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={12046--12055},
  year={2019}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
ircsn_ig65m_pretrained_r152_32x2x1_58e_kinetics400_rgb.py short-side 320 8x4 ResNet152 IG65M 80.14 94.93 x 8517 ckpt log json
ircsn_ig65m_pretrained_bnfrozen_r152_32x2x1_58e_kinetics400_rgb.py short-side 320 8x4 ResNet152 IG65M 82.76 95.68 x 8516 ckpt log json

Notes:

  1. The gpus indicates the number of gpu (32G V100) we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train CSN model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/csn/ircsn_ig65m_pretrained_r152_32x2x1_58e_kinetics400_rgb.py \
    --work-dir work_dirs/ircsn_ig65m_pretrained_r152_32x2x1_58e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test CSN model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/csn/ircsn_ig65m_pretrained_r152_32x2x1_58e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips prob

For more details, you can refer to Test a dataset part in getting_started.

I3D

Introduction

@inproceedings{inproceedings,
  author = {Carreira, J. and Zisserman, Andrew},
  year = {2017},
  month = {07},
  pages = {4724-4733},
  title = {Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset},
  doi = {10.1109/CVPR.2017.502}
}

@article{NonLocal2018,
  author =   {Xiaolong Wang and Ross Girshick and Abhinav Gupta and Kaiming He},
  title =    {Non-local Neural Networks},
  journal =  {CVPR},
  year =     {2018}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
i3d_r50_32x2x1_100e_kinetics400_rgb 340x256 8 ResNet50 ImageNet 72.68 90.78 1.7 (320x3 frames) 5170 ckpt log json
i3d_r50_32x2x1_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 73.27 90.92 x 5170 ckpt log json
i3d_r50_video_32x2x1_100e_kinetics400_rgb short-side 256p 8 ResNet50 ImageNet 72.85 90.75 x 5170 ckpt log json
i3d_r50_dense_32x2x1_100e_kinetics400_rgb 340x256 8x2 ResNet50 ImageNet 72.77 90.57 1.7 (320x3 frames) 5170 ckpt log json
i3d_r50_dense_32x2x1_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 73.48 91.00 x 5170 ckpt log json
i3d_r50_lazy_32x2x1_100e_kinetics400_rgb 340x256 8 ResNet50 ImageNet 72.32 90.72 1.8 (320x3 frames) 5170 ckpt log json
i3d_r50_lazy_32x2x1_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 73.24 90.99 x 5170 ckpt log json
i3d_nl_embedded_gaussian_r50_32x2x1_100e_kinetics400_rgb short-side 256p 8x4 ResNet50 ImageNet 74.71 91.81 x 6438 ckpt log json
i3d_nl_gaussian_r50_32x2x1_100e_kinetics400_rgb short-side 256p 8x4 ResNet50 ImageNet 73.37 91.26 x 4944 ckpt log json
i3d_nl_dot_product_r50_32x2x1_100e_kinetics400_rgb short-side 256p 8x4 ResNet50 ImageNet 73.92 91.59 x 4832 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train I3D model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/i3d/i3d_r50_32x2x1_100e_kinetics400_rgb.py \
    --work-dir work_dirs/i3d_r50_32x2x1_100e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test I3D model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/i3d/i3d_r50_32x2x1_100e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips prob

For more details, you can refer to Test a dataset part in getting_started.

Omni-sourced Webly-supervised Learning for Video Recognition

Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, Dahua Lin

In ECCV, 2020. Paper

https://github.com/open-mmlab/mmaction2/blob/master/configs/recognition/omnisource/pipeline.png?raw=truepipeline

Model Zoo

Kinetics-400 Model Release

We currently released 4 models trained with OmniSource framework, including both 2D and 3D architectures. We compare the performance of models trained with or without OmniSource in the following table.

Model Modality Pretrained Backbone Input Resolution Top-1 (Baseline / OmniSource (Delta)) Top-5 (Baseline / OmniSource (Delta))) Download
TSN RGB ImageNet ResNet50 3seg 340x256 70.6 / 73.6 (+ 3.0) 89.4 / 91.0 (+ 1.6) Baseline / OmniSource
TSN RGB IG-1B ResNet50 3seg short-side 320 73.1 / 75.7 (+ 2.6) 90.4 / 91.9 (+ 1.5) Baseline / OmniSource
SlowOnly RGB Scratch ResNet50 4x16 short-side 320 72.9 / 76.8 (+ 3.9) 90.9 / 92.5 (+ 1.6) Baseline / OmniSource
SlowOnly RGB Scratch ResNet101 8x8 short-side 320 76.5 / 80.4 (+ 3.9) 92.7 / 94.4 (+ 1.7) Baseline / OmniSource

Benchmark on Mini-Kinetics

We release a subset of web dataset used in the OmniSource paper. Specifically, we release the web data in the 200 classes of Mini-Kinetics. The statistics of those datasets is detailed in preparing_omnisource. To obtain those data, you need to fill in a data request form. After we received your request, the download link of these data will be send to you. For more details on the released OmniSource web dataset, please refer to preparing_omnisource.

We benchmark the OmniSource framework on the released subset, results are listed in the following table (we report the Top-1 and Top-5 accuracy on Mini-Kinetics validation). The benchmark can be used as a baseline for video recognition with web data.

TSN-8seg-ResNet50

Setting Top-1 Top-5 ckpt json log
Baseline 77.4 93.6 ckpt json log
+GG-img 78.0 93.6 ckpt json log
+[GG-IG]-img 78.6 93.6 ckpt json log
+IG-vid 80.6 95.0 ckpt json log
+KRaw 78.6 93.2 ckpt json log
OmniSource 81.3 94.8 ckpt json log

SlowOnly-8x8-ResNet50

Setting Top-1 Top-5 ckpt json log
Baseline 78.6 93.9 ckpt json log
+GG-img 80.8 95.0 ckpt json log
+[GG-IG]-img 81.3 95.2 ckpt json log
+IG-vid 82.4 95.6 ckpt json log
+KRaw 80.3 94.5 ckpt json log
OmniSource 82.9 95.8 ckpt json log

We also list the benchmark in the original paper which run on Kinetics-400 for comparison:

Model Baseline +GG-img +[GG-IG]-img +IG-vid +KRaw OmniSource
TSN-3seg-ResNet50 70.6 / 89.4 71.5 / 89.5 72.0 / 90.0 72.0 / 90.3 71.7 / 89.6 73.6 / 91.0
SlowOnly-4x16-ResNet50 73.8 / 90.9 74.5 / 91.4 75.2 / 91.6 75.2 / 91.7 74.5 / 91.1 76.6 / 92.5

Citing OmniSource

If you find OmniSource useful for your research, please consider citing the paper using the following BibTeX entry.

@article{duan2020omni,
  title={Omni-sourced Webly-supervised Learning for Video Recognition},
  author={Duan, Haodong and Zhao, Yue and Xiong, Yuanjun and Liu, Wentao and Lin, Dahua},
  journal={arXiv preprint arXiv:2003.13042},
  year={2020}
}

R2plus1D

Introduction

@inproceedings{tran2018closer,
  title={A closer look at spatiotemporal convolutions for action recognition},
  author={Tran, Du and Wang, Heng and Torresani, Lorenzo and Ray, Jamie and LeCun, Yann and Paluri, Manohar},
  booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition},
  pages={6450--6459},
  year={2018}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
r2plus1d_r34_8x8x1_180e_kinetics400_rgb short-side 256 8x4 ResNet34 None 67.30 87.65 x 5019 ckpt log json
r2plus1d_r34_video_8x8x1_180e_kinetics400_rgb short-side 256 8 ResNet34 None 67.3 87.8 x 5019 ckpt log json
r2plus1d_r34_8x8x1_180e_kinetics400_rgb short-side 320 8x2 ResNet34 None 68.68 88.36 1.6 (80x3 frames) 5019 ckpt log json
r2plus1d_r34_32x2x1_180e_kinetics400_rgb short-side 320 8x2 ResNet34 None 74.60 91.59 0.5 (320x3 frames) 12975 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train R(2+1)D model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/r2plus1d/r2plus1d_r34_8x8x1_180e_kinetics400_rgb.py \
    --work-dir work_dirs/r2plus1d_r34_3d_8x8x1_180e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test R(2+1)D model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/r2plus1d/r2plus1d_r34_8x8x1_180e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips=prob

For more details, you can refer to Test a dataset part in getting_started.

SlowFast

Introduction

@inproceedings{feichtenhofer2019slowfast,
  title={Slowfast networks for video recognition},
  author={Feichtenhofer, Christoph and Fan, Haoqi and Malik, Jitendra and He, Kaiming},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={6202--6211},
  year={2019}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
slowfast_r50_4x16x1_256e_kinetics400_rgb short-side 256 8x4 ResNet50 None 74.75 91.73 x 6203 ckpt log json
slowfast_r50_video_4x16x1_256e_kinetics400_rgb short-side 256 8 ResNet50 None 74.34 91.58 x 6203 ckpt log json
slowfast_r50_4x16x1_256e_kinetics400_rgb short-side 320 8x3 ResNet50 None 75.64 92.3 1.6 ((32+4)x10x3 frames) 6203 ckpt log json
slowfast_r50_8x8x1_256e_kinetics400_rgb short-side 256 8x4 ResNet50 None 75.61 92.34 x 9062 ckpt log json
slowfast_r50_8x8x1_256e_kinetics400_rgb short-side 320 8x3 ResNet50 None 76.94 92.8 1.3 ((32+8)x10x3 frames) 9062 ckpt log json
slowfast_r101_r50_4x16x1_256e_kinetics400_rgb short-side 256 8x1 ResNet101 + ResNet50 None 76.69 93.07 16628 ckpt log json
slowfast_r101_8x8x1_256e_kinetics400_rgb short-side 256 8x4 ResNet101 None 77.90 93.51 25994 ckpt log json
slowfast_r152_r50_4x16x1_256e_kinetics400_rgb short-side 256 8x1 ResNet152 + ResNet50 None 77.13 93.20 10077 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train SlowFast model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/slowfast/slowfast_r50_4x16x1_256e_kinetics400_rgb.py \
    --work-dir work_dirs/slowfast_r50_4x16x1_256e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test SlowFast model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/slowfast/slowfast_r50_4x16x1_256e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips=prob

For more details, you can refer to Test a dataset part in getting_started.

SlowOnly

Introduction

@inproceedings{feichtenhofer2019slowfast,
  title={Slowfast networks for video recognition},
  author={Feichtenhofer, Christoph and Fan, Haoqi and Malik, Jitendra and He, Kaiming},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={6202--6211},
  year={2019}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
slowonly_r50_4x16x1_256e_kinetics400_rgb short-side 256 8x4 ResNet50 None 72.76 90.51 x 3168 ckpt log json
slowonly_r50_video_4x16x1_256e_kinetics400_rgb short-side 320 8x2 ResNet50 None 72.90 90.82 x 8472 ckpt log json
slowonly_r50_8x8x1_256e_kinetics400_rgb short-side 256 8x4 ResNet50 None 74.42 91.49 x 5820 ckpt log json
slowonly_r50_4x16x1_256e_kinetics400_rgb short-side 320 8x2 ResNet50 None 73.02 90.77 4.0 (40x3 frames) 3168 ckpt log json
slowonly_r50_8x8x1_256e_kinetics400_rgb short-side 320 8x3 ResNet50 None 74.93 91.92 2.3 (80x3 frames) 5820 ckpt log json
slowonly_imagenet_pretrained_r50_4x16x1_150e_kinetics400_rgb short-side 320 8x2 ResNet50 ImageNet 73.39 91.12 x 3168 ckpt log json
slowonly_imagenet_pretrained_r50_8x8x1_150e_kinetics400_rgb short-side 320 8x4 ResNet50 ImageNet 75.55 92.04 x 5820 ckpt log json
slowonly_nl_embedded_gaussian_r50_4x16x1_150e_kinetics400_rgb short-side 320 8x2 ResNet50 ImageNet 74.54 91.73 x 4435 ckpt log json
slowonly_nl_embedded_gaussian_r50_8x8x1_150e_kinetics400_rgb short-side 320 8x4 ResNet50 ImageNet 76.07 92.42 x 8895 ckpt log json
slowonly_r50_4x16x1_256e_kinetics400_flow short-side 320 8x2 ResNet50 ImageNet 61.79 83.62 x 8450 ckpt log json
slowonly_r50_8x8x1_196e_kinetics400_flow short-side 320 8x4 ResNet50 ImageNet 65.76 86.25 x 8455 ckpt log json

Kinetics-400 Data Benchmark

In data benchmark, we compare two different data preprocessing methods: (1) Resize video to 340x256, (2) Resize the short edge of video to 320px, (3) Resize the short edge of video to 256px.

config resolution gpus backbone Input pretrain top1 acc top5 acc testing protocol ckpt log json
slowonly_r50_randomresizedcrop_340x256_4x16x1_256e_kinetics400_rgb 340x256 8x2 ResNet50 4x16 None 71.61 90.05 10 clips x 3 crops ckpt log json
slowonly_r50_randomresizedcrop_320p_4x16x1_256e_kinetics400_rgb short-side 320 8x2 ResNet50 4x16 None 73.02 90.77 10 clips x 3 crops ckpt log json
slowonly_r50_randomresizedcrop_256p_4x16x1_256e_kinetics400_rgb short-side 256 8x4 ResNet50 4x16 None 72.76 90.51 10 clips x 3 crops ckpt log json

Kinetics-400 OmniSource Experiments

config resolution backbone pretrain w. OmniSource top1 acc top5 acc ckpt log json
slowonly_r50_4x16x1_256e_kinetics400_rgb short-side 320 ResNet50 None :x: 73.0 90.8 ckpt log json
x x ResNet50 None :heavy_check_mark: 76.8 92.5 ckpt x x
slowonly_r101_8x8x1_196e_kinetics400_rgb x ResNet101 None :x: 76.5 92.7 ckpt x x
x x ResNet101 None :heavy_check_mark: 80.4 94.4 ckpt x x

Kinetics-600

config resolution gpus backbone pretrain top1 acc top5 acc ckpt log json
slowonly_r50_video_8x8x1_256e_kinetics600_rgb short-side 256 8x4 ResNet50 None 77.5 93.7 ckpt log json

Kinetics-700

config resolution gpus backbone pretrain top1 acc top5 acc ckpt log json
slowonly_r50_video_8x8x1_256e_kinetics700_rgb short-side 256 8x4 ResNet50 None 65.0 86.1 ckpt log json

GYM99

config resolution gpus backbone pretrain top1 acc mean class acc ckpt log json
slowonly_imagenet_pretrained_r50_4x16x1_120e_gym99_rgb short-side 256 8x2 ResNet50 ImageNet 79.3 70.2 ckpt log json
slowonly_kinetics_pretrained_r50_4x16x1_120e_gym99_flow short-side 256 8x2 ResNet50 Kinetics 80.3 71.0 ckpt log json
1: 1 Fusion 83.7 74.8

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train SlowOnly model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/slowonly/slowonly_r50_4x16x1_256e_kinetics400_rgb.py \
    --work-dir work_dirs/slowonly_r50_4x16x1_256e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test SlowOnly model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/slowonly/slowonly_r50_4x16x1_256e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips=prob

For more details, you can refer to Test a dataset part in getting_started.

TANet

Introduction

@article{liu2020tam,
  title={TAM: Temporal Adaptive Module for Video Recognition},
  author={Liu, Zhaoyang and Wang, Limin and Wu, Wayne and Qian, Chen and Lu, Tong},
  journal={arXiv preprint arXiv:2005.06803},
  year={2020}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tanet_r50_dense_1x1x8_100e_kinetics400_rgb short-side 320 8 TANet ImageNet 76.28 92.60 76.22 92.53 x 7124 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 8 GPUs x 8 videos/gpu and lr=0.04 for 16 GPUs x 16 videos/gpu.

  2. The values in columns named after “reference” are the results got by testing on our dataset, using the checkpoints provided by the author with same model settings. The checkpoints for reference repo can be downloaded here.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train TANet model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/tanet/tanet_r50_dense_1x1x8_100e_kinetics400_rgb.py \
    --work-dir work_dirs/tanet_r50_dense_1x1x8_100e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test TANet model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/tanet/tanet_r50_dense_1x1x8_100e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json

For more details, you can refer to Test a dataset part in getting_started.

TIN

Introduction

@article{shao2020temporal,
    title={Temporal Interlacing Network},
    author={Hao Shao and Shengju Qian and Yu Liu},
    year={2020},
    journal={AAAI},
}

Model Zoo

Something-Something V1

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc gpu_mem(M) ckpt log json
tin_r50_1x1x8_40e_sthv1_rgb height 100 8x4 ResNet50 ImageNet 44.25 73.94 44.04 72.72 6181 ckpt log json

Something-Something V2

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc gpu_mem(M) ckpt log json
tin_r50_1x1x8_40e_sthv2_rgb height 240 8x4 ResNet50 ImageNet 56.70 83.62 56.48 83.45 6185 ckpt log json

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc gpu_mem(M) ckpt log json
tin_tsm_finetune_r50_1x1x8_50e_kinetics400_rgb short-side 256 8x4 ResNet50 TSM-Kinetics400 70.89 89.89 6187 ckpt log json

Here, we use finetune to indicate that we use TSM model trained on Kinetics-400 to finetune the TIN model on Kinetics-400.

Notes:

  1. The reference topk acc are got by training the original repo ##1aacd0c with no AverageMeter issue. The AverageMeter issue will lead to incorrect performance, so we fix it before running.

  2. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  3. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

  4. The values in columns named after “reference” are the results got by training on the original repo, using the same model settings.

For more details on data preparation, you can refer to Kinetics400, Something-Something V1 and Something-Something V2 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train TIN model on Something-Something V1 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/tin/tin_r50_1x1x8_40e_sthv1_rgb.py \
    --work-dir work_dirs/tin_r50_1x1x8_40e_sthv1_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test TIN model on Something-Something V1 dataset and dump the result to a json file.

python tools/test.py configs/recognition/tin/tin_r50_1x1x8_40e_sthv1_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json

For more details, you can refer to Test a dataset part in getting_started.

TPN

Introduction

@inproceedings{yang2020tpn,
  title={Temporal Pyramid Network for Action Recognition},
  author={Yang, Ceyuan and Xu, Yinghao and Shi, Jianping and Dai, Bo and Zhou, Bolei},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tpn_slowonly_r50_8x8x1_150e_kinetics_rgb short-side 320 8x4 ResNet50 ImageNet 73.10 91.03 x x x 6916 ckpt log json
tpn_imagenet_pretrained_slowonly_r50_8x8x1_150e_kinetics_rgb short-side 320 8x4 ResNet50 ImageNet 76.20 92.44 75.49 92.05 x 6916 ckpt log json

Something-Something V1

config resolution gpus backbone pretrain top1 acc top5 acc gpu_mem(M) ckpt log json
tpn_tsm_r50_1x1x8_150e_sthv1_rgb height 100 8x6 ResNet50 TSM 50.80 79.05 8828 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

  3. The values in columns named after “reference” are the results got by testing the checkpoint released on the original repo and codes, using the same dataset with ours.

For more details on data preparation, you can refer to Kinetics400, Something-Something V1 and Something-Something V2 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train TPN model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/tpn/tpn_slowonly_r50_8x8x1_150e_kinetics_rgb.py \
    --work-dir work_dirs/tpn_slowonly_r50_8x8x1_150e_kinetics_rgb [--validate --seed 0 --deterministic]

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test TPN model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/tpn/tpn_slowonly_r50_8x8x1_150e_kinetics_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips prob

For more details, you can refer to Test a dataset part in getting_started.

TRN

Introduction

@article{zhou2017temporalrelation,
    title = {Temporal Relational Reasoning in Videos},
    author = {Zhou, Bolei and Andonian, Alex and Oliva, Aude and Torralba, Antonio},
    journal={European Conference on Computer Vision},
    year={2018}
}

Model Zoo

Something-Something V1

config resolution gpus backbone pretrain top1 acc (efficient/accurate) top5 acc (efficient/accurate) gpu_mem(M) ckpt log json
trn_r50_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 31.62 / 33.88 60.01 / 62.12 11010 ckpt log json

Something-Something V2

config resolution gpus backbone pretrain top1 acc (efficient/accurate) top5 acc (efficient/accurate) gpu_mem(M) ckpt log json
trn_r50_1x1x8_50e_sthv2_rgb height 100 8 ResNet50 ImageNet 45.14 / 47.96 73.21 / 75.97 11010 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. There are two kinds of test settings for Something-Something dataset, efficient setting (center crop x 1 clip) and accurate setting (Three crop x 2 clip).

  3. In the original repository, the author augments data with random flipping on something-something dataset, but the augmentation method may be wrong due to the direct actions, such as push left to right. So, we replaced flip with flip with label mapping, and change the testing method TenCrop, which has five flipped crops, to Twice Sample & ThreeCrop.

  4. We use ResNet50 instead of BNInception as the backbone of TRN. When Training TRN-ResNet50 on sthv1 dataset in the original repository, we get top1 (top5) accuracy 30.542 (58.627) vs. ours 31.62 (60.01).

For more details on data preparation, you can refer to

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train TRN model on sthv1 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/trn/trn_r50_1x1x8_50e_sthv1_rgb.py \
    --work-dir work_dirs/trn_r50_1x1x8_50e_sthv1_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test TRN model on sthv1 dataset and dump the result to a json file.

python tools/test.py configs/recognition/trn/trn_r50_1x1x8_50e_sthv1_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json

For more details, you can refer to Test a dataset part in getting_started.

TSM

Introduction

@inproceedings{lin2019tsm,
  title={TSM: Temporal Shift Module for Efficient Video Understanding},
  author={Lin, Ji and Gan, Chuang and Han, Song},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2019}
}

@article{NonLocal2018,
  author =   {Xiaolong Wang and Ross Girshick and Abhinav Gupta and Kaiming He},
  title =    {Non-local Neural Networks},
  journal =  {CVPR},
  year =     {2018}
}

Model Zoo

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tsm_r50_1x1x8_50e_kinetics400_rgb 340x256 8 ResNet50 ImageNet 70.24 89.56 70.36 89.49 74.0 (8x1 frames) 7079 ckpt log json
tsm_r50_1x1x8_50e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 70.59 89.52 x x x 7079 ckpt log json
tsm_r50_gpu_normalize_1x1x8_50e_kinetics400_rgb.py short-side 256 8 ResNet50 ImageNet 70.48 89.40 x x x 7076 ckpt log json
tsm_r50_video_1x1x8_50e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 70.25 89.66 70.36 89.49 74.0 (8x1 frames) 7077 ckpt log json
tsm_r50_dense_1x1x8_100e_kinetics400_rgb 340x256 8x4 ResNet50 ImageNet 72.9 90.44 72.22 90.37 11.5 (8x10 frames) 7079 ckpt log json
tsm_r50_dense_1x1x8_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 73.38 91.02 x x x 7079 ckpt log json
tsm_r50_1x1x16_50e_kinetics400_rgb 340x256 8 ResNet50 ImageNet 72.09 90.37 70.67 89.98 47.0 (16x1 frames) 10404 ckpt log json
tsm_r50_1x1x16_50e_kinetics400_rgb short-side 256 8x4 ResNet50 ImageNet 71.89 90.73 x x x 10398 ckpt log json
tsm_nl_embedded_gaussian_r50_1x1x8_50e_kinetics400_rgb short-side 320 8x4 ResNet50 ImageNet 72.03 90.25 71.81 90.36 x 8931 ckpt log json
tsm_nl_gaussian_r50_1x1x8_50e_kinetics400_rgb short-side 320 8x4 ResNet50 ImageNet 70.70 89.90 x x x 10125 ckpt log json
tsm_nl_dot_product_r50_1x1x8_50e_kinetics400_rgb short-side 320 8x4 ResNet50 ImageNet 71.60 90.34 x x x 8358 ckpt log json
tsm_mobilenetv2_dense_1x1x8_100e_kinetics400_rgb short-side 320 8 MobileNetV2 ImageNet 68.46 88.64 x x x 3385 ckpt log json

Something-Something V1

config resolution gpus backbone pretrain top1 acc (efficient/accurate) top5 acc (efficient/accurate) reference top1 acc (efficient/accurate) reference top5 acc (efficient/accurate) gpu_mem(M) ckpt log json
tsm_r50_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 45.58 / 47.70 75.02 / 76.12 45.50 / 47.33 74.34 / 76.60 7077 ckpt log json
tsm_r50_flip_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 47.10 / 48.51 76.02 / 77.56 45.50 / 47.33 74.34 / 76.60 7077 ckpt log json
tsm_r50_randaugment_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 47.16 / 48.90 76.07 / 77.92 45.50 / 47.33 74.34 / 76.60 7077 ckpt log json
tsm_r50_flip_randaugment_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 47.85 / 50.31 76.78 / 78.18 45.50 / 47.33 74.34 / 76.60 7077 ckpt log json
tsm_r50_1x1x16_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 47.62 / 49.28 76.63 / 77.82 47.05 / 48.61 76.40 / 77.96 10390 ckpt log json
tsm_r101_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 45.72 / 48.43 74.67 / 76.72 46.64 / 48.13 75.40 / 77.31 9800 ckpt log json

Something-Something V2

config resolution gpus backbone pretrain top1 acc (efficient/accurate) top5 acc (efficient/accurate) reference top1 acc (efficient/accurate) reference top5 acc (efficient/accurate) gpu_mem(M) ckpt log json
tsm_r50_1x1x8_50e_sthv2_rgb height 240 8 ResNet50 ImageNet 57.86 / 61.12 84.67 / 86.26 57.98 / 60.69 84.57 / 86.28 7069 ckpt log json
tsm_r50_1x1x8_50e_sthv2_rgb height 256 8 ResNet50 ImageNet 60.79 / 63.84 86.60 / 88.30 xx / 61.2 xx / xx 7069 ckpt log json
tsm_r50_1x1x16_50e_sthv2_rgb height 240 8 ResNet50 ImageNet 59.93 / 62.04 86.10 / 87.35 58.90 / 60.98 85.29 / 86.60 10400 ckpt log json
tsm_r50_1x1x16_50e_sthv2_rgb height 256 8 ResNet50 ImageNet 61.06 / 63.19 86.66 / 87.93 xx / 63.1 xx / xx 10400 ckpt log json
tsm_r101_1x1x8_50e_sthv2_rgb height 240 8 ResNet101 ImageNet 58.59 / 61.51 85.07 / 86.90 58.89 / 61.36 85.14 / 87.00 9784 ckpt log json

MixUp & CutMix on Something-Something V1

config resolution gpus backbone pretrain top1 acc (efficient/accurate) top5 acc (efficient/accurate) delta top1 acc (efficient/accurate) delta top5 acc (efficient/accurate) ckpt log json
tsm_r50_mixup_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 46.35 / 48.49 75.07 / 76.88 +0.77 / +0.79 +0.05 / +0.70 ckpt log json
tsm_r50_cutmix_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 45.92 / 47.46 75.23 / 76.71 +0.34 / -0.24 +0.21 / +0.59 ckpt log json

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

  3. The values in columns named after “reference” are the results got by training on the original repo, using the same model settings. The checkpoints for reference repo can be downloaded here.

  4. There are two kinds of test settings for Something-Something dataset, efficient setting (center crop x 1 clip) and accurate setting (Three crop x 2 clip), which is referred from the original repo. We use efficient setting as default provided in config files, and it can be changed to accurate setting by

...
test_pipeline = [
    dict(
        type='SampleFrames',
        clip_len=1,
        frame_interval=1,
        num_clips=16,   ## `num_clips = 8` when using 8 segments
        twice_sample=True,    ## set `twice_sample=True` for twice sample in accurate setting
        test_mode=True),
    dict(type='RawFrameDecode'),
    dict(type='Resize', scale=(-1, 256)),
    ## dict(type='CenterCrop', crop_size=224), it is used for efficient setting
    dict(type='ThreeCrop', crop_size=256),  ## it is used for accurate setting
    dict(type='Normalize', **img_norm_cfg),
    dict(type='FormatShape', input_format='NCHW'),
    dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
    dict(type='ToTensor', keys=['imgs'])
]

  1. When applying Mixup and CutMix, we use the hyper parameter alpha=0.2.

For more details on data preparation, you can refer to Kinetics400, Something-Something V1 and Something-Something V2 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train TSM model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/tsm/tsm_r50_1x1x8_50e_kinetics400_rgb.py \
    --work-dir work_dirs/tsm_r50_1x1x8_100e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test TSM model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/tsm/tsm_r50_1x1x8_50e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json

For more details, you can refer to Test a dataset part in getting_started.

TSN

Introduction

@inproceedings{wang2016temporal,
  title={Temporal segment networks: Towards good practices for deep action recognition},
  author={Wang, Limin and Xiong, Yuanjun and Wang, Zhe and Qiao, Yu and Lin, Dahua and Tang, Xiaoou and Van Gool, Luc},
  booktitle={European conference on computer vision},
  pages={20--36},
  year={2016},
  organization={Springer}
}

Model Zoo

UCF-101

config gpus backbone pretrain top1 acc top5 acc gpu_mem(M) ckpt log json
tsn_r50_1x1x3_75e_ucf101_rgb [1] 8 ResNet50 ImageNet 83.03 96.78 8332 ckpt log json

[1] We report the performance on UCF-101 split1.

HMDB51

config gpus backbone pretrain top1 acc top5 acc gpu_mem(M) ckpt log json
tsn_r50_1x1x8_50e_hmdb51_imagenet_rgb 8 ResNet50 ImageNet 48.95 80.19 21535 ckpt log json
tsn_r50_1x1x8_50e_hmdb51_kinetics400_rgb 8 ResNet50 Kinetics400 56.08 84.31 21535 ckpt log json
tsn_r50_1x1x8_50e_hmdb51_mit_rgb 8 ResNet50 Moments 54.25 83.86 21535 ckpt log json

Kinetics-400

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tsn_r50_1x1x3_100e_kinetics400_rgb 340x256 8 ResNet50 ImageNet 70.60 89.26 x x 4.3 (25x10 frames) 8344 ckpt log json
tsn_r50_1x1x3_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 70.42 89.03 x x x 8343 ckpt log json
tsn_r50_dense_1x1x5_50e_kinetics400_rgb 340x256 8x3 ResNet50 ImageNet 70.18 89.10 69.15 88.56 12.7 (8x10 frames) 7028 ckpt log json
tsn_r50_320p_1x1x3_100e_kinetics400_rgb short-side 320 8x2 ResNet50 ImageNet 70.91 89.51 x x 10.7 (25x3 frames) 8344 ckpt log json
tsn_r50_320p_1x1x3_110e_kinetics400_flow short-side 320 8x2 ResNet50 ImageNet 55.70 79.85 x x x 8471 ckpt log json
tsn_r50_320p_1x1x3_kinetics400_twostream [1: 1]* x x ResNet50 ImageNet 72.76 90.52 x x x x x x x
tsn_r50_1x1x8_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 71.80 90.17 x x x 8343 ckpt log json
tsn_r50_320p_1x1x8_100e_kinetics400_rgb short-side 320 8x3 ResNet50 ImageNet 72.41 90.55 x x 11.1 (25x3 frames) 8344 ckpt log json
tsn_r50_320p_1x1x8_110e_kinetics400_flow short-side 320 8x4 ResNet50 ImageNet 57.76 80.99 x x x 8473 ckpt log json
tsn_r50_320p_1x1x8_kinetics400_twostream [1: 1]* x x ResNet50 ImageNet 74.64 91.77 x x x x x x x
tsn_r50_video_320p_1x1x3_100e_kinetics400_rgb short-side 320 8 ResNet50 ImageNet 71.11 90.04 x x x 8343 ckpt log json
tsn_r50_dense_1x1x8_100e_kinetics400_rgb 340x256 8 ResNet50 ImageNet 70.77 89.3 68.75 88.42 12.2 (8x10 frames) 8344 ckpt log json
tsn_r50_video_1x1x8_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 71.79 90.25 x x x 21558 ckpt log json
tsn_r50_video_dense_1x1x8_100e_kinetics400_rgb short-side 256 8 ResNet50 ImageNet 70.40 89.12 x x x 21553 ckpt log json

Here, We use [1: 1] to indicate that we combine rgb and flow score with coefficients 1: 1 to get the two-stream prediction (without applying softmax).

Using backbones from 3rd-party in TSN

It’s possible and convenient to use a 3rd-party backbone for TSN under the framework of MMAction2, here we provide some examples for:

config resolution gpus backbone pretrain top1 acc top5 acc ckpt log json
tsn_rn101_32x4d_320p_1x1x3_100e_kinetics400_rgb short-side 320 8x2 ResNeXt101-32x4d [MMCls] ImageNet 73.43 91.01 ckpt log json
tsn_dense161_320p_1x1x3_100e_kinetics400_rgb short-side 320 8x2 ResNeXt101-32x4d [TorchVision] ImageNet 72.78 90.75 ckpt log json

Kinetics-400 Data Benchmark (8-gpus, ResNet50, ImageNet pretrain; 3 segments)

In data benchmark, we compare:

  1. Different data preprocessing methods: (1) Resize video to 340x256, (2) Resize the short edge of video to 320px, (3) Resize the short edge of video to 256px;

  2. Different data augmentation methods: (1) MultiScaleCrop, (2) RandomResizedCrop;

  3. Different testing protocols: (1) 25 frames x 10 crops, (2) 25 frames x 3 crops.

config resolution training augmentation testing protocol top1 acc top5 acc ckpt log json
tsn_r50_multiscalecrop_340x256_1x1x3_100e_kinetics400_rgb 340x256 MultiScaleCrop 25x10 frames 70.60 89.26 ckpt log json
x 340x256 MultiScaleCrop 25x3 frames 70.52 89.39 x x x
tsn_r50_randomresizedcrop_340x256_1x1x3_100e_kinetics400_rgb 340x256 RandomResizedCrop 25x10 frames 70.11 89.01 ckpt log json
x 340x256 RandomResizedCrop 25x3 frames 69.95 89.02 x x x
tsn_r50_multiscalecrop_320p_1x1x3_100e_kinetics400_rgb short-side 320 MultiScaleCrop 25x10 frames 70.32 89.25 ckpt log json
x short-side 320 MultiScaleCrop 25x3 frames 70.54 89.39 x x x
tsn_r50_randomresizedcrop_320p_1x1x3_100e_kinetics400_rgb short-side 320 RandomResizedCrop 25x10 frames 70.44 89.23 ckpt log json
x short-side 320 RandomResizedCrop 25x3 frames 70.91 89.51 x x x
tsn_r50_multiscalecrop_256p_1x1x3_100e_kinetics400_rgb short-side 256 MultiScaleCrop 25x10 frames 70.42 89.03 ckpt log json
x short-side 256 MultiScaleCrop 25x3 frames 70.79 89.42 x x x
tsn_r50_randomresizedcrop_256p_1x1x3_100e_kinetics400_rgb short-side 256 RandomResizedCrop 25x10 frames 69.80 89.06 ckpt log json
x short-side 256 RandomResizedCrop 25x3 frames 70.48 89.89 x x x

Kinetics-400 OmniSource Experiments

config resolution backbone pretrain w. OmniSource top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tsn_r50_1x1x3_100e_kinetics400_rgb 340x256 ResNet50 ImageNet :x: 70.6 89.3 4.3 (25x10 frames) 8344 ckpt log json
x 340x256 ResNet50 ImageNet :heavy_check_mark: 73.6 91.0 x 8344 ckpt x x
x short-side 320 ResNet50 IG-1B [1] :x: 73.1 90.4 x 8344 ckpt x x
x short-side 320 ResNet50 IG-1B [1] :heavy_check_mark: 75.7 91.9 x 8344 ckpt x x

[1] We obtain the pre-trained model from torch-hub, the pretrain model we used is resnet50_swsl

Kinetics-600

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tsn_r50_video_1x1x8_100e_kinetics600_rgb short-side 256 8x2 ResNet50 ImageNet 74.8 92.3 11.1 (25x3 frames) 8344 ckpt log json

Kinetics-700

config resolution gpus backbone pretrain top1 acc top5 acc inference_time(video/s) gpu_mem(M) ckpt log json
tsn_r50_video_1x1x8_100e_kinetics700_rgb short-side 256 8x2 ResNet50 ImageNet 61.7 83.6 11.1 (25x3 frames) 8344 ckpt log json

Something-Something V1

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc gpu_mem(M) ckpt log json
tsn_r50_1x1x8_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 18.55 44.80 17.53 44.29 10978 ckpt log json
tsn_r50_1x1x16_50e_sthv1_rgb height 100 8 ResNet50 ImageNet 15.77 39.85 13.33 35.58 5691 ckpt log json

Something-Something V2

config resolution gpus backbone pretrain top1 acc top5 acc reference top1 acc reference top5 acc gpu_mem(M) ckpt log json
tsn_r50_1x1x8_50e_sthv2_rgb height 240 8 ResNet50 ImageNet 32.97 63.62 30.56 58.49 10966 ckpt log json
tsn_r50_1x1x16_50e_sthv2_rgb height 240 8 ResNet50 ImageNet 27.21 55.84 21.91 46.87 8337 ckpt log json

Moments in Time

config resolution gpus backbone pretrain top1 acc top5 acc gpu_mem(M) ckpt log json
tsn_r50_1x1x6_100e_mit_rgb short-side 256 8x2 ResNet50 ImageNet 26.84 51.6 8339 ckpt log json

Multi-Moments in Time

config resolution gpus backbone pretrain mAP gpu_mem(M) ckpt log json
tsn_r101_1x1x5_50e_mmit_rgb short-side 256 8x2 ResNet101 ImageNet 61.09 10467 ckpt log json

ActivityNet v1.3

config resolution gpus backbone pretrain top1 acc top5 acc gpu_mem(M) ckpt log json
tsn_r50_320p_1x1x8_50e_activitynet_video_rgb short-side 320 8x1 ResNet50 Kinetics400 73.93 93.44 5692 ckpt log json
tsn_r50_320p_1x1x8_50e_activitynet_clip_rgb short-side 320 8x1 ResNet50 Kinetics400 76.90 94.47 5692 ckpt log json
tsn_r50_320p_1x1x8_150e_activitynet_video_flow 340x256 8x2 ResNet50 Kinetics400 57.51 83.02 5780 ckpt log json
tsn_r50_320p_1x1x8_150e_activitynet_clip_flow 340x256 8x2 ResNet50 Kinetics400 59.51 82.69 5780 ckpt log json

HVU

config[1] tag category resolution gpus backbone pretrain mAP HATNet[2] HATNet-multi[2] ckpt log json
tsn_r18_1x1x8_100e_hvu_action_rgb action short-side 256 8x2 ResNet18 ImageNet 57.5 51.8 53.5 ckpt log json
tsn_r18_1x1x8_100e_hvu_scene_rgb scene short-side 256 8 ResNet18 ImageNet 55.2 55.8 57.2 ckpt log json
tsn_r18_1x1x8_100e_hvu_object_rgb object short-side 256 8 ResNet18 ImageNet 45.7 34.2 35.1 ckpt log json
tsn_r18_1x1x8_100e_hvu_event_rgb event short-side 256 8 ResNet18 ImageNet 63.7 38.5 39.8 ckpt log json
tsn_r18_1x1x8_100e_hvu_concept_rgb concept short-side 256 8 ResNet18 ImageNet 47.5 26.1 27.3 ckpt log json
tsn_r18_1x1x8_100e_hvu_attribute_rgb attribute short-side 256 8 ResNet18 ImageNet 46.1 33.6 34.9 ckpt log json
- Overall short-side 256 - ResNet18 ImageNet 52.6 40.0 41.3 - - -

[1] For simplicity, we train a specific model for each tag category as the baselines for HVU.

[2] The performance of HATNet and HATNet-multi are from the paper Large Scale Holistic Video Understanding. The proposed HATNet is a 2 branch Convolution Network (one 2D branch, one 3D branch) and share the same backbone(ResNet18) with us. The inputs of HATNet are 16 or 32 frames long video clips (which is much larger than us), while the input resolution is coarser (112 instead of 224). HATNet is trained on each individual task (each tag category) while HATNet-multi is trained on multiple tasks. Since there is no released codes or models for the HATNet, we just include the performance reported by the original paper.

Notes:

  1. The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

  3. The values in columns named after “reference” are the results got by training on the original repo, using the same model settings.

For more details on data preparation, you can refer to

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train TSN model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/tsn/tsn_r50_1x1x3_100e_kinetics400_rgb.py \
    --work-dir work_dirs/tsn_r50_1x1x3_100e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test TSN model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/tsn/tsn_r50_1x1x3_100e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json

For more details, you can refer to Test a dataset part in getting_started.

X3D

Introduction

@misc{feichtenhofer2020x3d,
      title={X3D: Expanding Architectures for Efficient Video Recognition},
      author={Christoph Feichtenhofer},
      year={2020},
      eprint={2004.04730},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Model Zoo

Kinetics-400

config resolution backbone top1 10-view top1 30-view reference top1 10-view reference top1 30-view ckpt
x3d_s_13x6x1_facebook_kinetics400_rgb short-side 320 X3D_S 72.7 73.2 73.1 [SlowFast] 73.5 [SlowFast] ckpt[1]
x3d_m_16x5x1_facebook_kinetics400_rgb short-side 320 X3D_M 75.0 75.6 75.1 [SlowFast] 76.2 [SlowFast] ckpt[1]

[1] The models are ported from the repo SlowFast and tested on our data. Currently, we only support the testing of X3D models, training will be available soon.

Notes:

  1. The values in columns named after “reference” are the results got by testing the checkpoint released on the original repo and codes, using the same dataset with ours.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test X3D model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/x3d/x3d_s_13x6x1_facebook_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips prob

For more details, you can refer to Test a dataset part in getting_started.

ResNet for Audio

Introduction

@article{xiao2020audiovisual,
  title={Audiovisual SlowFast Networks for Video Recognition},
  author={Xiao, Fanyi and Lee, Yong Jae and Grauman, Kristen and Malik, Jitendra and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2001.08740},
  year={2020}
}

Model Zoo

Kinetics-400

config n_fft gpus backbone pretrain top1 acc/delta top5 acc/delta inference_time(video/s) gpu_mem(M) ckpt log json
tsn_r18_64x1x1_100e_kinetics400_audio_feature 1024 8 ResNet18 None 19.7 35.75 x 1897 ckpt log json
tsn_r18_64x1x1_100e_kinetics400_audio_feature + tsn_r50_video_320p_1x1x3_100e_kinetics400_rgb 1024 8 ResNet(18+50) None 71.50(+0.39) 90.18(+0.14) x x x x x

Notes:

  1. The gpus indicates the number of gpus we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default. According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU, e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.

  2. The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.

  3. The values in columns named after “reference” are the results got by training on the original repo, using the same model settings.

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train ResNet model on Kinetics-400 audio dataset in a deterministic option with periodic validation.

python tools/train.py configs/audio_recognition/tsn_r50_64x1x1_100e_kinetics400_audio_feature.py \
    --work-dir work_dirs/tsn_r50_64x1x1_100e_kinetics400_audio_feature \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test ResNet model on Kinetics-400 audio dataset and dump the result to a json file.

python tools/test.py configs/audio_recognition/tsn_r50_64x1x1_100e_kinetics400_audio_feature.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json

For more details, you can refer to Test a dataset part in getting_started.

Fusion

For multi-modality fusion, you can use the simple script, the standard usage is:

python tools/analysis/report_accuracy.py --scores ${AUDIO_RESULT_PKL} ${VISUAL_RESULT_PKL} --datalist data/kinetics400/kinetics400_val_list_rawframes.txt --coefficient 1 1

  • AUDIO_RESULT_PKL: The saved output file of tools/test.py by the argument --out.

  • VISUAL_RESULT_PKL: The saved output file of tools/test.py by the argument --out.