Initial commit

2023-07-17 17:49:06 -07:00
commit 6d62d873fa
41 changed files with 5989 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,94 @@
+# CoTracker: It is Better to Track Together
+
+**[Meta AI Research, FAIR](https://ai.facebook.com/research/)**; **[University of Oxford, VGG](https://www.robots.ox.ac.uk/~vgg/)**
+
+[Nikita Karaev](https://nikitakaraevv.github.io/), [Ignacio Rocco](https://www.irocco.info/), [Benjamin Graham](https://ai.facebook.com/people/benjamin-graham/), [Natalia Neverova](https://nneverova.github.io/), [Andrea Vedaldi](https://www.robots.ox.ac.uk/~vedaldi/), [Christian Rupprecht](https://chrirupp.github.io/)
+
+[[`Paper`]()] [[`Project`](https://co-tracker.github.io/)] [[`BibTeX`](#citing-cotracker)]
+
+![bmx-bumps](./assets/bmx-bumps.gif)
+
+**CoTracker** is a fast transformer-based model that can track any point in a video. It brings to tracking some of the benefits of Optical Flow.
+ 
+CoTracker can track:
+- **Every pixel** within a video
+- Points sampled on a regular grid on any video frame 
+- Manually selected points
+
+Try these tracking modes for yourself with our [Colab demo](https://github.com/facebookresearch/co-tracker/notebooks/demo.ipynb).
+
+
+
+## Installation Instructions
+Ensure you have both PyTorch and TorchVision installed on your system. Follow the instructions [here](https://pytorch.org/get-started/locally/) for the installation. We strongly recommend installing both PyTorch and TorchVision with CUDA support.
+
+## Steps to Install CoTracker and its dependencies:
+```
+git clone https://github.com/facebookresearch/co-tracker
+cd co-tracker
+pip install -e .
+pip install opencv-python einops timm matplotlib moviepy flow_vis
+```
+
+
+## Model Weights Download:
+```
+mkdir checkpoints
+cd checkpoints
+wget https://dl.fbaipublicfiles.com/cotracker/cotracker_stride_4_wind_8.pth
+wget https://dl.fbaipublicfiles.com/cotracker/cotracker_stride_4_wind_12.pth
+wget https://dl.fbaipublicfiles.com/cotracker/cotracker_stride_8_wind_16.pth
+cd ..
+```
+
+
+## Running the Demo:
+Try our [Colab demo](https://github.com/facebookresearch/co-tracker/notebooks/demo.ipynb) or run a local demo with 10*10 points sampled on a grid on the first frame of a video:
+```
+python demo.py --grid_size 10
+```
+
+## Evaluation
+To reproduce the results presented in the paper, download the following datasets:
+- [TAP-Vid](https://github.com/deepmind/tapnet)
+- [BADJA](https://github.com/benjiebob/BADJA)
+- [ZJU-Mocap (FastCapture)](https://arxiv.org/abs/2303.11898)
+
+And install the necessary dependencies:
+```
+pip install hydra-core==1.1.0 mediapy tensorboard 
+```
+Then, execute the following command to evaluate on BADJA:
+```
+python ./cotracker/evaluation/evaluate.py --config-name eval_badja exp_dir=./eval_outputs dataset_root=your/badja/path
+```
+
+## Training
+To train the CoTracker as described in our paper, you first need to generate annotations for [Google Kubric](https://github.com/google-research/kubric) MOVI-f dataset.  Instructions for annotation generation can be found [here](https://github.com/deepmind/tapnet).
+
+Once you have the annotated dataset, you need to make sure you followed the steps for evaluation setup and install the training dependencies:
+```
+pip install pytorch_lightning==1.6.0
+```
+ launch training on Kubric. Our model was trained using 32 GPUs, and you can adjust the parameters to best suit your hardware setup.
+```
+python train.py --batch_size 1 --num_workers 28 \
+--num_steps 50000 --ckpt_path ./ --model_name cotracker \
+--save_freq 200 --sequence_len 24 --eval_datasets tapvid_davis_first badja \
+--traj_per_sample 256 --sliding_window_len 8 --updateformer_space_depth 6 --updateformer_time_depth 6 \
+--save_every_n_epoch 10 --evaluate_every_n_epoch 10 --model_stride 4
+```
+
+## License
+The majority of CoTracker is licensed under CC-BY-NC, however portions of the project are available under separate license terms: Particle Video Revisited is licensed under the MIT license, TAP-Vid is licensed under the Apache 2.0 license.
+
+## Citing CoTracker
+If you find our repository useful, please consider giving it a star ⭐ and citing our paper in your work:
+```
+@article{karaev2023cotracker,
+  title={CoTracker: It is Better to Track Together},
+  author={Nikita Karaev and Ignacio Rocco and Benjamin Graham and Natalia Neverova and Andrea Vedaldi and Christian Rupprecht},
+  journal={arxiv},
+  year={2023}
+}
+```