102->201 / NAS->autoDL / more configs of TAS / reorganize docs / fix bugs in NAS baselines
This commit is contained in:
D-X-Y
229
README.md
229
README.md
@@ -1,159 +1,92 @@
|
||||
# Neural Architecture Search (NAS)
|
||||
# Auto Deep Learning (AutoDL)
|
||||
|
||||
This project contains the following neural architecture search (NAS) algorithms, implemented in [PyTorch](http://pytorch.org).
|
||||
More NAS resources can be found in [Awesome-NAS](https://github.com/D-X-Y/Awesome-NAS).
|
||||
---------
|
||||
[](LICENSE.md)
|
||||
|
||||
- NAS-Bench-102: Extending the Scope of Reproducible Neural Architecture Search, ICLR 2020
|
||||
- Network Pruning via Transformable Architecture Search, NeurIPS 2019
|
||||
- One-Shot Neural Architecture Search via Self-Evaluated Template Network, ICCV 2019
|
||||
- Searching for A Robust Neural Architecture in Four GPU Hours, CVPR 2019
|
||||
- 10 NAS algorithms for the neural topology in `exps/algos` (see [NAS-Bench-102.md](https://github.com/D-X-Y/NAS-Projects/blob/master/NAS-Bench-102.md) for more details)
|
||||
- Several typical classification models, e.g., ResNet and DenseNet (see [BASELINE.md](https://github.com/D-X-Y/NAS-Projects/blob/master/BASELINE.md))
|
||||
Auto Deep Learning by DXY (AutoDL-Projects) is an open source, lightweight, but useful project for researchers.
|
||||
In this project, Xuanyi Dong implemented several neural architecture search (NAS) and hyper-parameter optimization (HPO) algorithms.
|
||||
He hopes to build it as an easy-to-use AutoDL toolkit in future.
|
||||
|
||||
## **Who should consider using AutoDL-Projects**
|
||||
|
||||
- Beginner who want to **try different AutoDL algorithms** for study
|
||||
- Engineer who want to **try AutoDL** to investigate whether AutoDL works on your projects
|
||||
- Researchers who want to **easily** implement and experiement **new** AutoDL algorithms.
|
||||
|
||||
## **Why should we use AutoDL-Projects**
|
||||
- Simplest library dependencies: each examlpe is purely relied on PyTorch or Tensorflow (except for some basic libraries in Anaconda)
|
||||
- All algorithms are in the same codebase. If you implement new algorithms, it is easy to fairly compare with many other baselines.
|
||||
- I will actively support this project, because all my furture AutoDL research will be built upon this project.
|
||||
|
||||
|
||||
## AutoDL-Projects Capabilities
|
||||
|
||||
At the moment, this project provides the following algorithms and scripts to run them. Please see the details in the link provided in the description column.
|
||||
|
||||
|
||||
<table>
|
||||
<tbody>
|
||||
<tr align="center" valign="bottom">
|
||||
<th>Type</th>
|
||||
<th>Algorithms</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
<tr> <!-- (1-st row) -->
|
||||
<td rowspan="5" align="center" valign="middle" halign="middle"> NAS </td>
|
||||
<td align="center" valign="middle"> Network Pruning via Transformable Architecture Search </td>
|
||||
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NIPS-2019-TAS.md">NIPS-2019-TAS.md</a> </td>
|
||||
</tr>
|
||||
<tr> <!-- (2-nd row) -->
|
||||
<td align="center" valign="middle"> Searching for A Robust Neural Architecture in Four GPU Hours </td>
|
||||
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/CVPR-2019-GDAS.md">CVPR-2019-GDAS.md</a> </td>
|
||||
</tr>
|
||||
<tr> <!-- (3-rd row) -->
|
||||
<td align="center" valign="middle"> One-Shot Neural Architecture Search via Self-Evaluated Template Network </td>
|
||||
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/ICCV-2019-SETN.py">ICCV-2019-SETN.py</a> </td>
|
||||
</tr>
|
||||
<tr> <!-- (4-th row) -->
|
||||
<td align="center" valign="middle"> NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search </td>
|
||||
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> </td>
|
||||
</tr>
|
||||
<tr> <!-- (5-th row) -->
|
||||
<td align="center" valign="middle"> ENAS / DARTS / REA / REINFORCE / BOHB </td>
|
||||
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> </td>
|
||||
</tr>
|
||||
<tr> <!-- (start second block) -->
|
||||
<td rowspan="1" align="center" valign="middle" halign="middle"> HPO </td>
|
||||
<td align="center" valign="middle"> coming soon </td>
|
||||
<td align="center" valign="middle"> coming soon </a> </td>
|
||||
</tr>
|
||||
<tr> <!-- (start third block) -->
|
||||
<td rowspan="1" align="center" valign="middle" halign="middle"> Basic </td>
|
||||
<td align="center" valign="middle"> Deep Learning-based Image Classification </td>
|
||||
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/BASELINE.md">BASELINE.md</a> </a> </td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## History of this repo
|
||||
|
||||
At first, this repo is `GDAS`, which is used to reproduce results in Searching for A Robust Neural Architecture in Four GPU Hours.
|
||||
After that, more functions and more NAS algorithms are continuely added in this repo. After it supports more than five algorithms, it is upgraded from `GDAS` to `NAS-Project`.
|
||||
Now, since both HPO and NAS are supported in this repo, it is upgraded from `NAS-Project` to `AutoDL-Projects`.
|
||||
|
||||
|
||||
## Requirements and Preparation
|
||||
|
||||
Please install `PyTorch>=1.2.0`, `Python>=3.6`, and `opencv`.
|
||||
Please install `Python>=3.6` and `PyTorch>=1.3.0`. (You could also run this project in lower versions of Python and PyTorch, but may have bugs).
|
||||
Some visualization codes may require `opencv`.
|
||||
|
||||
CIFAR and ImageNet should be downloaded and extracted into `$TORCH_HOME`.
|
||||
Some methods use knowledge distillation (KD), which require pre-trained models. Please download these models from [Google Driver](https://drive.google.com/open?id=1ANmiYEGX-IQZTfH8w0aSpj-Wypg-0DR-) (or train by yourself) and save into `.latent-data`.
|
||||
Some methods use knowledge distillation (KD), which require pre-trained models. Please download these models from [Google Drive](https://drive.google.com/open?id=1ANmiYEGX-IQZTfH8w0aSpj-Wypg-0DR-) (or train by yourself) and save into `.latent-data`.
|
||||
|
||||
### Usefull tools
|
||||
1. Compute the number of parameters and FLOPs of a model:
|
||||
```
|
||||
from utils import get_model_infos
|
||||
flop, param = get_model_infos(net, (1,3,32,32))
|
||||
```
|
||||
|
||||
2. Different NAS-searched architectures are defined [here](https://github.com/D-X-Y/NAS-Projects/blob/master/lib/nas_infer_model/DXYs/genotypes.py).
|
||||
|
||||
|
||||
## [NAS-Bench-102: Extending the Scope of Reproducible Neural Architecture Search](https://openreview.net/forum?id=HJxyZkBKDr)
|
||||
|
||||
We build a new benchmark for neural architecture search, please see more details in [NAS-Bench-102.md](https://github.com/D-X-Y/NAS-Projects/blob/master/NAS-Bench-102.md).
|
||||
|
||||
The benchmark data file (v1.0) is `NAS-Bench-102-v1_0-e61699.pth`, which can be downloaded from [Google Drive](https://drive.google.com/open?id=1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs).
|
||||
|
||||
Now you can simply use our API by `pip install nas-bench-102`.
|
||||
|
||||
## [Network Pruning via Transformable Architecture Search](https://arxiv.org/abs/1905.09717)
|
||||
[](https://paperswithcode.com/sota/network-pruning-on-cifar-100?p=network-pruning-via-transformable)
|
||||
|
||||
In this paper, we proposed a differentiable searching strategy for transformable architectures, i.e., searching for the depth and width of a deep neural network.
|
||||
You could see the highlight of our Transformable Architecture Search (TAS) at our [project page](https://xuanyidong.com/assets/projects/NeurIPS-2019-TAS.html).
|
||||
|
||||
<p float="left">
|
||||
<img src="https://d-x-y.github.com/resources/paper-icon/NIPS-2019-TAS.png" width="680px"/>
|
||||
<img src="https://d-x-y.github.com/resources/videos/NeurIPS-2019-TAS/TAS-arch.gif?raw=true" width="180px"/>
|
||||
</p>
|
||||
|
||||
|
||||
### Usage
|
||||
|
||||
Use `bash ./scripts/prepare.sh` to prepare data splits for `CIFAR-10`, `CIFARR-100`, and `ILSVRC2012`.
|
||||
If you do not have `ILSVRC2012` data, pleasee comment L12 in `./scripts/prepare.sh`.
|
||||
|
||||
args: `cifar10` indicates the dataset name, `ResNet56` indicates the basemodel name, `CIFARX` indicates the searching hyper-parameters, `0.47/0.57` indicates the expected FLOP ratio, `-1` indicates the random seed.
|
||||
|
||||
#### Search for the depth configuration of ResNet:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0,1 bash ./scripts-search/search-depth-gumbel.sh cifar10 ResNet110 CIFARX 0.57 -1
|
||||
```
|
||||
|
||||
#### Search for the width configuration of ResNet:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0,1 bash ./scripts-search/search-width-gumbel.sh cifar10 ResNet110 CIFARX 0.57 -1
|
||||
```
|
||||
|
||||
#### Search for both depth and width configuration of ResNet:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0,1 bash ./scripts-search/search-shape-cifar.sh cifar10 ResNet56 CIFARX 0.47 -1
|
||||
```
|
||||
|
||||
#### Training the searched shape config from TAS
|
||||
If you want to directly train a model with searched configuration of TAS, try these:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0,1 bash ./scripts/tas-infer-train.sh cifar10 C010-ResNet32 -1
|
||||
CUDA_VISIBLE_DEVICES=0,1 bash ./scripts/tas-infer-train.sh cifar100 C100-ResNet32 -1
|
||||
```
|
||||
|
||||
### Model Configuration
|
||||
The searched shapes for ResNet-20/32/56/110/164 in Table 3 in the original paper are listed in [`configs/NeurIPS-2019`](https://github.com/D-X-Y/NAS-Projects/tree/master/configs/NeurIPS-2019).
|
||||
|
||||
|
||||
## [One-Shot Neural Architecture Search via Self-Evaluated Template Network](https://arxiv.org/abs/1910.05733)
|
||||
|
||||
<img align="right" src="https://d-x-y.github.com/resources/paper-icon/ICCV-2019-SETN.png" width="450">
|
||||
|
||||
<strong>Highlight</strong>: we equip one-shot NAS with an architecture sampler and train network weights using uniformly sampling.
|
||||
|
||||
|
||||
### Usage
|
||||
|
||||
Please use the following scripts to train the searched SETN-searched CNN on CIFAR-10, CIFAR-100, and ImageNet.
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar10 SETN 96 -1
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar100 SETN 96 -1
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 bash ./scripts/nas-infer-train.sh imagenet-1k SETN 256 -1
|
||||
```
|
||||
|
||||
The searching codes of SETN on a small search space:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/SETN.sh cifar10 -1
|
||||
```
|
||||
|
||||
|
||||
## [Searching for A Robust Neural Architecture in Four GPU Hours](https://arxiv.org/abs/1910.04465)
|
||||
|
||||
|
||||
<img align="right" src="https://d-x-y.github.com/resources/paper-icon/CVPR-2019-GDAS.png" width="300">
|
||||
|
||||
We proposed a Gradient-based searching algorithm using Differentiable Architecture Sampling (GDAS). GDAS is baseed on DARTS and improves it with Gumbel-softmax sampling.
|
||||
Experiments on CIFAR-10, CIFAR-100, ImageNet, PTB, and WT2 are reported.
|
||||
|
||||
|
||||
### Usage
|
||||
|
||||
#### Reproducing the results of our searched architecture in GDAS
|
||||
Please use the following scripts to train the searched GDAS-searched CNN on CIFAR-10, CIFAR-100, and ImageNet.
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar10 GDAS_V1 96 -1
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar100 GDAS_V1 96 -1
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 bash ./scripts/nas-infer-train.sh imagenet-1k GDAS_V1 256 -1
|
||||
```
|
||||
|
||||
#### Searching on the NASNet search space
|
||||
Please use the following scripts to use GDAS to search as in the original paper:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/GDAS-search-NASNet-space.sh cifar10 1 -1
|
||||
```
|
||||
|
||||
#### Searching on a small search space (NAS-Bench-102)
|
||||
The GDAS searching codes on a small search space:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/GDAS.sh cifar10 -1
|
||||
```
|
||||
|
||||
The baseline searching codes are DARTS:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/DARTS-V1.sh cifar10 -1
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/DARTS-V2.sh cifar10 -1
|
||||
```
|
||||
|
||||
#### Training the searched architecture
|
||||
To train the searched architecture found by the above scripts, please use the following codes:
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-102/train-a-net.sh '|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|skip_connect~1|skip_connect~2|' 16 5
|
||||
```
|
||||
`|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|skip_connect~1|skip_connect~2|` represents the structure of a searched architecture. My codes will automatically print it during the searching procedure.
|
||||
|
||||
|
||||
# Citation
|
||||
## Citation
|
||||
|
||||
If you find that this project helps your research, please consider citing some of the following papers:
|
||||
```
|
||||
@inproceedings{dong2020nasbench102,
|
||||
title = {NAS-Bench-102: Extending the Scope of Reproducible Neural Architecture Search},
|
||||
@inproceedings{dong2020nasbench201,
|
||||
title = {NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search},
|
||||
author = {Dong, Xuanyi and Yang, Yi},
|
||||
booktitle = {International Conference on Learning Representations (ICLR)},
|
||||
url = {https://openreview.net/forum?id=HJxyZkBKDr},
|
||||
@@ -180,3 +113,11 @@ If you find that this project helps your research, please consider citing some o
|
||||
year = {2019}
|
||||
}
|
||||
```
|
||||
|
||||
## Related Projects
|
||||
|
||||
- [Awesome-NAS](https://github.com/D-X-Y/Awesome-NAS) : A curated list of neural architecture search and related resources.
|
||||
- [AutoML Freiburg-Hannover](https://www.automl.org/) : A website maintained by Frank Hutter's team, containing many AutoML resources.
|
||||
|
||||
# License
|
||||
The entire codebase is under [MIT license](LICENSE.md)
|
||||
|
||||
Reference in New Issue
Block a user