Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2022
2191.
Ahn, Saehyun; Chang, Jung-Woo; Yoon, Hyeon-Seok; Kang, Suk-Ju
TouchNAS: Efficient Touch Detection Model Design Methodology for Resource-Constrained Devices Journal Article
In: IEEE Sensors Journal, pp. 1-1, 2022.
@article{9695433,
title = {TouchNAS: Efficient Touch Detection Model Design Methodology for Resource-Constrained Devices},
author = {Saehyun Ahn and Jung-Woo Chang and Hyeon-Seok Yoon and Suk-Ju Kang},
url = {https://ieeexplore.ieee.org/abstract/document/9695433},
doi = {10.1109/JSEN.2022.3147469},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Sensors Journal},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2190.
Hassantabar, Shayan; Dai, Xiaoliang; Jha, Niraj K.
CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search Journal Article
In: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1-1, 2022.
@article{9698855,
title = {CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search},
author = {Shayan Hassantabar and Xiaoliang Dai and Niraj K. Jha},
url = {https://ieeexplore.ieee.org/abstract/document/9698855},
doi = {10.1109/TCAD.2022.3148202},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2189.
Li, Wei; Wen, Shiping; Shi, Kaibo; Yang, Yin; Huang, Tingwen
Neural Architecture Search with a Lightweight Transformer for Text-to-Image Synthesis Journal Article
In: IEEE Transactions on Network Science and Engineering, pp. 1-1, 2022.
@article{9699403,
title = {Neural Architecture Search with a Lightweight Transformer for Text-to-Image Synthesis},
author = {Wei Li and Shiping Wen and Kaibo Shi and Yin Yang and Tingwen Huang},
url = {https://ieeexplore.ieee.org/abstract/document/9699403},
doi = {10.1109/TNSE.2022.3147787},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Network Science and Engineering},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2188.
Pauletto, Loïc; Amini, Massih-Reza; Winckler, Nicolas
Self Semi Supervised Neural Architecture Search for Semantic Segmentation Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12646,
title = {Self Semi Supervised Neural Architecture Search for Semantic Segmentation},
author = {Loïc Pauletto and Massih-Reza Amini and Nicolas Winckler},
url = {https://arxiv.org/abs/2201.12646},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12646},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2187.
Xu, Dongkuan; Mukherjee, Subhabrata; Liu, Xiaodong; Dey, Debadeepta; Wang, Wenhui; Zhang, Xiang; Awadallah, Ahmed Hassan; Gao, Jianfeng
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12507,
title = {AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models},
author = {Dongkuan Xu and Subhabrata Mukherjee and Xiaodong Liu and Debadeepta Dey and Wenhui Wang and Xiang Zhang and Ahmed Hassan Awadallah and Jianfeng Gao},
url = {https://arxiv.org/abs/2201.12507},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12507},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2186.
Wei, Hui; Lee, Feifei; Hu, Chunyan; Chen, Qiu
MOO-DNAS: Efficient Neural Network Design via Differentiable Architecture Search Based on Multi-Objective Optimization Journal Article
In: IEEE Access, vol. 10, pp. 14195-14207, 2022.
@article{9698215,
title = {MOO-DNAS: Efficient Neural Network Design via Differentiable Architecture Search Based on Multi-Objective Optimization},
author = {Hui Wei and Feifei Lee and Chunyan Hu and Qiu Chen},
url = {https://ieeexplore.ieee.org/abstract/document/9698215},
doi = {10.1109/ACCESS.2022.3148323},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Access},
volume = {10},
pages = {14195-14207},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2185.
Guo, Bicheng; He, Shibo; Chen, Tao; Chen, Jiming; Ye, Peng
Neural Architecture Ranker Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12725,
title = {Neural Architecture Ranker},
author = {Bicheng Guo and Shibo He and Tao Chen and Jiming Chen and Peng Ye},
url = {https://arxiv.org/abs/2201.12725},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12725},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2184.
Na, Byunggook; Mok, Jisoo; Park, Seongsik; Lee, Dongjin; Choe, Hyeokjun; Yoon, Sungroh
AutoSNN: Towards Energy-Efficient Spiking Neural Networks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12738,
title = {AutoSNN: Towards Energy-Efficient Spiking Neural Networks},
author = {Byunggook Na and Jisoo Mok and Seongsik Park and Dongjin Lee and Hyeokjun Choe and Sungroh Yoon},
url = {https://arxiv.org/abs/2201.12738},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12738},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2183.
Rguibi, Zakaria; Hajami, Abdelmajid; Dya, Zitouni
Äutomatic Searching of Deep Neural Networks for Medical Imaging Diagnostic Proceedings Article
In: Saidi, Rajaa; Bhiri, Brahim El; Maleh, Yassine; Mosallam, Ayman; Essaaidi, Mohammed (Ed.): Ädvanced Technologies for Humanity", pp. 129–140, Springer International Publishing, Cham, 2022, ISBN: 978-3-030-94188-8.
@inproceedings{10.1007/978-3-030-94188-8_13,
title = {Äutomatic Searching of Deep Neural Networks for Medical Imaging Diagnostic},
author = {Zakaria Rguibi and Abdelmajid Hajami and Zitouni Dya},
editor = {Rajaa Saidi and Brahim El Bhiri and Yassine Maleh and Ayman Mosallam and Mohammed Essaaidi},
isbn = {978-3-030-94188-8},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Ädvanced Technologies for Humanity"},
pages = {129--140},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Medical imaging diagnosis is the most assisted method to help physicians diagnose patient diseases using different imaging test modalities. But Imbalanced data is one of the biggest challenges in the field of medical imaging. To advance this field, this work proposes a framework that can be used to find the optimal DNN architectures for a database with the challenge of imbalanced datasets. In our paper, we present a framework for automatic deep neural network search for medical imaging diagnosis.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Medical imaging diagnosis is the most assisted method to help physicians diagnose patient diseases using different imaging test modalities. But Imbalanced data is one of the biggest challenges in the field of medical imaging. To advance this field, this work proposes a framework that can be used to find the optimal DNN architectures for a database with the challenge of imbalanced datasets. In our paper, we present a framework for automatic deep neural network search for medical imaging diagnosis.
2182.
Cardoso, Rui P.; Hart, Emma; Kurka, David Burth; Pitt, Jeremy V.
Augmenting Novelty Search with a Surrogate Model to Engineer Meta-Diversity in Ensembles of Classifiers Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12896,
title = {Augmenting Novelty Search with a Surrogate Model to Engineer Meta-Diversity in Ensembles of Classifiers},
author = {Rui P. Cardoso and Emma Hart and David Burth Kurka and Jeremy V. Pitt},
url = {https://arxiv.org/abs/2201.12896},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12896},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2181.
Mehta, Yash; White, Colin; Zela, Arber; Krishnakumar, Arjun; Zabergja, Guri; Moradian, Shakiba; Safari, Mahmoud; Yu, Kaicheng; Hutter, Frank
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy Proceedings Article
In: ICLR 2022, 2022.
@inproceedings{DBLP:journals/corr/abs-2201-13396,
title = {NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy},
author = {Yash Mehta and Colin White and Arber Zela and Arjun Krishnakumar and Guri Zabergja and Shakiba Moradian and Mahmoud Safari and Kaicheng Yu and Frank Hutter},
url = {https://arxiv.org/abs/2201.13396},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {ICLR 2022},
journal = {CoRR},
volume = {abs/2201.13396},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2180.
Kang, Ziyang; Wang, Shiying; Wang, Lei; Li, Shiming; Qu, Lianhua; Xu, Weixia
Hardware-aware liquid state machine generation for 2D/3D Network-on-Chip platforms Journal Article
In: Journal of Systems Architecture, vol. 124, pp. 102429, 2022, ISSN: 1383-7621.
@article{KANG2022102429,
title = {Hardware-aware liquid state machine generation for 2D/3D Network-on-Chip platforms},
author = {Ziyang Kang and Shiying Wang and Lei Wang and Shiming Li and Lianhua Qu and Weixia Xu},
url = {https://www.sciencedirect.com/science/article/pii/S1383762122000297},
doi = {https://doi.org/10.1016/j.sysarc.2022.102429},
issn = {1383-7621},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Journal of Systems Architecture},
volume = {124},
pages = {102429},
abstract = {The liquid state machine (LSM) is a spiking neural network (SNN) that usually is offline mapped to an NoC-based neuromorphic processor to perform a specific task. The creation of these LSM models does not consider the structure of Network on Chip (NoC) which results in heavy communication pressure on the NoC. This paper proposes a hardware-aware generation framework for the LSM network by considering the spatial distribution of neurons in the NoC. It is the first time for the LSM generation work with combining the characteristics of NoC. This framework also adopts the heuristic algorithm to search the hyperparameter for the LSM networks to achieve state-of-art accuracy. It also reduces the spikes generated by those LSM models. It keeps the communication between neurons within cores as much as possible, which could reduce the communication between cores effectively and improve the performance of NoC, including reducing the traffic flow, reducing the average latency, improving the throughput and reducing the total running time.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The liquid state machine (LSM) is a spiking neural network (SNN) that usually is offline mapped to an NoC-based neuromorphic processor to perform a specific task. The creation of these LSM models does not consider the structure of Network on Chip (NoC) which results in heavy communication pressure on the NoC. This paper proposes a hardware-aware generation framework for the LSM network by considering the spatial distribution of neurons in the NoC. It is the first time for the LSM generation work with combining the characteristics of NoC. This framework also adopts the heuristic algorithm to search the hyperparameter for the LSM networks to achieve state-of-art accuracy. It also reduces the spikes generated by those LSM models. It keeps the communication between neurons within cores as much as possible, which could reduce the communication between cores effectively and improve the performance of NoC, including reducing the traffic flow, reducing the average latency, improving the throughput and reducing the total running time.
2179.
Hu, Xing; Liang, Ling; Chen, Xiaobing; Deng, Lei; Ji, Yu; Ding, Yufei; Du, Zidong; Guo, Qi; Sherwood, Tim; Xie, Yuan
A Systematic View of Model Leakage Risks in Deep Neural Network Systems Journal Article
In: IEEE Transactions on Computers, pp. 1-1, 2022.
@article{9705069,
title = {A Systematic View of Model Leakage Risks in Deep Neural Network Systems},
author = {Xing Hu and Ling Liang and Xiaobing Chen and Lei Deng and Yu Ji and Yufei Ding and Zidong Du and Qi Guo and Tim Sherwood and Yuan Xie},
url = {https://ieeexplore.ieee.org/abstract/document/9705069},
doi = {10.1109/TC.2022.3148235},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Computers},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2178.
Dushatskiy, Arkadiy; Alderliesten, Tanja; Bosman, Peter A. N.
Heed the Noise in Performance Evaluations in Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-02078,
title = {Heed the Noise in Performance Evaluations in Neural Architecture Search},
author = {Arkadiy Dushatskiy and Tanja Alderliesten and Peter A. N. Bosman},
url = {https://arxiv.org/abs/2202.02078},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.02078},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2177.
Cho, Hyunghun; Shin, Jungwook; Rhee, Wonjong
B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-03005,
title = {B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search},
author = {Hyunghun Cho and Jungwook Shin and Wonjong Rhee},
url = {https://arxiv.org/abs/2202.03005},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.03005},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2176.
Cho, Minsu; Joshi, Ameya; Garg, Siddharth; Reagen, Brandon; Hegde, Chinmay
Selective Network Linearization for Efficient Private Inference Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-02340,
title = {Selective Network Linearization for Efficient Private Inference},
author = {Minsu Cho and Ameya Joshi and Siddharth Garg and Brandon Reagen and Chinmay Hegde},
url = {https://arxiv.org/abs/2202.02340},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.02340},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2175.
Mazumder, Arnab Neelim; Mohsenin, Tinoosh
A Fast Network Exploration Strategy to Profile Low Energy Consumption for Keyword Spotting Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-02361,
title = {A Fast Network Exploration Strategy to Profile Low Energy Consumption for Keyword Spotting},
author = {Arnab Neelim Mazumder and Tinoosh Mohsenin},
url = {https://arxiv.org/abs/2202.02361},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.02361},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2174.
Hassanzadeh, Tahereh; Essam, Daryl; Sarker, Ruhul
EvoDCNN: An Evolutionary Deep Convolutional Neural Network for Image Classification Journal Article
In: Neurocomputing, 2022, ISSN: 0925-2312.
@article{HASSANZADEH2022,
title = {EvoDCNN: An Evolutionary Deep Convolutional Neural Network for Image Classification},
author = {Tahereh Hassanzadeh and Daryl Essam and Ruhul Sarker},
url = {https://www.sciencedirect.com/science/article/pii/S092523122200145X},
doi = {https://doi.org/10.1016/j.neucom.2022.02.003},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
abstract = {Developing Deep Convolutional Neural Networks (DCNNs) for image classification is a complicated task that needs considerable effort and knowledge. By employing an evolutionary computation approach, one can automatically generate the network models. However, the Neuroevolution is computationally expensive, and in some cases it needs hundreds of GPU days for training. Therefore, there is a need to find optimum Neuroevolutionary models with minimum computation to deal with this problem. In this paper, by utilising a Genetic Algorithm (GA), we introduce EvoDCNN, as a block-based evolutionary model for developing an evolutionary deep convolutional network for image classification. Such that by using the proposed fixed-length encoding model, we can generate variable-length networks with high accuracy while using less computation. The proposed model by utilising a straightforward evolutionary framework is able to establish small networks with high classification accuracy. Eight datasets: CIFAR10, MNIST, and six versions of EMNIST, that include balanced and unbalanced datasets, are used for evaluation of the proposed model. We did a comprehensive evaluation where we compared the results with many previous works, and outperformed the previous state-of-the-art accuracy for classification of five of the datasets.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Developing Deep Convolutional Neural Networks (DCNNs) for image classification is a complicated task that needs considerable effort and knowledge. By employing an evolutionary computation approach, one can automatically generate the network models. However, the Neuroevolution is computationally expensive, and in some cases it needs hundreds of GPU days for training. Therefore, there is a need to find optimum Neuroevolutionary models with minimum computation to deal with this problem. In this paper, by utilising a Genetic Algorithm (GA), we introduce EvoDCNN, as a block-based evolutionary model for developing an evolutionary deep convolutional network for image classification. Such that by using the proposed fixed-length encoding model, we can generate variable-length networks with high accuracy while using less computation. The proposed model by utilising a straightforward evolutionary framework is able to establish small networks with high classification accuracy. Eight datasets: CIFAR10, MNIST, and six versions of EMNIST, that include balanced and unbalanced datasets, are used for evaluation of the proposed model. We did a comprehensive evaluation where we compared the results with many previous works, and outperformed the previous state-of-the-art accuracy for classification of five of the datasets.
2173.
Rabczuk, Timon; Guo, Hongwei; Zhuang, Xiaoying; Chen, Pengwan; Alajlan, Naif
Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media Journal Article
In: Engineering with Computers, vol. 2022, pp. 1 – 26, 2022.
@article{RabczukGuoZhuangetal.2022,
title = {Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media},
author = {Timon Rabczuk and Hongwei Guo and Xiaoying Zhuang and Pengwan Chen and Naif Alajlan},
url = {https://link.springer.com/article/10.1007/s00366-021-01586-2},
doi = {10.1007/s00366-021-01586-2},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Engineering with Computers},
volume = {2022},
pages = {1 -- 26},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2172.
Lee, Jemin; Yu, Misun; Kwon, Yongin; Kim, Taeho
Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-05048,
title = {Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment},
author = {Jemin Lee and Misun Yu and Yongin Kwon and Taeho Kim},
url = {https://arxiv.org/abs/2202.05048},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.05048},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2171.
Wang, Duo; Zhao, Yiren; Shumailov, Ilia; Mullins, Robert D.
Model Architecture Adaption for Bayesian Neural Networks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-04392,
title = {Model Architecture Adaption for Bayesian Neural Networks},
author = {Duo Wang and Yiren Zhao and Ilia Shumailov and Robert D. Mullins},
url = {https://arxiv.org/abs/2202.04392},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.04392},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2170.
Elsken, Thomas; Zela, Arber; Metzen, Jan Hendrik; Staffler, Benedikt; Brox, Thomas; Valada, Abhinav; Hutter, Frank
Neural Architecture Search for Dense Prediction Tasks in Computer Vision Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-07242,
title = {Neural Architecture Search for Dense Prediction Tasks in Computer Vision},
author = {Thomas Elsken and Arber Zela and Jan Hendrik Metzen and Benedikt Staffler and Thomas Brox and Abhinav Valada and Frank Hutter},
url = {https://arxiv.org/abs/2202.07242},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.07242},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2169.
Sun, Junding; Yao, Chong; Liu, Jie; Liu, Weifan; Yu, Zekuan
GNAS-U2Net: A new optic cup and optic disc segmentation architecture with genetic neural architecture search Journal Article
In: IEEE Signal Processing Letters, pp. 1-1, 2022.
@article{9713989,
title = {GNAS-U2Net: A new optic cup and optic disc segmentation architecture with genetic neural architecture search},
author = {Junding Sun and Chong Yao and Jie Liu and Weifan Liu and Zekuan Yu},
url = {https://ieeexplore.ieee.org/abstract/document/9713989},
doi = {10.1109/LSP.2022.3151549},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Signal Processing Letters},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2168.
Speckhard, Daniel T.; Misiunas, Karolis; Perel, Sagi; Zhu, Tenghui; Carlile, Simon; Slaney, Malcolm
Neural Architecture Search for Energy Efficient Always-on Audio Models Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-05397,
title = {Neural Architecture Search for Energy Efficient Always-on Audio Models},
author = {Daniel T. Speckhard and Karolis Misiunas and Sagi Perel and Tenghui Zhu and Simon Carlile and Malcolm Slaney},
url = {https://arxiv.org/abs/2202.05397},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.05397},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2167.
Jia, Liang; Tian, Ye; Zhang, Junguo
Domain-Aware Neural Architecture Search for Classifying Animals in Camera Trap Images Journal Article
In: Animals, vol. 12, no. 4, 2022, ISSN: 2076-2615.
@article{ani12040437,
title = {Domain-Aware Neural Architecture Search for Classifying Animals in Camera Trap Images},
author = {Liang Jia and Ye Tian and Junguo Zhang},
url = {https://www.mdpi.com/2076-2615/12/4/437},
doi = {10.3390/ani12040437},
issn = {2076-2615},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Animals},
volume = {12},
number = {4},
abstract = {Camera traps provide a feasible way for ecological researchers to observe wildlife, and they often produce millions of images of diverse species requiring classification. This classification can be automated via edge devices installed with convolutional neural networks, but networks may need to be customized per device because edge devices are highly heterogeneous and resource-limited. This can be addressed by a neural architecture search capable of automatically designing networks. However, search methods are usually developed based on benchmark datasets differing widely from camera trap images in many aspects including data distributions and aspect ratios. Therefore, we designed a novel search method conducted directly on camera trap images with lowered resolutions and maintained aspect ratios; the search is guided by a loss function whose hyper parameter is theoretically derived for finding lightweight networks. The search was applied to two datasets and led to lightweight networks tested on an edge device named NVIDIA Jetson X2. The resulting accuracies were competitive in comparison. Conclusively, researchers without knowledge of designing networks can obtain networks optimized for edge devices and thus establish or expand surveillance areas in a cost-effective way.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Camera traps provide a feasible way for ecological researchers to observe wildlife, and they often produce millions of images of diverse species requiring classification. This classification can be automated via edge devices installed with convolutional neural networks, but networks may need to be customized per device because edge devices are highly heterogeneous and resource-limited. This can be addressed by a neural architecture search capable of automatically designing networks. However, search methods are usually developed based on benchmark datasets differing widely from camera trap images in many aspects including data distributions and aspect ratios. Therefore, we designed a novel search method conducted directly on camera trap images with lowered resolutions and maintained aspect ratios; the search is guided by a loss function whose hyper parameter is theoretically derived for finding lightweight networks. The search was applied to two datasets and led to lightweight networks tested on an edge device named NVIDIA Jetson X2. The resulting accuracies were competitive in comparison. Conclusively, researchers without knowledge of designing networks can obtain networks optimized for edge devices and thus establish or expand surveillance areas in a cost-effective way.
2166.
Yang, Junhuan; Sheng, Yi; Zhang, Sizhe; Wang, Ruixuan; Foreman, Kenneth; Paige, Mikell; Jiao, Xun; Jiang, Weiwen; Yang, Lei
Automated Architecture Search for Brain-inspired Hyperdimensional Computing Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-05827,
title = {Automated Architecture Search for Brain-inspired Hyperdimensional Computing},
author = {Junhuan Yang and Yi Sheng and Sizhe Zhang and Ruixuan Wang and Kenneth Foreman and Mikell Paige and Xun Jiao and Weiwen Jiang and Lei Yang},
url = {https://arxiv.org/abs/2202.05827},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.05827},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2165.
Huang, Mingqiang; Liu, Yucen; Cheng, Quan; Yang, Shuxin; Li, Kai; Luo, Junyi; Yang, Zhengke; Li, Qiufeng; Yu, Hao; Man, Changhai
A High Throughput Multi-Bit-Width 3D Systolic Accelerator for NAS Optimized Deep Neural Networks on FPGA Proceedings Article
In: Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 50, Association for Computing Machinery, Virtual Event, USA, 2022, ISBN: 9781450391498.
@inproceedings{10.1145/3490422.3502343,
title = {A High Throughput Multi-Bit-Width 3D Systolic Accelerator for NAS Optimized Deep Neural Networks on FPGA},
author = {Mingqiang Huang and Yucen Liu and Quan Cheng and Shuxin Yang and Kai Li and Junyi Luo and Zhengke Yang and Qiufeng Li and Hao Yu and Changhai Man},
url = {https://doi.org/10.1145/3490422.3502343},
doi = {10.1145/3490422.3502343},
isbn = {9781450391498},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays},
pages = {50},
publisher = {Association for Computing Machinery},
address = {Virtual Event, USA},
series = {FPGA '22},
abstract = {Neural architecture search (NAS) optimized multi-bit-width convolutional neural network (CNN) maintains the balance between network performance and efficiency, thus enlightening a promising method for accurate yet energy-efficient edge computing. In this work, we propose a high throughput three-dimensional (3D) systolic accelerator for NAS optimized CNNs, in which the input feature matrix, weight matrix and output feature matrix are delivering vertically, horizontally and perpendicularly through the systolic array respectively. With 3D systolic data flow, the processing time and logic resources consumption can be both reduced compared to the classical non-stationary systolic array. Besides, Booth-based multi-bit-width (INT2/4/8) multiply-add-accumulation (MAC) unit is developed within the 3D systolic accelerator. Deployed on FPGA platform Xilinx ZCU102, peek performance of the convolutional layer can reach as high as 2775 GOPS for INT2, 1650 GOPS for INT4, and 816 GOPS for INT8 respectively. The average performance on accelerating full NAS VGG16 network is 647 GOPS.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural architecture search (NAS) optimized multi-bit-width convolutional neural network (CNN) maintains the balance between network performance and efficiency, thus enlightening a promising method for accurate yet energy-efficient edge computing. In this work, we propose a high throughput three-dimensional (3D) systolic accelerator for NAS optimized CNNs, in which the input feature matrix, weight matrix and output feature matrix are delivering vertically, horizontally and perpendicularly through the systolic array respectively. With 3D systolic data flow, the processing time and logic resources consumption can be both reduced compared to the classical non-stationary systolic array. Besides, Booth-based multi-bit-width (INT2/4/8) multiply-add-accumulation (MAC) unit is developed within the 3D systolic accelerator. Deployed on FPGA platform Xilinx ZCU102, peek performance of the convolutional layer can reach as high as 2775 GOPS for INT2, 1650 GOPS for INT4, and 816 GOPS for INT8 respectively. The average performance on accelerating full NAS VGG16 network is 647 GOPS.
2164.
Teso-Fz-Betoño, Daniel; Zulueta, Ekaitz; Sanchez-Chica, Ander; Fernandez-Gamiz, Unai; Teso-Fz-Betoño, Adrian; Lopez-Guede, Jose Manuel
Neural architecture search for the estimation of relative positioning of the autonomous mobile robot Journal Article
In: Logic Journal of the IGPL, 2022, ISSN: 1367-0751, (jzac030).
@article{10.1093/jigpal/jzac030,
title = {Neural architecture search for the estimation of relative positioning of the autonomous mobile robot},
author = {Daniel Teso-Fz-Betoño and Ekaitz Zulueta and Ander Sanchez-Chica and Unai Fernandez-Gamiz and Adrian Teso-Fz-Betoño and Jose Manuel Lopez-Guede},
url = {https://doi.org/10.1093/jigpal/jzac030},
doi = {10.1093/jigpal/jzac030},
issn = {1367-0751},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Logic Journal of the IGPL},
abstract = {In the present work, an artificial neural network (ANN) will be developed to estimate the relative rotation and translation of the autonomous mobile robot (AMR). The ANN will work as an iterative closed point, which is commonly used with the singular value decomposition algorithm. This development will provide better resolution for a relative positioning technique that is essential for the AMR localization. The ANN requires a specific architecture, although in the current work a neural architecture search will be adapted to select the best ANN for estimating the relative motion. At the end, these ANNs will be compared with conventional algorithms to check the good performance of adopting an intelligent method for relative positioning estimation.},
note = {jzac030},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In the present work, an artificial neural network (ANN) will be developed to estimate the relative rotation and translation of the autonomous mobile robot (AMR). The ANN will work as an iterative closed point, which is commonly used with the singular value decomposition algorithm. This development will provide better resolution for a relative positioning technique that is essential for the AMR localization. The ANN requires a specific architecture, although in the current work a neural architecture search will be adapted to select the best ANN for estimating the relative motion. At the end, these ANNs will be compared with conventional algorithms to check the good performance of adopting an intelligent method for relative positioning estimation.
2163.
Chen, Jiamin; Gao, Jianliang; Chen, Yibo; Oloulade, Babatounde MOCTARD; Lyu, Tengfei; Li, Zhao
Auto-GNAS: A Parallel Graph Neural Architecture Search Framework Journal Article
In: IEEE Transactions on Parallel and Distributed Systems, pp. 1-1, 2022.
@article{9714826,
title = {Auto-GNAS: A Parallel Graph Neural Architecture Search Framework},
author = {Jiamin Chen and Jianliang Gao and Yibo Chen and Babatounde MOCTARD Oloulade and Tengfei Lyu and Zhao Li},
url = {https://ieeexplore.ieee.org/abstract/document/9714826},
doi = {10.1109/TPDS.2022.3151895},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Parallel and Distributed Systems},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2162.
Kim, Youngkee; Yun, Won Joon; Lee, Youn Kyu; Kim, Joongheon
Two-Stage Architectural Fine-Tuning with Neural Architecture Search using Early-Stopping in Image Classification Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-08604,
title = {Two-Stage Architectural Fine-Tuning with Neural Architecture Search using Early-Stopping in Image Classification},
author = {Youngkee Kim and Won Joon Yun and Youn Kyu Lee and Joongheon Kim},
url = {https://arxiv.org/abs/2202.08604},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.08604},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2161.
Kim, Jae Kwan; Ahn, Wonbin; Park, Sangin; Lee, Soo-Hong; Kim, Laehyun
Early Prediction of Sepsis Onset Using Neural Architecture Search Based on Genetic Algorithms Journal Article
In: International Journal of Environmental Research and Public Health, vol. 19, no. 4, 2022, ISSN: 1660-4601.
@article{ijerph19042349,
title = {Early Prediction of Sepsis Onset Using Neural Architecture Search Based on Genetic Algorithms},
author = {Jae Kwan Kim and Wonbin Ahn and Sangin Park and Soo-Hong Lee and Laehyun Kim},
url = {https://www.mdpi.com/1660-4601/19/4/2349},
doi = {10.3390/ijerph19042349},
issn = {1660-4601},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {International Journal of Environmental Research and Public Health},
volume = {19},
number = {4},
abstract = {Sepsis is a life-threatening condition with a high mortality rate. Early prediction and treatment are the most effective strategies for increasing survival rates. This paper proposes a neural architecture search (NAS) model to predict the onset of sepsis with a low computational cost and high search performance by applying a genetic algorithm (GA). The proposed model shares the weights of all possible connection nodes internally within the neural network. Externally, the search cost is reduced through the weight-sharing effect between the genotypes of the GA. A predictive analysis was performed using the Medical Information Mart for Intensive Care III (MIMIC-III), a medical time-series dataset, with the primary objective of predicting sepsis onset 3 h before occurrence. In addition, experiments were conducted under various prediction times (0-12 h) for comparison. The proposed model exhibited an area under the receiver operating characteristic curve (AUROC) score of 0.94 (95% CI: 0.92-0.96) for 3 h, which is 0.31-0.26 higher than the scores obtained using the Sequential Organ Failure Assessment (SOFA), quick SOFA (qSOFA), and Simplified Acute Physiology Score (SAPS) II scoring systems. Furthermore, the proposed model exhibited a 12% improvement in the AUROC value over a simple model based on the long short-term memory neural network. Additionally, it is not only optimally searchable for sepsis onset prediction, but also outperforms conventional models that use similar predictive purposes and datasets. Notably, it is sufficiently robust to shape changes in the input data and has less structural dependence.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Sepsis is a life-threatening condition with a high mortality rate. Early prediction and treatment are the most effective strategies for increasing survival rates. This paper proposes a neural architecture search (NAS) model to predict the onset of sepsis with a low computational cost and high search performance by applying a genetic algorithm (GA). The proposed model shares the weights of all possible connection nodes internally within the neural network. Externally, the search cost is reduced through the weight-sharing effect between the genotypes of the GA. A predictive analysis was performed using the Medical Information Mart for Intensive Care III (MIMIC-III), a medical time-series dataset, with the primary objective of predicting sepsis onset 3 h before occurrence. In addition, experiments were conducted under various prediction times (0–12 h) for comparison. The proposed model exhibited an area under the receiver operating characteristic curve (AUROC) score of 0.94 (95% CI: 0.92–0.96) for 3 h, which is 0.31–0.26 higher than the scores obtained using the Sequential Organ Failure Assessment (SOFA), quick SOFA (qSOFA), and Simplified Acute Physiology Score (SAPS) II scoring systems. Furthermore, the proposed model exhibited a 12% improvement in the AUROC value over a simple model based on the long short-term memory neural network. Additionally, it is not only optimally searchable for sepsis onset prediction, but also outperforms conventional models that use similar predictive purposes and datasets. Notably, it is sufficiently robust to shape changes in the input data and has less structural dependence.
2160.
Dai, Liuyao; Cheng, Quan; Wang, Yuhang; Huang, Gengbin; Zhou, Junzhuo; Li, Kai; Mao, Wei; Yu, Hao
An Energy-Efficient Bit-Split-and-Combination Systolic Accelerator for NAS-Based Multi-Precision Convolution Neural Networks Proceedings Article
In: 2022 27th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 448-453, 2022.
@inproceedings{9712509,
title = {An Energy-Efficient Bit-Split-and-Combination Systolic Accelerator for NAS-Based Multi-Precision Convolution Neural Networks},
author = {Liuyao Dai and Quan Cheng and Yuhang Wang and Gengbin Huang and Junzhuo Zhou and Kai Li and Wei Mao and Hao Yu},
url = {https://ieeexplore.ieee.org/abstract/document/9712509},
doi = {10.1109/ASP-DAC52403.2022.9712509},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 27th Asia and South Pacific Design Automation Conference (ASP-DAC)},
pages = {448-453},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2159.
Lee, Jooyeon; Park, Junsang; Lee, Seunghyun; Kung, Jaeha
Implication of Optimizing NPU Dataflows on Neural Architecture Search for Mobile Devices Journal Article
In: ACM Trans. Des. Autom. Electron. Syst., 2022, ISSN: 1084-4309, (Just Accepted).
@article{10.1145/3513085,
title = {Implication of Optimizing NPU Dataflows on Neural Architecture Search for Mobile Devices},
author = {Jooyeon Lee and Junsang Park and Seunghyun Lee and Jaeha Kung},
url = {https://doi.org/10.1145/3513085},
doi = {10.1145/3513085},
issn = {1084-4309},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {ACM Trans. Des. Autom. Electron. Syst.},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Recent advances in deep learning have made it possible to implement artificial intelligence in mobile devices. Many studies have put a lot of effort into developing lightweight deep learning models optimized for mobile devices. To overcome the performance limitations of manually designed deep learning models, an automated search algorithm, called neural architecture search (NAS), has been proposed. However, studies on the effect of hardware architecture of the mobile device on the performance of NAS have been less explored. In this paper, we show the importance of optimizing a hardware architecture, namely NPU dataflow, when searching for a more accurate yet fast deep learning model. To do so, we first implement an optimization framework, named FlowOptimizer, for generating a best possible NPU dataflow for a given deep learning operator. Then, we utilize this framework during the latency-aware NAS to find the model with the highest accuracy satisfying the latency constraint. As a result, we show that the searched model with FlowOptimizer outperforms the performance by 87.1% and 92.3% on average compared to the searched model with NVDLA and Eyeriss, respectively, with better accuracy on a proxy dataset. We also show that the searched model can be transferred to a larger model to classify a more complex image dataset, i.e., ImageNet, achieving 0.2%/5.4% higher Top-1/Top-5 accuracy compared to MobileNetV2-1.0 with 3.6 texttimes lower latency.},
note = {Just Accepted},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recent advances in deep learning have made it possible to implement artificial intelligence in mobile devices. Many studies have put a lot of effort into developing lightweight deep learning models optimized for mobile devices. To overcome the performance limitations of manually designed deep learning models, an automated search algorithm, called neural architecture search (NAS), has been proposed. However, studies on the effect of hardware architecture of the mobile device on the performance of NAS have been less explored. In this paper, we show the importance of optimizing a hardware architecture, namely NPU dataflow, when searching for a more accurate yet fast deep learning model. To do so, we first implement an optimization framework, named FlowOptimizer, for generating a best possible NPU dataflow for a given deep learning operator. Then, we utilize this framework during the latency-aware NAS to find the model with the highest accuracy satisfying the latency constraint. As a result, we show that the searched model with FlowOptimizer outperforms the performance by 87.1% and 92.3% on average compared to the searched model with NVDLA and Eyeriss, respectively, with better accuracy on a proxy dataset. We also show that the searched model can be transferred to a larger model to classify a more complex image dataset, i.e., ImageNet, achieving 0.2%/5.4% higher Top-1/Top-5 accuracy compared to MobileNetV2-1.0 with 3.6 texttimes lower latency.
2158.
Zhang, Chunhui; Yuan, Xiaoming; Zhang, Qianyun; Zhu, Guangxu; Cheng, Lei; Zhang, Ning
Towards Tailored Models on Private AIoT Devices: Federated Direct Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-11490,
title = {Towards Tailored Models on Private AIoT Devices: Federated Direct Neural Architecture Search},
author = {Chunhui Zhang and Xiaoming Yuan and Qianyun Zhang and Guangxu Zhu and Lei Cheng and Ning Zhang},
url = {https://arxiv.org/abs/2202.11490},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.11490},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2157.
Bosma, Martijn M. A.; Dushatskiy, Arkadiy; Grewal, Monika; Alderliesten, Tanja; Bosman, Peter A. N.
Mixed-Block Neural Architecture Search for Medical Image Segmentation Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-11401,
title = {Mixed-Block Neural Architecture Search for Medical Image Segmentation},
author = {Martijn M. A. Bosma and Arkadiy Dushatskiy and Monika Grewal and Tanja Alderliesten and Peter A. N. Bosman},
url = {https://arxiv.org/abs/2202.11401},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.11401},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2156.
Sheng, Yi; Yang, Junhuan; Wu, Yawen; Mao, Kevin; Shi, Yiyu; Hu, Jingtong; Jiang, Weiwen; Yang, Lei
The Larger The Fairer? Small Neural Networks Can Achieve Fairness for Edge Devices Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-11317,
title = {The Larger The Fairer? Small Neural Networks Can Achieve Fairness for Edge Devices},
author = {Yi Sheng and Junhuan Yang and Yawen Wu and Kevin Mao and Yiyu Shi and Jingtong Hu and Weiwen Jiang and Lei Yang},
url = {https://arxiv.org/abs/2202.11317},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.11317},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2155.
Zhao, Shixiong; Li, Fanxin; Chen, Xusheng; Shen, Tianxiang; Chen, Li; Wang, Sen; Zhang, Nicholas; Li, Cheng; Cui, Heming
NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism Proceedings Article
In: Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 374–387, Association for Computing Machinery, Lausanne, Switzerland, 2022, ISBN: 9781450392051.
@inproceedings{10.1145/3503222.3507735,
title = {NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism},
author = {Shixiong Zhao and Fanxin Li and Xusheng Chen and Tianxiang Shen and Li Chen and Sen Wang and Nicholas Zhang and Cheng Li and Heming Cui},
url = {https://doi.org/10.1145/3503222.3507735},
doi = {10.1145/3503222.3507735},
isbn = {9781450392051},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems},
pages = {374–387},
publisher = {Association for Computing Machinery},
address = {Lausanne, Switzerland},
series = {ASPLOS 2022},
abstract = {Supernet training, a prevalent and important paradigm in Neural Architecture Search, embeds the whole DNN architecture search space into one monolithic supernet, iteratively activates a subset of the supernet (i.e., a subnet) for fitting each batch of data, and searches a high-quality subnet which meets specific requirements. Although training subnets in parallel on multiple GPUs is desirable for acceleration, there inherently exists a race hazard that concurrent subnets may access the same DNN layers. Existing systems support neither efficiently parallelizing subnets’ training executions, nor resolving the race hazard deterministically, leading to unreproducible training procedures and potentiallly non-trivial accuracy loss. We present NASPipe, the first high-performance and reproducible distributed supernet training system via causal synchronous parallel (CSP) pipeline scheduling abstraction: NASPipe partitions a supernet across GPUs and concurrently executes multiple generated sub-tasks (subnets) in a pipelined manner; meanwhile, it oversees the correlations between the subnets and deterministically resolves any causal dependency caused by subnets’ layer sharing. To obtain high performance, NASPipe’s CSP scheduler exploits the fact that the larger a supernet spans, the fewer dependencies manifest between chronologically close subnets; therefore, it aggressively schedules the subnets with larger chronological orders into execution, only if they are not causally dependent on unfinished precedent subnets. Moreover, to relieve the excessive GPU memory burden for holding the whole supernet’s parameters, NASPipe uses a context switch technique that stashes the whole supernet in CPU memory, precisely predicts the subnets’ schedule, and pre-fetches/evicts a subnet before/after its execution. The evaluation shows that NASPipe is the only system that retains supernet training reproducibility, while achieving a comparable and even higher performance (up to 7.8X) compared to three recent pipeline training systems (e.g., GPipe).},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Supernet training, a prevalent and important paradigm in Neural Architecture Search, embeds the whole DNN architecture search space into one monolithic supernet, iteratively activates a subset of the supernet (i.e., a subnet) for fitting each batch of data, and searches a high-quality subnet which meets specific requirements. Although training subnets in parallel on multiple GPUs is desirable for acceleration, there inherently exists a race hazard that concurrent subnets may access the same DNN layers. Existing systems support neither efficiently parallelizing subnets’ training executions, nor resolving the race hazard deterministically, leading to unreproducible training procedures and potentiallly non-trivial accuracy loss. We present NASPipe, the first high-performance and reproducible distributed supernet training system via causal synchronous parallel (CSP) pipeline scheduling abstraction: NASPipe partitions a supernet across GPUs and concurrently executes multiple generated sub-tasks (subnets) in a pipelined manner; meanwhile, it oversees the correlations between the subnets and deterministically resolves any causal dependency caused by subnets’ layer sharing. To obtain high performance, NASPipe’s CSP scheduler exploits the fact that the larger a supernet spans, the fewer dependencies manifest between chronologically close subnets; therefore, it aggressively schedules the subnets with larger chronological orders into execution, only if they are not causally dependent on unfinished precedent subnets. Moreover, to relieve the excessive GPU memory burden for holding the whole supernet’s parameters, NASPipe uses a context switch technique that stashes the whole supernet in CPU memory, precisely predicts the subnets’ schedule, and pre-fetches/evicts a subnet before/after its execution. The evaluation shows that NASPipe is the only system that retains supernet training reproducibility, while achieving a comparable and even higher performance (up to 7.8X) compared to three recent pipeline training systems (e.g., GPipe).
2154.
Zhang, Wentao; Shen, Yu; Lin, Zheyu; Li, Yang; Li, Xiaosen; Ouyang, Wen; Tao, Yangyu; Yang, Zhi; Cui, Bin
PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-00638,
title = {PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm},
author = {Wentao Zhang and Yu Shen and Zheyu Lin and Yang Li and Xiaosen Li and Wen Ouyang and Yangyu Tao and Zhi Yang and Bin Cui},
url = {https://doi.org/10.48550/arXiv.2203.00638},
doi = {10.48550/arXiv.2203.00638},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.00638},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2153.
Zhang, Tianning; Ang, Yee Sin; Li, Erping; Kee, Chun Yun; Ang, L. K.
SUTD-PRCM Dataset and Neural Architecture Search Approach for Complex Metasurface Design Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-00002,
title = {SUTD-PRCM Dataset and Neural Architecture Search Approach for Complex Metasurface Design},
author = {Tianning Zhang and Yee Sin Ang and Erping Li and Chun Yun Kee and L. K. Ang},
url = {https://doi.org/10.48550/arXiv.2203.00002},
doi = {10.48550/arXiv.2203.00002},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.00002},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2152.
Huang, Yongdong; Li, Yuanzhan; Cao, Xulong; Zhang, Siyu; Cai, Shen; Lu, Ting; Liu, Yuqi
An Efficient End-to-End 3D Model Reconstruction based on Neural Architecture Search Technical Report
2022.
@techreport{HuangNAS,
title = {An Efficient End-to-End 3D Model Reconstruction based on Neural Architecture Search},
author = {Yongdong Huang and Yuanzhan Li and Xulong Cao and Siyu Zhang and Shen Cai and Ting Lu and Yuqi Liu},
url = {https://arxiv.org/abs/2202.13313},
doi = {10.48550/ARXIV.2202.13313},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2151.
Seong, Jaeho; Lee, Chaehyun; Han, Dong Seog
Neural Architecture Search for Real-Time Driver Behavior Recognition Proceedings Article
In: 2022 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pp. 104-108, 2022.
@inproceedings{9722706,
title = {Neural Architecture Search for Real-Time Driver Behavior Recognition},
author = {Jaeho Seong and Chaehyun Lee and Dong Seog Han},
url = {https://ieeexplore.ieee.org/abstract/document/9722706},
doi = {10.1109/ICAIIC54071.2022.9722706},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 International Conference on Artificial Intelligence in Information and Communication (ICAIIC)},
pages = {104-108},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2150.
Cummings, Daniel; Sridhar, Sharath Nittur; Sarah, Anthony; Szankin, Maciej
Accelerating Neural Architecture Exploration Across Modalities Using Genetic Algorithms Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-12934,
title = {Accelerating Neural Architecture Exploration Across Modalities Using Genetic Algorithms},
author = {Daniel Cummings and Sharath Nittur Sridhar and Anthony Sarah and Maciej Szankin},
url = {https://arxiv.org/abs/2202.12934},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.12934},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2149.
Zhao, Yaqin; Feng, Liqi; Tang, Jiaxi; Zhao, Wenxuan; Ding, Zhipeng; Li, Ao; Zheng, Zhaoxiang
Automatically recognizing four-legged animal behaviors to enhance welfare using spatial temporal graph convolutional networks Journal Article
In: Applied Animal Behaviour Science, vol. 249, pp. 105594, 2022, ISSN: 0168-1591.
@article{ZHAO2022105594,
title = {Automatically recognizing four-legged animal behaviors to enhance welfare using spatial temporal graph convolutional networks},
author = {Yaqin Zhao and Liqi Feng and Jiaxi Tang and Wenxuan Zhao and Zhipeng Ding and Ao Li and Zhaoxiang Zheng},
url = {https://www.sciencedirect.com/science/article/pii/S0168159122000521},
doi = {https://doi.org/10.1016/j.applanim.2022.105594},
issn = {0168-1591},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Applied Animal Behaviour Science},
volume = {249},
pages = {105594},
abstract = {Automatically recognizing animal behaviors in zoos and in national natural reserves can provide valuable insight into their welfare for facilitating scientific decision-making processes in animal management. Due to the difficulty of capturing massive amounts of animal video footage, a few existing methods have identified the behaviors of several different animal species in static images, but little is known about video-based animal behavior recognition. An animal's behavior is carried out in consecutive frames rather than in a single image; thus, image-based animal behavior recognition methods have low recognition accuracy. To address this dilemma, we not only construct the first skeleton-based dynamic multispecies dataset (Animal-Skeleton) but also propose a novel scheme that automatically designs the best spatial-temporal graph convolutional network (GCN) architecture with neural architecture search (NAS) to perform animal behavior recognition, named Animal-Nas for short. This is the first time that GCNs with NAS have been introduced into the animal behavior recognition task. To alleviate the trial-and-error cost of manually designing the network structure, we turn to NAS and design a novel search space with graph-based cells. Furthermore, we adopt a differentiable architecture search strategy to automatically search the cost-efficient spatial-temporal graph convolutional network structure. To evaluate the performance of the proposed model, we conduct extensive experiments on Animal-Skeleton datasets from three perspectives: model accuracy, parameter amount and stability. The results show that our model can achieve state-of the-art performance with fewer parameters.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Automatically recognizing animal behaviors in zoos and in national natural reserves can provide valuable insight into their welfare for facilitating scientific decision-making processes in animal management. Due to the difficulty of capturing massive amounts of animal video footage, a few existing methods have identified the behaviors of several different animal species in static images, but little is known about video-based animal behavior recognition. An animal's behavior is carried out in consecutive frames rather than in a single image; thus, image-based animal behavior recognition methods have low recognition accuracy. To address this dilemma, we not only construct the first skeleton-based dynamic multispecies dataset (Animal-Skeleton) but also propose a novel scheme that automatically designs the best spatial-temporal graph convolutional network (GCN) architecture with neural architecture search (NAS) to perform animal behavior recognition, named Animal-Nas for short. This is the first time that GCNs with NAS have been introduced into the animal behavior recognition task. To alleviate the trial-and-error cost of manually designing the network structure, we turn to NAS and design a novel search space with graph-based cells. Furthermore, we adopt a differentiable architecture search strategy to automatically search the cost-efficient spatial-temporal graph convolutional network structure. To evaluate the performance of the proposed model, we conduct extensive experiments on Animal-Skeleton datasets from three perspectives: model accuracy, parameter amount and stability. The results show that our model can achieve state-of the-art performance with fewer parameters.
2148.
Sarah, Anthony; Cummings, Daniel; Sridhar, Sharath Nittur; Sundaresan, Sairam; Szankin, Maciej; Webb, Tristan; Munoz, J. Pablo
A Hardware-Aware System for Accelerating Deep Neural Network Optimization Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2202-12954,
title = {A Hardware-Aware System for Accelerating Deep Neural Network Optimization},
author = {Anthony Sarah and Daniel Cummings and Sharath Nittur Sridhar and Sairam Sundaresan and Maciej Szankin and Tristan Webb and J. Pablo Munoz},
url = {https://arxiv.org/abs/2202.12954},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2202.12954},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2147.
Sinha, Nilotpal; Chen, Kuan-Wen
Neural Architecture Search using Progressive Evolution Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-01559,
title = {Neural Architecture Search using Progressive Evolution},
author = {Nilotpal Sinha and Kuan-Wen Chen},
url = {https://doi.org/10.48550/arXiv.2203.01559},
doi = {10.48550/arXiv.2203.01559},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.01559},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2146.
Ye, Peng; Li, Baopu; Li, Yikang; Chen, Tao; Fan, Jiayuan; Ouyang, Wanli
(beta)-DARTS: Beta-Decay Regularization for Differentiable Architecture Search Proceedings Article
In: CVPR2022, 2022.
@inproceedings{DBLP:journals/corr/abs-2203-01665,
title = {(beta)-DARTS: Beta-Decay Regularization for Differentiable Architecture Search},
author = {Peng Ye and Baopu Li and Yikang Li and Tao Chen and Jiayuan Fan and Wanli Ouyang},
url = {https://openaccess.thecvf.com/content/CVPR2022/papers/Ye_b-DARTS_Beta-Decay_Regularization_for_Differentiable_Architecture_Search_CVPR_2022_paper.pdf},
doi = {10.48550/arXiv.2203.01665},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {CVPR2022},
journal = {CoRR},
volume = {abs/2203.01665},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2145.
Szwarcman, Daniela; Civitarese, Daniel; Vellasco, Marley
Quantum-inspired evolutionary algorithm applied to neural architecture search Journal Article
In: Applied Soft Computing, vol. 120, pp. 108674, 2022, ISSN: 1568-4946.
@article{SZWARCMAN2022108674,
title = {Quantum-inspired evolutionary algorithm applied to neural architecture search},
author = {Daniela Szwarcman and Daniel Civitarese and Marley Vellasco},
url = {https://www.sciencedirect.com/science/article/pii/S1568494622001478},
doi = {https://doi.org/10.1016/j.asoc.2022.108674},
issn = {1568-4946},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Applied Soft Computing},
volume = {120},
pages = {108674},
abstract = {The success of machine learning models over the last few years is mostly related to the significant progress of deep neural networks. These powerful and flexible models can even surpass human-level performance in tasks such as image recognition and strategy games. However, experts need to spend considerable time and resources to design the network structure. The demand for new architectures drives interest in automating this design process. Researchers have proposed new algorithms to address the neural architecture search (NAS) problem, including efforts to reduce the high computational cost of such methods. A common approach to improve efficiency is to reduce the search space with the help of expert knowledge, searching for cells rather than entire networks. Motivated by the faster convergence promoted by quantum-inspired evolutionary methods, the Q-NAS algorithm was proposed to address the NAS problem without relying on cell search. In this work, we consolidate Q-NAS, adding a new penalization feature, enhancing its retraining scheme, and also investigating more challenging search spaces than before. In CIFAR-10, we reached 93.85% of test accuracy in 67 GPU days, considering the addition of an early-stopping mechanism. We also applied Q-NAS to CIFAR-100, without modifying the parameters, and our best accuracy was 74.23%, which is comparable to ResNet164. The enhancements and results presented in this work show that Q-NAS can automatically generate network architectures that outperform hand-designed models for CIFAR-10 and CIFAR-100. Also, compared to other NAS methods, Q-NAS results are promising regarding the balance between performance, runtime efficiency, and automation. We believe that our results enrich the discussion on this balance, considering alternatives to the cell search approach.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The success of machine learning models over the last few years is mostly related to the significant progress of deep neural networks. These powerful and flexible models can even surpass human-level performance in tasks such as image recognition and strategy games. However, experts need to spend considerable time and resources to design the network structure. The demand for new architectures drives interest in automating this design process. Researchers have proposed new algorithms to address the neural architecture search (NAS) problem, including efforts to reduce the high computational cost of such methods. A common approach to improve efficiency is to reduce the search space with the help of expert knowledge, searching for cells rather than entire networks. Motivated by the faster convergence promoted by quantum-inspired evolutionary methods, the Q-NAS algorithm was proposed to address the NAS problem without relying on cell search. In this work, we consolidate Q-NAS, adding a new penalization feature, enhancing its retraining scheme, and also investigating more challenging search spaces than before. In CIFAR-10, we reached 93.85% of test accuracy in 67 GPU days, considering the addition of an early-stopping mechanism. We also applied Q-NAS to CIFAR-100, without modifying the parameters, and our best accuracy was 74.23%, which is comparable to ResNet164. The enhancements and results presented in this work show that Q-NAS can automatically generate network architectures that outperform hand-designed models for CIFAR-10 and CIFAR-100. Also, compared to other NAS methods, Q-NAS results are promising regarding the balance between performance, runtime efficiency, and automation. We believe that our results enrich the discussion on this balance, considering alternatives to the cell search approach.
2144.
Huynh, Lam; Rahtu, Esa; Matas, Jiri; Heikkilä, Janne
Fast Neural Architecture Search for Lightweight Dense Prediction Networks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-01994,
title = {Fast Neural Architecture Search for Lightweight Dense Prediction Networks},
author = {Lam Huynh and Esa Rahtu and Jiri Matas and Janne Heikkilä},
url = {https://doi.org/10.48550/arXiv.2203.01994},
doi = {10.48550/arXiv.2203.01994},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.01994},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2143.
Lin, Ke; A, Yong; Gan, Zhuoxin; Jiang, Yingying
WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-02086,
title = {WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor},
author = {Ke Lin and Yong A and Zhuoxin Gan and Yingying Jiang},
url = {https://doi.org/10.48550/arXiv.2203.02086},
doi = {10.48550/arXiv.2203.02086},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.02086},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2142.
Chen, Xuehui; Niu, Xin; Jiang, Jingfei; Pan, Hengyue; Dong, Peijie; Wei, Zimian
Influence of Initialization and Modularization on the Performance of Network Morphism-Based Neural Architecture Search Proceedings Article
In: Yao, Jian; Xiao, Yang; You, Peng; Sun, Guang (Ed.): The International Conference on Image, Vision and Intelligent Systems (ICIVIS 2021), pp. 875–887, Springer Singapore, Singapore, 2022, ISBN: 978-981-16-6963-7.
@inproceedings{10.1007/978-981-16-6963-7_77,
title = {Influence of Initialization and Modularization on the Performance of Network Morphism-Based Neural Architecture Search},
author = {Xuehui Chen and Xin Niu and Jingfei Jiang and Hengyue Pan and Peijie Dong and Zimian Wei},
editor = {Jian Yao and Yang Xiao and Peng You and Guang Sun},
url = {https://link.springer.com/chapter/10.1007/978-981-16-6963-7_77},
isbn = {978-981-16-6963-7},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {The International Conference on Image, Vision and Intelligent Systems (ICIVIS 2021)},
pages = {875--887},
publisher = {Springer Singapore},
address = {Singapore},
abstract = {Neural Architecture Search (NAS), the process of automatic network architecture design, has enabled remarkable progress over the last years on Computer Vision tasks. In this paper, we propose a novel and efficient NAS framework based on network morphism to further improve the performance of NAS algorithms. Firstly, we design four modular structures termed RBNC block, CBNR block, BNRC block and RCBN block which correspond to four initial neural network architectures and four modular network morphism methods. Each block is composed of a ReLU layer, a Batch-Norm layer and a convolutional layer. Then we introduce network morphism to correlate different modular structures for constructing network architectures. Moreover, we study the influence of different initial neural network architectures and modular network morphism methods on the performance of network morphism-based NAS algorithms through comparative experiments and ablation experiments. Finally, we find that the network morphism-based NAS algorithm that uses CBNR block for initialization and modularization is the best method to improve performance. Our proposed method achieves a test accuracy of 95.84% on CIFAR-10 with least parameters (only 2.72 M) and fewer search costs (2 GPU-days) for network architecture search.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural Architecture Search (NAS), the process of automatic network architecture design, has enabled remarkable progress over the last years on Computer Vision tasks. In this paper, we propose a novel and efficient NAS framework based on network morphism to further improve the performance of NAS algorithms. Firstly, we design four modular structures termed RBNC block, CBNR block, BNRC block and RCBN block which correspond to four initial neural network architectures and four modular network morphism methods. Each block is composed of a ReLU layer, a Batch-Norm layer and a convolutional layer. Then we introduce network morphism to correlate different modular structures for constructing network architectures. Moreover, we study the influence of different initial neural network architectures and modular network morphism methods on the performance of network morphism-based NAS algorithms through comparative experiments and ablation experiments. Finally, we find that the network morphism-based NAS algorithm that uses CBNR block for initialization and modularization is the best method to improve performance. Our proposed method achieves a test accuracy of 95.84% on CIFAR-10 with least parameters (only 2.72 M) and fewer search costs (2 GPU-days) for network architecture search.