Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2021
1132.
Ji, Zexin; Dong, Xin; Li, Zhendong; Yu, Zekuan; Liu, Hao
Non-local Network Routing for Perceptual Image Super-Resolution Proceedings Article
In: Ma, Huimin; Wang, Liang; Zhang, Changshui; Wu, Fei; Tan, Tieniu; Wang, Yaonan; Lai, Jianhuang; Zhao, Yao (Ed.): Pattern Recognition and Computer Vision, pp. 164–176, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-88010-1.
@inproceedings{10.1007/978-3-030-88010-1_14,
title = {Non-local Network Routing for Perceptual Image Super-Resolution},
author = {Zexin Ji and Xin Dong and Zhendong Li and Zekuan Yu and Hao Liu},
editor = {Huimin Ma and Liang Wang and Changshui Zhang and Fei Wu and Tieniu Tan and Yaonan Wang and Jianhuang Lai and Yao Zhao},
url = {https://link.springer.com/chapter/10.1007/978-3-030-88010-1_14},
isbn = {978-3-030-88010-1},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Pattern Recognition and Computer Vision},
pages = {164--176},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In this paper, we propose a non-local network routing (NNR) approach for perceptual image super-resolution. Unlike conventional methods which generate visually-faked textures due to exiting hand-designed losses, our approach aims to globally optimize both procedures of learning an optimal perceptual loss and routing a spatial-adaptive network architecture in a unified reinforcement learning framework. To this end, we introduce a reward function to teach our objective to pay more attention on the visual quality of the super-resolved image. Moreover, we carefully design an offset operation inside the neural architecture search space, which typically deforms the receptive field on boundary refinement in a non-local manner. Experimentally, our proposed method surpasses the perceptual performance over state-of-the-art methods on several widely-evaluated benchmark datasets.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper, we propose a non-local network routing (NNR) approach for perceptual image super-resolution. Unlike conventional methods which generate visually-faked textures due to exiting hand-designed losses, our approach aims to globally optimize both procedures of learning an optimal perceptual loss and routing a spatial-adaptive network architecture in a unified reinforcement learning framework. To this end, we introduce a reward function to teach our objective to pay more attention on the visual quality of the super-resolved image. Moreover, we carefully design an offset operation inside the neural architecture search space, which typically deforms the receptive field on boundary refinement in a non-local manner. Experimentally, our proposed method surpasses the perceptual performance over state-of-the-art methods on several widely-evaluated benchmark datasets.
1131.
Feng, Qiantai; Xu, Ke; Li, Yuhai; Sun, Yuxin; Wang, Dong
Edge-Wise One-Level Global Pruning on NAS Generated Networks Proceedings Article
In: Ma, Huimin; Wang, Liang; Zhang, Changshui; Wu, Fei; Tan, Tieniu; Wang, Yaonan; Lai, Jianhuang; Zhao, Yao (Ed.): Pattern Recognition and Computer Vision, pp. 3–15, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-88013-2.
@inproceedings{10.1007/978-3-030-88013-2_1,
title = {Edge-Wise One-Level Global Pruning on NAS Generated Networks},
author = {Qiantai Feng and Ke Xu and Yuhai Li and Yuxin Sun and Dong Wang},
editor = {Huimin Ma and Liang Wang and Changshui Zhang and Fei Wu and Tieniu Tan and Yaonan Wang and Jianhuang Lai and Yao Zhao},
url = {https://link.springer.com/chapter/10.1007/978-3-030-88013-2_1},
isbn = {978-3-030-88013-2},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Pattern Recognition and Computer Vision},
pages = {3--15},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In recent years, there has been a lot of studies in neural architecture search (NAS) in the field of deep learning. Among them, the cell-based search method, such as [23, 27, 32, 36], is one of the most popular and widely discussed topics, which usually stacks less cells in search process and more in evaluation. Although this method can reduce the resource consumption in the process of search, the difference in the number of cells may inevitably cause a certain degree of redundancy in network evaluation. In order to mitigate the computational cost, we propose a novel algorithm called Edge-Wise One-Level Global Pruning (EOG-Pruning). The proposed approach can prune out weak edges from the cell-based network generated by NAS globally, by introducing an edge factor to represent the importance of each edge, which can not only greatly improve the inference speed of the model with reducing the number of edges, but also promote the model accuracy. Experimental results show that networks pruned by EOG-Pruning achieve significant improvement in accuracy and speedup rate on CPU in common with 50% pruning rate on CIFAR. Specifically, we reduced the test error rate by 1.58% and 1.34% on CIFAR-100 for DARTS (2nd-order) and PC-DARTS.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In recent years, there has been a lot of studies in neural architecture search (NAS) in the field of deep learning. Among them, the cell-based search method, such as [23, 27, 32, 36], is one of the most popular and widely discussed topics, which usually stacks less cells in search process and more in evaluation. Although this method can reduce the resource consumption in the process of search, the difference in the number of cells may inevitably cause a certain degree of redundancy in network evaluation. In order to mitigate the computational cost, we propose a novel algorithm called Edge-Wise One-Level Global Pruning (EOG-Pruning). The proposed approach can prune out weak edges from the cell-based network generated by NAS globally, by introducing an edge factor to represent the importance of each edge, which can not only greatly improve the inference speed of the model with reducing the number of edges, but also promote the model accuracy. Experimental results show that networks pruned by EOG-Pruning achieve significant improvement in accuracy and speedup rate on CPU in common with 50% pruning rate on CIFAR. Specifically, we reduced the test error rate by 1.58% and 1.34% on CIFAR-100 for DARTS (2nd-order) and PC-DARTS.
1130.
Xue, Xizhe; Zhang, Haokui; Bai, Zongwen; Li, Ying
3D-ANAS v2: Grafting Transformer Module on Automatically Designed ConvNet for Hyperspectral Image Classification Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-11084,
title = {3D-ANAS v2: Grafting Transformer Module on Automatically Designed ConvNet for Hyperspectral Image Classification},
author = {Xizhe Xue and Haokui Zhang and Zongwen Bai and Ying Li},
url = {https://arxiv.org/abs/2110.11084},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.11084},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1129.
Nayman, Niv; Aflalo, Yonathan; Noy, Asaf; Jin, Rong; Zelnik-Manor, Lihi
IQNAS: Interpretable Integer Quadratic Programming Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-12399,
title = {IQNAS: Interpretable Integer Quadratic Programming Neural Architecture Search},
author = {Niv Nayman and Yonathan Aflalo and Asaf Noy and Rong Jin and Lihi Zelnik-Manor},
url = {https://arxiv.org/abs/2110.12399},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.12399},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1128.
Simon, Christian; Koniusz, Piotr; Petersson, Lars; Han, Yan; Harandi, Mehrtash
Towards a Robust Differentiable Architecture Search under Label Noise Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-12197,
title = {Towards a Robust Differentiable Architecture Search under Label Noise},
author = {Christian Simon and Piotr Koniusz and Lars Petersson and Yan Han and Mehrtash Harandi},
url = {https://arxiv.org/abs/2110.12197},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.12197},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1127.
Geiping, Jonas; Lukasik, Jovita; Keuper, Margret; Moeller, Michael
DARTS for Inverse Problems: a Study on Stability Proceedings Article
In: NeurIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021.
@inproceedings{<LineBreak>geiping2021darts,
title = {DARTS for Inverse Problems: a Study on Stability},
author = {Jonas Geiping and Jovita Lukasik and Margret Keuper and Michael Moeller},
url = {https://openreview.net/forum?id=ty5XCitJfLA},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {NeurIPS 2021 Workshop on Deep Learning and Inverse Problems},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1126.
Gadikar, Pranav U; Ganesan, Vinod; Panda, Pratyush Kumar
Generalized Weight Agnostic Neural Networks for Configurable and Continual Autonomous Systems Proceedings Article
In: The First International Conference on AI-ML-Systems, Association for Computing Machinery, Bangalore, India, 2021, ISBN: 9781450385947.
@inproceedings{10.1145/3486001.3486226,
title = {Generalized Weight Agnostic Neural Networks for Configurable and Continual Autonomous Systems},
author = {Pranav U Gadikar and Vinod Ganesan and Pratyush Kumar Panda},
url = {https://doi.org/10.1145/3486001.3486226},
doi = {10.1145/3486001.3486226},
isbn = {9781450385947},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {The First International Conference on AI-ML-Systems},
publisher = {Association for Computing Machinery},
address = {Bangalore, India},
series = {AIMLSystems 2021},
abstract = {Implementing pervasive intelligence faces the challenges of efficient, continual, and configurable learning on embedded devices. We address these challenges with two novel extensions to Weight Agnostic Neural Networks (WANNs) namely – (i) Multi-weight extension and (ii) Multi-objective extension. In the multi-weight extension, we extend the idea of single shared weight WANNs to multiple weights to enable efficient continual learning. Our results across four different tasks implemented on Raspberry Pi demonstrate that the multi-weight WANNs achieve higher reward as compared to single shared weight WANNs and can be fine-tuned on device in orders of magnitude smaller time. In the multi-objective extension of WANNs, we extend the idea of a single reward function to multiple competing rewards and demonstrate sensitive and monotonic trade-off between multiple competing rewards to enable efficient configurable learning. We also demonstrate robustness of WANNs for changes in the weight parameters and changes in the environmental conditions as compared to the Proximal Policy Optimization(PPO) algorithm. These significant results open the possibility of truly autonomous agents using WANNs on low compute and power budget devices.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Implementing pervasive intelligence faces the challenges of efficient, continual, and configurable learning on embedded devices. We address these challenges with two novel extensions to Weight Agnostic Neural Networks (WANNs) namely – (i) Multi-weight extension and (ii) Multi-objective extension. In the multi-weight extension, we extend the idea of single shared weight WANNs to multiple weights to enable efficient continual learning. Our results across four different tasks implemented on Raspberry Pi demonstrate that the multi-weight WANNs achieve higher reward as compared to single shared weight WANNs and can be fine-tuned on device in orders of magnitude smaller time. In the multi-objective extension of WANNs, we extend the idea of a single reward function to multiple competing rewards and demonstrate sensitive and monotonic trade-off between multiple competing rewards to enable efficient configurable learning. We also demonstrate robustness of WANNs for changes in the weight parameters and changes in the environmental conditions as compared to the Proximal Policy Optimization(PPO) algorithm. These significant results open the possibility of truly autonomous agents using WANNs on low compute and power budget devices.
1125.
Knyazev, Boris; Drozdzal, Michal; Taylor, Graham W.; Romero-Soriano, Adriana
Parameter Prediction for Unseen Deep Architectures Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-13100,
title = {Parameter Prediction for Unseen Deep Architectures},
author = {Boris Knyazev and Michal Drozdzal and Graham W. Taylor and Adriana Romero-Soriano},
url = {https://arxiv.org/abs/2110.13100},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13100},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1124.
Yuan, Gonglin; Xue, Bing; Zhang, Mengjie
Ä Two-Stage Efficient Evolutionary Neural Architecture Search Method for Image Classification Proceedings Article
In: Pham, Duc Nghia; Theeramunkong, Thanaruk; Governatori, Guido; Liu, Fenrong (Ed.): PRICAI 2021: Trends in Artificial Intelligence, pp. 469–484, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-89188-6.
@inproceedings{10.1007/978-3-030-89188-6_35,
title = {Ä Two-Stage Efficient Evolutionary Neural Architecture Search Method for Image Classification},
author = {Gonglin Yuan and Bing Xue and Mengjie Zhang},
editor = {Duc Nghia Pham and Thanaruk Theeramunkong and Guido Governatori and Fenrong Liu},
url = {https://link.springer.com/chapter/10.1007/978-3-030-89188-6_35},
isbn = {978-3-030-89188-6},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {PRICAI 2021: Trends in Artificial Intelligence},
pages = {469--484},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Deep convolutional neural networks (DCNNs) have achieved promising performance in different computer vision tasks in recent years. Conventionally, deep learning experts are needed to design convolutional neural network's (CNN's) architectures when facing new tasks. Neural architecture search (NAS) is to automatically find suitable architectures; however, NAS suffers from the tremendous computational cost. This paper employs a genetic algorithm (GA) and a grid search (GS) strategy to search for the micro-architecture and adjust the macro-architecture efficiently and effectively, named TSCNN. We propose two mutation operations to explore the search space comprehensively. Furthermore, the micro-architecture searched on one dataset is transferred to another dataset to verify its transferability. The proposed algorithm is evaluated on two widely used datasets. The experimental results show that TSCNN achieves very competitive accuracy. On the CIFAR10 dataset, the computational cost is reduced from hundreds or even thousands to only 2.5 GPU-days, and the number of parameters is reduced from thirty more million to only 1.25 M.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Deep convolutional neural networks (DCNNs) have achieved promising performance in different computer vision tasks in recent years. Conventionally, deep learning experts are needed to design convolutional neural network's (CNN's) architectures when facing new tasks. Neural architecture search (NAS) is to automatically find suitable architectures; however, NAS suffers from the tremendous computational cost. This paper employs a genetic algorithm (GA) and a grid search (GS) strategy to search for the micro-architecture and adjust the macro-architecture efficiently and effectively, named TSCNN. We propose two mutation operations to explore the search space comprehensively. Furthermore, the micro-architecture searched on one dataset is transferred to another dataset to verify its transferability. The proposed algorithm is evaluated on two widely used datasets. The experimental results show that TSCNN achieves very competitive accuracy. On the CIFAR10 dataset, the computational cost is reduced from hundreds or even thousands to only 2.5 GPU-days, and the number of parameters is reduced from thirty more million to only 1.25 M.
1123.
Cai, Rongshen; Tao, Qian; Tang, Yufei; Shi, Min
ALGNN: Auto-Designed Lightweight Graph Neural Network Proceedings Article
In: Pham, Duc Nghia; Theeramunkong, Thanaruk; Governatori, Guido; Liu, Fenrong (Ed.): PRICAI 2021: Trends in Artificial Intelligence, pp. 500–512, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-89188-6.
@inproceedings{10.1007/978-3-030-89188-6_37,
title = {ALGNN: Auto-Designed Lightweight Graph Neural Network},
author = {Rongshen Cai and Qian Tao and Yufei Tang and Min Shi},
editor = {Duc Nghia Pham and Thanaruk Theeramunkong and Guido Governatori and Fenrong Liu},
isbn = {978-3-030-89188-6},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {PRICAI 2021: Trends in Artificial Intelligence},
pages = {500--512},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Graph neural networks (GNNs) are widely used on graph-structured data, and its research has made substantial progress in recent years. However, given the various number of choices and combinations of components such as aggregator and activation function, designing GNNs for specific tasks is very heavy manual work. Recently, neural architecture search (NAS) was proposed with the aim of automating the GNN design process and generating task-dependent architectures. While existing approaches have achieved competitive performance, they are not well suited to practical application scenarios where the computational budget is limited. In this paper, we propose an auto-designed lightweight graph neural network (ALGNN) method to automatically design lightweight, task-dependent GNN architectures. ALGNN uses multi-objective optimization to optimize the architecture constrained by the computation cost and complexity of the model. We define, for the first time, an evaluation standard for consumption cost with the analysis of the message passing process in GNNs. Experiments on real-world datasets demonstrate that ALGNN can generate a lightweight GNN model that has much fewer parameters and GPU hours, meanwhile has comparable performance with state-of-the-art approaches.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Graph neural networks (GNNs) are widely used on graph-structured data, and its research has made substantial progress in recent years. However, given the various number of choices and combinations of components such as aggregator and activation function, designing GNNs for specific tasks is very heavy manual work. Recently, neural architecture search (NAS) was proposed with the aim of automating the GNN design process and generating task-dependent architectures. While existing approaches have achieved competitive performance, they are not well suited to practical application scenarios where the computational budget is limited. In this paper, we propose an auto-designed lightweight graph neural network (ALGNN) method to automatically design lightweight, task-dependent GNN architectures. ALGNN uses multi-objective optimization to optimize the architecture constrained by the computation cost and complexity of the model. We define, for the first time, an evaluation standard for consumption cost with the analysis of the message passing process in GNNs. Experiments on real-world datasets demonstrate that ALGNN can generate a lightweight GNN model that has much fewer parameters and GPU hours, meanwhile has comparable performance with state-of-the-art approaches.
1122.
Egele, Romain; Maulik, Romit; Raghavan, Krishnan; Balaprakash, Prasanna; Lusch, Bethany
AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-13511,
title = {AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification},
author = {Romain Egele and Romit Maulik and Krishnan Raghavan and Prasanna Balaprakash and Bethany Lusch},
url = {https://arxiv.org/abs/2110.13511},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13511},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1121.
Song, Siyang; Shao, Zilong; Jaiswal, Shashank; Shen, Linlin; Valstar, Michel F.; Gunes, Hatice
Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-13570,
title = {Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition},
author = {Siyang Song and Zilong Shao and Shashank Jaiswal and Linlin Shen and Michel F. Valstar and Hatice Gunes},
url = {https://arxiv.org/abs/2110.13570},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13570},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1120.
Lopes, Vasco; Santos, Miguel; Degardin, Bruno; Alexandre, Luís A.
Guided Evolution for Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-15232,
title = {Guided Evolution for Neural Architecture Search},
author = {Vasco Lopes and Miguel Santos and Bruno Degardin and Luís A. Alexandre},
url = {https://arxiv.org/abs/2110.15232},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.15232},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1119.
Xue, Yu; Yuan, Ziming; Slowik, Adam
A Novel Sleep Stage Classification Using CNN Generated by an Efficient Neural Architecture Search with a New Data Processing Trick Technical Report
2021.
@techreport{xue2021novel,
title = {A Novel Sleep Stage Classification Using CNN Generated by an Efficient Neural Architecture Search with a New Data Processing Trick},
author = {Yu Xue and Ziming Yuan and Adam Slowik},
url = {https://arxiv.org/abs/2110.15277},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1118.
Kong, Qi; Xu, Xin; Zhang, Liangliang
ODNASSD: An End-to-end Object Detection Neural Architecture Search Space Design Proceedings Article
In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3075-3080, 2021.
@inproceedings{9565112,
title = {ODNASSD: An End-to-end Object Detection Neural Architecture Search Space Design},
author = {Qi Kong and Xin Xu and Liangliang Zhang},
url = {https://ieeexplore.ieee.org/abstract/document/9565112},
doi = {10.1109/ITSC48978.2021.9565112},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 IEEE International Intelligent Transportation Systems Conference (ITSC)},
pages = {3075-3080},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1117.
Krishna, Ravi; Kalaiah, Aravind; Wu, Bichen; Naumov, Maxim; Mudigere, Dheevatsa; Smelyanskiy, Misha; Keutzer, Kurt
Differentiable NAS Framework and Application to Ads CTR Prediction Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-14812,
title = {Differentiable NAS Framework and Application to Ads CTR Prediction},
author = {Ravi Krishna and Aravind Kalaiah and Bichen Wu and Maxim Naumov and Dheevatsa Mudigere and Misha Smelyanskiy and Kurt Keutzer},
url = {https://arxiv.org/abs/2110.14812},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.14812},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1116.
Cozma, Adriana-Eliza; Morgan, Lisa; Stolz, Martin; Stoeckel, David; Rambach, Kilian
DeepHybrid: Deep Learning on Automotive Radar Spectra and Reflections for Object Classification Proceedings Article
In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 2682-2687, 2021.
@inproceedings{9564526,
title = {DeepHybrid: Deep Learning on Automotive Radar Spectra and Reflections for Object Classification},
author = {Adriana-Eliza Cozma and Lisa Morgan and Martin Stolz and David Stoeckel and Kilian Rambach},
url = {https://ieeexplore.ieee.org/abstract/document/9564526},
doi = {10.1109/ITSC48978.2021.9564526},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 IEEE International Intelligent Transportation Systems Conference (ITSC)},
pages = {2682-2687},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1115.
Xie, Bangquan; Yang, Zongming; Yang, Liang; Luo, Ruifa; Wei, Ailin; Weng, Xiaoxiong; Li, Bing
Multi-Scale Fusion With Matching Attention Model: A Novel Decoding Network Cooperated With NAS for Real-Time Semantic Segmentation Journal Article
In: IEEE Transactions on Intelligent Transportation Systems, pp. 1-11, 2021.
@article{9585519,
title = {Multi-Scale Fusion With Matching Attention Model: A Novel Decoding Network Cooperated With NAS for Real-Time Semantic Segmentation},
author = {Bangquan Xie and Zongming Yang and Liang Yang and Ruifa Luo and Ailin Wei and Xiaoxiong Weng and Bing Li},
url = {https://ieeexplore.ieee.org/abstract/document/9585519},
doi = {10.1109/TITS.2021.3115705},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {IEEE Transactions on Intelligent Transportation Systems},
pages = {1-11},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1114.
Traoré, Kalifou René; Camero, Andrés; Zhu, Xiao Xiang
Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-01584,
title = {Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems},
author = {Kalifou René Traoré and Andrés Camero and Xiao Xiang Zhu},
url = {https://arxiv.org/abs/2111.01584},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.01584},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1113.
Lu, Bingqian; Yang, Jianyi; Jiang, Weiwen; Shi, Yiyu; Ren, Shaolei
One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search Technical Report
2021.
@techreport{lu2021one,
title = {One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search},
author = {Bingqian Lu and Jianyi Yang and Weiwen Jiang and Yiyu Shi and Shaolei Ren},
url = {https://arxiv.org/abs/2111.01203},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {arXiv preprint arXiv:2111.01203},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1112.
Deng, Xiaoqing; Luo, Weiqi; Fang, Yanmei
Spatial Steganalysis Based on Gradient-Based Neural Architecture Search Proceedings Article
In: Huang, Qiong; Yu, Yu (Ed.): Provable and Practical Security, pp. 365–375, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-90402-9.
@inproceedings{10.1007/978-3-030-90402-9_20,
title = {Spatial Steganalysis Based on Gradient-Based Neural Architecture Search},
author = {Xiaoqing Deng and Weiqi Luo and Yanmei Fang},
editor = {Qiong Huang and Yu Yu},
url = {https://link.springer.com/chapter/10.1007/978-3-030-90402-9_20},
isbn = {978-3-030-90402-9},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Provable and Practical Security},
pages = {365--375},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Most existing steganalytic networks are designed empirically, which probably limits their performances. Neural architecture search (NAS) is a technology that can automatically find the optimal network architecture in the search space without excessive manual intervention. In this paper, we introduce a gradient-based NAS method called PC-DARTS in steganalysis. We firstly define the overall network architecture, and the search spaces of the corresponding cells in the network. We then use softmax over all candidate operations to construct an over-parameterized network. By updating the parameters of such a network based on gradient descent, the optimal operations, i.e., the high-pass filters in pre-processing module and operations in feature extraction module, can be obtained. Experimental results show that the resulting steganalytic network via NAS can achieve competitive performance with some advanced well-designed steganalytic networks, while the searching time is relatively short.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Most existing steganalytic networks are designed empirically, which probably limits their performances. Neural architecture search (NAS) is a technology that can automatically find the optimal network architecture in the search space without excessive manual intervention. In this paper, we introduce a gradient-based NAS method called PC-DARTS in steganalysis. We firstly define the overall network architecture, and the search spaces of the corresponding cells in the network. We then use softmax over all candidate operations to construct an over-parameterized network. By updating the parameters of such a network based on gradient descent, the optimal operations, i.e., the high-pass filters in pre-processing module and operations in feature extraction module, can be obtained. Experimental results show that the resulting steganalytic network via NAS can achieve competitive performance with some advanced well-designed steganalytic networks, while the searching time is relatively short.
1111.
Li, Yanxi; Dong, Minjing; Xu, Yixing; Wang, Yunhe; Xu, Chang
Neural architecture tuning with policy adaptation Journal Article
In: Neurocomputing, 2021, ISSN: 0925-2312.
@article{LI2021e,
title = {Neural architecture tuning with policy adaptation},
author = {Yanxi Li and Minjing Dong and Yixing Xu and Yunhe Wang and Chang Xu},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221016325},
doi = {https://doi.org/10.1016/j.neucom.2021.10.095},
issn = {0925-2312},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Neurocomputing},
abstract = {Neural architecture search (NAS) is to automatically design task-specific neural architectures, whose performance has already surpassed those of many manually designed neural networks. Existing NAS techniques focus on searching for the neural architecture and training the optimal network weights from the scratch. Nevertheless, it could be essential to study how to tune a given neural architecture instead of producing a completely new neural architecture in some scenarios, which may lead to a more optimal solution by combining human experience and the advantages of the machine’s automatic searching. This paper proposes to learn to tune the architectures at hand to achieve better performance. The proposed Neural Architecture Tuning (NAT) algorithm trains a deep Q-network to tune neural architectures given a random architecture so that we can achieve better performance on a reduced space. We then apply adversarial autoencoder to make the learned policy be generalized to a different searching space in real-world applications. The proposed algorithm is evaluated on the NAS-Bench-101 dataset. The results indicate that our NAT framework can achieve state-of-the-art performance on the NAS-Bench-101 benchmark, and the learned policy can be adapted to a different search space while maintaining the performance.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) is to automatically design task-specific neural architectures, whose performance has already surpassed those of many manually designed neural networks. Existing NAS techniques focus on searching for the neural architecture and training the optimal network weights from the scratch. Nevertheless, it could be essential to study how to tune a given neural architecture instead of producing a completely new neural architecture in some scenarios, which may lead to a more optimal solution by combining human experience and the advantages of the machine’s automatic searching. This paper proposes to learn to tune the architectures at hand to achieve better performance. The proposed Neural Architecture Tuning (NAT) algorithm trains a deep Q-network to tune neural architectures given a random architecture so that we can achieve better performance on a reduced space. We then apply adversarial autoencoder to make the learned policy be generalized to a different searching space in real-world applications. The proposed algorithm is evaluated on the NAS-Bench-101 dataset. The results indicate that our NAT framework can achieve state-of-the-art performance on the NAS-Bench-101 benchmark, and the learned policy can be adapted to a different search space while maintaining the performance.
1110.
Hou, Wenxuan; Liu, Longjun; Zhang, Haonan; Sun, Hongbin; Zheng, Nanning
DFSNet: Dividing-Fuse Deep Neural Networks with Searching Strategy for Distributed DNN Architecture Journal Article
In: Neurocomputing, 2021, ISSN: 0925-2312.
@article{HOU2021,
title = {DFSNet: Dividing-Fuse Deep Neural Networks with Searching Strategy for Distributed DNN Architecture},
author = {Wenxuan Hou and Longjun Liu and Haonan Zhang and Hongbin Sun and Nanning Zheng},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221016076},
doi = {https://doi.org/10.1016/j.neucom.2021.08.144},
issn = {0925-2312},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Neurocomputing},
abstract = {The overwhelming parameters and computation consumption of deep neural networks limit their applicability to a single computing node with poor computing power, such as edge and mobile devices. Most previous works leverage model pruning and compression strategies to reduce DNN parameters for resource-constrained devices. However, most model compression methods may suffer from accuracy loss. Recently, we find that combine many weak computing nodes as a distributed system to run large and sophisticated DNN models is a promising solution for the issue. However, it is essential for the distributed system to design distributed DNN models and inference schemes, one of the great challenges of distributed system is how to design an efficient distributed DNN model for data parallelism and model parallelism, and communication overhead is also another critical performance bottleneck for distributed DNN model. Therefore, in this article, we propose DFSNet framework (Dividing-Fuse neural Network with Searching Strategy) for distributed DNN architecture. Firstly, the DFSNet framework includes a joint ”dividing-fusing” method to convert regular DNN models into distributed models that are friendly for distributed systems. This method divides the conventional DNN model in the channel dimension, and sets a few special layers to fuse feature-map information from different channel groups for accuracy improvement. Since the fusion layers are sparse in the network, they do not increase too much extra inference time and communication overhead on the distributed nodes, but they can maintain the accuracy of distributed neural networks significantly. Secondly, considering the architecture of distributed computing nodes, we propose a parallel fusion topology to improve the utilization of different computing nodes. Lastly, the popular weight-sharing neural architecture search (NAS) technique is leveraged to search the position of fusion layers in the distributed DNN model for high accuracy and finally generate an efficient distributed DNN model. Compared with the original network, our converted distributed DNN achieves better performance (e.g. 1.88% precision boosting in ResNet56 on CIFAR-100 dataset, and 1.25% precision improving in MobileNetV2 on ImageNet dataset). In addition, most layers of DNN have been divided into different distributed nodes on channel dimension, which is particularly suitable for distributed DNN architecture with very low communication overhead.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The overwhelming parameters and computation consumption of deep neural networks limit their applicability to a single computing node with poor computing power, such as edge and mobile devices. Most previous works leverage model pruning and compression strategies to reduce DNN parameters for resource-constrained devices. However, most model compression methods may suffer from accuracy loss. Recently, we find that combine many weak computing nodes as a distributed system to run large and sophisticated DNN models is a promising solution for the issue. However, it is essential for the distributed system to design distributed DNN models and inference schemes, one of the great challenges of distributed system is how to design an efficient distributed DNN model for data parallelism and model parallelism, and communication overhead is also another critical performance bottleneck for distributed DNN model. Therefore, in this article, we propose DFSNet framework (Dividing-Fuse neural Network with Searching Strategy) for distributed DNN architecture. Firstly, the DFSNet framework includes a joint ”dividing-fusing” method to convert regular DNN models into distributed models that are friendly for distributed systems. This method divides the conventional DNN model in the channel dimension, and sets a few special layers to fuse feature-map information from different channel groups for accuracy improvement. Since the fusion layers are sparse in the network, they do not increase too much extra inference time and communication overhead on the distributed nodes, but they can maintain the accuracy of distributed neural networks significantly. Secondly, considering the architecture of distributed computing nodes, we propose a parallel fusion topology to improve the utilization of different computing nodes. Lastly, the popular weight-sharing neural architecture search (NAS) technique is leveraged to search the position of fusion layers in the distributed DNN model for high accuracy and finally generate an efficient distributed DNN model. Compared with the original network, our converted distributed DNN achieves better performance (e.g. 1.88% precision boosting in ResNet56 on CIFAR-100 dataset, and 1.25% precision improving in MobileNetV2 on ImageNet dataset). In addition, most layers of DNN have been divided into different distributed nodes on channel dimension, which is particularly suitable for distributed DNN architecture with very low communication overhead.
1109.
Wei, Zhikun; Wang, Xin; Zhu, Wenwu
AutoIAS: Automatic Integrated Architecture Searcher for Click-Trough Rate Prediction Proceedings Article
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2101–2110, Association for Computing Machinery, Virtual Event, Queensland, Australia, 2021, ISBN: 9781450384469.
@inproceedings{10.1145/3459637.3482234,
title = {AutoIAS: Automatic Integrated Architecture Searcher for Click-Trough Rate Prediction},
author = {Zhikun Wei and Xin Wang and Wenwu Zhu},
url = {https://doi.org/10.1145/3459637.3482234},
doi = {10.1145/3459637.3482234},
isbn = {9781450384469},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
pages = {2101–2110},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Queensland, Australia},
series = {CIKM '21},
abstract = {Automating architecture design for recommendation tasks becomes a trending topic because expert efforts are saved, and better performance is expected. Neural Architecture Search (NAS) is introduced to discover powerful CTR prediction model architectures in recent works. CTR prediction model usually consists of three components: embedding layer, interaction layer, and deep neural network. However, existing automation works focus on searching single component and leaving other components hand-crafted. The isolated searching will cause incompatibility among components and lead to weak generalization ability. Moreover, there is not a unified framework for integrated CTR prediction model architecture searching. This paper presents Automatic Integrated Architecture Searcher (AutoIAS), a framework that provides a practical and general method to find optimal CTR prediction model architecture in an automatic manner. In AutoIAS, we unify existing interaction-based CTR prediction model architectures and propose an integrated search space for a complete CTR prediction model. We utilize a supernet to predict the performance of sub-architectures, and the supernet is trained with Knowledge Distillation(KD) to enhance consistency among sub-architectures. To efficiently explore the search space, we design an architecture generator network that explicitly models the architecture dependencies among components and generates conditioned architectures distribution for each component. Experiments on public datasets show the outstanding performance and generalization ability of AutoIAS. Ablation study shows the effectiveness of the KD-based supernet training method and the Architecture Generator Network.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Automating architecture design for recommendation tasks becomes a trending topic because expert efforts are saved, and better performance is expected. Neural Architecture Search (NAS) is introduced to discover powerful CTR prediction model architectures in recent works. CTR prediction model usually consists of three components: embedding layer, interaction layer, and deep neural network. However, existing automation works focus on searching single component and leaving other components hand-crafted. The isolated searching will cause incompatibility among components and lead to weak generalization ability. Moreover, there is not a unified framework for integrated CTR prediction model architecture searching. This paper presents Automatic Integrated Architecture Searcher (AutoIAS), a framework that provides a practical and general method to find optimal CTR prediction model architecture in an automatic manner. In AutoIAS, we unify existing interaction-based CTR prediction model architectures and propose an integrated search space for a complete CTR prediction model. We utilize a supernet to predict the performance of sub-architectures, and the supernet is trained with Knowledge Distillation(KD) to enhance consistency among sub-architectures. To efficiently explore the search space, we design an architecture generator network that explicitly models the architecture dependencies among components and generates conditioned architectures distribution for each component. Experiments on public datasets show the outstanding performance and generalization ability of AutoIAS. Ablation study shows the effectiveness of the KD-based supernet training method and the Architecture Generator Network.
1108.
Li, Yaliang; Wang, Zhen; Xie, Yuexiang; Ding, Bolin; Zeng, Kai; Zhang, Ce
AutoML: From Methodology to Application Proceedings Article
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 4853–4856, Association for Computing Machinery, Virtual Event, Queensland, Australia, 2021, ISBN: 9781450384469.
@inproceedings{10.1145/3459637.3483279,
title = {AutoML: From Methodology to Application},
author = {Yaliang Li and Zhen Wang and Yuexiang Xie and Bolin Ding and Kai Zeng and Ce Zhang},
url = {https://doi.org/10.1145/3459637.3483279},
doi = {10.1145/3459637.3483279},
isbn = {9781450384469},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
pages = {4853–4856},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Queensland, Australia},
series = {CIKM '21},
abstract = {Machine Learning methods have been adopted for a wide range of real-world applications, ranging from social networks, online image/video-sharing platforms, and e-commerce to education, healthcare, etc. However, in practice, a large amount of effort is required to tune several components of machine learning methods, including data representation, hyperparameter, and model architecture, in order to achieve a good performance. To alleviate the required tunning efforts, Automated Machine Learning (AutoML), which can automate the process of applying machine learning methods, has been studied in both academy and industry recently. In this tutorial, we will introduce the main research topics of AutoML, including Hyperparameter Optimization, Neural Architecture Search, and Meta-Learning. Two emerging topics of AutoML, Automatic Feature Generation and Machine Learning Guided Database, will also be discussed since they are important components for real-world applications. For each topic, we will motivate it with application examples from industry, illustrate the state-of-the-art methodologies, and discuss some future research directions based on our experience from industry and the trends in academy.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Machine Learning methods have been adopted for a wide range of real-world applications, ranging from social networks, online image/video-sharing platforms, and e-commerce to education, healthcare, etc. However, in practice, a large amount of effort is required to tune several components of machine learning methods, including data representation, hyperparameter, and model architecture, in order to achieve a good performance. To alleviate the required tunning efforts, Automated Machine Learning (AutoML), which can automate the process of applying machine learning methods, has been studied in both academy and industry recently. In this tutorial, we will introduce the main research topics of AutoML, including Hyperparameter Optimization, Neural Architecture Search, and Meta-Learning. Two emerging topics of AutoML, Automatic Feature Generation and Machine Learning Guided Database, will also be discussed since they are important components for real-world applications. For each topic, we will motivate it with application examples from industry, illustrate the state-of-the-art methodologies, and discuss some future research directions based on our experience from industry and the trends in academy.
1107.
Wei, Penghui; Zhang, Weimin; Xu, Zixuan; Liu, Shaoguo; Lee, Kuang-chih; Zheng, Bo
AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation Proceedings Article
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3528–3532, Association for Computing Machinery, Virtual Event, Queensland, Australia, 2021, ISBN: 9781450384469.
@inproceedings{10.1145/3459637.3482061,
title = {AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation},
author = {Penghui Wei and Weimin Zhang and Zixuan Xu and Shaoguo Liu and Kuang-chih Lee and Bo Zheng},
url = {https://doi.org/10.1145/3459637.3482061},
doi = {10.1145/3459637.3482061},
isbn = {9781450384469},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
pages = {3528–3532},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Queensland, Australia},
series = {CIKM '21},
abstract = {Post-click conversion rate (CVR) estimation is a crucial task in online advertising and recommendation systems. To address the sample selection bias problem in traditional CVR models trained in click space, recent studies perform entire space multi-task learning based on the probability of events in user behavior funnels like "impression-click-conversion". However, those models learn the feature representation of each task independently, and omit potential inter-task correlations that can help improve the CVR estimation performance. In this paper, we propose AutoHERI, an entire space CVR model with automated hierarchical representation integration, which leverages the interplay across multi-tasks' representation learning. It performs neural architecture search to learn optimal connections between layer-wise representations of different tasks. Besides, AutoHERI achieves better search efficiency with one-shot search algorithm, and thus it can be easily extended to new scenarios that have more complex user behaviors. Both offline and online experimental results on large-scale real-world datasets verify that AutoHERI outperforms previous entire space models significantly.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Post-click conversion rate (CVR) estimation is a crucial task in online advertising and recommendation systems. To address the sample selection bias problem in traditional CVR models trained in click space, recent studies perform entire space multi-task learning based on the probability of events in user behavior funnels like "impression-click-conversion". However, those models learn the feature representation of each task independently, and omit potential inter-task correlations that can help improve the CVR estimation performance. In this paper, we propose AutoHERI, an entire space CVR model with automated hierarchical representation integration, which leverages the interplay across multi-tasks' representation learning. It performs neural architecture search to learn optimal connections between layer-wise representations of different tasks. Besides, AutoHERI achieves better search efficiency with one-shot search algorithm, and thus it can be easily extended to new scenarios that have more complex user behaviors. Both offline and online experimental results on large-scale real-world datasets verify that AutoHERI outperforms previous entire space models significantly.
1106.
Lin, Jun-Liang; Wang, Sheng-De
Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices Technical Report
2021.
@techreport{lin2021communicationefficient,
title = {Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices},
author = {Jun-Liang Lin and Sheng-De Wang},
url = {https://arxiv.org/abs/2111.02489},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1105.
Cao, Jie; Ma, Jialin; Huang, Dailin; Yu, Ping
Finding the optimal multilayer network structure through reinforcement learning in fault diagnosis Journal Article
In: Measurement, pp. 110377, 2021, ISSN: 0263-2241.
@article{CAO2021110377,
title = {Finding the optimal multilayer network structure through reinforcement learning in fault diagnosis},
author = {Jie Cao and Jialin Ma and Dailin Huang and Ping Yu},
url = {https://www.sciencedirect.com/science/article/pii/S0263224121012707},
doi = {https://doi.org/10.1016/j.measurement.2021.110377},
issn = {0263-2241},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Measurement},
pages = {110377},
abstract = {Deep learning (DL) is an important method in industrial fault diagnosis. However, DL’s network structure needs to be designed with experience. To simplify the design of network structures, we propose the neural architecture search network with Pareto efficiency reward and insert replay buffer (NAS-PERIRB) algorithm. In this paper, the early stopping and insert replay buffer (IRB) are used to improving the training efficiency of the samples. In addition, we design the Pareto efficiency reward function to optimize the goals and design a network search space to perform effective searches. What is more, we evaluate the NAS-PERIRB under two datasets. Results show that the two datasets have reached 99% accuracy in various situations, which means the NAS-PERIRB can achieve the purpose of designing the network structure independently.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Deep learning (DL) is an important method in industrial fault diagnosis. However, DL’s network structure needs to be designed with experience. To simplify the design of network structures, we propose the neural architecture search network with Pareto efficiency reward and insert replay buffer (NAS-PERIRB) algorithm. In this paper, the early stopping and insert replay buffer (IRB) are used to improving the training efficiency of the samples. In addition, we design the Pareto efficiency reward function to optimize the goals and design a network search space to perform effective searches. What is more, we evaluate the NAS-PERIRB under two datasets. Results show that the two datasets have reached 99% accuracy in various situations, which means the NAS-PERIRB can achieve the purpose of designing the network structure independently.
1104.
Chen, Zhe; Wang, Wenhai; Xie, Enze; Yang, Zhibo; Lu, Tong; Luo, Ping
FAST: Searching for a Faster Arbitrarily-Shaped Text Detector with Minimalist Kernel Representation Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-02394,
title = {FAST: Searching for a Faster Arbitrarily-Shaped Text Detector with Minimalist Kernel Representation},
author = {Zhe Chen and Wenhai Wang and Enze Xie and Zhibo Yang and Tong Lu and Ping Luo},
url = {https://arxiv.org/abs/2111.02394},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.02394},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1103.
Traoré, Kalifou René; Camero, Andrés; Zhu, Xiao Xiang
A Data-driven Approach to Neural Architecture Search Initialization Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03524,
title = {A Data-driven Approach to Neural Architecture Search Initialization},
author = {Kalifou René Traoré and Andrés Camero and Xiao Xiang Zhu},
url = {https://arxiv.org/abs/2111.03524},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03524},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1102.
Eyono, Roy Henha; Carlucci, Fabio Maria; Esperança, Pedro M.; Ru, Binxin; Torr, Phillip
AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03555,
title = {AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family},
author = {Roy Henha Eyono and Fabio Maria Carlucci and Pedro M. Esperança and Binxin Ru and Phillip Torr},
url = {https://arxiv.org/abs/2111.03555},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03555},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1101.
Xu, Zhimeng; Mai, Yuting; Liu, Denghui; He, Wenjun; Lin, Xinyuan; Xu, Chi; Zhang, Lei; Meng, Xin; Mafofo, Joseph; Zaher, Walid Abbas; Koshy, Ashish; Li, Yi; Qiao, Nan
Fast-bonito: A faster deep learning based basecaller for nanopore sequencing Journal Article
In: Artificial Intelligence in the Life Sciences, vol. 1, pp. 100011, 2021, ISSN: 2667-3185.
@article{XU2021100011,
title = {Fast-bonito: A faster deep learning based basecaller for nanopore sequencing},
author = {Zhimeng Xu and Yuting Mai and Denghui Liu and Wenjun He and Xinyuan Lin and Chi Xu and Lei Zhang and Xin Meng and Joseph Mafofo and Walid Abbas Zaher and Ashish Koshy and Yi Li and Nan Qiao},
url = {https://www.sciencedirect.com/science/article/pii/S2667318521000118},
doi = {https://doi.org/10.1016/j.ailsci.2021.100011},
issn = {2667-3185},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Artificial Intelligence in the Life Sciences},
volume = {1},
pages = {100011},
abstract = {Nanopore sequencing from Oxford Nanopore Technologies (ONT) is a promising third-generation sequencing (TGS) technology that generates relatively longer sequencing reads compared to the next-generation sequencing (NGS) technology. A basecaller is a piece of software that translates the original electrical current signals into nucleotide sequences. The accuracy of the basecaller is crucially important to downstream analysis. Bonito is a deep learning-based basecaller recently developed by ONT. Its neural network architecture is composed of a single convolutional layer followed by three stacked bidirectional gated recurrent unit (GRU) layers. Although Bonito has achieved state-of-the-art base calling accuracy, its speed is too slow to be used in production. We therefore developed Fast-Bonito, by using the neural architecture search (NAS) technique to search for a brand-new neural network backbone, and trained it from scratch using several advanced deep learning model training techniques. The new Fast-Bonito model balanced performance in terms of speed and accuracy. Fast-Bonito was 153.8% faster than the original Bonito on NVIDIA V100 GPU. When running on HUAWEI Ascend 910 NPU, Fast-Bonito was 565% faster than the original Bonito. The accuracy of Fast-Bonito was also slightly higher than that of Bonito. We have made Fast-Bonito open source, hoping it will boost the adoption of TGS in both academia and industry.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Nanopore sequencing from Oxford Nanopore Technologies (ONT) is a promising third-generation sequencing (TGS) technology that generates relatively longer sequencing reads compared to the next-generation sequencing (NGS) technology. A basecaller is a piece of software that translates the original electrical current signals into nucleotide sequences. The accuracy of the basecaller is crucially important to downstream analysis. Bonito is a deep learning-based basecaller recently developed by ONT. Its neural network architecture is composed of a single convolutional layer followed by three stacked bidirectional gated recurrent unit (GRU) layers. Although Bonito has achieved state-of-the-art base calling accuracy, its speed is too slow to be used in production. We therefore developed Fast-Bonito, by using the neural architecture search (NAS) technique to search for a brand-new neural network backbone, and trained it from scratch using several advanced deep learning model training techniques. The new Fast-Bonito model balanced performance in terms of speed and accuracy. Fast-Bonito was 153.8% faster than the original Bonito on NVIDIA V100 GPU. When running on HUAWEI Ascend 910 NPU, Fast-Bonito was 565% faster than the original Bonito. The accuracy of Fast-Bonito was also slightly higher than that of Bonito. We have made Fast-Bonito open source, hoping it will boost the adoption of TGS in both academia and industry.
1100.
Yan, Shen; White, Colin; Savani, Yash; Hutter, Frank
NAS-Bench-x11 and the Power of Learning Curves Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03602,
title = {NAS-Bench-x11 and the Power of Learning Curves},
author = {Shen Yan and Colin White and Yash Savani and Frank Hutter},
url = {https://arxiv.org/abs/2111.03602},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03602},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1099.
Ding, Xinyi; Han, Tao; Fang, Yili; Larson, Eric C.
An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-04497,
title = {An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing},
author = {Xinyi Ding and Tao Han and Yili Fang and Eric C. Larson},
url = {https://arxiv.org/abs/2111.04497},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.04497},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1098.
Wan, Xingchen; Ru, Binxin; Esperança, Pedro M.; Carlucci, Fabio Maria
Approximate Neural Architecture Search via Operation Distribution Learning Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-04670,
title = {Approximate Neural Architecture Search via Operation Distribution Learning},
author = {Xingchen Wan and Binxin Ru and Pedro M. Esperança and Fabio Maria Carlucci},
url = {https://arxiv.org/abs/2111.04670},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.04670},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1097.
Lyu, Bo; Wen, Shiping; Yan, Zheng; Shi, Kaibo; Li, Ke; Huang, Tingwen
TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03892,
title = {TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework},
author = {Bo Lyu and Shiping Wen and Zheng Yan and Kaibo Shi and Ke Li and Tingwen Huang},
url = {https://arxiv.org/abs/2111.03892},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03892},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1096.
Garg, Bhanu; Zhang, Li; Sridhara, Pradyumna; Hosseini, Ramtin; Xing, Eric P.; Xie, Pengtao
Learning from Mistakes - A Framework for Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-06353,
title = {Learning from Mistakes - A Framework for Neural Architecture Search},
author = {Bhanu Garg and Li Zhang and Pradyumna Sridhara and Ramtin Hosseini and Eric P. Xing and Pengtao Xie},
url = {https://arxiv.org/abs/2111.06353},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.06353},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1095.
Zhao, Tianli; Zhang, Xi Sheryl; Zhu, Wentao; Wang, Jiaxing; Liu, Ji; Cheng, Jian
Joint Channel and Weight Pruning for Model Acceleration on Moblie Devices Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-08013,
title = {Joint Channel and Weight Pruning for Model Acceleration on Moblie Devices},
author = {Tianli Zhao and Xi Sheryl Zhang and Wentao Zhu and Jiaxing Wang and Ji Liu and Jian Cheng},
url = {https://arxiv.org/abs/2110.08013},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.08013},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1094.
Bhardwaj, Kartikeya; Li, Guihong; Marculescu, Radu
How Does Topology Influence Gradient Propagation and Model Performance of Deep Networks With DenseNet-Type Skip Connections? Proceedings Article
In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 13498–13507, Computer Vision Foundation / IEEE, 2021.
@inproceedings{DBLP:conf/cvpr/BhardwajLM21,
title = {How Does Topology Influence Gradient Propagation and Model Performance of Deep Networks With DenseNet-Type Skip Connections?},
author = {Kartikeya Bhardwaj and Guihong Li and Radu Marculescu},
url = {https://openaccess.thecvf.com/content/CVPR2021/html/Bhardwaj_How_Does_Topology_Influence_Gradient_Propagation_and_Model_Performance_of_CVPR_2021_paper.html},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2021, virtual, June 19-25, 2021},
pages = {13498--13507},
publisher = {Computer Vision Foundation / IEEE},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1093.
Gao, Chen; Li, Yinfeng; yao,; Jin, Depeng; Li, Yong
Progressive Feature Interaction Search for Deep Sparse Network Proceedings Article
In: Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
@inproceedings{<LineBreak>gao2021progressive,
title = {Progressive Feature Interaction Search for Deep Sparse Network},
author = {Chen Gao and Yinfeng Li and yao and Depeng Jin and Yong Li},
url = {https://openreview.net/forum?id=rl2FreDHTb0},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Thirty-Fifth Conference on Neural Information Processing Systems},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1092.
Chen, Minghao; Wu, Kan; Ni, Bolin; Peng, Houwen; Liu, Bei; Fu, Jianlong; Chao, Hongyang; Ling, Haibin
Searching the Search Space of Vision Transformer Proceedings Article
In: Advances in Neural Information Processing Systems 34 pre-proceedings (NeurIPS 2021) , 2021.
@inproceedings{DBLP:journals/corr/abs-2111-14725,
title = {Searching the Search Space of Vision Transformer},
author = {Minghao Chen and Kan Wu and Bolin Ni and Houwen Peng and Bei Liu and Jianlong Fu and Hongyang Chao and Haibin Ling},
url = {https://openreview.net/pdf?id=AVS8CamBecS},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = { Advances in Neural Information Processing Systems 34 pre-proceedings (NeurIPS 2021) },
journal = {CoRR},
volume = {abs/2111.14725},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1091.
Jeong, Wonyong; Lee, Hayeon; Park, Geon; Hyung, Eunyoung; Baek, Jinheon; Hwang, Sung Ju
Task-Adaptive Neural Network Retrieval with Meta-Contrastive Learning Proceedings Article
In: Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
@inproceedings{<LineBreak>jeong2021taskadaptive,
title = {Task-Adaptive Neural Network Retrieval with Meta-Contrastive Learning},
author = {Wonyong Jeong and Hayeon Lee and Geon Park and Eunyoung Hyung and Jinheon Baek and Sung Ju Hwang},
url = {https://openreview.net/forum?id=U68DvXABbJ3},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Thirty-Fifth Conference on Neural Information Processing Systems},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1090.
Lu, Haodong; Du, Miao; He, Xiaoming; Qian, Kai; Chen, Jianli; Sun, Yanfei; Wang, Kun
An Adaptive Neural Architecture Search Design for Collaborative Edge-Cloud Computing Journal Article
In: IEEE Network, vol. 35, no. 5, pp. 83-89, 2021.
@article{9606812,
title = {An Adaptive Neural Architecture Search Design for Collaborative Edge-Cloud Computing},
author = {Haodong Lu and Miao Du and Xiaoming He and Kai Qian and Jianli Chen and Yanfei Sun and Kun Wang},
url = {https://ieeexplore.ieee.org/abstract/document/9606812},
doi = {10.1109/MNET.201.2100069},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {IEEE Network},
volume = {35},
number = {5},
pages = {83-89},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1089.
Loni, Mohammad; Zoljodi, Ali; Majd, Amin; Ahn, Byung Hoon; Daneshtalab, Masoud; Sjödin, Mikael; Esmaeilzadeh, Hadi
FastStereoNet: A Fast Neural Architecture Search for Improving the Inference of Disparity Estimation on Resource-Limited Platforms Journal Article
In: IEEE Transactions on Systems, Man, and Cybernetics: Systems, pp. 1-13, 2021.
@article{9609009b,
title = {FastStereoNet: A Fast Neural Architecture Search for Improving the Inference of Disparity Estimation on Resource-Limited Platforms},
author = {Mohammad Loni and Ali Zoljodi and Amin Majd and Byung Hoon Ahn and Masoud Daneshtalab and Mikael Sjödin and Hadi Esmaeilzadeh},
url = {https://ieeexplore.ieee.org/abstract/document/9609009},
doi = {10.1109/TSMC.2021.3123136},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {IEEE Transactions on Systems, Man, and Cybernetics: Systems},
pages = {1-13},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1088.
Ding, Zixiang; Chen, Yaran; Li, Nannan; Zhao, Dongbin
Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-07722,
title = {Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search},
author = {Zixiang Ding and Yaran Chen and Nannan Li and Dongbin Zhao},
url = {https://arxiv.org/abs/2111.07722},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.07722},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1087.
Zhou, Yuan; Wang, Haiyang; Huo, Shuwei; Wang, Boyu
Full-attention based Neural Architecture Search using Context Auto-regression Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-07139,
title = {Full-attention based Neural Architecture Search using Context Auto-regression},
author = {Yuan Zhou and Haiyang Wang and Shuwei Huo and Boyu Wang},
url = {https://arxiv.org/abs/2111.07139},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.07139},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1086.
Tian, Yuqing; Zhang, Zhaoyang; Yang, Zhaohui; Yang, Qianqian
JMSNAS: Joint Model Split and Neural Architecture Search for Learning over Mobile Edge Networks Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-08206,
title = {JMSNAS: Joint Model Split and Neural Architecture Search for Learning over Mobile Edge Networks},
author = {Yuqing Tian and Zhaoyang Zhang and Zhaohui Yang and Qianqian Yang},
url = {https://arxiv.org/abs/2111.08206},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.08206},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1085.
Ahn, Joon Young; Cho, Nam Ik
Multi-Branch Neural Architecture Search for Lightweight Image Super-Resolution Journal Article
In: IEEE Access, vol. 9, pp. 153633-153646, 2021.
@article{9612202,
title = {Multi-Branch Neural Architecture Search for Lightweight Image Super-Resolution},
author = {Joon Young Ahn and Nam Ik Cho},
url = {https://ieeexplore.ieee.org/abstract/document/9612202},
doi = {10.1109/ACCESS.2021.3127437},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {IEEE Access},
volume = {9},
pages = {153633-153646},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1084.
Lee, Kyung-Chae; Linh, Le Vu; Kim, Heejae; Youn, Chan-Hyun
Neural Architecture Search for Computation Offloading of DNNs from Mobile Devices to the Edge Server Proceedings Article
In: International Conference on Information and Communication Technology Convergence, ICTC 2021, Jeju Island, Korea, Republic of, October 20-22, 2021, pp. 134–139, IEEE, 2021.
@inproceedings{DBLP:conf/ictc/LeeLKY21,
title = {Neural Architecture Search for Computation Offloading of DNNs from Mobile Devices to the Edge Server},
author = {Kyung-Chae Lee and Le Vu Linh and Heejae Kim and Chan-Hyun Youn},
url = {https://doi.org/10.1109/ICTC52510.2021.9621012},
doi = {10.1109/ICTC52510.2021.9621012},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {International Conference on Information and Communication Technology
Convergence, ICTC 2021, Jeju Island, Korea, Republic of, October
20-22, 2021},
pages = {134--139},
publisher = {IEEE},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1083.
Jing, Kun; Xu, Jungang; Zhang, Zhen
A neural architecture generator for efficient search space Journal Article
In: Neurocomputing, 2021, ISSN: 0925-2312.
@article{JING2021,
title = {A neural architecture generator for efficient search space},
author = {Kun Jing and Jungang Xu and Zhen Zhang},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221016994},
doi = {https://doi.org/10.1016/j.neucom.2021.10.118},
issn = {0925-2312},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Neurocomputing},
abstract = {Neural architecture search (NAS) has made significant progress in recent years. However, the existing methods usually search architectures in a small-scale, well-designed architecture space, discover only one architecture in a single search, and hardly rework, which severely limits their potential. In this paper, we propose a novel neural architecture generator (NAG) that can efficiently sample architectures in a large-scale architecture space. Like a generative adversarial network (GAN), our model consists of two components: (1) a generator that can generate directed acyclic graphs (DAGs) as cells or blocks of neural architectures and (2) a discriminator that can estimate the probability that a DAG comes from cells of real architectures rather than the generator. Furthermore, we employ a random search with NAG (RS-NAG) to discover the optimal architecture according to the customized requirements. Experimental results show that the NAG can generate diverse architectures with our customized requirements multiple times after one adversary training. Furthermore, compared with the existing methods, our RS-NAG achieves the competitive results with 2.50% and 25.5% top-1 accuracies on two benchmark datasets – CIFAR-10 and ImageNet.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) has made significant progress in recent years. However, the existing methods usually search architectures in a small-scale, well-designed architecture space, discover only one architecture in a single search, and hardly rework, which severely limits their potential. In this paper, we propose a novel neural architecture generator (NAG) that can efficiently sample architectures in a large-scale architecture space. Like a generative adversarial network (GAN), our model consists of two components: (1) a generator that can generate directed acyclic graphs (DAGs) as cells or blocks of neural architectures and (2) a discriminator that can estimate the probability that a DAG comes from cells of real architectures rather than the generator. Furthermore, we employ a random search with NAG (RS-NAG) to discover the optimal architecture according to the customized requirements. Experimental results show that the NAG can generate diverse architectures with our customized requirements multiple times after one adversary training. Furthermore, compared with the existing methods, our RS-NAG achieves the competitive results with 2.50% and 25.5% top-1 accuracies on two benchmark datasets – CIFAR-10 and ImageNet.