Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2022
2126.
Mi, Jian-Xun; Feng, Jie; Huang, Ke-Yang
Designing efficient convolutional neural network structure: A survey Journal Article
In: Neurocomputing, vol. 489, pp. 139-156, 2022, ISSN: 0925-2312.
@article{MI2022139,
title = {Designing efficient convolutional neural network structure: A survey},
author = {Jian-Xun Mi and Jie Feng and Ke-Yang Huang},
url = {https://www.sciencedirect.com/science/article/pii/S0925231222003162},
doi = {https://doi.org/10.1016/j.neucom.2021.08.158},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
volume = {489},
pages = {139-156},
abstract = {As a powerful machine learning method, deep learning has attracted the attention of numerous researchers. While exploring a high-performance neural network model, the floating-point operations of a neural network model are also increasing. In recent years, many researchers have noticed that efficiency is also one of important indicators to measure the property of neural network models. Obviously, the efficient neural network model is more helpful to deploy on mobile and embedded devices. Therefore, the efficient neural network model becomes a hot research spot. In this paper, we review the methods related to the structural design of efficient convolution neural networks in recent years. According to the characteristics of these methods, we divide them into three kinds of methods: model pruning, efficient architecture, and neural architecture search. Detailed analyses of each method are presented to demonstrate their advantages and disadvantages. Then, we comprehensively compare them in detail and propose many suggestions about the design of the efficient convolution neural network model structure. Inspired by these suggestions, we built a new efficient neural network model, SharedNet. And the SharedNet obtains the best accuracy of manually-designed efficient CNN models on the ImageNet dataset.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
As a powerful machine learning method, deep learning has attracted the attention of numerous researchers. While exploring a high-performance neural network model, the floating-point operations of a neural network model are also increasing. In recent years, many researchers have noticed that efficiency is also one of important indicators to measure the property of neural network models. Obviously, the efficient neural network model is more helpful to deploy on mobile and embedded devices. Therefore, the efficient neural network model becomes a hot research spot. In this paper, we review the methods related to the structural design of efficient convolution neural networks in recent years. According to the characteristics of these methods, we divide them into three kinds of methods: model pruning, efficient architecture, and neural architecture search. Detailed analyses of each method are presented to demonstrate their advantages and disadvantages. Then, we comprehensively compare them in detail and propose many suggestions about the design of the efficient convolution neural network model structure. Inspired by these suggestions, we built a new efficient neural network model, SharedNet. And the SharedNet obtains the best accuracy of manually-designed efficient CNN models on the ImageNet dataset.
2125.
Blumberg, Stefano B.; Lin, Hongxiang; Grussu, Francesco; Zhou, Yukun; Figini, Matteo; Alexander, Daniel C.
Progressive Subsampling for Oversampled Data - Application to Quantitative MRI Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-09268,
title = {Progressive Subsampling for Oversampled Data - Application to Quantitative MRI},
author = {Stefano B. Blumberg and Hongxiang Lin and Francesco Grussu and Yukun Zhou and Matteo Figini and Daniel C. Alexander},
url = {https://doi.org/10.48550/arXiv.2203.09268},
doi = {10.48550/arXiv.2203.09268},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.09268},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2124.
Vo-Ho, Viet-Khoa; Yamazaki, Kashu; Hoang, Hieu; Tran, Minh-Triet; Le, Ngan
Meta-Learning of NAS for Few-shot Learning in Medical Image Applications Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-08951,
title = {Meta-Learning of NAS for Few-shot Learning in Medical Image Applications},
author = {Viet-Khoa Vo-Ho and Kashu Yamazaki and Hieu Hoang and Minh-Triet Tran and Ngan Le},
url = {https://doi.org/10.48550/arXiv.2203.08951},
doi = {10.48550/arXiv.2203.08951},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.08951},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2123.
Chang, Qing; Peng, Junran; Xie, Lingxi; Sun, Jiajun; Yin, Haoran; Tian, Qi; Zhang, Zhaoxiang
DATA: Domain-Aware and Task-Aware Self-supervised Learning Proceedings Article
In: CVPR2022, 2022.
@inproceedings{DBLP:journals/corr/abs-2203-09041,
title = {DATA: Domain-Aware and Task-Aware Self-supervised Learning},
author = {Qing Chang and Junran Peng and Lingxi Xie and Jiajun Sun and Haoran Yin and Qi Tian and Zhaoxiang Zhang},
url = {https://openaccess.thecvf.com/content/CVPR2022/papers/Chang_DATA_Domain-Aware_and_Task-Aware_Self-Supervised_Learning_CVPR_2022_paper.pdf},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {CVPR2022},
journal = {CoRR},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2122.
Lu, Zhenyu; Liang, Shaoyang; Yang, Qiang; Du, Bo
Evolving Block-Based Convolutional Neural Network for Hyperspectral Image Classification Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-21, 2022.
@article{9737511,
title = {Evolving Block-Based Convolutional Neural Network for Hyperspectral Image Classification},
author = {Zhenyu Lu and Shaoyang Liang and Qiang Yang and Bo Du},
url = {https://ieeexplore.ieee.org/abstract/document/9737511},
doi = {10.1109/TGRS.2022.3160513},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
volume = {60},
pages = {1-21},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2121.
Lukasik, Jovita; Jung, Steffen; Keuper, Margret
Learning Where To Look - Generative NAS is Surprisingly Efficient Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-08734,
title = {Learning Where To Look - Generative NAS is Surprisingly Efficient},
author = {Jovita Lukasik and Steffen Jung and Margret Keuper},
url = {https://doi.org/10.48550/arXiv.2203.08734},
doi = {10.48550/arXiv.2203.08734},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.08734},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2120.
Yan, Chenqian; Zhang, Yuge; Zhang, Quanlu; Yang, Yaming; Jiang, Xinyang; Yang, Yuqing; Wang, Baoyuan
Privacy-preserving Online AutoML for Domain-Specific Face Detection Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-08399,
title = {Privacy-preserving Online AutoML for Domain-Specific Face Detection},
author = {Chenqian Yan and Yuge Zhang and Quanlu Zhang and Yaming Yang and Xinyang Jiang and Yuqing Yang and Baoyuan Wang},
url = {https://doi.org/10.48550/arXiv.2203.08399},
doi = {10.48550/arXiv.2203.08399},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.08399},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2119.
Yang, Sen; Yang, Wankou; Cui, Zhen
Searching part-specific neural fabrics for human pose estimation Journal Article
In: Pattern Recognition, vol. 128, pp. 108652, 2022, ISSN: 0031-3203.
@article{YANG2022108652,
title = {Searching part-specific neural fabrics for human pose estimation},
author = {Sen Yang and Wankou Yang and Zhen Cui},
url = {https://www.sciencedirect.com/science/article/pii/S0031320322001339},
doi = {https://doi.org/10.1016/j.patcog.2022.108652},
issn = {0031-3203},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Pattern Recognition},
volume = {128},
pages = {108652},
abstract = {Neural architecture search (NAS) has emerged in many domains to jointly learn the architectures and weights of neural networks. The core spirit behind NAS is to automatically search neural architectures for target tasks with better performance-efficiency trade-offs. However, existing approaches emphasize on only searching a single architecture with less human intervention to replace a human-designed neural network, yet making the search process almost independent of the domain knowledge. In this paper, we aim to apply NAS for human pose estimation and we ask: when NAS meets this localization task, can the articulated human body structure help to search better task-specific architectures? To this end, we first design a new neural architecture search space, Cell-based Neural Fabric (CNF), to learn micro as well as macro neural architecture using a differentiable search strategy. Then, by viewing locating human parts as multiple disentangled prediction sub-tasks, we exploit the compositionality of human body structure as guidance to search multiple part-specific CNFs specialized for different human parts. After the search, all these part-specific neural fabrics have been tailored with distinct micro and macro architecture parameters. The results show that such knowledge-guided NAS-based model outperforms a hand-crafted part-based baseline model, and the resulting multiple part-specific architectures gain significant performance improvement against a single NAS-based architecture for the whole body. The experiments on MPII and COCO datasets show that our models11Code is available at https://github.com/yangsenius/PoseNFS. achieve comparable performance against the state-of-the-art methods while being relatively lightweight.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) has emerged in many domains to jointly learn the architectures and weights of neural networks. The core spirit behind NAS is to automatically search neural architectures for target tasks with better performance-efficiency trade-offs. However, existing approaches emphasize on only searching a single architecture with less human intervention to replace a human-designed neural network, yet making the search process almost independent of the domain knowledge. In this paper, we aim to apply NAS for human pose estimation and we ask: when NAS meets this localization task, can the articulated human body structure help to search better task-specific architectures? To this end, we first design a new neural architecture search space, Cell-based Neural Fabric (CNF), to learn micro as well as macro neural architecture using a differentiable search strategy. Then, by viewing locating human parts as multiple disentangled prediction sub-tasks, we exploit the compositionality of human body structure as guidance to search multiple part-specific CNFs specialized for different human parts. After the search, all these part-specific neural fabrics have been tailored with distinct micro and macro architecture parameters. The results show that such knowledge-guided NAS-based model outperforms a hand-crafted part-based baseline model, and the resulting multiple part-specific architectures gain significant performance improvement against a single NAS-based architecture for the whole body. The experiments on MPII and COCO datasets show that our models11Code is available at https://github.com/yangsenius/PoseNFS. achieve comparable performance against the state-of-the-art methods while being relatively lightweight.
2118.
Zhang, Haichao; Hao, Kuangrong; Pedrycz, Witold; Gao, Lei; Tang, Xue-Song; Wei, Bing
Vision Transformer with Convolutions Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-10435,
title = {Vision Transformer with Convolutions Architecture Search},
author = {Haichao Zhang and Kuangrong Hao and Witold Pedrycz and Lei Gao and Xue-Song Tang and Bing Wei},
url = {https://doi.org/10.48550/arXiv.2203.10435},
doi = {10.48550/arXiv.2203.10435},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.10435},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2117.
Hu, Yiming; Wang, Xingang; Gu, Qingyi
PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework Journal Article
In: IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2022.
@article{9739130,
title = {PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework},
author = {Yiming Hu and Xingang Wang and Qingyi Gu},
url = {https://ieeexplore.ieee.org/abstract/document/9739130},
doi = {10.1109/TNNLS.2022.3156373},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Neural Networks and Learning Systems},
pages = {1-14},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2116.
Wang, Xiaoxing; Lin, Jiale; Yan, Junchi; Zhao, Juanping; Yang, Xiaokang
EAutoDet: Efficient Architecture Search for Object Detection Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-10747,
title = {EAutoDet: Efficient Architecture Search for Object Detection},
author = {Xiaoxing Wang and Jiale Lin and Junchi Yan and Juanping Zhao and Xiaokang Yang},
url = {https://doi.org/10.48550/arXiv.2203.10747},
doi = {10.48550/arXiv.2203.10747},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.10747},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2115.
Habibian, Amirhossein; Yahia, Haitam Ben; Abati, Davide; Gavves, Efstratios; Porikli, Fatih
Delta Distillation for Efficient Video Processing Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-09594,
title = {Delta Distillation for Efficient Video Processing},
author = {Amirhossein Habibian and Haitam Ben Yahia and Davide Abati and Efstratios Gavves and Fatih Porikli},
url = {https://doi.org/10.48550/arXiv.2203.09594},
doi = {10.48550/arXiv.2203.09594},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.09594},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2114.
Arora, Parul; Jalali, Seyed Mohammad Jafar; Ahmadian, Sajad; Panigrahi, Bijaya Ketan; Suganthan, Pn; Khosravi, Abbas
Probabilistic Wind Power Forecasting Using Optimised Deep Auto-Regressive Recurrent Neural Networks Journal Article
In: IEEE Transactions on Industrial Informatics, pp. 1-1, 2022.
@article{9739990,
title = {Probabilistic Wind Power Forecasting Using Optimised Deep Auto-Regressive Recurrent Neural Networks},
author = {Parul Arora and Seyed Mohammad Jafar Jalali and Sajad Ahmadian and Bijaya Ketan Panigrahi and Pn Suganthan and Abbas Khosravi},
url = {https://ieeexplore.ieee.org/abstract/document/9739990},
doi = {10.1109/TII.2022.3160696},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Industrial Informatics},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2113.
Yüzügüler, Ahmet Caner; Dimitriadis, Nikolaos; Frossard, Pascal
U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search Technical Report
2022.
@techreport{yuzuguler2022u,
title = {U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search},
author = {Ahmet Caner Yüzügüler and Nikolaos Dimitriadis and Pascal Frossard},
url = {https://arxiv.org/abs/2203.12412},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {arXiv preprint arXiv:2203.12412},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2112.
Xie, Yirong; Chen, Hong; Ma, Yongjie; Xu, Yang
Automated design of CNN architecture based on efficient evolutionary search Journal Article
In: Neurocomputing, vol. 491, pp. 160-171, 2022, ISSN: 0925-2312.
@article{XIE2022160,
title = {Automated design of CNN architecture based on efficient evolutionary search},
author = {Yirong Xie and Hong Chen and Yongjie Ma and Yang Xu},
url = {https://www.sciencedirect.com/science/article/pii/S092523122200340X},
doi = {https://doi.org/10.1016/j.neucom.2022.03.046},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
volume = {491},
pages = {160-171},
abstract = {Evolutionary Neural Architecture Search (ENAS) is a promising method for the automated design of deep network architecture, which has attracted extensive attention in the field of automated machine learning. However, the existing ENAS methods often need a lot of computing resources to design CNN architecture automatically. In order to achieve efficient and automated design of CNNs, this paper focuses on two aspects to improve efficiency. On the one hand, efficient CNN-based building blocks are introduced to ensure the effectiveness of the generated architectures and a triplet attention mechanism is incorporated into the architectures to further improve the classification performance. On the other hand, a random forest-based performance predictor is used in the fitness evaluation to reduce the amount of computation required to train each individual from scratch. Experimental results show that the proposed algorithm can significantly reduce the computational resources required and achieve competitive classification performance on the CIFAR dataset. Also, the architecture designed for the traffic sign recognition task exceeds the accuracy of manual expert design.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Evolutionary Neural Architecture Search (ENAS) is a promising method for the automated design of deep network architecture, which has attracted extensive attention in the field of automated machine learning. However, the existing ENAS methods often need a lot of computing resources to design CNN architecture automatically. In order to achieve efficient and automated design of CNNs, this paper focuses on two aspects to improve efficiency. On the one hand, efficient CNN-based building blocks are introduced to ensure the effectiveness of the generated architectures and a triplet attention mechanism is incorporated into the architectures to further improve the classification performance. On the other hand, a random forest-based performance predictor is used in the fitness evaluation to reduce the amount of computation required to train each individual from scratch. Experimental results show that the proposed algorithm can significantly reduce the computational resources required and achieve competitive classification performance on the CIFAR dataset. Also, the architecture designed for the traffic sign recognition task exceeds the accuracy of manual expert design.
2111.
Benmeziane, Hadjer; Ouarnoughi, Hamza; Maghraoui, Kaoutar El; Niar, Smail
Real-Time Style Transfer with Efficient Vision Transformers Proceedings Article
In: Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking, pp. 31–36, Association for Computing Machinery, Rennes, France, 2022, ISBN: 9781450392532.
@inproceedings{10.1145/3517206.3526271,
title = {Real-Time Style Transfer with Efficient Vision Transformers},
author = {Hadjer Benmeziane and Hamza Ouarnoughi and Kaoutar El Maghraoui and Smail Niar},
url = {https://doi.org/10.1145/3517206.3526271},
doi = {10.1145/3517206.3526271},
isbn = {9781450392532},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking},
pages = {31–36},
publisher = {Association for Computing Machinery},
address = {Rennes, France},
series = {EdgeSys '22},
abstract = {Style Transfer aims at transferring the artistic style from a reference image to a content image. While Deep Learning (DL) has achieved state-of-the-art Style Transfer performance using Convolutional Neural Networks (CNN), its real-time application still requires powerful hardware such as GPU-accelerated systems. This paper leverages transformer-based models to accelerate real-time Style Transfer on mobile and embedded hardware platforms. We designed a Neural Architecture Search (NAS) algorithm dedicated to vision transformers to find the best set of architecture hyperparameters that maximizes the Style Transfer performance, expressed in Frame/seconds (FPS). Our approach has been evaluated and validated on the Xiaomi Redmi 7 mobile phone and the Raspberry Pi 3 platform. Experimental evaluation shows that our approach allows to achieve a 3.5x and 2.1x speedups compared to CNN-based Style Transfer models and Transformer-based models respectively1.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Style Transfer aims at transferring the artistic style from a reference image to a content image. While Deep Learning (DL) has achieved state-of-the-art Style Transfer performance using Convolutional Neural Networks (CNN), its real-time application still requires powerful hardware such as GPU-accelerated systems. This paper leverages transformer-based models to accelerate real-time Style Transfer on mobile and embedded hardware platforms. We designed a Neural Architecture Search (NAS) algorithm dedicated to vision transformers to find the best set of architecture hyperparameters that maximizes the Style Transfer performance, expressed in Frame/seconds (FPS). Our approach has been evaluated and validated on the Xiaomi Redmi 7 mobile phone and the Raspberry Pi 3 platform. Experimental evaluation shows that our approach allows to achieve a 3.5x and 2.1x speedups compared to CNN-based Style Transfer models and Transformer-based models respectively1.
2110.
Rajesh, Chilukamari; Kumar, Sushil
An evolutionary block based network for medical image denoising using Differential Evolution Journal Article
In: Applied Soft Computing, vol. 121, pp. 108776, 2022, ISSN: 1568-4946.
@article{RAJESH2022108776,
title = {An evolutionary block based network for medical image denoising using Differential Evolution},
author = {Chilukamari Rajesh and Sushil Kumar},
url = {https://www.sciencedirect.com/science/article/pii/S1568494622002022},
doi = {https://doi.org/10.1016/j.asoc.2022.108776},
issn = {1568-4946},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Applied Soft Computing},
volume = {121},
pages = {108776},
abstract = {Image denoising is the key component in several computer vision and image processing operations due to unavoidable noise in the image generation process. For medical image processing, deep convolutional neural networks (CNN) gives a state-of-the-art performance. However, network structures are manually constructed for specific tasks and require several trials to tune a large number of hyperparameters, which can take a long time to construct a network. Additionally, the fittest hyperparameters which may be suitable for source data properties like noisy features cannot be easily found to target data. The realistic noise is generally mixed, complex, and unpredictable in medical images, which makes it difficult to design an efficient denoising network. We developed a Differential Evolution (DE) based automatic network evolution model in this paper to optimize the network architectures and hyperparameters by exploring the fittest parameters. Furthermore, we adopted a transfer learning technique to accelerate the training process. The proposed evolutionary algorithm is flexible and finds optimistic network architectures using well-known methods including residual and dense blocks. Finally, the proposed model was evaluated on four different medical image datasets. The obtained results at different noise levels show the potentiality of the proposed model named DEvoNet for identifying the optimal parameters to develop a high-performance denoising network structure.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Image denoising is the key component in several computer vision and image processing operations due to unavoidable noise in the image generation process. For medical image processing, deep convolutional neural networks (CNN) gives a state-of-the-art performance. However, network structures are manually constructed for specific tasks and require several trials to tune a large number of hyperparameters, which can take a long time to construct a network. Additionally, the fittest hyperparameters which may be suitable for source data properties like noisy features cannot be easily found to target data. The realistic noise is generally mixed, complex, and unpredictable in medical images, which makes it difficult to design an efficient denoising network. We developed a Differential Evolution (DE) based automatic network evolution model in this paper to optimize the network architectures and hyperparameters by exploring the fittest parameters. Furthermore, we adopted a transfer learning technique to accelerate the training process. The proposed evolutionary algorithm is flexible and finds optimistic network architectures using well-known methods including residual and dense blocks. Finally, the proposed model was evaluated on four different medical image datasets. The obtained results at different noise levels show the potentiality of the proposed model named DEvoNet for identifying the optimal parameters to develop a high-performance denoising network structure.
2109.
Zhou, Qinqin; Sheng, Kekai; Zheng, Xiawu; Li, Ke; Sun, Xing; Tian, Yonghong; Chen, Jie; Ji, Rongrong
Training-free Transformer Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-12217,
title = {Training-free Transformer Architecture Search},
author = {Qinqin Zhou and Kekai Sheng and Xiawu Zheng and Ke Li and Xing Sun and Yonghong Tian and Jie Chen and Rongrong Ji},
url = {https://doi.org/10.48550/arXiv.2203.12217},
doi = {10.48550/arXiv.2203.12217},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.12217},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2108.
Mok, Jisoo; Na, Byunggook; Kim, Ji-Hoon; Han, Dongyoon; Yoon, Sungroh
Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training? Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-14577,
title = {Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?},
author = {Jisoo Mok and Byunggook Na and Ji-Hoon Kim and Dongyoon Han and Sungroh Yoon},
url = {https://doi.org/10.48550/arXiv.2203.14577},
doi = {10.48550/arXiv.2203.14577},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.14577},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2107.
Das, Mayukh; Singh, Brijraj; Chheda, Harsh Kanti; Sharma, Pawan; NS, Pradeep
AutoCoMet: Smart Neural Architecture Search via Co-Regulated Shaping Reinforcement Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-15408,
title = {AutoCoMet: Smart Neural Architecture Search via Co-Regulated Shaping Reinforcement},
author = {Mayukh Das and Brijraj Singh and Harsh Kanti Chheda and Pawan Sharma and Pradeep NS},
url = {https://doi.org/10.48550/arXiv.2203.15408},
doi = {10.48550/arXiv.2203.15408},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.15408},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2106.
Yang, Jin; Huang, Yingying; Jiang, Guangxin; Chen, Ying
An Intelligent End-to-End Neural Architecture Search Framework for Electricity Forecasting Model Development Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-13563,
title = {An Intelligent End-to-End Neural Architecture Search Framework for Electricity Forecasting Model Development},
author = {Jin Yang and Yingying Huang and Guangxin Jiang and Ying Chen},
url = {https://doi.org/10.48550/arXiv.2203.13563},
doi = {10.48550/arXiv.2203.13563},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.13563},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2105.
Sun, Haiyang; Lian, Zheng; Liu, Bin; Li, Ying; Sun, Licai; Cai, Cong; Tao, Jianhua; Wang, Meng; Cheng, Yuan
EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-13617,
title = {EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition},
author = {Haiyang Sun and Zheng Lian and Bin Liu and Ying Li and Licai Sun and Cong Cai and Jianhua Tao and Meng Wang and Yuan Cheng},
url = {https://doi.org/10.48550/arXiv.2203.13617},
doi = {10.48550/arXiv.2203.13617},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.13617},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2104.
Lu, Bingqian; Yan, Zheyu; Shi, Yiyu; Ren, Shaolei
A Semi-Decoupled Approach to Fast and Optimal Hardware-Software Co-Design of Neural Accelerators Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-13921,
title = {A Semi-Decoupled Approach to Fast and Optimal Hardware-Software Co-Design of Neural Accelerators},
author = {Bingqian Lu and Zheyu Yan and Yiyu Shi and Shaolei Ren},
url = {https://doi.org/10.48550/arXiv.2203.13921},
doi = {10.48550/arXiv.2203.13921},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.13921},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2103.
M., Abishai Ebenezer; Arya, Arti
An Atypical Metaheuristic Approach to Recognize an Optimal Architecture of a Neural Network Technical Report
2022.
@techreport{DBLP:conf/icaart/MA22,
title = {An Atypical Metaheuristic Approach to Recognize an Optimal Architecture of a Neural Network},
author = {Abishai Ebenezer M. and Arti Arya},
editor = {Ana Paula Rocha and Luc Steels and H. Jaap Herik},
url = {https://doi.org/10.5220/0010951600003116},
doi = {10.5220/0010951600003116},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 14th International Conference on Agents and Artificial
Intelligence, ICAART 2022, Volume 3, Online Streaming, February
3-5, 2022},
pages = {917--925},
publisher = {SCITEPRESS},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2102.
Wang, Chunnan; Chen, Xingyu; Wu, Chengyue; Wang, Hongzhi
AutoTS: Automatic Time Series Forecasting Model Design Based on Two-Stage Pruning Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-14169,
title = {AutoTS: Automatic Time Series Forecasting Model Design Based on Two-Stage Pruning},
author = {Chunnan Wang and Xingyu Chen and Chengyue Wu and Hongzhi Wang},
url = {https://doi.org/10.48550/arXiv.2203.14169},
doi = {10.48550/arXiv.2203.14169},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.14169},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2101.
Zaman, Khalid; Sun, Zhaoyun; Shah, Sayyed Mudassar; Shoaib, Muhammad; Pei, Lili; Hussain, Altaf
Driver Emotions Recognition Based on Improved Faster R-CNN and Neural Architectural Search Network Journal Article
In: Symmetry, vol. 14, no. 4, pp. 687, 2022.
@article{DBLP:journals/symmetry/ZamanSSSPH22,
title = {Driver Emotions Recognition Based on Improved Faster R-CNN and Neural Architectural Search Network},
author = {Khalid Zaman and Zhaoyun Sun and Sayyed Mudassar Shah and Muhammad Shoaib and Lili Pei and Altaf Hussain},
url = {https://doi.org/10.3390/sym14040687},
doi = {10.3390/sym14040687},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Symmetry},
volume = {14},
number = {4},
pages = {687},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2100.
Zheng, Ruiqi; Qu, Liang; Cui, Bin; Shi, Yuhui; Yin, Hongzhi
AutoML for Deep Recommender Systems: A Survey Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-13922,
title = {AutoML for Deep Recommender Systems: A Survey},
author = {Ruiqi Zheng and Liang Qu and Bin Cui and Yuhui Shi and Hongzhi Yin},
url = {https://doi.org/10.48550/arXiv.2203.13922},
doi = {10.48550/arXiv.2203.13922},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.13922},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2099.
Raychaudhuri, Dripta S.; Suh, Yumin; Schulter, Samuel; Yu, Xiang; Faraki, Masoud; Roy-Chowdhury, Amit K.; Chandraker, Manmohan
Controllable Dynamic Multi-Task Architectures Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-14949,
title = {Controllable Dynamic Multi-Task Architectures},
author = {Dripta S. Raychaudhuri and Yumin Suh and Samuel Schulter and Xiang Yu and Masoud Faraki and Amit K. Roy-Chowdhury and Manmohan Chandraker},
url = {https://doi.org/10.48550/arXiv.2203.14949},
doi = {10.48550/arXiv.2203.14949},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.14949},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2098.
Wang, Rui; Bai, Qibing; Ao, Junyi; Zhou, Long; Xiong, Zhixiang; Wei, Zhihua; Zhang, Yu; Ko, Tom; Li, Haizhou
LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-15610,
title = {LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT},
author = {Rui Wang and Qibing Bai and Junyi Ao and Long Zhou and Zhixiang Xiong and Zhihua Wei and Yu Zhang and Tom Ko and Haizhou Li},
url = {https://doi.org/10.48550/arXiv.2203.15610},
doi = {10.48550/arXiv.2203.15610},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.15610},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2097.
Chang, Yangyang; Sobelman, Gerald E.
Lightweight CNN Frameworks and their Optimization using Evolutionary Algorithms Proceedings Article
In: 2022 International Electrical Engineering Congress (iEECON), pp. 1-4, 2022.
@inproceedings{9741692,
title = {Lightweight CNN Frameworks and their Optimization using Evolutionary Algorithms},
author = {Yangyang Chang and Gerald E. Sobelman},
url = {https://ieeexplore.ieee.org/abstract/document/9741692},
doi = {10.1109/iEECON53204.2022.9741692},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 International Electrical Engineering Congress (iEECON)},
pages = {1-4},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2096.
Park, Gunju; Yi, Youngmin
CondNAS: Neural Architecture Search for Conditional CNNs Journal Article
In: Electronics, vol. 11, no. 7, 2022, ISSN: 2079-9292.
@article{electronics11071101,
title = {CondNAS: Neural Architecture Search for Conditional CNNs},
author = {Gunju Park and Youngmin Yi},
url = {https://www.mdpi.com/2079-9292/11/7/1101},
doi = {10.3390/electronics11071101},
issn = {2079-9292},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Electronics},
volume = {11},
number = {7},
abstract = {As deep learning has become prevalent and adopted in various application domains, the need for efficient convolution neural network (CNN) inference on diverse target platforms has increased. To address the need, a neural architecture search (NAS) technique called once-for-all, or OFA, which aims to efficiently find the optimal CNN architecture for the given target platform using genetic algorithm (GA), has recently been proposed. Meanwhile, a conditional CNN architecture, which allows early exits with auxiliary classifiers in the middle of a network to achieve efficient inference without accuracy loss or with negligible loss, has been proposed. In this paper, we propose a NAS technique for the conditional CNN architecture, CondNAS, which efficiently finds a near-optimal conditional CNN architecture for the target platform using GA. By attaching auxiliary classifiers through adaptive pooling, OFA’s SuperNet is successfully extended, such that it incorporates the various conditional CNN sub-networks. In addition, we devise machine learning-based prediction models for the accuracy and latency of an arbitrary conditional CNN, which are used in the GA of CondNAS to efficiently explore the large search space. The experimental results show that the conditional CNNs from CondNAS is 2.52× and 1.75× faster than the CNNs from OFA for Galaxy Note10+ GPU and CPU, respectively.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
As deep learning has become prevalent and adopted in various application domains, the need for efficient convolution neural network (CNN) inference on diverse target platforms has increased. To address the need, a neural architecture search (NAS) technique called once-for-all, or OFA, which aims to efficiently find the optimal CNN architecture for the given target platform using genetic algorithm (GA), has recently been proposed. Meanwhile, a conditional CNN architecture, which allows early exits with auxiliary classifiers in the middle of a network to achieve efficient inference without accuracy loss or with negligible loss, has been proposed. In this paper, we propose a NAS technique for the conditional CNN architecture, CondNAS, which efficiently finds a near-optimal conditional CNN architecture for the target platform using GA. By attaching auxiliary classifiers through adaptive pooling, OFA’s SuperNet is successfully extended, such that it incorporates the various conditional CNN sub-networks. In addition, we devise machine learning-based prediction models for the accuracy and latency of an arbitrary conditional CNN, which are used in the GA of CondNAS to efficiently explore the large search space. The experimental results show that the conditional CNNs from CondNAS is 2.52× and 1.75× faster than the CNNs from OFA for Galaxy Note10+ GPU and CPU, respectively.
2095.
Zhou, Qinghua; Gorban, Alexander N.; Mirkes, Evgeny M.; Bac, Jonathan; Zinovyev, Andrei Yu.; Tyukin, Ivan Yu.
Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2203-16687,
title = {Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation},
author = {Qinghua Zhou and Alexander N. Gorban and Evgeny M. Mirkes and Jonathan Bac and Andrei Yu. Zinovyev and Ivan Yu. Tyukin},
url = {https://doi.org/10.48550/arXiv.2203.16687},
doi = {10.48550/arXiv.2203.16687},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2203.16687},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2094.
Li, Yawei
Towards Efficient Deep Neural Networks PhD Thesis
ETH Zurich, 2022.
@phdthesis{20.500.11850/540498,
title = {Towards Efficient Deep Neural Networks},
author = {Yawei Li},
doi = {10.3929/ethz-b-000540498},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
publisher = {ETH Zurich},
address = {Zurich},
school = {ETH Zurich},
abstract = {Computational efficiency is an essential factor that influences the applicability of computer vision algorithms. Although deep neural networks have reached state-of-the-art performances in a variety of computer vision tasks, there are a couple of efficiency related problems of the deep learning based solutions. First, the overparameterization of deep neural networks results in models with millions of parameters, which lowers the parameter efficiency of the designed networks. To store the parameters and intermediate feature maps during the computation, a large device memory footprint is required. Secondly, the massive computation in deep neural networks slows down their training and inference. This limits the application of deep neural networks to latency-demanding scenarios and low-end devices. Thirdly, the massive computation consumes significant amount of energy, which leaves a large carbon footprint of deep learning models.The aim of this thesis is to improve the computational efficiency of current deep neural networks. This problem is tackled from three perspective including neural network compression, neural architecture optimization, and computational procedure optimization.In the first part of the thesis, we reduce the model complexity of neural networks by network compression techniques including filter decomposition and filter pruning. The basic assumption for filter decomposition is that the ensemble of filters in deep neural networks constitutes an overcomplete set. Instead of using the original filters directly during the computation, they can be approximated by a linear combination of a set of basis filters. The contribution of this thesis is to provide a unified analysis of previous filter decomposition methods. On the other hand, a differentiable filter pruning method is proposed. To achieve differentiability, the layers of neural networks is reparameterized by a meta network. Sparsity regularization is applied to the input of the meta network, i.e. latent vectors. Optimizing with the introduced regularization leads to an automatic network pruning method. Additionally, a joint analysis of filter decomposition and filter pruning is presented from the perspective of compact tensor approximation. The hinge of the two techniques is the introduced sparsity inducing matrix. By simply changing the way the group sparsity regularization is enforced to the matrix, the two techniques can be derived accordingly.Secondly, we try to improve the performance of a baseline network by a fine-grained neural architecture optimization method. Different from network compression methods, the aim of this method is to improve the prediction accuracy of neural networks while reducing their model complexity at the same time. Achieving the two targets simultaneously makes the problem more challenging. In addition, a nearly cost-free constraint is enforced during the architecture optimization, which differs from current neural architecture search methods with bulky computation. This can be regarded as another efficiency-improving technique.Thirdly, we optimize the computational procedure of graph neural networks. By mathematically analyzing the operations in graph neural network, two methods are proposed to improve the computational efficiency. The first method is related to the simplification of neighbor querying in graph neural network while the second involves shuffling the order of graph feature gathering and an feature extraction operations. To summarize, this thesis contributes to multiple aspects of improving the computational efficiency of neural networks during the optimization, training, and test phase.},
keywords = {},
pubstate = {published},
tppubtype = {phdthesis}
}
Computational efficiency is an essential factor that influences the applicability of computer vision algorithms. Although deep neural networks have reached state-of-the-art performances in a variety of computer vision tasks, there are a couple of efficiency related problems of the deep learning based solutions. First, the overparameterization of deep neural networks results in models with millions of parameters, which lowers the parameter efficiency of the designed networks. To store the parameters and intermediate feature maps during the computation, a large device memory footprint is required. Secondly, the massive computation in deep neural networks slows down their training and inference. This limits the application of deep neural networks to latency-demanding scenarios and low-end devices. Thirdly, the massive computation consumes significant amount of energy, which leaves a large carbon footprint of deep learning models.The aim of this thesis is to improve the computational efficiency of current deep neural networks. This problem is tackled from three perspective including neural network compression, neural architecture optimization, and computational procedure optimization.In the first part of the thesis, we reduce the model complexity of neural networks by network compression techniques including filter decomposition and filter pruning. The basic assumption for filter decomposition is that the ensemble of filters in deep neural networks constitutes an overcomplete set. Instead of using the original filters directly during the computation, they can be approximated by a linear combination of a set of basis filters. The contribution of this thesis is to provide a unified analysis of previous filter decomposition methods. On the other hand, a differentiable filter pruning method is proposed. To achieve differentiability, the layers of neural networks is reparameterized by a meta network. Sparsity regularization is applied to the input of the meta network, i.e. latent vectors. Optimizing with the introduced regularization leads to an automatic network pruning method. Additionally, a joint analysis of filter decomposition and filter pruning is presented from the perspective of compact tensor approximation. The hinge of the two techniques is the introduced sparsity inducing matrix. By simply changing the way the group sparsity regularization is enforced to the matrix, the two techniques can be derived accordingly.Secondly, we try to improve the performance of a baseline network by a fine-grained neural architecture optimization method. Different from network compression methods, the aim of this method is to improve the prediction accuracy of neural networks while reducing their model complexity at the same time. Achieving the two targets simultaneously makes the problem more challenging. In addition, a nearly cost-free constraint is enforced during the architecture optimization, which differs from current neural architecture search methods with bulky computation. This can be regarded as another efficiency-improving technique.Thirdly, we optimize the computational procedure of graph neural networks. By mathematically analyzing the operations in graph neural network, two methods are proposed to improve the computational efficiency. The first method is related to the simplification of neighbor querying in graph neural network while the second involves shuffling the order of graph feature gathering and an feature extraction operations. To summarize, this thesis contributes to multiple aspects of improving the computational efficiency of neural networks during the optimization, training, and test phase.
2093.
Zang, Huaijuan; Cheng, Guoan; Duan, Zhipeng; Zhao, Ying; Zhan, Shu
Automatic Search Dense Connection Module for Super-Resolution Journal Article
In: Entropy, vol. 24, no. 4, 2022, ISSN: 1099-4300.
@article{e24040489,
title = {Automatic Search Dense Connection Module for Super-Resolution},
author = {Huaijuan Zang and Guoan Cheng and Zhipeng Duan and Ying Zhao and Shu Zhan},
url = {https://www.mdpi.com/1099-4300/24/4/489},
doi = {10.3390/e24040489},
issn = {1099-4300},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Entropy},
volume = {24},
number = {4},
abstract = {The development of display technology has continuously increased the requirements for image resolution. However, the imaging systems of many cameras are limited by their physical conditions, and the image resolution is often restrictive. Recently, several models based on deep convolutional neural network (CNN) have gained significant performance for image super-resolution (SR), while extensive memory consumption and computation overhead hinder practical applications. For this purpose, we present a lightweight network that automatically searches dense connection (ASDCN) for image super-resolution (SR), which effectively reduces redundancy in dense connection and focuses on more valuable features. We employ neural architecture search (NAS) to model the searching of dense connections. Qualitative and quantitative experiments on five public datasets show that our derived model achieves superior performance over the state-of-the-art models.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The development of display technology has continuously increased the requirements for image resolution. However, the imaging systems of many cameras are limited by their physical conditions, and the image resolution is often restrictive. Recently, several models based on deep convolutional neural network (CNN) have gained significant performance for image super-resolution (SR), while extensive memory consumption and computation overhead hinder practical applications. For this purpose, we present a lightweight network that automatically searches dense connection (ASDCN) for image super-resolution (SR), which effectively reduces redundancy in dense connection and focuses on more valuable features. We employ neural architecture search (NAS) to model the searching of dense connections. Qualitative and quantitative experiments on five public datasets show that our derived model achieves superior performance over the state-of-the-art models.
2092.
Sinha, Nilotpal; Chen, Kuan-Wen
Novelty Driven Evolutionary Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-00188,
title = {Novelty Driven Evolutionary Neural Architecture Search},
author = {Nilotpal Sinha and Kuan-Wen Chen},
url = {https://doi.org/10.48550/arXiv.2204.00188},
doi = {10.48550/arXiv.2204.00188},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.00188},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2091.
Bansal, Kanishk; Singh, Amar; Verma, Sahil; Kavita,; Jhanjhi, Noor Zaman; Shorfuzzaman, Mohammad; Masud, Mehedi
Evolving CNN with Paddy Field Algorithm for Geographical Landmark Recognition Journal Article
In: Electronics, vol. 11, no. 7, 2022, ISSN: 2079-9292.
@article{electronics11071075,
title = {Evolving CNN with Paddy Field Algorithm for Geographical Landmark Recognition},
author = {Kanishk Bansal and Amar Singh and Sahil Verma and Kavita and Noor Zaman Jhanjhi and Mohammad Shorfuzzaman and Mehedi Masud},
url = {https://www.mdpi.com/2079-9292/11/7/1075},
doi = {10.3390/electronics11071075},
issn = {2079-9292},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Electronics},
volume = {11},
number = {7},
abstract = {Convolutional Neural Networks (CNNs) operate within a wide variety of hyperparameters, the optimization of which can greatly improve the performance of CNNs when performing the task at hand. However, these hyperparameters can be very difficult to optimize, either manually or by brute force. Neural architecture search or NAS methods have been developed to address this problem and are used to find the best architectures for the deep learning paradigm. In this article, a CNN has been evolved with a well-known nature-inspired metaheuristic paddy field algorithm (PFA). It can be seen that PFA can evolve the neural architecture using the Google Landmarks Dataset V2, which is one of the toughest datasets available in the literature. The CNN’s performance, when evaluated based on the accuracy benchmark, increases from an accuracy of 0.53 to 0.76, which is an improvement of more than 40%. The evolved architecture also shows some major improvements in hyperparameters that are normally considered to be the best suited for the task.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Convolutional Neural Networks (CNNs) operate within a wide variety of hyperparameters, the optimization of which can greatly improve the performance of CNNs when performing the task at hand. However, these hyperparameters can be very difficult to optimize, either manually or by brute force. Neural architecture search or NAS methods have been developed to address this problem and are used to find the best architectures for the deep learning paradigm. In this article, a CNN has been evolved with a well-known nature-inspired metaheuristic paddy field algorithm (PFA). It can be seen that PFA can evolve the neural architecture using the Google Landmarks Dataset V2, which is one of the toughest datasets available in the literature. The CNN’s performance, when evaluated based on the accuracy benchmark, increases from an accuracy of 0.53 to 0.76, which is an improvement of more than 40%. The evolved architecture also shows some major improvements in hyperparameters that are normally considered to be the best suited for the task.
2090.
Chen, Bo; Zhao, Xiangyu; Wang, Yejing; Fan, Wenqi; Guo, Huifeng; Tang, Ruiming
Automated Machine Learning for Deep Recommender Systems: A Survey Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-01390,
title = {Automated Machine Learning for Deep Recommender Systems: A Survey},
author = {Bo Chen and Xiangyu Zhao and Yejing Wang and Wenqi Fan and Huifeng Guo and Ruiming Tang},
url = {https://doi.org/10.48550/arXiv.2204.01390},
doi = {10.48550/arXiv.2204.01390},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.01390},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2089.
Xu, Zhen; Wei, Lanning; Zhao, Huan; Ying, Rex; Yao, Quanming; Tu, Wei-Wei; Guyon, Isabelle
Bridging the Gap of AutoGraph between Academia and Industry: Analysing AutoGraph Challenge at KDD Cup 2020 Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-02625,
title = {Bridging the Gap of AutoGraph between Academia and Industry: Analysing AutoGraph Challenge at KDD Cup 2020},
author = {Zhen Xu and Lanning Wei and Huan Zhao and Rex Ying and Quanming Yao and Wei-Wei Tu and Isabelle Guyon},
url = {https://doi.org/10.48550/arXiv.2204.02625},
doi = {10.48550/arXiv.2204.02625},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.02625},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2088.
Fang, Mengjie; Tian, Jie; Dong, Di
Non-invasively predicting response to neoadjuvant chemotherapy in gastric cancer via deep learning radiomics Journal Article
In: eClinicalMedicine, vol. 46, pp. 101380, 2022, ISSN: 2589-5370.
@article{FANG2022101380,
title = {Non-invasively predicting response to neoadjuvant chemotherapy in gastric cancer via deep learning radiomics},
author = {Mengjie Fang and Jie Tian and Di Dong},
url = {https://www.sciencedirect.com/science/article/pii/S2589537022001109},
doi = {https://doi.org/10.1016/j.eclinm.2022.101380},
issn = {2589-5370},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {eClinicalMedicine},
volume = {46},
pages = {101380},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2087.
Felicetti, Matthew J.; Wang, Dianhui
Deep stochastic configuration networks with optimised model and hyper-parameters Journal Article
In: Information Sciences, vol. 600, pp. 431-441, 2022, ISSN: 0020-0255.
@article{FELICETTI2022431,
title = {Deep stochastic configuration networks with optimised model and hyper-parameters},
author = {Matthew J. Felicetti and Dianhui Wang},
url = {https://www.sciencedirect.com/science/article/pii/S0020025522003498},
doi = {https://doi.org/10.1016/j.ins.2022.04.013},
issn = {0020-0255},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Information Sciences},
volume = {600},
pages = {431-441},
abstract = {The selection of hyper-parameters in building neural networks is often left to end-users. However, this may lead to poor performance regardless of the amount of experience of the user. With deep implementation, this becomes even more challenging due to a large number of hyper-parameters. Deep stochastic configuration networks (DeepSCNs) have become increasingly popular over the past years because of their universal approximation property, fast learning and easy implementation. So far, understanding and setting the hyper-parameters for this type of network is still unexplored. This paper defines a suitable search space for DeepSCN, a performance estimation strategy for finding suitable hyper-parameters with search strategies Monte-Carlo tree search (MCTS) and random search. Simulations are performed using both searches over four benchmark datasets, and results indicate some significant improvements in modelling performance. Furthermore, a case study is presented to demonstrate that an optimised model and hyper-parameters can be found using MCTS.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The selection of hyper-parameters in building neural networks is often left to end-users. However, this may lead to poor performance regardless of the amount of experience of the user. With deep implementation, this becomes even more challenging due to a large number of hyper-parameters. Deep stochastic configuration networks (DeepSCNs) have become increasingly popular over the past years because of their universal approximation property, fast learning and easy implementation. So far, understanding and setting the hyper-parameters for this type of network is still unexplored. This paper defines a suitable search space for DeepSCN, a performance estimation strategy for finding suitable hyper-parameters with search strategies Monte-Carlo tree search (MCTS) and random search. Simulations are performed using both searches over four benchmark datasets, and results indicate some significant improvements in modelling performance. Furthermore, a case study is presented to demonstrate that an optimised model and hyper-parameters can be found using MCTS.
2086.
Qian, Guocheng; Zhang, Xuanyang; Li, Guohao; Zhao, Chen; Chen, Yukang; Zhang, Xiangyu; Ghanem, Bernard; Sun, Jian
When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-04918,
title = {When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search},
author = {Guocheng Qian and Xuanyang Zhang and Guohao Li and Chen Zhao and Yukang Chen and Xiangyu Zhang and Bernard Ghanem and Jian Sun},
url = {https://doi.org/10.48550/arXiv.2204.04918},
doi = {10.48550/arXiv.2204.04918},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.04918},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2085.
Cha, Stephen; Kim, Taehyeon; Lee, Hayeon; Yun, Se-Young
SuperNet in Neural Architecture Search: A Taxonomic Survey Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-03916,
title = {SuperNet in Neural Architecture Search: A Taxonomic Survey},
author = {Stephen Cha and Taehyeon Kim and Hayeon Lee and Se-Young Yun},
url = {https://doi.org/10.48550/arXiv.2204.03916},
doi = {10.48550/arXiv.2204.03916},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.03916},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2084.
Huang, Minbin; Huang, Zhijian; Li, Changlin; Chen, Xin; Xu, Hang; Li, Zhenguo; Liang, Xiaodan
Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-05941,
title = {Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search},
author = {Minbin Huang and Zhijian Huang and Changlin Li and Xin Chen and Hang Xu and Zhenguo Li and Xiaodan Liang},
url = {https://doi.org/10.48550/arXiv.2204.05941},
doi = {10.48550/arXiv.2204.05941},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.05941},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2083.
Shi, Min; Tang, Yufei; Zhu, Xingquan; Huang, Yu; Wilson, David; Zhuang, Yuan; Liu, Jianxun
Genetic-GNN: Evolutionary architecture search for Graph Neural Networks Journal Article
In: Knowledge-Based Systems, vol. 247, pp. 108752, 2022, ISSN: 0950-7051.
@article{SHI2022108752,
title = {Genetic-GNN: Evolutionary architecture search for Graph Neural Networks},
author = {Min Shi and Yufei Tang and Xingquan Zhu and Yu Huang and David Wilson and Yuan Zhuang and Jianxun Liu},
url = {https://www.sciencedirect.com/science/article/pii/S0950705122003525},
doi = {https://doi.org/10.1016/j.knosys.2022.108752},
issn = {0950-7051},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Knowledge-Based Systems},
volume = {247},
pages = {108752},
abstract = {Neural architecture search (NAS) has seen significant attention throughout the computational intelligence research community and has pushed forward the state-of-the-art of many neural models to address grid-like data such as texts and images. However, little work has been done on Graph Neural Network (GNN) models dedicated to unstructured network data. Given the huge number of choices and combinations of components such as aggregators and activation functions, determining the suitable GNN model for a specific problem normally necessitates tremendous expert knowledge and laborious trials. In addition, the moderate change of hyperparameters such as the learning rate and dropout rate would dramatically impact the learning capacity of a GNN model. In this paper, we propose a novel framework through the evolution of individual models in a large GNN architecture searching space. Instead of simply optimizing the model structures, an alternating evolution process is performed between GNN model structures and hyperparameters to dynamically approach the optimal fit of each other. Experiments and validations demonstrate that evolutionary NAS is capable of matching existing state-of-the-art reinforcement learning methods for both transductive and inductive graph representation learning and node classification.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) has seen significant attention throughout the computational intelligence research community and has pushed forward the state-of-the-art of many neural models to address grid-like data such as texts and images. However, little work has been done on Graph Neural Network (GNN) models dedicated to unstructured network data. Given the huge number of choices and combinations of components such as aggregators and activation functions, determining the suitable GNN model for a specific problem normally necessitates tremendous expert knowledge and laborious trials. In addition, the moderate change of hyperparameters such as the learning rate and dropout rate would dramatically impact the learning capacity of a GNN model. In this paper, we propose a novel framework through the evolution of individual models in a large GNN architecture searching space. Instead of simply optimizing the model structures, an alternating evolution process is performed between GNN model structures and hyperparameters to dynamically approach the optimal fit of each other. Experiments and validations demonstrate that evolutionary NAS is capable of matching existing state-of-the-art reinforcement learning methods for both transductive and inductive graph representation learning and node classification.
2082.
Lou, Xiaoxuan; Xu, Guowen; Chen, Kangjie; Li, Guanlin; Li, Jiwei; Zhang, Tianwei
ShiftNAS: Towards Automatic Generation of Advanced Mulitplication-Less Neural Networks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-05113,
title = {ShiftNAS: Towards Automatic Generation of Advanced Mulitplication-Less Neural Networks},
author = {Xiaoxuan Lou and Guowen Xu and Kangjie Chen and Guanlin Li and Jiwei Li and Tianwei Zhang},
url = {https://doi.org/10.48550/arXiv.2204.05113},
doi = {10.48550/arXiv.2204.05113},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.05113},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2081.
Dong, Xin; Salvo, Barbara De; Li, Meng; Liu, Chiao; Qu, Zhongnan; Kung, H. T.; Li, Ziyun
SplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-04705,
title = {SplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems},
author = {Xin Dong and Barbara De Salvo and Meng Li and Chiao Liu and Zhongnan Qu and H. T. Kung and Ziyun Li},
url = {https://doi.org/10.48550/arXiv.2204.04705},
doi = {10.48550/arXiv.2204.04705},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.04705},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2080.
Wang, Yihan; Li, Muyang; Cai, Han; Chen, Wei-Ming; Han, Song
Lite Pose: Efficient Architecture Design for 2D Human Pose Estimation Journal Article
In: CVPR2022, vol. abs/2205.0127, 2022.
@article{DBLP:journals/corr/abs-2205-01271,
title = {Lite Pose: Efficient Architecture Design for 2D Human Pose Estimation},
author = {Yihan Wang and Muyang Li and Han Cai and Wei-Ming Chen and Song Han},
url = {https://tinyml.mit.edu/wp-content/uploads/2022/04/CVPR2022__Lite_Pose.pdf},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CVPR2022},
volume = {abs/2205.0127},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2079.
Wei, Xin; Zhang, Ning; Liu, Wenchao; Chen, He
NAS-Based CNN Channel Pruning for Remote Sensing Scene Classification Journal Article
In: IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1-5, 2022.
@article{9751693,
title = {NAS-Based CNN Channel Pruning for Remote Sensing Scene Classification},
author = {Xin Wei and Ning Zhang and Wenchao Liu and He Chen},
url = {https://ieeexplore.ieee.org/abstract/document/9751693},
doi = {10.1109/LGRS.2022.3165841},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Geoscience and Remote Sensing Letters},
volume = {19},
pages = {1-5},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2078.
Li, Chao; Ning, Jia; Hu, Han; He, Kun
Enhancing the Robustness, Efficiency, and Diversity of Differentiable Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-04681,
title = {Enhancing the Robustness, Efficiency, and Diversity of Differentiable Architecture Search},
author = {Chao Li and Jia Ning and Han Hu and Kun He},
url = {https://doi.org/10.48550/arXiv.2204.04681},
doi = {10.48550/arXiv.2204.04681},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.04681},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2077.
Busia, Paola; Deriu, Gianfranco; Rinelli, Luca; Chesta, Cristina; Raffo, Luigi; Meloni, Paolo
Target-Aware Neural Architecture Search and Deployment for Keyword Spotting Journal Article
In: IEEE Access, vol. 10, pp. 40687-40700, 2022.
@article{9755991,
title = {Target-Aware Neural Architecture Search and Deployment for Keyword Spotting},
author = {Paola Busia and Gianfranco Deriu and Luca Rinelli and Cristina Chesta and Luigi Raffo and Paolo Meloni},
url = {https://ieeexplore.ieee.org/abstract/document/9755991},
doi = {10.1109/ACCESS.2022.3166939},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Access},
volume = {10},
pages = {40687-40700},
keywords = {},
pubstate = {published},
tppubtype = {article}
}