Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2023
2843.
Eisenbach, Markus; Lübberstedt, Jannik; Aganian, Dustin; Gross, Horst-Michael
A Little Bit Attention Is All You Need for Person Re-Identification Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2302-14574,
title = {A Little Bit Attention Is All You Need for Person Re-Identification},
author = {Markus Eisenbach and Jannik Lübberstedt and Dustin Aganian and Horst-Michael Gross},
url = {https://doi.org/10.48550/arXiv.2302.14574},
doi = {10.48550/arXiv.2302.14574},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2302.14574},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2842.
Hasana, Md. Mehedi; Ibrahim, Muhammad; Ali, Md. Sawkat
Speeding Up EfficientNet: Selecting Update Blocks of Convolutional Neural Networks using Genetic Algorithm in Transfer Learning Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-00261,
title = {Speeding Up EfficientNet: Selecting Update Blocks of Convolutional Neural Networks using Genetic Algorithm in Transfer Learning},
author = {Md. Mehedi Hasana and Muhammad Ibrahim and Md. Sawkat Ali},
url = {https://doi.org/10.48550/arXiv.2303.00261},
doi = {10.48550/arXiv.2303.00261},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.00261},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2841.
Lin, Shih-Ping; Wang, Sheng-De
SGAS-es: Avoiding Performance Collapse by Sequential Greedy Architecture Search with the Early Stopping Indicator Proceedings Article
In: Ärai, Kohei" (Ed.): Ädvances in Information and Communication", pp. 135–154, Springer Nature Switzerland, Cham, 2023, ISBN: 978-3-031-28073-3.
@inproceedings{10.1007/978-3-031-28073-3_10,
title = {SGAS-es: Avoiding Performance Collapse by Sequential Greedy Architecture Search with the Early Stopping Indicator},
author = {Shih-Ping Lin and Sheng-De Wang},
editor = {Kohei" Ärai},
url = {https://link.springer.com/chapter/10.1007/978-3-031-28073-3_10},
isbn = {978-3-031-28073-3},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Ädvances in Information and Communication"},
pages = {135--154},
publisher = {Springer Nature Switzerland},
address = {Cham},
abstract = {Sequential Greedy Architecture Search (SGAS) reduces the discretization loss of Differentiable Architecture Search (DARTS). However, we observed that SGAS may lead to unstable searched results as DARTS. We referred to this problem as the cascade performance collapse issue. Therefore, we proposed Sequential Greedy Architecture Search with the Early Stopping Indicator (SGAS-es). We adopted the early stopping mechanism in each phase of SGAS to stabilize searched results and further improve the searching ability. The early stopping mechanism is based on the relation among Flat Minima, the largest eigenvalue of the Hessian matrix of the loss function, and performance collapse. We devised a mathematical derivation to show the relation between Flat Minima and the largest eigenvalue. The moving averaged largest eigenvalue is used as an early stopping indicator. Finally, we used NAS-Bench-201 and Fashion-MNIST to confirm the performance and stability of SGAS-es. Moreover, we used EMNIST-Balanced to verify the transferability of searched results. These experiments show that SGAS-es is a robust method and can derive the architecture with good performance and transferability.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Sequential Greedy Architecture Search (SGAS) reduces the discretization loss of Differentiable Architecture Search (DARTS). However, we observed that SGAS may lead to unstable searched results as DARTS. We referred to this problem as the cascade performance collapse issue. Therefore, we proposed Sequential Greedy Architecture Search with the Early Stopping Indicator (SGAS-es). We adopted the early stopping mechanism in each phase of SGAS to stabilize searched results and further improve the searching ability. The early stopping mechanism is based on the relation among Flat Minima, the largest eigenvalue of the Hessian matrix of the loss function, and performance collapse. We devised a mathematical derivation to show the relation between Flat Minima and the largest eigenvalue. The moving averaged largest eigenvalue is used as an early stopping indicator. Finally, we used NAS-Bench-201 and Fashion-MNIST to confirm the performance and stability of SGAS-es. Moreover, we used EMNIST-Balanced to verify the transferability of searched results. These experiments show that SGAS-es is a robust method and can derive the architecture with good performance and transferability.
2840.
Lu, Shun; Hu, Yu; Yang, Longxing; Sun, Zihao; Mei, Jilin; Tan, Jianchao; Song, Chengru
PA&DA: Jointly Sampling PAth and DAta for Consistent NAS Conference
CVPR2023, vol. abs/2302.14772, 2023.
@conference{DBLP:journals/corr/abs-2302-14772,
title = {PA&DA: Jointly Sampling PAth and DAta for Consistent NAS},
author = {Shun Lu and Yu Hu and Longxing Yang and Zihao Sun and Jilin Mei and Jianchao Tan and Chengru Song},
url = {https://doi.org/10.48550/arXiv.2302.14772},
doi = {10.48550/arXiv.2302.14772},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {CVPR2023},
journal = {CoRR},
volume = {abs/2302.14772},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
2839.
Njor, Emil; Madsen, Jan; Fafoutis, Xenofon
Data Aware Neural Architecture Search Proceedings Article
In: Proceedings of tinyML Research Symposium, 2023, (tinyML Research Symposium’23 ; Conference date: 27-03-2023 Through 27-03-2023).
@inproceedings{40cb879dbb9a4fd9bd92ad27b617056f,
title = {Data Aware Neural Architecture Search},
author = {Emil Njor and Jan Madsen and Xenofon Fafoutis},
url = {https://orbit.dtu.dk/en/publications/data-aware-neural-architecture-search},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Proceedings of tinyML Research Symposium},
abstract = {Neural Architecture Search (NAS) is a popular tool for automatically generating Neural Network (NN) architectures. In early NAS works, these tools typically optimized NN architectures for a single metric, such as accuracy. However, in the case of resource constrained Machine Learning, one single metric is not enough to evaluate a NN architecture. For example, a NN model achieving a high accuracy is not useful if it does not fit inside the flash memory of a given system. Therefore, recent works on NAS for resource constrained systems have investigated various approaches to optimize for multiple metrics. In this paper, we propose that, on top of these approaches, it could be beneficial for NAS optimization of resource constrained systems to also consider input data granularity. We name such a system “Data Aware NAS”, and we provide experimental evidence of its benefits by comparing it to traditional NAS.},
note = {tinyML Research Symposium’23 ; Conference date: 27-03-2023 Through 27-03-2023},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural Architecture Search (NAS) is a popular tool for automatically generating Neural Network (NN) architectures. In early NAS works, these tools typically optimized NN architectures for a single metric, such as accuracy. However, in the case of resource constrained Machine Learning, one single metric is not enough to evaluate a NN architecture. For example, a NN model achieving a high accuracy is not useful if it does not fit inside the flash memory of a given system. Therefore, recent works on NAS for resource constrained systems have investigated various approaches to optimize for multiple metrics. In this paper, we propose that, on top of these approaches, it could be beneficial for NAS optimization of resource constrained systems to also consider input data granularity. We name such a system “Data Aware NAS”, and we provide experimental evidence of its benefits by comparing it to traditional NAS.
2838.
Wang, Kun; Han, Ling; Li, Liangzhi
A decoupled search deep network framework for high-resolution remote sensing image classification Journal Article
In: Remote Sensing Letters, vol. 14, no. 3, pp. 243-253, 2023.
@article{doi:10.1080/2150704X.2023.2185110,
title = {A decoupled search deep network framework for high-resolution remote sensing image classification},
author = {Kun Wang and Ling Han and Liangzhi Li},
url = {https://doi.org/10.1080/2150704X.2023.2185110},
doi = {10.1080/2150704X.2023.2185110},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Remote Sensing Letters},
volume = {14},
number = {3},
pages = {243-253},
publisher = {Taylor & Francis},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2837.
Xue, Yu; Chen, Chen; Słowik, Adam
Neural Architecture Search Based on A Multi-objective Evolutionary Algorithm With Probability Stack Journal Article
In: IEEE Transactions on Evolutionary Computation, pp. 1-1, 2023.
@article{10059145,
title = {Neural Architecture Search Based on A Multi-objective Evolutionary Algorithm With Probability Stack},
author = {Yu Xue and Chen Chen and Adam Słowik},
doi = {10.1109/TEVC.2023.3252612},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Transactions on Evolutionary Computation},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2836.
Cao, Chunhong; Xiang, Han; Song, Wei; Yi, Hongbo; Xiao, Fen; Gao, Xieping
Lightweight Multiscale Neural Architecture Search With Spectral–Spatial Attention for Hyperspectral Image Classification Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-15, 2023.
@article{10061276,
title = {Lightweight Multiscale Neural Architecture Search With Spectral–Spatial Attention for Hyperspectral Image Classification},
author = {Chunhong Cao and Han Xiang and Wei Song and Hongbo Yi and Fen Xiao and Xieping Gao},
doi = {10.1109/TGRS.2023.3253247},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
volume = {61},
pages = {1-15},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2835.
Kus, Zeki; Akkan, Can; Gülcü, Ayla
Novel Surrogate Measures Based on a Similarity Network for Neural Architecture Search Journal Article
In: IEEE Access, vol. 11, pp. 22596–22613, 2023.
@article{DBLP:journals/access/KusAG23,
title = {Novel Surrogate Measures Based on a Similarity Network for Neural Architecture Search},
author = {Zeki Kus and Can Akkan and Ayla Gülcü},
url = {https://doi.org/10.1109/ACCESS.2023.3252887},
doi = {10.1109/ACCESS.2023.3252887},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Access},
volume = {11},
pages = {22596--22613},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2834.
Cereda, Elia; Crupi, Luca; Risso, Matteo; Burrello, Alessio; Benini, Luca; Giusti, Alessandro; Pagliari, Daniele Jahier; Palossi, Daniele
Deep Neural Network Architecture Search for Accurate Visual Pose Estimation aboard Nano-UAVs Proceedings Article
In: IEEE ICRA 2023, 2023.
@inproceedings{DBLP:journals/corr/abs-2303-01931,
title = {Deep Neural Network Architecture Search for Accurate Visual Pose Estimation aboard Nano-UAVs},
author = {Elia Cereda and Luca Crupi and Matteo Risso and Alessio Burrello and Luca Benini and Alessandro Giusti and Daniele Jahier Pagliari and Daniele Palossi},
url = {https://doi.org/10.48550/arXiv.2303.01931},
doi = {10.48550/arXiv.2303.01931},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {IEEE ICRA 2023},
journal = {CoRR},
volume = {abs/2303.01931},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2833.
Ke, Songyu; Pan, Zheyi; He, Tianfu; Liang, Yuxuan; Zhang, Junbo; Zheng, Yu
AutoSTG+: An automatic framework to discover the optimal network for spatio-temporal graph prediction Journal Article
In: Artificial Intelligence, vol. 318, pp. 103899, 2023, ISSN: 0004-3702.
@article{KE2023103899,
title = {AutoSTG+: An automatic framework to discover the optimal network for spatio-temporal graph prediction},
author = {Songyu Ke and Zheyi Pan and Tianfu He and Yuxuan Liang and Junbo Zhang and Yu Zheng},
url = {https://www.sciencedirect.com/science/article/pii/S0004370223000450},
doi = {https://doi.org/10.1016/j.artint.2023.103899},
issn = {0004-3702},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Artificial Intelligence},
volume = {318},
pages = {103899},
abstract = {Spatio-temporal graphs (STGs) are important structures to describe urban sensory data, e.g., traffic speed and air quality. Predicting over spatio-temporal graphs enables many essential applications in intelligent cities, such as traffic management and environment analysis. Recently, many deep learning models have been proposed for spatio-temporal graph prediction and achieved significant results. However, manually designing neural networks requires rich domain knowledge and heavy expert efforts, making it impractical for real-world deployments. Therefore, we study automated neural architecture search for spatio-temporal graphs, which meets three challenges: 1) how to define search space for capturing complex spatio-temporal correlations; 2) how to jointly model the explicit and implicit relationships between nodes of an STG; and 3) how to learn network weight parameters related to meta graphs of STGs. To tackle these challenges, we propose a novel neural architecture search framework, entitled AutoSTG+, for automated spatio-temporal graph prediction. In our AutoSTG+, spatial graph convolution and temporal convolution operations are adopted in the search space of AutoSTG+ to capture complex spatio-temporal correlations. Besides, we propose to employ the meta-learning technique to learn the adjacency matrices of spatial graph convolution layers and kernels of temporal convolution layers from the meta knowledge of meta graphs. And specifically, such meta-knowledge is learned by graph meta-knowledge learners, which iteratively aggregate knowledge on the attributed graphs and the similarity graphs. Finally, extensive experiments have been conducted on multiple real-world datasets to demonstrate that AutoSTG+ can find effective network architectures and achieve up to about 20% relative improvements compared to human-designed networks.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Spatio-temporal graphs (STGs) are important structures to describe urban sensory data, e.g., traffic speed and air quality. Predicting over spatio-temporal graphs enables many essential applications in intelligent cities, such as traffic management and environment analysis. Recently, many deep learning models have been proposed for spatio-temporal graph prediction and achieved significant results. However, manually designing neural networks requires rich domain knowledge and heavy expert efforts, making it impractical for real-world deployments. Therefore, we study automated neural architecture search for spatio-temporal graphs, which meets three challenges: 1) how to define search space for capturing complex spatio-temporal correlations; 2) how to jointly model the explicit and implicit relationships between nodes of an STG; and 3) how to learn network weight parameters related to meta graphs of STGs. To tackle these challenges, we propose a novel neural architecture search framework, entitled AutoSTG+, for automated spatio-temporal graph prediction. In our AutoSTG+, spatial graph convolution and temporal convolution operations are adopted in the search space of AutoSTG+ to capture complex spatio-temporal correlations. Besides, we propose to employ the meta-learning technique to learn the adjacency matrices of spatial graph convolution layers and kernels of temporal convolution layers from the meta knowledge of meta graphs. And specifically, such meta-knowledge is learned by graph meta-knowledge learners, which iteratively aggregate knowledge on the attributed graphs and the similarity graphs. Finally, extensive experiments have been conducted on multiple real-world datasets to demonstrate that AutoSTG+ can find effective network architectures and achieve up to about 20% relative improvements compared to human-designed networks.
2832.
Brown, Austin; Gupta, Maanak; Abdelsalam, Mahmoud
Automated Machine Learning for Deep Learning based Malware Detection Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-01679,
title = {Automated Machine Learning for Deep Learning based Malware Detection},
author = {Austin Brown and Maanak Gupta and Mahmoud Abdelsalam},
url = {https://doi.org/10.48550/arXiv.2303.01679},
doi = {10.48550/arXiv.2303.01679},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.01679},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2831.
Santana, Roberto; Hidalgo-Cenalmor, Ivan; Garciarena, Unai; Mendiburu, Alexander; Lozano, José Antonio
Neuroevolutionary algorithms driven by neuron coverage metrics for semi-supervised classification Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-02801,
title = {Neuroevolutionary algorithms driven by neuron coverage metrics for semi-supervised classification},
author = {Roberto Santana and Ivan Hidalgo-Cenalmor and Unai Garciarena and Alexander Mendiburu and José Antonio Lozano},
url = {https://doi.org/10.48550/arXiv.2303.02801},
doi = {10.48550/arXiv.2303.02801},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.02801},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2830.
Lee, Jong-Ryul; Moon, Yong-Hyuk
Bespoke: A Block-Level Neural Network Optimization Framework for Low-Cost Deployment Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-01913,
title = {Bespoke: A Block-Level Neural Network Optimization Framework for Low-Cost Deployment},
author = {Jong-Ryul Lee and Yong-Hyuk Moon},
url = {https://doi.org/10.48550/arXiv.2303.01913},
doi = {10.48550/arXiv.2303.01913},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.01913},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2829.
Shen, Xuan; Wang, Yaohua; Lin, Ming; Huang, Yilun; Tang, Hao; Sun, Xiuyu; Wang, Yanzhi
DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-02165,
title = {DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network},
author = {Xuan Shen and Yaohua Wang and Ming Lin and Yilun Huang and Hao Tang and Xiuyu Sun and Yanzhi Wang},
url = {https://doi.org/10.48550/arXiv.2303.02165},
doi = {10.48550/arXiv.2303.02165},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.02165},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2828.
Ali, Mohamed Nabih; Paissan, Francesco; Falavigna, Daniele; Brutti, Alessio
Scaling strategies for on-device low-complexity source separation with Conv-Tasnet Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-03005,
title = {Scaling strategies for on-device low-complexity source separation with Conv-Tasnet},
author = {Mohamed Nabih Ali and Francesco Paissan and Daniele Falavigna and Alessio Brutti},
url = {https://doi.org/10.48550/arXiv.2303.03005},
doi = {10.48550/arXiv.2303.03005},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.03005},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2827.
Chitty-Venkata, Krishna Teja; Emani, Murali; Vishwanath, Venkatram; Somani, Arun K.
Neural Architecture Search Benchmarks: Insights and Survey Journal Article
In: IEEE Access, vol. 11, pp. 25217–25236, 2023.
@article{DBLP:journals/access/ChittyVenkataEVS23,
title = {Neural Architecture Search Benchmarks: Insights and Survey},
author = {Krishna Teja Chitty-Venkata and Murali Emani and Venkatram Vishwanath and Arun K. Somani},
url = {https://doi.org/10.1109/ACCESS.2023.3253818},
doi = {10.1109/ACCESS.2023.3253818},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Access},
volume = {11},
pages = {25217--25236},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2826.
Qin, Dalin; Wang, Chenxi; Wen, Qingsong; Chen, Weiqi; Sun, Liang; Wang, Yi
Personalized Federated DARTS for Electricity Load Forecasting of Individual Buildings Journal Article
In: IEEE Transactions on Smart Grid, pp. 1-1, 2023.
@article{10063999,
title = {Personalized Federated DARTS for Electricity Load Forecasting of Individual Buildings},
author = {Dalin Qin and Chenxi Wang and Qingsong Wen and Weiqi Chen and Liang Sun and Yi Wang},
doi = {10.1109/TSG.2023.3253855},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Transactions on Smart Grid},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2825.
Alaiad, Ahmad; Migdady, Aya; Al-Khatib, Ra’ed M.; Alzoubi, Omar; Zitar, Raed Abu; Abualigah, Laith
Autokeras Approach: A Robust Automated Deep Learning Network for Diagnosis Disease Cases in Medical Images Journal Article
In: Journal of Imaging, vol. 9, no. 3, 2023, ISSN: 2313-433X.
@article{jimaging9030064,
title = {Autokeras Approach: A Robust Automated Deep Learning Network for Diagnosis Disease Cases in Medical Images},
author = {Ahmad Alaiad and Aya Migdady and Ra’ed M. Al-Khatib and Omar Alzoubi and Raed Abu Zitar and Laith Abualigah},
url = {https://www.mdpi.com/2313-433X/9/3/64},
doi = {10.3390/jimaging9030064},
issn = {2313-433X},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Journal of Imaging},
volume = {9},
number = {3},
abstract = {Automated deep learning is promising in artificial intelligence (AI). However, a few applications of automated deep learning networks have been made in the clinical medical fields. Therefore, we studied the application of an open-source automated deep learning framework, Autokeras, for detecting smear blood images infected with malaria parasites. Autokeras is able to identify the optimal neural network to perform the classification task. Hence, the robustness of the adopted model is due to it not needing any prior knowledge from deep learning. In contrast, the traditional deep neural network methods still require more construction to identify the best convolutional neural network (CNN). The dataset used in this study consisted of 27,558 blood smear images. A comparative process proved the superiority of our proposed approach over other traditional neural networks. The evaluation results of our proposed model achieved high efficiency with impressive accuracy, reaching 95.6% when compared with previous competitive models.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Automated deep learning is promising in artificial intelligence (AI). However, a few applications of automated deep learning networks have been made in the clinical medical fields. Therefore, we studied the application of an open-source automated deep learning framework, Autokeras, for detecting smear blood images infected with malaria parasites. Autokeras is able to identify the optimal neural network to perform the classification task. Hence, the robustness of the adopted model is due to it not needing any prior knowledge from deep learning. In contrast, the traditional deep neural network methods still require more construction to identify the best convolutional neural network (CNN). The dataset used in this study consisted of 27,558 blood smear images. A comparative process proved the superiority of our proposed approach over other traditional neural networks. The evaluation results of our proposed model achieved high efficiency with impressive accuracy, reaching 95.6% when compared with previous competitive models.
2824.
Wang, Zhipeng; Liu, Rengkui; Gao, Yi; Tang, Yuanjie
Metro Track Geometry Defect Identification Model Based on Car-Body Vibration Data and Differentiable Architecture Search Journal Article
In: Applied Sciences, vol. 13, no. 6, 2023, ISSN: 2076-3417.
@article{app13063457,
title = {Metro Track Geometry Defect Identification Model Based on Car-Body Vibration Data and Differentiable Architecture Search},
author = {Zhipeng Wang and Rengkui Liu and Yi Gao and Yuanjie Tang},
url = {https://www.mdpi.com/2076-3417/13/6/3457},
doi = {10.3390/app13063457},
issn = {2076-3417},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Applied Sciences},
volume = {13},
number = {6},
abstract = {Efficient and low-cost modes for detecting metro track geometry defects (TGDs) are essential for condition-prediction-based preventive maintenance, which can help improve the safety of metro operations and reduce the maintenance cost of metro tracks. Compared with the traditional TGD detection method that utilizes the track geometry car, the method that uses a portable detector to acquire the car-body vibration data (CVD) can be used on an ordinary in-service train without occupying the metro schedule line, thereby improving efficiency and reducing the cost. A convolutional neural network-based identification model for TGD, built on a differentiable architecture search, is proposed in this study to employ only the CVD acquired by a portable detector for integrated identification of the type and severity level of TGDs. Second, the random oversampling method is introduced, and a strategy for applying this method is proposed to improve the poor training effect of the model caused by the natural class-imbalance problem arising from the TGD dataset. Subsequently, a comprehensive performance-evaluation metric (track geometry defect F-score) is designed by considering the actual management needs of the metro infrastructure. Finally, a case study is conducted using actual field data collected from Beijing Subway to validate the proposed model.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Efficient and low-cost modes for detecting metro track geometry defects (TGDs) are essential for condition-prediction-based preventive maintenance, which can help improve the safety of metro operations and reduce the maintenance cost of metro tracks. Compared with the traditional TGD detection method that utilizes the track geometry car, the method that uses a portable detector to acquire the car-body vibration data (CVD) can be used on an ordinary in-service train without occupying the metro schedule line, thereby improving efficiency and reducing the cost. A convolutional neural network-based identification model for TGD, built on a differentiable architecture search, is proposed in this study to employ only the CVD acquired by a portable detector for integrated identification of the type and severity level of TGDs. Second, the random oversampling method is introduced, and a strategy for applying this method is proposed to improve the poor training effect of the model caused by the natural class-imbalance problem arising from the TGD dataset. Subsequently, a comprehensive performance-evaluation metric (track geometry defect F-score) is designed by considering the actual management needs of the metro infrastructure. Finally, a case study is conducted using actual field data collected from Beijing Subway to validate the proposed model.
2823.
Mecharbat, Lotfi Abdelkrim; Benmeziane, Hadjer; Ouranoughi, Hamza; Niar, Smaïl
HyT-NAS: Hybrid Transformers Neural Architecture Search for Edge Devices Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-04440b,
title = {HyT-NAS: Hybrid Transformers Neural Architecture Search for Edge Devices},
author = {Lotfi Abdelkrim Mecharbat and Hadjer Benmeziane and Hamza Ouranoughi and Smaïl Niar},
url = {https://doi.org/10.48550/arXiv.2303.04440},
doi = {10.48550/arXiv.2303.04440},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.04440},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2822.
Wendlinger, Lorenz; Granitzer, Michael; Fellicious, Christofer
Pooling Graph Convolutional Networks for Structural Performance Prediction Proceedings Article
In: Nicosia, Giuseppe; Ojha, Varun; Malfa, Emanuele La; Malfa, Gabriele La; Pardalos, Panos; Fatta, Giuseppe Di; Giuffrida, Giovanni; Umeton, Renato (Ed.): Machine Learning, Optimization, and Data Science, pp. 1–16, Springer Nature Switzerland, Cham, 2023, ISBN: 978-3-031-25891-6.
@inproceedings{10.1007/978-3-031-25891-6_1,
title = {Pooling Graph Convolutional Networks for Structural Performance Prediction},
author = {Lorenz Wendlinger and Michael Granitzer and Christofer Fellicious},
editor = {Giuseppe Nicosia and Varun Ojha and Emanuele La Malfa and Gabriele La Malfa and Panos Pardalos and Giuseppe Di Fatta and Giovanni Giuffrida and Renato Umeton},
url = {https://link.springer.com/chapter/10.1007/978-3-031-25891-6_1},
isbn = {978-3-031-25891-6},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Machine Learning, Optimization, and Data Science},
pages = {1--16},
publisher = {Springer Nature Switzerland},
address = {Cham},
abstract = {Neural Architecture Search can help in finding high-performance task specific neural network architectures. However, the training of architectures that is required for fitness computation can be prohibitively expensive. Employing surrogate models as performance predictors can reduce or remove the need for these costly evaluations. We present a deep graph learning approach that achieves state-of-the-art performance in multiple NAS performance prediction benchmarks. In contrast to other methods, this model is purely supervised, which can be a methodologic advantage, as it does not rely on unlabeled instances sampled from complex search spaces.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural Architecture Search can help in finding high-performance task specific neural network architectures. However, the training of architectures that is required for fitness computation can be prohibitively expensive. Employing surrogate models as performance predictors can reduce or remove the need for these costly evaluations. We present a deep graph learning approach that achieves state-of-the-art performance in multiple NAS performance prediction benchmarks. In contrast to other methods, this model is purely supervised, which can be a methodologic advantage, as it does not rely on unlabeled instances sampled from complex search spaces.
2821.
Wen, Hao; Li, Yuanchun; Zhang, Zunshuai; Jiang, Shiqi; Ye, Xiaozhou; Ouyang, Ye; Zhang, Ya-Qin; Liu, Yunxin
AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-07129,
title = {AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments},
author = {Hao Wen and Yuanchun Li and Zunshuai Zhang and Shiqi Jiang and Xiaozhou Ye and Ye Ouyang and Ya-Qin Zhang and Yunxin Liu},
url = {https://doi.org/10.48550/arXiv.2303.07129},
doi = {10.48550/arXiv.2303.07129},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.07129},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2820.
Gu, Jianyang; Wang, Kai; Luo, Hao; Chen, Chen; Jiang, Wei; Fang, Yuqiang; Zhang, Shanghang; You, Yang; Zhao, Jian
MSINet: Twins Contrastive Search of Multi-Scale Interaction for Object ReID Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-07065,
title = {MSINet: Twins Contrastive Search of Multi-Scale Interaction for Object ReID},
author = {Jianyang Gu and Kai Wang and Hao Luo and Chen Chen and Wei Jiang and Yuqiang Fang and Shanghang Zhang and Yang You and Jian Zhao},
url = {https://doi.org/10.48550/arXiv.2303.07065},
doi = {10.48550/arXiv.2303.07065},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.07065},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2819.
Choi, Wonhyeok; Im, Sunghoon
Dynamic Neural Network for Multi-Task Learning Searching across Diverse Network Topologies Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-06856,
title = {Dynamic Neural Network for Multi-Task Learning Searching across Diverse Network Topologies},
author = {Wonhyeok Choi and Sunghoon Im},
url = {https://doi.org/10.48550/arXiv.2303.06856},
doi = {10.48550/arXiv.2303.06856},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.06856},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2818.
Zhang, Li Lyna; Wang, Xudong; Xu, Jiahang; Zhang, Quanlu; Wang, Yujing; Yang, Yuqing; Zheng, Ningxin; Cao, Ting; Yang, Mao
SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-08308,
title = {SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference},
author = {Li Lyna Zhang and Xudong Wang and Jiahang Xu and Quanlu Zhang and Yujing Wang and Yuqing Yang and Ningxin Zheng and Ting Cao and Mao Yang},
url = {https://doi.org/10.48550/arXiv.2303.08308},
doi = {10.48550/arXiv.2303.08308},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.08308},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2817.
Yang, Yuguang; Pan, Yu; Yin, Jingjing; Han, Jiangyu; Ma, Lei; Lu, Heng
HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-08636,
title = {HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism},
author = {Yuguang Yang and Yu Pan and Jingjing Yin and Jiangyu Han and Lei Ma and Heng Lu},
url = {https://doi.org/10.48550/arXiv.2303.08636},
doi = {10.48550/arXiv.2303.08636},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.08636},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2816.
Savadikar, Chinmay; Dai, Michelle; Wu, Tianfu
Learning to Grow Artificial Hippocampi in Vision Transformers for Resilient Lifelong Learning Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-08250,
title = {Learning to Grow Artificial Hippocampi in Vision Transformers for Resilient Lifelong Learning},
author = {Chinmay Savadikar and Michelle Dai and Tianfu Wu},
url = {https://doi.org/10.48550/arXiv.2303.08250},
doi = {10.48550/arXiv.2303.08250},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.08250},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2815.
Hamid, Saad; Wan, Xingchen; Jørgensen, Martin; Ru, Binxin; Osborne, Michael A.
Bayesian Quadrature for Neural Ensemble Search Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-08874,
title = {Bayesian Quadrature for Neural Ensemble Search},
author = {Saad Hamid and Xingchen Wan and Martin Jørgensen and Binxin Ru and Michael A. Osborne},
url = {https://doi.org/10.48550/arXiv.2303.08874},
doi = {10.48550/arXiv.2303.08874},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.08874},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2814.
Lyu, Bo; Wen, Shiping; Yang, Yin; Chang, Xiaojun; Sun, Junwei; Chen, Yiran; Huang, Tingwen
Designing Efficient Bit-Level Sparsity-Tolerant Memristive Networks Journal Article
In: IEEE Transactions on Neural Networks and Learning Systems, pp. 1-10, 2023.
@article{10075408,
title = {Designing Efficient Bit-Level Sparsity-Tolerant Memristive Networks},
author = {Bo Lyu and Shiping Wen and Yin Yang and Xiaojun Chang and Junwei Sun and Yiran Chen and Tingwen Huang},
doi = {10.1109/TNNLS.2023.3250437},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Transactions on Neural Networks and Learning Systems},
pages = {1-10},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2813.
Montes, Charles; Morehouse, Todd; Kasilingam, Dayalan; Zhou, Ruolin
Optimized CNN Auto-Generator Using GA With Stopping Criterion: Design and a Use Case Proceedings Article
In: 2023 IEEE 20th Consumer Communications & Networking Conference (CCNC), pp. 1009-1014, 2023.
@inproceedings{10060860,
title = {Optimized CNN Auto-Generator Using GA With Stopping Criterion: Design and a Use Case},
author = {Charles Montes and Todd Morehouse and Dayalan Kasilingam and Ruolin Zhou},
doi = {10.1109/CCNC51644.2023.10060860},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {2023 IEEE 20th Consumer Communications & Networking Conference (CCNC)},
pages = {1009-1014},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2812.
Trivedi, Aashka; Udagawa, Takuma; Merler, Michele; Panda, Rameswar; El-Kurdi, Yousef; Bhattacharjee, Bishwaranjan
Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-09639,
title = {Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models},
author = {Aashka Trivedi and Takuma Udagawa and Michele Merler and Rameswar Panda and Yousef El-Kurdi and Bishwaranjan Bhattacharjee},
url = {https://doi.org/10.48550/arXiv.2303.09639},
doi = {10.48550/arXiv.2303.09639},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.09639},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2811.
Li, Chaojian; Chen, Wenwan; Yuan, Jiayi; Lin, Yingyan; Sabharwal, Ashutosh
ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-10727,
title = {ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement},
author = {Chaojian Li and Wenwan Chen and Jiayi Yuan and Yingyan Lin and Ashutosh Sabharwal},
url = {https://doi.org/10.48550/arXiv.2303.10727},
doi = {10.48550/arXiv.2303.10727},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.10727},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2810.
Zhang, Li Lyna; Wang, Xudong; Xu, Jiahang; Zhang, Quanlu; Wang, Yujing; Yang, Yuqing; Zheng, Ningxin; Cao, Ting; Yang, Mao
SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-08308b,
title = {SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference},
author = {Li Lyna Zhang and Xudong Wang and Jiahang Xu and Quanlu Zhang and Yujing Wang and Yuqing Yang and Ningxin Zheng and Ting Cao and Mao Yang},
url = {https://doi.org/10.48550/arXiv.2303.08308},
doi = {10.48550/arXiv.2303.08308},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.08308},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2809.
Zhou, Ao; Yang, Jianlei; Qi, Yingjie; Shi, Yumeng; Qiao, Tong; Zhao, Weisheng; Hu, Chunming
Hardware-Aware Graph Neural Network Automated Design for Edge Computing Platforms Journal Article
In: CoRR, vol. abs/2303.10875, 2023.
@article{DBLP:journals/corr/abs-2303-10875,
title = {Hardware-Aware Graph Neural Network Automated Design for Edge Computing Platforms},
author = {Ao Zhou and Jianlei Yang and Yingjie Qi and Yumeng Shi and Tong Qiao and Weisheng Zhao and Chunming Hu},
url = {https://doi.org/10.48550/arXiv.2303.10875},
doi = {10.48550/arXiv.2303.10875},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.10875},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2808.
Lin, Guan-Ting; Tang, Qingming; Kao, Chieh-Chi; Rozgic, Viktor; Wang, Chao
Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-10351,
title = {Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints},
author = {Guan-Ting Lin and Qingming Tang and Chieh-Chi Kao and Viktor Rozgic and Chao Wang},
url = {https://doi.org/10.48550/arXiv.2303.10351},
doi = {10.48550/arXiv.2303.10351},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.10351},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2807.
Tang, Chen; Zhang, Li Lyna; Jiang, Huiqiang; Xu, Jiahang; Cao, Ting; Zhang, Quanlu; Yang, Yuqing; Wang, Zhi; Yang, Mao
ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devices Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-09730,
title = {ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devices},
author = {Chen Tang and Li Lyna Zhang and Huiqiang Jiang and Jiahang Xu and Ting Cao and Quanlu Zhang and Yuqing Yang and Zhi Wang and Mao Yang},
url = {https://doi.org/10.48550/arXiv.2303.09730},
doi = {10.48550/arXiv.2303.09730},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.09730},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2806.
Shi, Huihong; You, Haoran; Wang, Zhongfeng; Lin, Yingyan
NASA $+$ : Neural Architecture Search and Acceleration for Multiplication-Reduced Hybrid Networks Journal Article
In: IEEE Transactions on Circuits and Systems I: Regular Papers, pp. 1-14, 2023.
@article{10078392,
title = {NASA $+$ : Neural Architecture Search and Acceleration for Multiplication-Reduced Hybrid Networks},
author = {Huihong Shi and Haoran You and Zhongfeng Wang and Yingyan Lin},
url = {https://ieeexplore.ieee.org/abstract/document/10078392},
doi = {10.1109/TCSI.2023.3256700},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Transactions on Circuits and Systems I: Regular Papers},
pages = {1-14},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2805.
Li, Sheng; Andersen, Garrett; Chen, Tao; Cheng, Liqun; Grady, Julian; Huang, Da; Le, Quoc V.; Li, Andrew; Li, Xin; Li, Yang; Liang, Chen; Lu, Yifeng; Ni, Yun; Pang, Ruoming; Tan, Mingxing; Wicke, Martin; Wu, Gang; Zhu, Shengqi; Ranganathan, Parthasarathy; Jouppi, Norman P.
Hyperscale Hardware Optimized Neural Architecture Search Proceedings Article
In: Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 343–358, Association for Computing Machinery, Vancouver, BC, Canada, 2023, ISBN: 9781450399180.
@inproceedings{10.1145/3582016.3582049,
title = {Hyperscale Hardware Optimized Neural Architecture Search},
author = {Sheng Li and Garrett Andersen and Tao Chen and Liqun Cheng and Julian Grady and Da Huang and Quoc V. Le and Andrew Li and Xin Li and Yang Li and Chen Liang and Yifeng Lu and Yun Ni and Ruoming Pang and Mingxing Tan and Martin Wicke and Gang Wu and Shengqi Zhu and Parthasarathy Ranganathan and Norman P. Jouppi},
url = {https://doi.org/10.1145/3582016.3582049},
doi = {10.1145/3582016.3582049},
isbn = {9781450399180},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3},
pages = {343–358},
publisher = {Association for Computing Machinery},
address = {Vancouver, BC, Canada},
series = {ASPLOS 2023},
abstract = {Recent advances in machine learning have leveraged dramatic increases in computational power, a trend expected to continue in the future. This paper introduces the first Hyperscale Hardware Optimized Neural Architecture Search (H2O-NAS) to automatically design accurate and performant machine learning models tailored to the underlying hardware architecture. H2O-NAS consists of three key components: a new massively parallel “one-shot” search algorithm with intelligent weight sharing, which can scale to search spaces of O(10280) and handle large volumes of production traffic; hardware-optimized search spaces for diverse ML models on heterogeneous hardware; and a novel two-phase hybrid performance model and a multi-objective reward function optimized for large scale deployments. H2O-NAS has been implemented around state-of-the-art machine learning models (e.g. convolutional models, vision transformers, and deep learning recommendation models) and deployed at zettaflop scale in production. Our results demonstrate significant improvements in performance (22% ∼ 56%) and energy efficiency (17% ∼25%) at same or better quality. Our solution is designed for largescale deployment, streamlining privacy and security processes and reducing manual overhead. This facilitates a smooth and automated transition from research to production.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent advances in machine learning have leveraged dramatic increases in computational power, a trend expected to continue in the future. This paper introduces the first Hyperscale Hardware Optimized Neural Architecture Search (H2O-NAS) to automatically design accurate and performant machine learning models tailored to the underlying hardware architecture. H2O-NAS consists of three key components: a new massively parallel “one-shot” search algorithm with intelligent weight sharing, which can scale to search spaces of O(10280) and handle large volumes of production traffic; hardware-optimized search spaces for diverse ML models on heterogeneous hardware; and a novel two-phase hybrid performance model and a multi-objective reward function optimized for large scale deployments. H2O-NAS has been implemented around state-of-the-art machine learning models (e.g. convolutional models, vision transformers, and deep learning recommendation models) and deployed at zettaflop scale in production. Our results demonstrate significant improvements in performance (22% ∼ 56%) and energy efficiency (17% ∼25%) at same or better quality. Our solution is designed for largescale deployment, streamlining privacy and security processes and reducing manual overhead. This facilitates a smooth and automated transition from research to production.
2804.
Yin, Shibai; Hu, Shuhao; Wang, Yibin; Yang, Yee-Hong
High-order Adams Network (HIAN) for image dehazing Journal Article
In: Applied Soft Computing, vol. 139, pp. 110204, 2023, ISSN: 1568-4946.
@article{YIN2023110204,
title = {High-order Adams Network (HIAN) for image dehazing},
author = {Shibai Yin and Shuhao Hu and Yibin Wang and Yee-Hong Yang},
url = {https://www.sciencedirect.com/science/article/pii/S1568494623002223},
doi = {https://doi.org/10.1016/j.asoc.2023.110204},
issn = {1568-4946},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Applied Soft Computing},
volume = {139},
pages = {110204},
abstract = {Convolutional Neural Networks (CNN) are widely used in image dehazing. However, existing network frameworks are built based on manual design from practical experience, lacking interpretable result or theoretical guidelines. Recently, residual networks are regarded as the explicit Euler forward approximation of the ODE (Ordinary Differential Equation), and several ODE-inspired networks are proposed based on the low-order explicit Euler schemes. However, on the issues of system stability and training convergence, high-order Implicit Adams Predictor–Corrector (IAPC) methods have proven to be better than low-order explicit Euler methods. Hence, we extend the IAPC method to the High-order Implicit Adams Network (HIAN). To do so, we design a series of Implicit Adams Predictor–Corrector Blocks (IABs) based on the high-order IAPC methods, all of which give better stability and accuracy than the ones designed using the low-order Euler methods. Given that, we further propose the Implicit Adams Predictor–Corrector Module (IAM) by combining the Non-local Sparse Attention (NSA) and Attention Feature Fusion (AFF) with stacked IABs where the NSA explores the mutual-correlation among intermediate features with low computation cost via a sparse constraint, while the AFF fuses intermediate features by reweighting the features from stacked IABs adaptively. Moreover, because manual network design with IABs limits dehazing performance, the Neural Architecture Search (NAS) is used to find an optimal architecture automatically. This resulting design not only is interpretable for image dehazing but also provides a reliable guideline on future network designs. The experiments demonstrate that the proposed method outperforms most existing methods on both synthetic and real images.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Convolutional Neural Networks (CNN) are widely used in image dehazing. However, existing network frameworks are built based on manual design from practical experience, lacking interpretable result or theoretical guidelines. Recently, residual networks are regarded as the explicit Euler forward approximation of the ODE (Ordinary Differential Equation), and several ODE-inspired networks are proposed based on the low-order explicit Euler schemes. However, on the issues of system stability and training convergence, high-order Implicit Adams Predictor–Corrector (IAPC) methods have proven to be better than low-order explicit Euler methods. Hence, we extend the IAPC method to the High-order Implicit Adams Network (HIAN). To do so, we design a series of Implicit Adams Predictor–Corrector Blocks (IABs) based on the high-order IAPC methods, all of which give better stability and accuracy than the ones designed using the low-order Euler methods. Given that, we further propose the Implicit Adams Predictor–Corrector Module (IAM) by combining the Non-local Sparse Attention (NSA) and Attention Feature Fusion (AFF) with stacked IABs where the NSA explores the mutual-correlation among intermediate features with low computation cost via a sparse constraint, while the AFF fuses intermediate features by reweighting the features from stacked IABs adaptively. Moreover, because manual network design with IABs limits dehazing performance, the Neural Architecture Search (NAS) is used to find an optimal architecture automatically. This resulting design not only is interpretable for image dehazing but also provides a reliable guideline on future network designs. The experiments demonstrate that the proposed method outperforms most existing methods on both synthetic and real images.
2803.
Zhou, Yuan; Wang, Haiyang; Huo, Shuwei; Wang, Boyu
Hierarchical full-attention neural architecture search based on search space compression Journal Article
In: Knowledge-Based Systems, vol. 269, pp. 110507, 2023, ISSN: 0950-7051.
@article{ZHOU2023110507,
title = {Hierarchical full-attention neural architecture search based on search space compression},
author = {Yuan Zhou and Haiyang Wang and Shuwei Huo and Boyu Wang},
url = {https://www.sciencedirect.com/science/article/pii/S0950705123002575},
doi = {https://doi.org/10.1016/j.knosys.2023.110507},
issn = {0950-7051},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Knowledge-Based Systems},
volume = {269},
pages = {110507},
abstract = {Neural architecture search (NAS) has significantly advanced the automatic design of convolutional neural architectures. However, it is challenging to directly extend existing NAS methods to attention networks because of the uniform structure of the search space and the lack of long-range feature extraction. To address these issues, we construct a hierarchical search space that allows various attention operations to be adopted for different layers of a network. To reduce the complexity of the search, a low-cost search space compression method is proposed to automatically remove the unpromising candidate operations for each layer. Furthermore, we propose a novel search strategy combining a self-supervised search with a supervised one to simultaneously capture long-range and short-range dependencies. To verify the effectiveness of the proposed methods, we conduct extensive experiments on various learning tasks, including image classification, fine-grained image recognition, and zero-shot image retrieval. The empirical results show strong evidence that our method is capable of discovering high-performance full-attention architectures while guaranteeing the required search efficiency.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) has significantly advanced the automatic design of convolutional neural architectures. However, it is challenging to directly extend existing NAS methods to attention networks because of the uniform structure of the search space and the lack of long-range feature extraction. To address these issues, we construct a hierarchical search space that allows various attention operations to be adopted for different layers of a network. To reduce the complexity of the search, a low-cost search space compression method is proposed to automatically remove the unpromising candidate operations for each layer. Furthermore, we propose a novel search strategy combining a self-supervised search with a supervised one to simultaneously capture long-range and short-range dependencies. To verify the effectiveness of the proposed methods, we conduct extensive experiments on various learning tasks, including image classification, fine-grained image recognition, and zero-shot image retrieval. The empirical results show strong evidence that our method is capable of discovering high-performance full-attention architectures while guaranteeing the required search efficiency.
2802.
Huang, Jia-Cheng; Zeng, Guo-Qiang; Geng, Guang-Gang; Weng, Jian; Lu, Kang-Di
In: IET Cyber-Systems and Robotics, vol. 5, no. 1, pp. e12085, 2023.
@article{https://doi.org/10.1049/csy2.12085,
title = {SOPA-GA-CNN: Synchronous optimisation of parameters and architectures by genetic algorithms with convolutional neural network blocks for securing Industrial Internet-of-Things},
author = {Jia-Cheng Huang and Guo-Qiang Zeng and Guang-Gang Geng and Jian Weng and Kang-Di Lu},
url = {https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/csy2.12085},
doi = {https://doi.org/10.1049/csy2.12085},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IET Cyber-Systems and Robotics},
volume = {5},
number = {1},
pages = {e12085},
abstract = {Abstract In recent years, deep learning has been applied to a variety of scenarios in Industrial Internet of Things (IIoT), including enhancing the security of IIoT. However, the existing deep learning methods utilised in IIoT security are manually designed by heavily relying on the experience of the designers. The authors have made the first contribution concerning the joint optimisation of neural architecture search and hyper-parameters optimisation for securing IIoT. A novel automated deep learning method called synchronous optimisation of parameters and architectures by GA with CNN blocks (SOPA-GA-CNN) is proposed to synchronously optimise the hyperparameters and block-based architectures in convolutional neural networks (CNNs) by genetic algorithms (GA) for the intrusion detection issue of IIoT. An efficient hybrid encoding strategy and the corresponding GA-based evolutionary operations are designed to characterise and evolve both the hyperparameters, including batch size, learning rate, weight optimiser and weight regularisation, and the architectures, such as the block-based network topology and the parameters of each CNN block. The experimental results on five intrusion detection datasets in IIoT, including secure water treatment, water distribution, Gas Pipeline, Botnet in Internet of Things and Power System Attack Dataset, have demonstrated the superiority of the proposed SOPA-GA-CNN to the state-of-the-art manually designed models and neuron-evolutionary methods in terms of accuracy, precision, recall, F1-score, and the number of parameters of the deep learning models.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Abstract In recent years, deep learning has been applied to a variety of scenarios in Industrial Internet of Things (IIoT), including enhancing the security of IIoT. However, the existing deep learning methods utilised in IIoT security are manually designed by heavily relying on the experience of the designers. The authors have made the first contribution concerning the joint optimisation of neural architecture search and hyper-parameters optimisation for securing IIoT. A novel automated deep learning method called synchronous optimisation of parameters and architectures by GA with CNN blocks (SOPA-GA-CNN) is proposed to synchronously optimise the hyperparameters and block-based architectures in convolutional neural networks (CNNs) by genetic algorithms (GA) for the intrusion detection issue of IIoT. An efficient hybrid encoding strategy and the corresponding GA-based evolutionary operations are designed to characterise and evolve both the hyperparameters, including batch size, learning rate, weight optimiser and weight regularisation, and the architectures, such as the block-based network topology and the parameters of each CNN block. The experimental results on five intrusion detection datasets in IIoT, including secure water treatment, water distribution, Gas Pipeline, Botnet in Internet of Things and Power System Attack Dataset, have demonstrated the superiority of the proposed SOPA-GA-CNN to the state-of-the-art manually designed models and neuron-evolutionary methods in terms of accuracy, precision, recall, F1-score, and the number of parameters of the deep learning models.
2801.
Sakuma, Yuiko; Ishii, Masato; Narihira, Takuya
DetOFA: Efficient Training of Once-for-All Networks for Object Detection by Using Pre-trained Supernet and Path Filter Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-13121,
title = {DetOFA: Efficient Training of Once-for-All Networks for Object Detection by Using Pre-trained Supernet and Path Filter},
author = {Yuiko Sakuma and Masato Ishii and Takuya Narihira},
url = {https://doi.org/10.48550/arXiv.2303.13121},
doi = {10.48550/arXiv.2303.13121},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.13121},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2800.
Xu, Ying; Cheng, Long; Cai, Xuyi; Ma, Xiaohan; Chen, Weiwei; Zhang, Lei; Wang, Ying
Efficient Supernet Training Using Path Parallelism Proceedings Article
In: 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 1249-1261, 2023.
@inproceedings{10071099,
title = {Efficient Supernet Training Using Path Parallelism},
author = {Ying Xu and Long Cheng and Xuyi Cai and Xiaohan Ma and Weiwei Chen and Lei Zhang and Ying Wang},
doi = {10.1109/HPCA56546.2023.10071099},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)},
pages = {1249-1261},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2799.
Verma, Monu; Mandal, Murari; Reddy, M. Satish Kumar; Meedimale, Yashwanth Reddy; Vipparthi, Santosh Kumar
Efficient Neural Architecture Search for Emotion Recognition Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-13653,
title = {Efficient Neural Architecture Search for Emotion Recognition},
author = {Monu Verma and Murari Mandal and M. Satish Kumar Reddy and Yashwanth Reddy Meedimale and Santosh Kumar Vipparthi},
url = {https://doi.org/10.48550/arXiv.2303.13653},
doi = {10.48550/arXiv.2303.13653},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.13653},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2798.
Ito, Rafael C.; Zuben, Fernando J. Von
OFA(^mbox2): A Multi-Objective Perspective for the Once-for-All Neural Architecture Search Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-13683,
title = {OFA(^mbox2): A Multi-Objective Perspective for the Once-for-All Neural Architecture Search},
author = {Rafael C. Ito and Fernando J. Von Zuben},
url = {https://doi.org/10.48550/arXiv.2303.13683},
doi = {10.48550/arXiv.2303.13683},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.13683},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2797.
Benmeziane, Hadjer; Ounnoughene, Amine Ziad; Hamzaoui, Imane; Bouhadjar, Younes
Skip Connections in Spiking Neural Networks: An Analysis of Their Effect on Network Training Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-13563,
title = {Skip Connections in Spiking Neural Networks: An Analysis of Their Effect on Network Training},
author = {Hadjer Benmeziane and Amine Ziad Ounnoughene and Imane Hamzaoui and Younes Bouhadjar},
url = {https://doi.org/10.48550/arXiv.2303.13563},
doi = {10.48550/arXiv.2303.13563},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.13563},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2796.
Xiong, Zhuoran; Amein, Marihan; Therrien, Olivier; Gross, Warren J.; Meyer, Brett H.
FMAS: Fast Multi-Objective SuperNet Architecture Search for Semantic Segmentation Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-16322,
title = {FMAS: Fast Multi-Objective SuperNet Architecture Search for Semantic Segmentation},
author = {Zhuoran Xiong and Marihan Amein and Olivier Therrien and Warren J. Gross and Brett H. Meyer},
url = {https://doi.org/10.48550/arXiv.2303.16322},
doi = {10.48550/arXiv.2303.16322},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.16322},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2795.
Dong, Peijie; Li, Lujun; Wei, Zimian
DisWOT: Student Architecture Search for Distillation WithOut Training Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-15678,
title = {DisWOT: Student Architecture Search for Distillation WithOut Training},
author = {Peijie Dong and Lujun Li and Zimian Wei},
url = {https://doi.org/10.48550/arXiv.2303.15678},
doi = {10.48550/arXiv.2303.15678},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.15678},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2794.
Naumov, A.; Melnikov, Artem; Abronin, V.; Oxanichenko, F.; Izmailov, K.; Pflitsch, Markus; Melnikov, A.; Perelshtein, Michael
Tetra-AML: Automatic Machine Learning via Tensor Networks Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2303-16214,
title = {Tetra-AML: Automatic Machine Learning via Tensor Networks},
author = {A. Naumov and Artem Melnikov and V. Abronin and F. Oxanichenko and K. Izmailov and Markus Pflitsch and A. Melnikov and Michael Perelshtein},
url = {https://doi.org/10.48550/arXiv.2303.16214},
doi = {10.48550/arXiv.2303.16214},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2303.16214},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}