Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2021
1164.
Cheng, Anda; Wang, Jiaxing; Zhang, Xi Sheryl; Chen, Qiang; Wang, Peisong; Cheng, Jian
DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-08557,
title = {DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy},
author = {Anda Cheng and Jiaxing Wang and Xi Sheryl Zhang and Qiang Chen and Peisong Wang and Jian Cheng},
url = {https://arxiv.org/abs/2110.08557},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.08557},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1163.
Pang, Dong; Le, Xinyi; Guan, Xinping
RL-DARTS: Differentiable neural architecture search via reinforcement-learning-based meta-optimizer Journal Article
In: Knowledge-Based Systems, vol. 234, pp. 107585, 2021, ISSN: 0950-7051.
@article{PANG2021107585,
title = {RL-DARTS: Differentiable neural architecture search via reinforcement-learning-based meta-optimizer},
author = {Dong Pang and Xinyi Le and Xinping Guan},
url = {https://www.sciencedirect.com/science/article/pii/S0950705121008479},
doi = {https://doi.org/10.1016/j.knosys.2021.107585},
issn = {0950-7051},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Knowledge-Based Systems},
volume = {234},
pages = {107585},
abstract = {Differentiable search approaches have attracted extensive attention recently due to their advantages in effectively finding novel neural architectures. However, these methods suffer from shortcomings on heavy computation consumption and low robustness in some cases. In this work, we propose a novel differentiable search method based on reinforcement learning, to further improve the computation efficiency, network precision, and robustness in the neural architecture search area. Our method constructs a reinforcement learning-based meta-optimizer to solve the architecture-parameter optimization problem, which is superior in properties of adaptability and robustness to fixed optimizers. This learnable meta-optimizer can alter its model parameters along with the search process to adapt the optimization procedure, making it possible to find out better structures and parameters with less time. Specifically, we formulate a double-loop algorithm to address the optimization problem in the searched super-network. Through switching between the external and internal loops, our method alternately optimizes the super-network and the meta-optimizer, which converges to the optimal location more rapidly and robustly.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Differentiable search approaches have attracted extensive attention recently due to their advantages in effectively finding novel neural architectures. However, these methods suffer from shortcomings on heavy computation consumption and low robustness in some cases. In this work, we propose a novel differentiable search method based on reinforcement learning, to further improve the computation efficiency, network precision, and robustness in the neural architecture search area. Our method constructs a reinforcement learning-based meta-optimizer to solve the architecture-parameter optimization problem, which is superior in properties of adaptability and robustness to fixed optimizers. This learnable meta-optimizer can alter its model parameters along with the search process to adapt the optimization procedure, making it possible to find out better structures and parameters with less time. Specifically, we formulate a double-loop algorithm to address the optimization problem in the searched super-network. Through switching between the external and internal loops, our method alternately optimizes the super-network and the meta-optimizer, which converges to the optimal location more rapidly and robustly.
1162.
Wu, Robert; Saxena, Nayan; Jain, Rohan
NeuralArTS: Structuring Neural Architecture Search with Type Theory Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-08710,
title = {NeuralArTS: Structuring Neural Architecture Search with Type Theory},
author = {Robert Wu and Nayan Saxena and Rohan Jain},
url = {https://arxiv.org/abs/2110.08710},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.08710},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1161.
Lin, Xuanxiang; Chen, Ke; Jia, Kui
Object Point Cloud Classification via Poly-Convolutional Architecture Search Book Chapter
In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 807–815, Association for Computing Machinery, New York, NY, USA, 2021, ISBN: 9781450386517.
@inbook{10.1145/3474085.3475252,
title = {Object Point Cloud Classification via Poly-Convolutional Architecture Search},
author = {Xuanxiang Lin and Ke Chen and Kui Jia},
url = {https://doi.org/10.1145/3474085.3475252},
isbn = {9781450386517},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 29th ACM International Conference on Multimedia},
pages = {807–815},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Existing point cloud classifiers concern on handling irregular data structures to discover a global and discriminative configuration of local geometries. These classification methods design a number of effective permutation-invariant feature encoding kernels, but still suffer from the intrinsic challenge of large geometric feature variations caused by inconsistent point distributions along object surface. In this paper, point cloud classification can be addressed via deep graph representation learning on aggregating multiple convolutional feature kernels (namely, a poly convolutional operation) anchored on each point with its local neighbours. Inspired by recent success of neural architecture search, we introduce a novel concept of poly-convolutional architecture search (PolyConv search in short) to model local geometric patterns in a more flexible manner.To this end, the Monte Carlo Tree Search (MCTS) method is adopted, which can be formulated into a Markov Decision Process problem to cast decisions for dependently selecting layer-wise aggregation kernels. Experiments on the popular ModelNet40 benchmark have verified that superior performance can be achieved by constructing networks via the MCTS method, with aggregation kernels in our PolyConv search space.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Existing point cloud classifiers concern on handling irregular data structures to discover a global and discriminative configuration of local geometries. These classification methods design a number of effective permutation-invariant feature encoding kernels, but still suffer from the intrinsic challenge of large geometric feature variations caused by inconsistent point distributions along object surface. In this paper, point cloud classification can be addressed via deep graph representation learning on aggregating multiple convolutional feature kernels (namely, a poly convolutional operation) anchored on each point with its local neighbours. Inspired by recent success of neural architecture search, we introduce a novel concept of poly-convolutional architecture search (PolyConv search in short) to model local geometric patterns in a more flexible manner.To this end, the Monte Carlo Tree Search (MCTS) method is adopted, which can be formulated into a Markov Decision Process problem to cast decisions for dependently selecting layer-wise aggregation kernels. Experiments on the popular ModelNet40 benchmark have verified that superior performance can be achieved by constructing networks via the MCTS method, with aggregation kernels in our PolyConv search space.
1160.
Zhang, Miao; Liu, Tingwei; Piao, Yongri; Yao, Shunyu; Lu, Huchuan
Auto-MSFNet: Search Multi-Scale Fusion Network for Salient Object Detection Book Chapter
In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 667–676, Association for Computing Machinery, New York, NY, USA, 2021, ISBN: 9781450386517.
@inbook{10.1145/3474085.3475231,
title = {Auto-MSFNet: Search Multi-Scale Fusion Network for Salient Object Detection},
author = {Miao Zhang and Tingwei Liu and Yongri Piao and Shunyu Yao and Huchuan Lu},
url = {https://doi.org/10.1145/3474085.3475231},
isbn = {9781450386517},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 29th ACM International Conference on Multimedia},
pages = {667–676},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Multi-scale features fusion plays a critical role in salient object detection. Most of existing methods have achieved remarkable performance by exploiting various multi-scale features fusion strategies. However, an elegant fusion framework requires expert knowledge and experience, heavily relying on laborious trial and error. In this paper, we propose a multi-scale features fusion framework based on Neural Architecture Search (NAS), named Auto-MSFNet. First, we design a novel search cell, named FusionCell to automatically decide multi-scale features aggregation. Rather than searching one repeatable cell stacked, we allow different FusionCells to flexibly integrate multi-level features. Simultaneously, considering features generated from CNNs are naturally spatial and channel-wise, we propose a new search space for efficiently focusing on the most relevant information. The search space mitigates incomplete object structures or over-predicted foreground regions caused by progressive fusion. Second, we propose a progressive polishing loss to further obtain exquisite boundaries by penalizing misalignment of salient object boundaries. Extensive experiments on five benchmark datasets demonstrate the effectiveness of the proposed method and achieve state-of-the-art performance on four evaluation metrics. The code and results of our method are available at https://github.com/OIPLab-DUT/Auto-MSFNet.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Multi-scale features fusion plays a critical role in salient object detection. Most of existing methods have achieved remarkable performance by exploiting various multi-scale features fusion strategies. However, an elegant fusion framework requires expert knowledge and experience, heavily relying on laborious trial and error. In this paper, we propose a multi-scale features fusion framework based on Neural Architecture Search (NAS), named Auto-MSFNet. First, we design a novel search cell, named FusionCell to automatically decide multi-scale features aggregation. Rather than searching one repeatable cell stacked, we allow different FusionCells to flexibly integrate multi-level features. Simultaneously, considering features generated from CNNs are naturally spatial and channel-wise, we propose a new search space for efficiently focusing on the most relevant information. The search space mitigates incomplete object structures or over-predicted foreground regions caused by progressive fusion. Second, we propose a progressive polishing loss to further obtain exquisite boundaries by penalizing misalignment of salient object boundaries. Extensive experiments on five benchmark datasets demonstrate the effectiveness of the proposed method and achieve state-of-the-art performance on four evaluation metrics. The code and results of our method are available at https://github.com/OIPLab-DUT/Auto-MSFNet.
1159.
Selg, Hardi; Jenihhin, Maksim; Ellervee, Peeter
JÄNES: A NAS Framework for ML-based EDA Applications Proceedings Article
In: 2021 IEEE International Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT), pp. 1-6, 2021.
@inproceedings{9568321,
title = {JÄNES: A NAS Framework for ML-based EDA Applications},
author = {Hardi Selg and Maksim Jenihhin and Peeter Ellervee},
url = {https://ieeexplore.ieee.org/abstract/document/9568321},
doi = {10.1109/DFT52944.2021.9568321},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 IEEE International Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT)},
pages = {1-6},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1158.
Zhang, Zhihao; Jia, Zhihao
GradSign: Model Performance Inference with Theoretical Insights Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-08616,
title = {GradSign: Model Performance Inference with Theoretical Insights},
author = {Zhihao Zhang and Zhihao Jia},
url = {https://arxiv.org/abs/2110.08616},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.08616},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1157.
Li, Jian; Zhang, Bin; Wang, Yabiao; Tai, Ying; Zhang, Zhenyu; Wang, Chengjie; Li, Jilin; Huang, Xiaoming; Xia, Yili
ASFD: Automatic and Scalable Face Detector Book Chapter
In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 2139–2147, Association for Computing Machinery, New York, NY, USA, 2021, ISBN: 9781450386517.
@inbook{10.1145/3474085.3475372,
title = {ASFD: Automatic and Scalable Face Detector},
author = {Jian Li and Bin Zhang and Yabiao Wang and Ying Tai and Zhenyu Zhang and Chengjie Wang and Jilin Li and Xiaoming Huang and Yili Xia},
url = {https://doi.org/10.1145/3474085.3475372},
isbn = {9781450386517},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 29th ACM International Conference on Multimedia},
pages = {2139–2147},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Along with current multi-scale based detectors, Feature Aggregation and Enhancement (FAE) modules have shown superior performance gains for cutting-edge object detection. However, these hand-crafted FAE modules show inconsistent improvements on face detection, which is mainly due to the significant distribution difference between its training and applying corpus, i.e. COCO vs. WIDER Face. To tackle this problem, we essentially analyse the effect of data distribution, and consequently propose to search an effective FAE architecture, termed AutoFAE by a differentiable architecture search, which outperforms all existing FAE modules in face detection with a considerable margin. Upon the found AutoFAE and existing backbones, a supernet is further built and trained, which automatically obtains a family of detectors under the different complexity constraints. Extensive experiments conducted on popular benchmarks, i.e. WIDER Face and FDDB, demonstrate the state-of-the-art performance-efficiency trade-off for the proposed automatic and scalable face detector (ASFD) family. In particular, our strong ASFD-D6 outperforms the best competitor with AP 96.7/96.2/92.1 on WIDER Face test, and the lightweight ASFD-D0 costs about 3.1 ms, i.e. more than 320 FPS, on the V100 GPU with VGA-resolution images.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Along with current multi-scale based detectors, Feature Aggregation and Enhancement (FAE) modules have shown superior performance gains for cutting-edge object detection. However, these hand-crafted FAE modules show inconsistent improvements on face detection, which is mainly due to the significant distribution difference between its training and applying corpus, i.e. COCO vs. WIDER Face. To tackle this problem, we essentially analyse the effect of data distribution, and consequently propose to search an effective FAE architecture, termed AutoFAE by a differentiable architecture search, which outperforms all existing FAE modules in face detection with a considerable margin. Upon the found AutoFAE and existing backbones, a supernet is further built and trained, which automatically obtains a family of detectors under the different complexity constraints. Extensive experiments conducted on popular benchmarks, i.e. WIDER Face and FDDB, demonstrate the state-of-the-art performance-efficiency trade-off for the proposed automatic and scalable face detector (ASFD) family. In particular, our strong ASFD-D6 outperforms the best competitor with AP 96.7/96.2/92.1 on WIDER Face test, and the lightweight ASFD-D0 costs about 3.1 ms, i.e. more than 320 FPS, on the V100 GPU with VGA-resolution images.
1156.
Papalexopoulos, Theodore; Tjandraatmadja, Christian; Anderson, Ross; Vielma, Juan Pablo; Belanger, David
Constrained Discrete Black-Box Optimization using Mixed-Integer Programming Technical Report
2021.
@techreport{papalexopoulos2021constrained,
title = {Constrained Discrete Black-Box Optimization using Mixed-Integer Programming},
author = {Theodore Papalexopoulos and Christian Tjandraatmadja and Ross Anderson and Juan Pablo Vielma and David Belanger},
url = {https://arxiv.org/abs/2110.09569},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1155.
Kim, Dahyun; Singh, Kunal Pratap; Choi, Jonghyun
BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-08562,
title = {BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements},
author = {Dahyun Kim and Kunal Pratap Singh and Jonghyun Choi},
url = {https://arxiv.org/abs/2110.08562},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.08562},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1154.
Shen, Yu; Li, Yang; Zheng, Jian; Zhang, Wentao; Yao, Peng; Li, Jixiang; Yang, Sen; Liu, Ji; Cui, Bin
ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-10423,
title = {ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies},
author = {Yu Shen and Yang Li and Jian Zheng and Wentao Zhang and Peng Yao and Jixiang Li and Sen Yang and Ji Liu and Bin Cui},
url = {https://arxiv.org/abs/2110.10423},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.10423},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1153.
Kong, Qi; Xu, Xin; Zhang, Liangliang
MEMA-NAS: Memory-Efficient Multi-Agent Neural Architecture Search Proceedings Article
In: Ma, Huimin; Wang, Liang; Zhang, Changshui; Wu, Fei; Tan, Tieniu; Wang, Yaonan; Lai, Jianhuang; Zhao, Yao (Ed.): Pattern Recognition and Computer Vision, pp. 176–187, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-88013-2.
@inproceedings{10.1007/978-3-030-88013-2_15,
title = {MEMA-NAS: Memory-Efficient Multi-Agent Neural Architecture Search},
author = {Qi Kong and Xin Xu and Liangliang Zhang},
editor = {Huimin Ma and Liang Wang and Changshui Zhang and Fei Wu and Tieniu Tan and Yaonan Wang and Jianhuang Lai and Yao Zhao},
isbn = {978-3-030-88013-2},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Pattern Recognition and Computer Vision},
pages = {176--187},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Öbject detection is a core computer vision task that aims to localize and classify categories for various objects in an image. With the development of convolutional neural networks, deep learning methods have been widely used in the object detection task, achieving promising performance compared to traditional methods. However, designing a well-performing detection network is inefficient. It consumes too much hardware resources and time to trial, and it also heavily relies on expert knowledge. To efficiently design the neural network architecture, there has been a growing interest in automatically designing neural network architecture by Neural Architecture Search (NAS). In this paper, we propose a Memory-Efficient Multi-Agent Neural Architecture Search (MEMA-NAS) framework in end-to-end object detection neural network. Specifically, we introduce the multi-agent learning to search holistic architecture of the detection network. In this way, a lot of GPU memory is saved, allowing us to search each module's architecture of the detection network simultaneously. To find a better tradeoff between the precision and computational costs, we add the resource constraint in our method. Search experiments on multiple datasets show that MEMA-NAS achieves state-of-the-art results in search efficiency and precision."},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Öbject detection is a core computer vision task that aims to localize and classify categories for various objects in an image. With the development of convolutional neural networks, deep learning methods have been widely used in the object detection task, achieving promising performance compared to traditional methods. However, designing a well-performing detection network is inefficient. It consumes too much hardware resources and time to trial, and it also heavily relies on expert knowledge. To efficiently design the neural network architecture, there has been a growing interest in automatically designing neural network architecture by Neural Architecture Search (NAS). In this paper, we propose a Memory-Efficient Multi-Agent Neural Architecture Search (MEMA-NAS) framework in end-to-end object detection neural network. Specifically, we introduce the multi-agent learning to search holistic architecture of the detection network. In this way, a lot of GPU memory is saved, allowing us to search each module's architecture of the detection network simultaneously. To find a better tradeoff between the precision and computational costs, we add the resource constraint in our method. Search experiments on multiple datasets show that MEMA-NAS achieves state-of-the-art results in search efficiency and precision."
1152.
Ho, T. Y.; Guo, D.; Jin, D.; Zhu, Z.; Hung, T. M.; Xiao, J.; Lu, L.; Lin, C. Y.
Comprehensive Head and Neck Organs at Risk Segmentation Using Stratified Learning and Neural Architecture Search Journal Article
In: International Journal of Radiation Oncology*Biology*Physics, vol. 111, no. 3, Supplement, pp. e369-e370, 2021, ISSN: 0360-3016, (2021 Proceedings of the ASTRO 63rd Annual Meeting).
@article{HO2021e369,
title = {Comprehensive Head and Neck Organs at Risk Segmentation Using Stratified Learning and Neural Architecture Search},
author = {T. Y. Ho and D. Guo and D. Jin and Z. Zhu and T. M. Hung and J. Xiao and L. Lu and C. Y. Lin},
url = {https://www.sciencedirect.com/science/article/pii/S0360301621019635},
doi = {https://doi.org/10.1016/j.ijrobp.2021.07.1093},
issn = {0360-3016},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {International Journal of Radiation Oncology*Biology*Physics},
volume = {111},
number = {3, Supplement},
pages = {e369-e370},
abstract = {Purpose/Objective(s)
Organs at risk (OAR) segmentation is an essential step in the radiotherapy of head and neck (H&N) cancer. Automated approaches benefit physicians in significantly reducing manual work and improving the annotation quality and consistency. Current standard deep learning methods face challenges when the number of OARs becomes large, e.g., > 40. Physicians often refer to easy OARs when delineating harder ones, e.g., using mandibles to help identify adjacent salivary glands. This study aims to emulate this process and develop a new approach that stratifies 42 head and neck OARs into anchor, mid-level, and small & hard (S&H) levels to ensure high segmentation quality.
Materials/Methods
We curated two datasets of CT scans, each annotated with 42 OAR masks: one from 142 oropharyngeal cancer (OPX) patients and the other from 31 nasopharyngeal cancer (NPC) patients. Our segmentation method is first developed and evaluated using the OPX dataset and then further tested using the NPC dataset. Emulating clinical practice, we first stratify the 42 OARs into 3 levels. Anchor OARs are high in intensity contrast and low in inter- and intra-reader variability. Mid-level OARs are low in contrast, but not inordinately small. S&H OARs are poor in contrast and very small. For each level, a tailored deep learning segmentation network is developed using the automated network architecture search (NAS). NAS allows the network to choose among 2D, 3D, or Pseudo-3D convolutions. by considering three levels of complexities. Anchor OARs are used to infer the mid-level and S&H OARs segmentation.
Results
With 4-fold cross-validation on the OPX dataset, our method has achieved an average 75.1% Dice score (DSC) and 1.1mm average surface distance (ASD). It outperforms the previous leading method, UaNet, on mid-level OAR segmentation by 3.4% DSC increase 0.4mm, ASD reduction; and on S&H OAR of 10.1% DSC increase, 1.0mm ASD reduction, respectively. Using the NPC patients as an unseen testing set, our method has achieved an average DSC of 76.3% and 1.3mm ASD, which is consistent as in the OPX dataset. This result demonstrates the robustness and generalizability of our method in patients, even with various cancer types.
Conclusion
We introduced a new stratified method for segmenting a large comprehensive set of H&N OARs. Our method integrates multi-stage segmentation and NAS in a synergy for the first time. It was trained using OPX patients and achieved state-of-the-art performance and generalized well to patients of NPC. Our method is a critical step towards an automated, accurate, and dependable OAR segmentation system in various H&N cancers.},
note = {2021 Proceedings of the ASTRO 63rd Annual Meeting},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Purpose/Objective(s)
Organs at risk (OAR) segmentation is an essential step in the radiotherapy of head and neck (H&N) cancer. Automated approaches benefit physicians in significantly reducing manual work and improving the annotation quality and consistency. Current standard deep learning methods face challenges when the number of OARs becomes large, e.g., > 40. Physicians often refer to easy OARs when delineating harder ones, e.g., using mandibles to help identify adjacent salivary glands. This study aims to emulate this process and develop a new approach that stratifies 42 head and neck OARs into anchor, mid-level, and small & hard (S&H) levels to ensure high segmentation quality.
Materials/Methods
We curated two datasets of CT scans, each annotated with 42 OAR masks: one from 142 oropharyngeal cancer (OPX) patients and the other from 31 nasopharyngeal cancer (NPC) patients. Our segmentation method is first developed and evaluated using the OPX dataset and then further tested using the NPC dataset. Emulating clinical practice, we first stratify the 42 OARs into 3 levels. Anchor OARs are high in intensity contrast and low in inter- and intra-reader variability. Mid-level OARs are low in contrast, but not inordinately small. S&H OARs are poor in contrast and very small. For each level, a tailored deep learning segmentation network is developed using the automated network architecture search (NAS). NAS allows the network to choose among 2D, 3D, or Pseudo-3D convolutions. by considering three levels of complexities. Anchor OARs are used to infer the mid-level and S&H OARs segmentation.
Results
With 4-fold cross-validation on the OPX dataset, our method has achieved an average 75.1% Dice score (DSC) and 1.1mm average surface distance (ASD). It outperforms the previous leading method, UaNet, on mid-level OAR segmentation by 3.4% DSC increase 0.4mm, ASD reduction; and on S&H OAR of 10.1% DSC increase, 1.0mm ASD reduction, respectively. Using the NPC patients as an unseen testing set, our method has achieved an average DSC of 76.3% and 1.3mm ASD, which is consistent as in the OPX dataset. This result demonstrates the robustness and generalizability of our method in patients, even with various cancer types.
Conclusion
We introduced a new stratified method for segmenting a large comprehensive set of H&N OARs. Our method integrates multi-stage segmentation and NAS in a synergy for the first time. It was trained using OPX patients and achieved state-of-the-art performance and generalized well to patients of NPC. Our method is a critical step towards an automated, accurate, and dependable OAR segmentation system in various H&N cancers.
Organs at risk (OAR) segmentation is an essential step in the radiotherapy of head and neck (H&N) cancer. Automated approaches benefit physicians in significantly reducing manual work and improving the annotation quality and consistency. Current standard deep learning methods face challenges when the number of OARs becomes large, e.g., > 40. Physicians often refer to easy OARs when delineating harder ones, e.g., using mandibles to help identify adjacent salivary glands. This study aims to emulate this process and develop a new approach that stratifies 42 head and neck OARs into anchor, mid-level, and small & hard (S&H) levels to ensure high segmentation quality.
Materials/Methods
We curated two datasets of CT scans, each annotated with 42 OAR masks: one from 142 oropharyngeal cancer (OPX) patients and the other from 31 nasopharyngeal cancer (NPC) patients. Our segmentation method is first developed and evaluated using the OPX dataset and then further tested using the NPC dataset. Emulating clinical practice, we first stratify the 42 OARs into 3 levels. Anchor OARs are high in intensity contrast and low in inter- and intra-reader variability. Mid-level OARs are low in contrast, but not inordinately small. S&H OARs are poor in contrast and very small. For each level, a tailored deep learning segmentation network is developed using the automated network architecture search (NAS). NAS allows the network to choose among 2D, 3D, or Pseudo-3D convolutions. by considering three levels of complexities. Anchor OARs are used to infer the mid-level and S&H OARs segmentation.
Results
With 4-fold cross-validation on the OPX dataset, our method has achieved an average 75.1% Dice score (DSC) and 1.1mm average surface distance (ASD). It outperforms the previous leading method, UaNet, on mid-level OAR segmentation by 3.4% DSC increase 0.4mm, ASD reduction; and on S&H OAR of 10.1% DSC increase, 1.0mm ASD reduction, respectively. Using the NPC patients as an unseen testing set, our method has achieved an average DSC of 76.3% and 1.3mm ASD, which is consistent as in the OPX dataset. This result demonstrates the robustness and generalizability of our method in patients, even with various cancer types.
Conclusion
We introduced a new stratified method for segmenting a large comprehensive set of H&N OARs. Our method integrates multi-stage segmentation and NAS in a synergy for the first time. It was trained using OPX patients and achieved state-of-the-art performance and generalized well to patients of NPC. Our method is a critical step towards an automated, accurate, and dependable OAR segmentation system in various H&N cancers.
1151.
Akhauri, Yash; Munoz, Juan Pablo; Iyer, Ravishankar; Jain, Nilesh
A Genetic Programming Approach To Zero-Shot Neural Architecture Ranking Proceedings Article
In: Advances in Programming Languages and Neurosymbolic Systems Workshop, 2021.
@inproceedings{<LineBreak>akhauri2021a,
title = {A Genetic Programming Approach To Zero-Shot Neural Architecture Ranking},
author = {Yash Akhauri and Juan Pablo Munoz and Ravishankar Iyer and Nilesh Jain},
url = {https://openreview.net/forum?id=xuVVuLcqBP5},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Advances in Programming Languages and Neurosymbolic Systems Workshop},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1150.
Hirose, Yoichi; Yoshinari, Nozomu; Shirakawa, Shinichi
NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters Proceedings Article
In: ACML2021, 2021.
@inproceedings{DBLP:journals/corr/abs-2110-10165,
title = {NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters},
author = {Yoichi Hirose and Nozomu Yoshinari and Shinichi Shirakawa},
url = {https://arxiv.org/abs/2110.10165},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {ACML2021},
journal = {CoRR},
volume = {abs/2110.10165},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1149.
Liu, Zhu; Ma, Long; Liu, Risheng; Fan, Xin; Luo, Zhongxuan; Zhang, Yuduo
Latency-Constrained Spatial-Temporal Aggregated Architecture Search for Video Deraining Proceedings Article
In: Ma, Huimin; Wang, Liang; Zhang, Changshui; Wu, Fei; Tan, Tieniu; Wang, Yaonan; Lai, Jianhuang; Zhao, Yao (Ed.): Pattern Recognition and Computer Vision, pp. 16–28, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-88010-1.
@inproceedings{10.1007/978-3-030-88010-1_2,
title = {Latency-Constrained Spatial-Temporal Aggregated Architecture Search for Video Deraining},
author = {Zhu Liu and Long Ma and Risheng Liu and Xin Fan and Zhongxuan Luo and Yuduo Zhang},
editor = {Huimin Ma and Liang Wang and Changshui Zhang and Fei Wu and Tieniu Tan and Yaonan Wang and Jianhuang Lai and Yao Zhao},
url = {https://link.springer.com/chapter/10.1007/978-3-030-88010-1_2},
isbn = {978-3-030-88010-1},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Pattern Recognition and Computer Vision},
pages = {16--28},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Existing deep learning-based video deraining techniques have achieved remarkable processes. However, there exist some fundamental issues including plentiful engineering experiences for architecture design and slow hardware-insensitive inference speed. To settle these issues, we develop a highly efficient spatial-temporal aggregated video deraining architecture, derived from the architecture search procedure under a newly-defined flexible search space and latency-constrained search strategy. To be specific, we establish an inter-frame aggregation module to fully integrate temporal correlation according to a set division perspective. Subsequently, we construct an intra-frame enhancement module to eliminate the residual rain streaks by introducing rain kernels that characterize the rain locations. A flexible search space for defining architectures of these two modules is built to avert the demand for expensive engineering skills. Further, we design a latency-constrained differentiable search strategy to automatically discover a hardware-sensitive high-efficient video deraining architecture. Extensive experiments demonstrate that our method can obtain best performance against other state-of-the-art methods.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Existing deep learning-based video deraining techniques have achieved remarkable processes. However, there exist some fundamental issues including plentiful engineering experiences for architecture design and slow hardware-insensitive inference speed. To settle these issues, we develop a highly efficient spatial-temporal aggregated video deraining architecture, derived from the architecture search procedure under a newly-defined flexible search space and latency-constrained search strategy. To be specific, we establish an inter-frame aggregation module to fully integrate temporal correlation according to a set division perspective. Subsequently, we construct an intra-frame enhancement module to eliminate the residual rain streaks by introducing rain kernels that characterize the rain locations. A flexible search space for defining architectures of these two modules is built to avert the demand for expensive engineering skills. Further, we design a latency-constrained differentiable search strategy to automatically discover a hardware-sensitive high-efficient video deraining architecture. Extensive experiments demonstrate that our method can obtain best performance against other state-of-the-art methods.
1148.
Ji, Zexin; Dong, Xin; Li, Zhendong; Yu, Zekuan; Liu, Hao
Non-local Network Routing for Perceptual Image Super-Resolution Proceedings Article
In: Ma, Huimin; Wang, Liang; Zhang, Changshui; Wu, Fei; Tan, Tieniu; Wang, Yaonan; Lai, Jianhuang; Zhao, Yao (Ed.): Pattern Recognition and Computer Vision, pp. 164–176, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-88010-1.
@inproceedings{10.1007/978-3-030-88010-1_14,
title = {Non-local Network Routing for Perceptual Image Super-Resolution},
author = {Zexin Ji and Xin Dong and Zhendong Li and Zekuan Yu and Hao Liu},
editor = {Huimin Ma and Liang Wang and Changshui Zhang and Fei Wu and Tieniu Tan and Yaonan Wang and Jianhuang Lai and Yao Zhao},
url = {https://link.springer.com/chapter/10.1007/978-3-030-88010-1_14},
isbn = {978-3-030-88010-1},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Pattern Recognition and Computer Vision},
pages = {164--176},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In this paper, we propose a non-local network routing (NNR) approach for perceptual image super-resolution. Unlike conventional methods which generate visually-faked textures due to exiting hand-designed losses, our approach aims to globally optimize both procedures of learning an optimal perceptual loss and routing a spatial-adaptive network architecture in a unified reinforcement learning framework. To this end, we introduce a reward function to teach our objective to pay more attention on the visual quality of the super-resolved image. Moreover, we carefully design an offset operation inside the neural architecture search space, which typically deforms the receptive field on boundary refinement in a non-local manner. Experimentally, our proposed method surpasses the perceptual performance over state-of-the-art methods on several widely-evaluated benchmark datasets.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper, we propose a non-local network routing (NNR) approach for perceptual image super-resolution. Unlike conventional methods which generate visually-faked textures due to exiting hand-designed losses, our approach aims to globally optimize both procedures of learning an optimal perceptual loss and routing a spatial-adaptive network architecture in a unified reinforcement learning framework. To this end, we introduce a reward function to teach our objective to pay more attention on the visual quality of the super-resolved image. Moreover, we carefully design an offset operation inside the neural architecture search space, which typically deforms the receptive field on boundary refinement in a non-local manner. Experimentally, our proposed method surpasses the perceptual performance over state-of-the-art methods on several widely-evaluated benchmark datasets.
1147.
Feng, Qiantai; Xu, Ke; Li, Yuhai; Sun, Yuxin; Wang, Dong
Edge-Wise One-Level Global Pruning on NAS Generated Networks Proceedings Article
In: Ma, Huimin; Wang, Liang; Zhang, Changshui; Wu, Fei; Tan, Tieniu; Wang, Yaonan; Lai, Jianhuang; Zhao, Yao (Ed.): Pattern Recognition and Computer Vision, pp. 3–15, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-88013-2.
@inproceedings{10.1007/978-3-030-88013-2_1,
title = {Edge-Wise One-Level Global Pruning on NAS Generated Networks},
author = {Qiantai Feng and Ke Xu and Yuhai Li and Yuxin Sun and Dong Wang},
editor = {Huimin Ma and Liang Wang and Changshui Zhang and Fei Wu and Tieniu Tan and Yaonan Wang and Jianhuang Lai and Yao Zhao},
url = {https://link.springer.com/chapter/10.1007/978-3-030-88013-2_1},
isbn = {978-3-030-88013-2},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Pattern Recognition and Computer Vision},
pages = {3--15},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In recent years, there has been a lot of studies in neural architecture search (NAS) in the field of deep learning. Among them, the cell-based search method, such as [23, 27, 32, 36], is one of the most popular and widely discussed topics, which usually stacks less cells in search process and more in evaluation. Although this method can reduce the resource consumption in the process of search, the difference in the number of cells may inevitably cause a certain degree of redundancy in network evaluation. In order to mitigate the computational cost, we propose a novel algorithm called Edge-Wise One-Level Global Pruning (EOG-Pruning). The proposed approach can prune out weak edges from the cell-based network generated by NAS globally, by introducing an edge factor to represent the importance of each edge, which can not only greatly improve the inference speed of the model with reducing the number of edges, but also promote the model accuracy. Experimental results show that networks pruned by EOG-Pruning achieve significant improvement in accuracy and speedup rate on CPU in common with 50% pruning rate on CIFAR. Specifically, we reduced the test error rate by 1.58% and 1.34% on CIFAR-100 for DARTS (2nd-order) and PC-DARTS.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In recent years, there has been a lot of studies in neural architecture search (NAS) in the field of deep learning. Among them, the cell-based search method, such as [23, 27, 32, 36], is one of the most popular and widely discussed topics, which usually stacks less cells in search process and more in evaluation. Although this method can reduce the resource consumption in the process of search, the difference in the number of cells may inevitably cause a certain degree of redundancy in network evaluation. In order to mitigate the computational cost, we propose a novel algorithm called Edge-Wise One-Level Global Pruning (EOG-Pruning). The proposed approach can prune out weak edges from the cell-based network generated by NAS globally, by introducing an edge factor to represent the importance of each edge, which can not only greatly improve the inference speed of the model with reducing the number of edges, but also promote the model accuracy. Experimental results show that networks pruned by EOG-Pruning achieve significant improvement in accuracy and speedup rate on CPU in common with 50% pruning rate on CIFAR. Specifically, we reduced the test error rate by 1.58% and 1.34% on CIFAR-100 for DARTS (2nd-order) and PC-DARTS.
1146.
Xue, Xizhe; Zhang, Haokui; Bai, Zongwen; Li, Ying
3D-ANAS v2: Grafting Transformer Module on Automatically Designed ConvNet for Hyperspectral Image Classification Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-11084,
title = {3D-ANAS v2: Grafting Transformer Module on Automatically Designed ConvNet for Hyperspectral Image Classification},
author = {Xizhe Xue and Haokui Zhang and Zongwen Bai and Ying Li},
url = {https://arxiv.org/abs/2110.11084},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.11084},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1145.
Nayman, Niv; Aflalo, Yonathan; Noy, Asaf; Jin, Rong; Zelnik-Manor, Lihi
IQNAS: Interpretable Integer Quadratic Programming Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-12399,
title = {IQNAS: Interpretable Integer Quadratic Programming Neural Architecture Search},
author = {Niv Nayman and Yonathan Aflalo and Asaf Noy and Rong Jin and Lihi Zelnik-Manor},
url = {https://arxiv.org/abs/2110.12399},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.12399},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1144.
Simon, Christian; Koniusz, Piotr; Petersson, Lars; Han, Yan; Harandi, Mehrtash
Towards a Robust Differentiable Architecture Search under Label Noise Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-12197,
title = {Towards a Robust Differentiable Architecture Search under Label Noise},
author = {Christian Simon and Piotr Koniusz and Lars Petersson and Yan Han and Mehrtash Harandi},
url = {https://arxiv.org/abs/2110.12197},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.12197},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1143.
Geiping, Jonas; Lukasik, Jovita; Keuper, Margret; Moeller, Michael
DARTS for Inverse Problems: a Study on Stability Proceedings Article
In: NeurIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021.
@inproceedings{<LineBreak>geiping2021darts,
title = {DARTS for Inverse Problems: a Study on Stability},
author = {Jonas Geiping and Jovita Lukasik and Margret Keuper and Michael Moeller},
url = {https://openreview.net/forum?id=ty5XCitJfLA},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {NeurIPS 2021 Workshop on Deep Learning and Inverse Problems},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1142.
Gadikar, Pranav U; Ganesan, Vinod; Panda, Pratyush Kumar
Generalized Weight Agnostic Neural Networks for Configurable and Continual Autonomous Systems Proceedings Article
In: The First International Conference on AI-ML-Systems, Association for Computing Machinery, Bangalore, India, 2021, ISBN: 9781450385947.
@inproceedings{10.1145/3486001.3486226,
title = {Generalized Weight Agnostic Neural Networks for Configurable and Continual Autonomous Systems},
author = {Pranav U Gadikar and Vinod Ganesan and Pratyush Kumar Panda},
url = {https://doi.org/10.1145/3486001.3486226},
doi = {10.1145/3486001.3486226},
isbn = {9781450385947},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {The First International Conference on AI-ML-Systems},
publisher = {Association for Computing Machinery},
address = {Bangalore, India},
series = {AIMLSystems 2021},
abstract = {Implementing pervasive intelligence faces the challenges of efficient, continual, and configurable learning on embedded devices. We address these challenges with two novel extensions to Weight Agnostic Neural Networks (WANNs) namely – (i) Multi-weight extension and (ii) Multi-objective extension. In the multi-weight extension, we extend the idea of single shared weight WANNs to multiple weights to enable efficient continual learning. Our results across four different tasks implemented on Raspberry Pi demonstrate that the multi-weight WANNs achieve higher reward as compared to single shared weight WANNs and can be fine-tuned on device in orders of magnitude smaller time. In the multi-objective extension of WANNs, we extend the idea of a single reward function to multiple competing rewards and demonstrate sensitive and monotonic trade-off between multiple competing rewards to enable efficient configurable learning. We also demonstrate robustness of WANNs for changes in the weight parameters and changes in the environmental conditions as compared to the Proximal Policy Optimization(PPO) algorithm. These significant results open the possibility of truly autonomous agents using WANNs on low compute and power budget devices.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Implementing pervasive intelligence faces the challenges of efficient, continual, and configurable learning on embedded devices. We address these challenges with two novel extensions to Weight Agnostic Neural Networks (WANNs) namely – (i) Multi-weight extension and (ii) Multi-objective extension. In the multi-weight extension, we extend the idea of single shared weight WANNs to multiple weights to enable efficient continual learning. Our results across four different tasks implemented on Raspberry Pi demonstrate that the multi-weight WANNs achieve higher reward as compared to single shared weight WANNs and can be fine-tuned on device in orders of magnitude smaller time. In the multi-objective extension of WANNs, we extend the idea of a single reward function to multiple competing rewards and demonstrate sensitive and monotonic trade-off between multiple competing rewards to enable efficient configurable learning. We also demonstrate robustness of WANNs for changes in the weight parameters and changes in the environmental conditions as compared to the Proximal Policy Optimization(PPO) algorithm. These significant results open the possibility of truly autonomous agents using WANNs on low compute and power budget devices.
1141.
Knyazev, Boris; Drozdzal, Michal; Taylor, Graham W.; Romero-Soriano, Adriana
Parameter Prediction for Unseen Deep Architectures Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-13100,
title = {Parameter Prediction for Unseen Deep Architectures},
author = {Boris Knyazev and Michal Drozdzal and Graham W. Taylor and Adriana Romero-Soriano},
url = {https://arxiv.org/abs/2110.13100},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13100},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1140.
Yuan, Gonglin; Xue, Bing; Zhang, Mengjie
Ä Two-Stage Efficient Evolutionary Neural Architecture Search Method for Image Classification Proceedings Article
In: Pham, Duc Nghia; Theeramunkong, Thanaruk; Governatori, Guido; Liu, Fenrong (Ed.): PRICAI 2021: Trends in Artificial Intelligence, pp. 469–484, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-89188-6.
@inproceedings{10.1007/978-3-030-89188-6_35,
title = {Ä Two-Stage Efficient Evolutionary Neural Architecture Search Method for Image Classification},
author = {Gonglin Yuan and Bing Xue and Mengjie Zhang},
editor = {Duc Nghia Pham and Thanaruk Theeramunkong and Guido Governatori and Fenrong Liu},
url = {https://link.springer.com/chapter/10.1007/978-3-030-89188-6_35},
isbn = {978-3-030-89188-6},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {PRICAI 2021: Trends in Artificial Intelligence},
pages = {469--484},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Deep convolutional neural networks (DCNNs) have achieved promising performance in different computer vision tasks in recent years. Conventionally, deep learning experts are needed to design convolutional neural network's (CNN's) architectures when facing new tasks. Neural architecture search (NAS) is to automatically find suitable architectures; however, NAS suffers from the tremendous computational cost. This paper employs a genetic algorithm (GA) and a grid search (GS) strategy to search for the micro-architecture and adjust the macro-architecture efficiently and effectively, named TSCNN. We propose two mutation operations to explore the search space comprehensively. Furthermore, the micro-architecture searched on one dataset is transferred to another dataset to verify its transferability. The proposed algorithm is evaluated on two widely used datasets. The experimental results show that TSCNN achieves very competitive accuracy. On the CIFAR10 dataset, the computational cost is reduced from hundreds or even thousands to only 2.5 GPU-days, and the number of parameters is reduced from thirty more million to only 1.25 M.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Deep convolutional neural networks (DCNNs) have achieved promising performance in different computer vision tasks in recent years. Conventionally, deep learning experts are needed to design convolutional neural network's (CNN's) architectures when facing new tasks. Neural architecture search (NAS) is to automatically find suitable architectures; however, NAS suffers from the tremendous computational cost. This paper employs a genetic algorithm (GA) and a grid search (GS) strategy to search for the micro-architecture and adjust the macro-architecture efficiently and effectively, named TSCNN. We propose two mutation operations to explore the search space comprehensively. Furthermore, the micro-architecture searched on one dataset is transferred to another dataset to verify its transferability. The proposed algorithm is evaluated on two widely used datasets. The experimental results show that TSCNN achieves very competitive accuracy. On the CIFAR10 dataset, the computational cost is reduced from hundreds or even thousands to only 2.5 GPU-days, and the number of parameters is reduced from thirty more million to only 1.25 M.
1139.
Cai, Rongshen; Tao, Qian; Tang, Yufei; Shi, Min
ALGNN: Auto-Designed Lightweight Graph Neural Network Proceedings Article
In: Pham, Duc Nghia; Theeramunkong, Thanaruk; Governatori, Guido; Liu, Fenrong (Ed.): PRICAI 2021: Trends in Artificial Intelligence, pp. 500–512, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-89188-6.
@inproceedings{10.1007/978-3-030-89188-6_37,
title = {ALGNN: Auto-Designed Lightweight Graph Neural Network},
author = {Rongshen Cai and Qian Tao and Yufei Tang and Min Shi},
editor = {Duc Nghia Pham and Thanaruk Theeramunkong and Guido Governatori and Fenrong Liu},
isbn = {978-3-030-89188-6},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {PRICAI 2021: Trends in Artificial Intelligence},
pages = {500--512},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Graph neural networks (GNNs) are widely used on graph-structured data, and its research has made substantial progress in recent years. However, given the various number of choices and combinations of components such as aggregator and activation function, designing GNNs for specific tasks is very heavy manual work. Recently, neural architecture search (NAS) was proposed with the aim of automating the GNN design process and generating task-dependent architectures. While existing approaches have achieved competitive performance, they are not well suited to practical application scenarios where the computational budget is limited. In this paper, we propose an auto-designed lightweight graph neural network (ALGNN) method to automatically design lightweight, task-dependent GNN architectures. ALGNN uses multi-objective optimization to optimize the architecture constrained by the computation cost and complexity of the model. We define, for the first time, an evaluation standard for consumption cost with the analysis of the message passing process in GNNs. Experiments on real-world datasets demonstrate that ALGNN can generate a lightweight GNN model that has much fewer parameters and GPU hours, meanwhile has comparable performance with state-of-the-art approaches.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Graph neural networks (GNNs) are widely used on graph-structured data, and its research has made substantial progress in recent years. However, given the various number of choices and combinations of components such as aggregator and activation function, designing GNNs for specific tasks is very heavy manual work. Recently, neural architecture search (NAS) was proposed with the aim of automating the GNN design process and generating task-dependent architectures. While existing approaches have achieved competitive performance, they are not well suited to practical application scenarios where the computational budget is limited. In this paper, we propose an auto-designed lightweight graph neural network (ALGNN) method to automatically design lightweight, task-dependent GNN architectures. ALGNN uses multi-objective optimization to optimize the architecture constrained by the computation cost and complexity of the model. We define, for the first time, an evaluation standard for consumption cost with the analysis of the message passing process in GNNs. Experiments on real-world datasets demonstrate that ALGNN can generate a lightweight GNN model that has much fewer parameters and GPU hours, meanwhile has comparable performance with state-of-the-art approaches.
1138.
Egele, Romain; Maulik, Romit; Raghavan, Krishnan; Balaprakash, Prasanna; Lusch, Bethany
AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-13511,
title = {AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification},
author = {Romain Egele and Romit Maulik and Krishnan Raghavan and Prasanna Balaprakash and Bethany Lusch},
url = {https://arxiv.org/abs/2110.13511},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13511},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1137.
Song, Siyang; Shao, Zilong; Jaiswal, Shashank; Shen, Linlin; Valstar, Michel F.; Gunes, Hatice
Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-13570,
title = {Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition},
author = {Siyang Song and Zilong Shao and Shashank Jaiswal and Linlin Shen and Michel F. Valstar and Hatice Gunes},
url = {https://arxiv.org/abs/2110.13570},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13570},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1136.
Lopes, Vasco; Santos, Miguel; Degardin, Bruno; Alexandre, Luís A.
Guided Evolution for Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-15232,
title = {Guided Evolution for Neural Architecture Search},
author = {Vasco Lopes and Miguel Santos and Bruno Degardin and Luís A. Alexandre},
url = {https://arxiv.org/abs/2110.15232},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.15232},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1135.
Xue, Yu; Yuan, Ziming; Slowik, Adam
A Novel Sleep Stage Classification Using CNN Generated by an Efficient Neural Architecture Search with a New Data Processing Trick Technical Report
2021.
@techreport{xue2021novel,
title = {A Novel Sleep Stage Classification Using CNN Generated by an Efficient Neural Architecture Search with a New Data Processing Trick},
author = {Yu Xue and Ziming Yuan and Adam Slowik},
url = {https://arxiv.org/abs/2110.15277},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1134.
Kong, Qi; Xu, Xin; Zhang, Liangliang
ODNASSD: An End-to-end Object Detection Neural Architecture Search Space Design Proceedings Article
In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3075-3080, 2021.
@inproceedings{9565112,
title = {ODNASSD: An End-to-end Object Detection Neural Architecture Search Space Design},
author = {Qi Kong and Xin Xu and Liangliang Zhang},
url = {https://ieeexplore.ieee.org/abstract/document/9565112},
doi = {10.1109/ITSC48978.2021.9565112},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 IEEE International Intelligent Transportation Systems Conference (ITSC)},
pages = {3075-3080},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1133.
Krishna, Ravi; Kalaiah, Aravind; Wu, Bichen; Naumov, Maxim; Mudigere, Dheevatsa; Smelyanskiy, Misha; Keutzer, Kurt
Differentiable NAS Framework and Application to Ads CTR Prediction Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2110-14812,
title = {Differentiable NAS Framework and Application to Ads CTR Prediction},
author = {Ravi Krishna and Aravind Kalaiah and Bichen Wu and Maxim Naumov and Dheevatsa Mudigere and Misha Smelyanskiy and Kurt Keutzer},
url = {https://arxiv.org/abs/2110.14812},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.14812},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1132.
Cozma, Adriana-Eliza; Morgan, Lisa; Stolz, Martin; Stoeckel, David; Rambach, Kilian
DeepHybrid: Deep Learning on Automotive Radar Spectra and Reflections for Object Classification Proceedings Article
In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 2682-2687, 2021.
@inproceedings{9564526,
title = {DeepHybrid: Deep Learning on Automotive Radar Spectra and Reflections for Object Classification},
author = {Adriana-Eliza Cozma and Lisa Morgan and Martin Stolz and David Stoeckel and Kilian Rambach},
url = {https://ieeexplore.ieee.org/abstract/document/9564526},
doi = {10.1109/ITSC48978.2021.9564526},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 IEEE International Intelligent Transportation Systems Conference (ITSC)},
pages = {2682-2687},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1131.
Xie, Bangquan; Yang, Zongming; Yang, Liang; Luo, Ruifa; Wei, Ailin; Weng, Xiaoxiong; Li, Bing
Multi-Scale Fusion With Matching Attention Model: A Novel Decoding Network Cooperated With NAS for Real-Time Semantic Segmentation Journal Article
In: IEEE Transactions on Intelligent Transportation Systems, pp. 1-11, 2021.
@article{9585519,
title = {Multi-Scale Fusion With Matching Attention Model: A Novel Decoding Network Cooperated With NAS for Real-Time Semantic Segmentation},
author = {Bangquan Xie and Zongming Yang and Liang Yang and Ruifa Luo and Ailin Wei and Xiaoxiong Weng and Bing Li},
url = {https://ieeexplore.ieee.org/abstract/document/9585519},
doi = {10.1109/TITS.2021.3115705},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {IEEE Transactions on Intelligent Transportation Systems},
pages = {1-11},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1130.
Traoré, Kalifou René; Camero, Andrés; Zhu, Xiao Xiang
Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-01584,
title = {Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems},
author = {Kalifou René Traoré and Andrés Camero and Xiao Xiang Zhu},
url = {https://arxiv.org/abs/2111.01584},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.01584},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1129.
Lu, Bingqian; Yang, Jianyi; Jiang, Weiwen; Shi, Yiyu; Ren, Shaolei
One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search Technical Report
2021.
@techreport{lu2021one,
title = {One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search},
author = {Bingqian Lu and Jianyi Yang and Weiwen Jiang and Yiyu Shi and Shaolei Ren},
url = {https://arxiv.org/abs/2111.01203},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {arXiv preprint arXiv:2111.01203},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1128.
Deng, Xiaoqing; Luo, Weiqi; Fang, Yanmei
Spatial Steganalysis Based on Gradient-Based Neural Architecture Search Proceedings Article
In: Huang, Qiong; Yu, Yu (Ed.): Provable and Practical Security, pp. 365–375, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-90402-9.
@inproceedings{10.1007/978-3-030-90402-9_20,
title = {Spatial Steganalysis Based on Gradient-Based Neural Architecture Search},
author = {Xiaoqing Deng and Weiqi Luo and Yanmei Fang},
editor = {Qiong Huang and Yu Yu},
url = {https://link.springer.com/chapter/10.1007/978-3-030-90402-9_20},
isbn = {978-3-030-90402-9},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Provable and Practical Security},
pages = {365--375},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Most existing steganalytic networks are designed empirically, which probably limits their performances. Neural architecture search (NAS) is a technology that can automatically find the optimal network architecture in the search space without excessive manual intervention. In this paper, we introduce a gradient-based NAS method called PC-DARTS in steganalysis. We firstly define the overall network architecture, and the search spaces of the corresponding cells in the network. We then use softmax over all candidate operations to construct an over-parameterized network. By updating the parameters of such a network based on gradient descent, the optimal operations, i.e., the high-pass filters in pre-processing module and operations in feature extraction module, can be obtained. Experimental results show that the resulting steganalytic network via NAS can achieve competitive performance with some advanced well-designed steganalytic networks, while the searching time is relatively short.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Most existing steganalytic networks are designed empirically, which probably limits their performances. Neural architecture search (NAS) is a technology that can automatically find the optimal network architecture in the search space without excessive manual intervention. In this paper, we introduce a gradient-based NAS method called PC-DARTS in steganalysis. We firstly define the overall network architecture, and the search spaces of the corresponding cells in the network. We then use softmax over all candidate operations to construct an over-parameterized network. By updating the parameters of such a network based on gradient descent, the optimal operations, i.e., the high-pass filters in pre-processing module and operations in feature extraction module, can be obtained. Experimental results show that the resulting steganalytic network via NAS can achieve competitive performance with some advanced well-designed steganalytic networks, while the searching time is relatively short.
1127.
Li, Yanxi; Dong, Minjing; Xu, Yixing; Wang, Yunhe; Xu, Chang
Neural architecture tuning with policy adaptation Journal Article
In: Neurocomputing, 2021, ISSN: 0925-2312.
@article{LI2021e,
title = {Neural architecture tuning with policy adaptation},
author = {Yanxi Li and Minjing Dong and Yixing Xu and Yunhe Wang and Chang Xu},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221016325},
doi = {https://doi.org/10.1016/j.neucom.2021.10.095},
issn = {0925-2312},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Neurocomputing},
abstract = {Neural architecture search (NAS) is to automatically design task-specific neural architectures, whose performance has already surpassed those of many manually designed neural networks. Existing NAS techniques focus on searching for the neural architecture and training the optimal network weights from the scratch. Nevertheless, it could be essential to study how to tune a given neural architecture instead of producing a completely new neural architecture in some scenarios, which may lead to a more optimal solution by combining human experience and the advantages of the machine’s automatic searching. This paper proposes to learn to tune the architectures at hand to achieve better performance. The proposed Neural Architecture Tuning (NAT) algorithm trains a deep Q-network to tune neural architectures given a random architecture so that we can achieve better performance on a reduced space. We then apply adversarial autoencoder to make the learned policy be generalized to a different searching space in real-world applications. The proposed algorithm is evaluated on the NAS-Bench-101 dataset. The results indicate that our NAT framework can achieve state-of-the-art performance on the NAS-Bench-101 benchmark, and the learned policy can be adapted to a different search space while maintaining the performance.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) is to automatically design task-specific neural architectures, whose performance has already surpassed those of many manually designed neural networks. Existing NAS techniques focus on searching for the neural architecture and training the optimal network weights from the scratch. Nevertheless, it could be essential to study how to tune a given neural architecture instead of producing a completely new neural architecture in some scenarios, which may lead to a more optimal solution by combining human experience and the advantages of the machine’s automatic searching. This paper proposes to learn to tune the architectures at hand to achieve better performance. The proposed Neural Architecture Tuning (NAT) algorithm trains a deep Q-network to tune neural architectures given a random architecture so that we can achieve better performance on a reduced space. We then apply adversarial autoencoder to make the learned policy be generalized to a different searching space in real-world applications. The proposed algorithm is evaluated on the NAS-Bench-101 dataset. The results indicate that our NAT framework can achieve state-of-the-art performance on the NAS-Bench-101 benchmark, and the learned policy can be adapted to a different search space while maintaining the performance.
1126.
Hou, Wenxuan; Liu, Longjun; Zhang, Haonan; Sun, Hongbin; Zheng, Nanning
DFSNet: Dividing-Fuse Deep Neural Networks with Searching Strategy for Distributed DNN Architecture Journal Article
In: Neurocomputing, 2021, ISSN: 0925-2312.
@article{HOU2021,
title = {DFSNet: Dividing-Fuse Deep Neural Networks with Searching Strategy for Distributed DNN Architecture},
author = {Wenxuan Hou and Longjun Liu and Haonan Zhang and Hongbin Sun and Nanning Zheng},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221016076},
doi = {https://doi.org/10.1016/j.neucom.2021.08.144},
issn = {0925-2312},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Neurocomputing},
abstract = {The overwhelming parameters and computation consumption of deep neural networks limit their applicability to a single computing node with poor computing power, such as edge and mobile devices. Most previous works leverage model pruning and compression strategies to reduce DNN parameters for resource-constrained devices. However, most model compression methods may suffer from accuracy loss. Recently, we find that combine many weak computing nodes as a distributed system to run large and sophisticated DNN models is a promising solution for the issue. However, it is essential for the distributed system to design distributed DNN models and inference schemes, one of the great challenges of distributed system is how to design an efficient distributed DNN model for data parallelism and model parallelism, and communication overhead is also another critical performance bottleneck for distributed DNN model. Therefore, in this article, we propose DFSNet framework (Dividing-Fuse neural Network with Searching Strategy) for distributed DNN architecture. Firstly, the DFSNet framework includes a joint ”dividing-fusing” method to convert regular DNN models into distributed models that are friendly for distributed systems. This method divides the conventional DNN model in the channel dimension, and sets a few special layers to fuse feature-map information from different channel groups for accuracy improvement. Since the fusion layers are sparse in the network, they do not increase too much extra inference time and communication overhead on the distributed nodes, but they can maintain the accuracy of distributed neural networks significantly. Secondly, considering the architecture of distributed computing nodes, we propose a parallel fusion topology to improve the utilization of different computing nodes. Lastly, the popular weight-sharing neural architecture search (NAS) technique is leveraged to search the position of fusion layers in the distributed DNN model for high accuracy and finally generate an efficient distributed DNN model. Compared with the original network, our converted distributed DNN achieves better performance (e.g. 1.88% precision boosting in ResNet56 on CIFAR-100 dataset, and 1.25% precision improving in MobileNetV2 on ImageNet dataset). In addition, most layers of DNN have been divided into different distributed nodes on channel dimension, which is particularly suitable for distributed DNN architecture with very low communication overhead.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The overwhelming parameters and computation consumption of deep neural networks limit their applicability to a single computing node with poor computing power, such as edge and mobile devices. Most previous works leverage model pruning and compression strategies to reduce DNN parameters for resource-constrained devices. However, most model compression methods may suffer from accuracy loss. Recently, we find that combine many weak computing nodes as a distributed system to run large and sophisticated DNN models is a promising solution for the issue. However, it is essential for the distributed system to design distributed DNN models and inference schemes, one of the great challenges of distributed system is how to design an efficient distributed DNN model for data parallelism and model parallelism, and communication overhead is also another critical performance bottleneck for distributed DNN model. Therefore, in this article, we propose DFSNet framework (Dividing-Fuse neural Network with Searching Strategy) for distributed DNN architecture. Firstly, the DFSNet framework includes a joint ”dividing-fusing” method to convert regular DNN models into distributed models that are friendly for distributed systems. This method divides the conventional DNN model in the channel dimension, and sets a few special layers to fuse feature-map information from different channel groups for accuracy improvement. Since the fusion layers are sparse in the network, they do not increase too much extra inference time and communication overhead on the distributed nodes, but they can maintain the accuracy of distributed neural networks significantly. Secondly, considering the architecture of distributed computing nodes, we propose a parallel fusion topology to improve the utilization of different computing nodes. Lastly, the popular weight-sharing neural architecture search (NAS) technique is leveraged to search the position of fusion layers in the distributed DNN model for high accuracy and finally generate an efficient distributed DNN model. Compared with the original network, our converted distributed DNN achieves better performance (e.g. 1.88% precision boosting in ResNet56 on CIFAR-100 dataset, and 1.25% precision improving in MobileNetV2 on ImageNet dataset). In addition, most layers of DNN have been divided into different distributed nodes on channel dimension, which is particularly suitable for distributed DNN architecture with very low communication overhead.
1125.
Wei, Zhikun; Wang, Xin; Zhu, Wenwu
AutoIAS: Automatic Integrated Architecture Searcher for Click-Trough Rate Prediction Proceedings Article
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2101–2110, Association for Computing Machinery, Virtual Event, Queensland, Australia, 2021, ISBN: 9781450384469.
@inproceedings{10.1145/3459637.3482234,
title = {AutoIAS: Automatic Integrated Architecture Searcher for Click-Trough Rate Prediction},
author = {Zhikun Wei and Xin Wang and Wenwu Zhu},
url = {https://doi.org/10.1145/3459637.3482234},
doi = {10.1145/3459637.3482234},
isbn = {9781450384469},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
pages = {2101–2110},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Queensland, Australia},
series = {CIKM '21},
abstract = {Automating architecture design for recommendation tasks becomes a trending topic because expert efforts are saved, and better performance is expected. Neural Architecture Search (NAS) is introduced to discover powerful CTR prediction model architectures in recent works. CTR prediction model usually consists of three components: embedding layer, interaction layer, and deep neural network. However, existing automation works focus on searching single component and leaving other components hand-crafted. The isolated searching will cause incompatibility among components and lead to weak generalization ability. Moreover, there is not a unified framework for integrated CTR prediction model architecture searching. This paper presents Automatic Integrated Architecture Searcher (AutoIAS), a framework that provides a practical and general method to find optimal CTR prediction model architecture in an automatic manner. In AutoIAS, we unify existing interaction-based CTR prediction model architectures and propose an integrated search space for a complete CTR prediction model. We utilize a supernet to predict the performance of sub-architectures, and the supernet is trained with Knowledge Distillation(KD) to enhance consistency among sub-architectures. To efficiently explore the search space, we design an architecture generator network that explicitly models the architecture dependencies among components and generates conditioned architectures distribution for each component. Experiments on public datasets show the outstanding performance and generalization ability of AutoIAS. Ablation study shows the effectiveness of the KD-based supernet training method and the Architecture Generator Network.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Automating architecture design for recommendation tasks becomes a trending topic because expert efforts are saved, and better performance is expected. Neural Architecture Search (NAS) is introduced to discover powerful CTR prediction model architectures in recent works. CTR prediction model usually consists of three components: embedding layer, interaction layer, and deep neural network. However, existing automation works focus on searching single component and leaving other components hand-crafted. The isolated searching will cause incompatibility among components and lead to weak generalization ability. Moreover, there is not a unified framework for integrated CTR prediction model architecture searching. This paper presents Automatic Integrated Architecture Searcher (AutoIAS), a framework that provides a practical and general method to find optimal CTR prediction model architecture in an automatic manner. In AutoIAS, we unify existing interaction-based CTR prediction model architectures and propose an integrated search space for a complete CTR prediction model. We utilize a supernet to predict the performance of sub-architectures, and the supernet is trained with Knowledge Distillation(KD) to enhance consistency among sub-architectures. To efficiently explore the search space, we design an architecture generator network that explicitly models the architecture dependencies among components and generates conditioned architectures distribution for each component. Experiments on public datasets show the outstanding performance and generalization ability of AutoIAS. Ablation study shows the effectiveness of the KD-based supernet training method and the Architecture Generator Network.
1124.
Li, Yaliang; Wang, Zhen; Xie, Yuexiang; Ding, Bolin; Zeng, Kai; Zhang, Ce
AutoML: From Methodology to Application Proceedings Article
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 4853–4856, Association for Computing Machinery, Virtual Event, Queensland, Australia, 2021, ISBN: 9781450384469.
@inproceedings{10.1145/3459637.3483279,
title = {AutoML: From Methodology to Application},
author = {Yaliang Li and Zhen Wang and Yuexiang Xie and Bolin Ding and Kai Zeng and Ce Zhang},
url = {https://doi.org/10.1145/3459637.3483279},
doi = {10.1145/3459637.3483279},
isbn = {9781450384469},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
pages = {4853–4856},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Queensland, Australia},
series = {CIKM '21},
abstract = {Machine Learning methods have been adopted for a wide range of real-world applications, ranging from social networks, online image/video-sharing platforms, and e-commerce to education, healthcare, etc. However, in practice, a large amount of effort is required to tune several components of machine learning methods, including data representation, hyperparameter, and model architecture, in order to achieve a good performance. To alleviate the required tunning efforts, Automated Machine Learning (AutoML), which can automate the process of applying machine learning methods, has been studied in both academy and industry recently. In this tutorial, we will introduce the main research topics of AutoML, including Hyperparameter Optimization, Neural Architecture Search, and Meta-Learning. Two emerging topics of AutoML, Automatic Feature Generation and Machine Learning Guided Database, will also be discussed since they are important components for real-world applications. For each topic, we will motivate it with application examples from industry, illustrate the state-of-the-art methodologies, and discuss some future research directions based on our experience from industry and the trends in academy.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Machine Learning methods have been adopted for a wide range of real-world applications, ranging from social networks, online image/video-sharing platforms, and e-commerce to education, healthcare, etc. However, in practice, a large amount of effort is required to tune several components of machine learning methods, including data representation, hyperparameter, and model architecture, in order to achieve a good performance. To alleviate the required tunning efforts, Automated Machine Learning (AutoML), which can automate the process of applying machine learning methods, has been studied in both academy and industry recently. In this tutorial, we will introduce the main research topics of AutoML, including Hyperparameter Optimization, Neural Architecture Search, and Meta-Learning. Two emerging topics of AutoML, Automatic Feature Generation and Machine Learning Guided Database, will also be discussed since they are important components for real-world applications. For each topic, we will motivate it with application examples from industry, illustrate the state-of-the-art methodologies, and discuss some future research directions based on our experience from industry and the trends in academy.
1123.
Wei, Penghui; Zhang, Weimin; Xu, Zixuan; Liu, Shaoguo; Lee, Kuang-chih; Zheng, Bo
AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation Proceedings Article
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3528–3532, Association for Computing Machinery, Virtual Event, Queensland, Australia, 2021, ISBN: 9781450384469.
@inproceedings{10.1145/3459637.3482061,
title = {AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation},
author = {Penghui Wei and Weimin Zhang and Zixuan Xu and Shaoguo Liu and Kuang-chih Lee and Bo Zheng},
url = {https://doi.org/10.1145/3459637.3482061},
doi = {10.1145/3459637.3482061},
isbn = {9781450384469},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the 30th ACM International Conference on Information & Knowledge Management},
pages = {3528–3532},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Queensland, Australia},
series = {CIKM '21},
abstract = {Post-click conversion rate (CVR) estimation is a crucial task in online advertising and recommendation systems. To address the sample selection bias problem in traditional CVR models trained in click space, recent studies perform entire space multi-task learning based on the probability of events in user behavior funnels like "impression-click-conversion". However, those models learn the feature representation of each task independently, and omit potential inter-task correlations that can help improve the CVR estimation performance. In this paper, we propose AutoHERI, an entire space CVR model with automated hierarchical representation integration, which leverages the interplay across multi-tasks' representation learning. It performs neural architecture search to learn optimal connections between layer-wise representations of different tasks. Besides, AutoHERI achieves better search efficiency with one-shot search algorithm, and thus it can be easily extended to new scenarios that have more complex user behaviors. Both offline and online experimental results on large-scale real-world datasets verify that AutoHERI outperforms previous entire space models significantly.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Post-click conversion rate (CVR) estimation is a crucial task in online advertising and recommendation systems. To address the sample selection bias problem in traditional CVR models trained in click space, recent studies perform entire space multi-task learning based on the probability of events in user behavior funnels like "impression-click-conversion". However, those models learn the feature representation of each task independently, and omit potential inter-task correlations that can help improve the CVR estimation performance. In this paper, we propose AutoHERI, an entire space CVR model with automated hierarchical representation integration, which leverages the interplay across multi-tasks' representation learning. It performs neural architecture search to learn optimal connections between layer-wise representations of different tasks. Besides, AutoHERI achieves better search efficiency with one-shot search algorithm, and thus it can be easily extended to new scenarios that have more complex user behaviors. Both offline and online experimental results on large-scale real-world datasets verify that AutoHERI outperforms previous entire space models significantly.
1122.
Lin, Jun-Liang; Wang, Sheng-De
Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices Technical Report
2021.
@techreport{lin2021communicationefficient,
title = {Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices},
author = {Jun-Liang Lin and Sheng-De Wang},
url = {https://arxiv.org/abs/2111.02489},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1121.
Cao, Jie; Ma, Jialin; Huang, Dailin; Yu, Ping
Finding the optimal multilayer network structure through reinforcement learning in fault diagnosis Journal Article
In: Measurement, pp. 110377, 2021, ISSN: 0263-2241.
@article{CAO2021110377,
title = {Finding the optimal multilayer network structure through reinforcement learning in fault diagnosis},
author = {Jie Cao and Jialin Ma and Dailin Huang and Ping Yu},
url = {https://www.sciencedirect.com/science/article/pii/S0263224121012707},
doi = {https://doi.org/10.1016/j.measurement.2021.110377},
issn = {0263-2241},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Measurement},
pages = {110377},
abstract = {Deep learning (DL) is an important method in industrial fault diagnosis. However, DL’s network structure needs to be designed with experience. To simplify the design of network structures, we propose the neural architecture search network with Pareto efficiency reward and insert replay buffer (NAS-PERIRB) algorithm. In this paper, the early stopping and insert replay buffer (IRB) are used to improving the training efficiency of the samples. In addition, we design the Pareto efficiency reward function to optimize the goals and design a network search space to perform effective searches. What is more, we evaluate the NAS-PERIRB under two datasets. Results show that the two datasets have reached 99% accuracy in various situations, which means the NAS-PERIRB can achieve the purpose of designing the network structure independently.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Deep learning (DL) is an important method in industrial fault diagnosis. However, DL’s network structure needs to be designed with experience. To simplify the design of network structures, we propose the neural architecture search network with Pareto efficiency reward and insert replay buffer (NAS-PERIRB) algorithm. In this paper, the early stopping and insert replay buffer (IRB) are used to improving the training efficiency of the samples. In addition, we design the Pareto efficiency reward function to optimize the goals and design a network search space to perform effective searches. What is more, we evaluate the NAS-PERIRB under two datasets. Results show that the two datasets have reached 99% accuracy in various situations, which means the NAS-PERIRB can achieve the purpose of designing the network structure independently.
1120.
Chen, Zhe; Wang, Wenhai; Xie, Enze; Yang, Zhibo; Lu, Tong; Luo, Ping
FAST: Searching for a Faster Arbitrarily-Shaped Text Detector with Minimalist Kernel Representation Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-02394,
title = {FAST: Searching for a Faster Arbitrarily-Shaped Text Detector with Minimalist Kernel Representation},
author = {Zhe Chen and Wenhai Wang and Enze Xie and Zhibo Yang and Tong Lu and Ping Luo},
url = {https://arxiv.org/abs/2111.02394},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.02394},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1119.
Traoré, Kalifou René; Camero, Andrés; Zhu, Xiao Xiang
A Data-driven Approach to Neural Architecture Search Initialization Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03524,
title = {A Data-driven Approach to Neural Architecture Search Initialization},
author = {Kalifou René Traoré and Andrés Camero and Xiao Xiang Zhu},
url = {https://arxiv.org/abs/2111.03524},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03524},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1118.
Eyono, Roy Henha; Carlucci, Fabio Maria; Esperança, Pedro M.; Ru, Binxin; Torr, Phillip
AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03555,
title = {AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family},
author = {Roy Henha Eyono and Fabio Maria Carlucci and Pedro M. Esperança and Binxin Ru and Phillip Torr},
url = {https://arxiv.org/abs/2111.03555},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03555},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1117.
Xu, Zhimeng; Mai, Yuting; Liu, Denghui; He, Wenjun; Lin, Xinyuan; Xu, Chi; Zhang, Lei; Meng, Xin; Mafofo, Joseph; Zaher, Walid Abbas; Koshy, Ashish; Li, Yi; Qiao, Nan
Fast-bonito: A faster deep learning based basecaller for nanopore sequencing Journal Article
In: Artificial Intelligence in the Life Sciences, vol. 1, pp. 100011, 2021, ISSN: 2667-3185.
@article{XU2021100011,
title = {Fast-bonito: A faster deep learning based basecaller for nanopore sequencing},
author = {Zhimeng Xu and Yuting Mai and Denghui Liu and Wenjun He and Xinyuan Lin and Chi Xu and Lei Zhang and Xin Meng and Joseph Mafofo and Walid Abbas Zaher and Ashish Koshy and Yi Li and Nan Qiao},
url = {https://www.sciencedirect.com/science/article/pii/S2667318521000118},
doi = {https://doi.org/10.1016/j.ailsci.2021.100011},
issn = {2667-3185},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {Artificial Intelligence in the Life Sciences},
volume = {1},
pages = {100011},
abstract = {Nanopore sequencing from Oxford Nanopore Technologies (ONT) is a promising third-generation sequencing (TGS) technology that generates relatively longer sequencing reads compared to the next-generation sequencing (NGS) technology. A basecaller is a piece of software that translates the original electrical current signals into nucleotide sequences. The accuracy of the basecaller is crucially important to downstream analysis. Bonito is a deep learning-based basecaller recently developed by ONT. Its neural network architecture is composed of a single convolutional layer followed by three stacked bidirectional gated recurrent unit (GRU) layers. Although Bonito has achieved state-of-the-art base calling accuracy, its speed is too slow to be used in production. We therefore developed Fast-Bonito, by using the neural architecture search (NAS) technique to search for a brand-new neural network backbone, and trained it from scratch using several advanced deep learning model training techniques. The new Fast-Bonito model balanced performance in terms of speed and accuracy. Fast-Bonito was 153.8% faster than the original Bonito on NVIDIA V100 GPU. When running on HUAWEI Ascend 910 NPU, Fast-Bonito was 565% faster than the original Bonito. The accuracy of Fast-Bonito was also slightly higher than that of Bonito. We have made Fast-Bonito open source, hoping it will boost the adoption of TGS in both academia and industry.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Nanopore sequencing from Oxford Nanopore Technologies (ONT) is a promising third-generation sequencing (TGS) technology that generates relatively longer sequencing reads compared to the next-generation sequencing (NGS) technology. A basecaller is a piece of software that translates the original electrical current signals into nucleotide sequences. The accuracy of the basecaller is crucially important to downstream analysis. Bonito is a deep learning-based basecaller recently developed by ONT. Its neural network architecture is composed of a single convolutional layer followed by three stacked bidirectional gated recurrent unit (GRU) layers. Although Bonito has achieved state-of-the-art base calling accuracy, its speed is too slow to be used in production. We therefore developed Fast-Bonito, by using the neural architecture search (NAS) technique to search for a brand-new neural network backbone, and trained it from scratch using several advanced deep learning model training techniques. The new Fast-Bonito model balanced performance in terms of speed and accuracy. Fast-Bonito was 153.8% faster than the original Bonito on NVIDIA V100 GPU. When running on HUAWEI Ascend 910 NPU, Fast-Bonito was 565% faster than the original Bonito. The accuracy of Fast-Bonito was also slightly higher than that of Bonito. We have made Fast-Bonito open source, hoping it will boost the adoption of TGS in both academia and industry.
1116.
Yan, Shen; White, Colin; Savani, Yash; Hutter, Frank
NAS-Bench-x11 and the Power of Learning Curves Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-03602,
title = {NAS-Bench-x11 and the Power of Learning Curves},
author = {Shen Yan and Colin White and Yash Savani and Frank Hutter},
url = {https://arxiv.org/abs/2111.03602},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.03602},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1115.
Ding, Xinyi; Han, Tao; Fang, Yili; Larson, Eric C.
An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2111-04497,
title = {An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing},
author = {Xinyi Ding and Tao Han and Yili Fang and Eric C. Larson},
url = {https://arxiv.org/abs/2111.04497},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {CoRR},
volume = {abs/2111.04497},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}