TinyM²Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices
| dc.contributor.author | Rashid, Hasib-Al | |
| dc.contributor.author | Ovi, Pretom Roy | |
| dc.contributor.author | Gangopadhyay, Aryya | |
| dc.contributor.author | Mohsenin, Tinoosh | |
| dc.date.accessioned | 2022-03-15T16:23:28Z | |
| dc.date.available | 2022-03-15T16:23:28Z | |
| dc.date.issued | 2022-02-09 | |
| dc.description | tinyML Research Symposium’22, March 2022, San Jose, CA | |
| dc.description.abstract | With the emergence of Artificial Intelligence (AI), new attention has been given to implement AI algorithms on resource constrained tiny devices to expand the application domain of IoT. Multimodal Learning has recently become very popular with the classification task due to its impressive performance for both image and audio event classification. This paper presents TinyM²Net -- a flexible system algorithm co-designed multimodal learning framework for resource constrained tiny devices. The framework was designed to be evaluated on two different case-studies: COVID-19 detection from multimodal audio recordings and battle field object detection from multimodal images and audios. In order to compress the model to implement on tiny devices, substantial network architecture optimization and mixed precision quantization were performed (mixed 8-bit and 4-bit). TinyM²Net shows that even a tiny multimodal learning model can improve the classification performance than that of any unimodal frameworks. The most compressed TinyM²Net achieves 88.4% COVID-19 detection accuracy (14.5% improvement from unimodal base model) and 96.8\% battle field object detection accuracy (3.9% improvement from unimodal base model). Finally, we test our TinyM²Net models on a Raspberry Pi 4 to see how they perform when deployed to a resource constrained tiny device. | en_US |
| dc.description.sponsorship | We acknowledge the support of the U.S. Army Grant No. W911NF21- 20076. We also acknowledge the support of the University of Maryland, Baltimore, Institute for Clinical and Translational Research (ICTR) and the National Center for Advancing Translational Sciences (NCATS) Clinical Translational Science Award (CTSA) grant number UL1TR003098 | en_US |
| dc.description.uri | https://arxiv.org/abs/2202.04303 | en_US |
| dc.description.uri | https://www.youtube.com/watch?v=qD8mxEhFBcI | |
| dc.format.extent | 7 pages | en_US |
| dc.genre | conference papers and proceedings | en_US |
| dc.genre | preprints | en_US |
| dc.genre | video recordings | |
| dc.identifier | doi:10.13016/m2k9xf-fqm5 | |
| dc.identifier.uri | https://doi.org/10.48550/arXiv.2202.04303 | |
| dc.identifier.uri | http://hdl.handle.net/11603/24393 | |
| dc.language.iso | en_US | en_US |
| dc.relation.isAvailableAt | The University of Maryland, Baltimore County (UMBC) | |
| dc.relation.ispartof | UMBC Computer Science and Electrical Engineering Department Collection | |
| dc.relation.ispartof | UMBC Faculty Collection | |
| dc.relation.ispartof | UMBC Student Collection | |
| dc.relation.ispartof | UMBC Information Systems Department | |
| dc.rights | This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author. | en_US |
| dc.rights | Attribution 4.0 International (CC BY 4.0) | |
| dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | * |
| dc.title | TinyM²Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices | en_US |
| dc.type | Text | en_US |
| dcterms.creator | https://orcid.org/0000-0002-9983-6929 | en_US |