TinyM²Net-V2: A Compact Low Power Software Hardware Architecture for Multimodal Deep Neural Networks

Files

3595633.pdf (2.51 MB)

Links to Files

https://dl.acm.org/doi/abs/10.1145/3595633

Permanent Link

https://doi.org/10.1145/3595633
 http://hdl.handle.net/11603/28089

Collections

UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection
UMBC Information Systems Department
UMBC Student Collection

Author/Creator

Author/Creator ORCID

https://orcid.org/0000-0002-9983-6929

Date

2023-05-03

Type of Work

journal articles
postprints
Text

Department

Program

Citation of Original Publication

Rashid,Hasib-Al, Utteja Kallakuri, Tinoosh Mohsenin. "TinyM²Net-V2: A Compact Low Power Software Hardware Architecture for Multimodal Deep Neural Networks" ACM Transactions on Embedded Computing Systems (3 May,2023). https://doi.org/10.1145/3595633.

Rights

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

Subjects

Abstract

With the evaluation of Artificial Intelligence (AI), there has been a resurgence of interest in how to use AI algorithms on low-power embedded systems to broaden potential use cases of the Internet of Things (IoT). To mimic multimodal human perception, multimodal deep neural networks (M-DNN) have recently become very popular with the classification task due to their impressive performance for computer vision and audio processing tasks. This paper presents TinyM²Net-V2 - a compact low-power software hardware architecture for multimodal deep neural networks for resource-constrained tiny devices. In order to compress the models to implement on tiny devices, cyclicly sparsification and hybrid quantization (4-bits weights and 8-bits activations) methods are used. Although model compression techniques are an active research area, we are the first to demonstrate their efficacy for multimodal deep neural networks, using cyclicly sparsification and hybrid quantization of weights/activations. TinyM²Net-V2 shows that even a tiny multimodal deep neural network model can improve the classification accuracy more than that of any unimodal counterparts. Parameterized M-DNN model architecture was designed to be evaluated in two different case-studies: vehicle detection from multimodal images and audios and COVID-19 detection from multimodal audio recordings. The most compressed TinyM²Net-V2 achieves 92.5% COVID-19 detection accuracy (6.8% improvement from the unimodal full precision model) and 90.6% vehicle classification accuracy (7.7% improvement from the unimodal full precision model). A parameterized and flexible FPGA hardware accelerator was designed as well for TinyM²Net-V2 models. To the best of our knowledge, this is the first work accelerating multimodal deep neural network models on low power Artix-7 FPGA hardware. We achieved energy efficiency of 9.04 GOP/s/W and 15.38 GOP/s/W for case-study 1 and case-study 2 respectively which is comparable to the state-of-the-art results. Finally, we compared our tiny FPGA hardware implementation results with off-the-shelf resource-constrained devices and showed our implementation is faster and consumed less power compared to the off-the-shelf resource-constrained devices.