• Login
    View Item 
    •   Maryland Shared Open Access Repository Home
    • ScholarWorks@UMBC
    • UMBC College of Engineering and Information Technology
    • UMBC Computer Science and Electrical Engineering Department
    • View Item
    •   Maryland Shared Open Access Repository Home
    • ScholarWorks@UMBC
    • UMBC College of Engineering and Information Technology
    • UMBC Computer Science and Electrical Engineering Department
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    TinyM²Net-V2: A Compact Low Power Software Hardware Architecture for Multimodal Deep Neural Networks

    Thumbnail
    Files
    3595633.pdf (2.509Mb)
    Links to Files
    https://dl.acm.org/doi/abs/10.1145/3595633
    Permanent Link
    https://doi.org/10.1145/3595633
    http://hdl.handle.net/11603/28089
    Collections
    • UMBC Computer Science and Electrical Engineering Department
    • UMBC Faculty Collection
    • UMBC Information Systems Department
    • UMBC Student Collection
    Metadata
    Show full item record
    Author/Creator
    Rashid, Hasib-Al
    Kallakuri, Utteja
    Mohsenin, Tinoosh
    Author/Creator ORCID
    https://orcid.org/0000-0002-9983-6929
    Date
    2023-05-03
    Type of Work
    22 pages
    Text
    journal articles
    postprints
    Citation of Original Publication
    Rashid,Hasib-Al, Utteja Kallakuri, Tinoosh Mohsenin. "TinyM²Net-V2: A Compact Low Power Software Hardware Architecture for Multimodal Deep Neural Networks" ACM Transactions on Embedded Computing Systems (3 May,2023). https://doi.org/10.1145/3595633.
    Rights
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.
    Abstract
    With the evaluation of Artificial Intelligence (AI), there has been a resurgence of interest in how to use AI algorithms on low-power embedded systems to broaden potential use cases of the Internet of Things (IoT). To mimic multimodal human perception, multimodal deep neural networks (M-DNN) have recently become very popular with the classification task due to their impressive performance for computer vision and audio processing tasks. This paper presents TinyM²Net-V2 - a compact low-power software hardware architecture for multimodal deep neural networks for resource-constrained tiny devices. In order to compress the models to implement on tiny devices, cyclicly sparsification and hybrid quantization (4-bits weights and 8-bits activations) methods are used. Although model compression techniques are an active research area, we are the first to demonstrate their efficacy for multimodal deep neural networks, using cyclicly sparsification and hybrid quantization of weights/activations. TinyM²Net-V2 shows that even a tiny multimodal deep neural network model can improve the classification accuracy more than that of any unimodal counterparts. Parameterized M-DNN model architecture was designed to be evaluated in two different case-studies: vehicle detection from multimodal images and audios and COVID-19 detection from multimodal audio recordings. The most compressed TinyM²Net-V2 achieves 92.5% COVID-19 detection accuracy (6.8% improvement from the unimodal full precision model) and 90.6% vehicle classification accuracy (7.7% improvement from the unimodal full precision model). A parameterized and flexible FPGA hardware accelerator was designed as well for TinyM²Net-V2 models. To the best of our knowledge, this is the first work accelerating multimodal deep neural network models on low power Artix-7 FPGA hardware. We achieved energy efficiency of 9.04 GOP/s/W and 15.38 GOP/s/W for case-study 1 and case-study 2 respectively which is comparable to the state-of-the-art results. Finally, we compared our tiny FPGA hardware implementation results with off-the-shelf resource-constrained devices and showed our implementation is faster and consumed less power compared to the off-the-shelf resource-constrained devices.


    Albin O. Kuhn Library & Gallery
    University of Maryland, Baltimore County
    1000 Hilltop Circle
    Baltimore, MD 21250
    www.umbc.edu/scholarworks

    Contact information:
    Email: scholarworks-group@umbc.edu
    Phone: 410-455-3544


    If you wish to submit a copyright complaint or withdrawal request, please email mdsoar-help@umd.edu.

     

     

    My Account

    LoginRegister

    Browse

    This CollectionBy Issue DateTitlesAuthorsSubjectsType

    Statistics

    View Usage Statistics


    Albin O. Kuhn Library & Gallery
    University of Maryland, Baltimore County
    1000 Hilltop Circle
    Baltimore, MD 21250
    www.umbc.edu/scholarworks

    Contact information:
    Email: scholarworks-group@umbc.edu
    Phone: 410-455-3544


    If you wish to submit a copyright complaint or withdrawal request, please email mdsoar-help@umd.edu.