A Tutorial on Neural Networks and Gradient-free Training

Thumbnail Image

Files

2211.17217.pdf (385.32 KB)

Links to Files

https://arxiv.org/abs/2211.17217

Permanent Link

https://doi.org/10.48550/arXiv.2211.17217
 http://hdl.handle.net/11603/26531

Collections

UMBC Mechanical Engineering Department
UMBC Faculty Collection
UMBC Student Collection

Author/Creator

Author/Creator ORCID

https://orcid.org/0000-0002-4146-6275

Date

2022-11-26

Type of Work

journal articles
preprints
Text

Department

Program

Citation of Original Publication

Rights

This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)

Subjects

Abstract

This paper presents a compact, matrix-based representation of neural networks in a self-contained tutorial fashion. Although neural networks are well-understood pictorially in terms of interconnected neurons, neural networks are mathematical nonlinear functions constructed by composing several vector-valued functions. Using basic results from linear algebra, we represent a neural network as an alternating sequence of linear maps and scalar nonlinear functions, also known as activation functions. The training of neural networks requires the minimization of a cost function, which in turn requires the computation of a gradient. Using basic multivariable calculus results, the cost gradient is also shown to be a function composed of a sequence of linear maps and nonlinear functions. In addition to the analytical gradient computation, we consider two gradient-free training methods and compare the three training methods in terms of convergence rate and prediction accuracy