Energy Efficient Deep Spiking Recurrent Neural Networks: A Reservoir Computing-Based Approach
| dc.contributor.author | Hamedani, Kian | en |
| dc.contributor.committeechair | Yi, Yang | en |
| dc.contributor.committeechair | Liu, Lingjia | en |
| dc.contributor.committeemember | Ha, Dong S. | en |
| dc.contributor.committeemember | Reed, Jeffrey H. | en |
| dc.contributor.committeemember | Butt, Ali | en |
| dc.contributor.department | Electrical Engineering | en |
| dc.date.accessioned | 2021-12-11T07:00:19Z | en |
| dc.date.available | 2021-12-11T07:00:19Z | en |
| dc.date.issued | 2020-06-18 | en |
| dc.description.abstract | Recurrent neural networks (RNNs) have been widely used for supervised pattern recognition and exploring the underlying spatio-temporal correlation. However, due to the vanishing/exploding gradient problem, training a fully connected RNN in many cases is very difficult or even impossible. The difficulties of training traditional RNNs, led us to reservoir computing (RC) which recently attracted a lot of attention due to its simple training methods and fixed weights at its recurrent layer. There are three different categories of RC systems, namely, echo state networks (ESNs), liquid state machines (LSMs), and delayed feedback reservoirs (DFRs). In this dissertation a novel structure of RNNs which is inspired by dynamic delayed feedback loops is introduced. In the reservoir (recurrent) layer of DFR, only one neuron is required which makes DFRs extremely suitable for hardware implementations. The main motivation of this dissertation is to introduce an energy efficient, and easy to train RNN while this model achieves high performances in different tasks compared to the state-of-the-art. To improve the energy efficiency of our model, we propose to adopt spiking neurons as the information processing unit of DFR. Spiking neural networks (SNNs) are the most biologically plausible and energy efficient class of artificial neural networks (ANNs). The traditional analog ANNs have marginal similarity with the brain-like information processing. It is clear that the biological neurons communicate together through spikes. Therefore, artificial SNNs have been introduced to mimic the biological neurons. On the other hand, the hardware implementation of SNNs have shown to be extremely energy efficient. Towards achieving this overarching goal, this dissertation presents a spiking DFR (SDFR) with novel encoding schemes, and defense mechanisms against adversarial attacks. To verify the effectiveness and performance of the SDFR, it is adopted in three different applications where there exists a significant Spatio-temporal correlations. These three applications are attack detection in smart grids, spectrum sensing of multi-input-multi-output(MIMO)-orthogonal frequency division multiplexing (OFDM) Dynamic Spectrum Sharing (DSS) systems, and video-based face recognition. In this dissertation, the performance of SDFR is first verified in cyber attack detection in Smart grids. Smart grids are a new generation of power grids which guarantee a more reliable and efficient transmission and delivery of power to the costumers. A more reliable and efficient power generation and distribution can be realized through the integration of internet, telecommunication, and energy technologies. The convergence of different technologies, brings up opportunities, but the challenges are also inevitable. One of the major challenges that pose threat to the smart grids is cyber-attacks. A novel method is developed to detect false data injection (FDI) attacks in smart grids. The second novel application of SDFR is the spectrum sensing of MIMO-OFDM DSS systems. DSS is being implemented in the fifth generation of wireless communication systems (5G) to improve the spectrum efficiency. In a MIMO-OFDM system, not all the subcarriers are utilized simultaneously by the primary user (PU). Therefore, it is essential to sense the idle frequency bands and assign them to the secondary user (SU). The effectiveness of SDFR in capturing the spatio-temporal correlation of MIMO-OFDM time-series and predicting the availability of frequency bands in the future time slots is studied as well. In the third application, the SDFR is modified to be adopted in video-based face recognition. In this task, the SDFR is leveraged to recognize the identities of different subjects while they rotate their heads in different angles. Another contribution of this dissertation is to propose a novel encoding scheme of spiking neurons which is inspired by the cognitive studies of rats. For the first time, the multiplexing of multiple neural codes is introduced and it is shown that the robustness and resilience of the spiking neurons is increased against noisy data, and adversarial attacks, respectively. Adversarial attacks are small and imperceptible perturbations of the input data, which have shown to be able to fool deep learning (DL) models. So far, many adversarial attack and defense mechanisms have been introduced for DL models. Compromising the security and reliability of artificial intelligence (AI) systems is a major concern of government, industry and cyber-security researchers, in that insufficient protections can compromise the security and privacy of everyone in society. Finally, a defense mechanism to protect spiking neurons against adversarial attacks is introduced for the first time. In a nutshell, this dissertation presents a novel energy efficient deep spiking recurrent neural network which is inspired by delayed dynamic loops. The effectiveness of the introduced model is verified in several different applications. At the end, novel encoding and defense mechanisms are introduced which improve the robustness of the model against noise and adversarial attacks. | en |
| dc.description.abstractgeneral | The ultimate goal of artificial intelligence (AI) is to mimic the human brain. Artificial neural networks (ANN) are an attempt to realize that goal. However, traditional ANNs are very far from mimicking biological neurons. It is well-known that biological neurons communicate with one another through signals in the format of spikes. Therefore, artificial spiking neural networks (SNNs) have been introduced which behave more similarly to biological neurons. Moreover, SNNs are very energy efficient which makes them a suitable choice for hardware implementation of ANNs (neuromporphic computing). Despite the many benefits that are brought about by SNNs, they are still behind traditional ANNs in terms of performance. Therefore, in this dissertation, a new structure of SNNs is introduced which outperforms the traditional ANNs in three different applications. This new structure is inspired by delayed dynamic loops which exist in biological brains. The main objective of this novel structure is to capture the spatio-temporal correlation which exists in time-series while the training overhead and power consumption is reduced. Another contribution of this dissertation is to introduce novel encoding schemes for spiking neurons. It is clear that biological neurons leverage spikes, but the language that they use to communicate is not clear. Hence, the spikes require to be encoded in a certain language which is called neural spike encoding scheme. Inspired by the cognitive studies of rats, a novel encoding scheme is presented. Lastly, it is shown that the introduced encoding scheme increases the robustness of SNNs against noisy data and adversarial attacks. AI models including SNNs have shown to be vulnerable to adversarial attacks. Adversarial attacks are minor perturbations of the input data that can cause the AI model to misscalassify the data. For the first time, a defense mechanism is introduced which can protect SNNs against such attacks. | en |
| dc.description.degree | Doctor of Philosophy | en |
| dc.format.medium | ETD | en |
| dc.identifier.other | vt_gsexam:25571 | en |
| dc.identifier.uri | http://hdl.handle.net/10919/106938 | en |
| dc.publisher | Virginia Tech | en |
| dc.rights | In Copyright | en |
| dc.rights.uri | http://rightsstatements.org/vocab/InC/1.0/ | en |
| dc.subject | Recurrent Neural Network | en |
| dc.subject | Reservoir Computing | en |
| dc.subject | Spiking Neural Networks | en |
| dc.subject | Smart Grids | en |
| dc.subject | Spectrum Sensing | en |
| dc.subject | Adversarial Attacks. | en |
| dc.title | Energy Efficient Deep Spiking Recurrent Neural Networks: A Reservoir Computing-Based Approach | en |
| dc.type | Dissertation | en |
| thesis.degree.discipline | Electrical Engineering | en |
| thesis.degree.grantor | Virginia Polytechnic Institute and State University | en |
| thesis.degree.level | doctoral | en |
| thesis.degree.name | Doctor of Philosophy | en |