Neurosymbolic Representation Learning with Holographic Reduced Representations

Abstract

Artificial intelligence has dramatically impacted our modern world, as rapid advancements in model capabilities have opened opportunities for AI to be used in various aspects of our everyday lives, finding applications in fields ranging from robotics and engineering, to creative endeavors such as writing and art. However, despite the big impact of these models, and our growing reliance on them, they still largely lack many of the key facets of human intelligence, such as the ability to reason under certainty, and to leverage novel combinations of concepts that were previously encountered. This has led to an interest in developing ways to augment the powerful feature extraction capabilities of deep neural networks (DNNs) with explicit rule-based symbolic methods, a combination which is often known as a neurosymbolic AI system. Neurosymbolic methods can provide DNNs with the missing tools they need to symbolically represent and manipulate ideas, potentially enabling more rigorous reasoning while also introducing more transparency into the reasoning process itself, through interpretable reasoning traces on discrete objects. Although neurosymbolic methods hold a lot of promise to enhance the capabilities of DNNs, many implementations are application-specific and cannot be broadly applied. Thus, the ability to represent arbitrary data in a neurosymbolic fashion could be an important step towards enabling these enhanced capabilities. This thesis provides a step towards addressing this problem of neurosymbolic representation learning through a novel integration of DNNs with holographic reduced representations (HRRs), a framework for implementing symbolic processing with continuous vectors. In particular, this thesis develops an HRR method that can be applied to the general problem of disentanglement, that focuses on separating the factors of variation in a dataset, something which comes naturally to humans. Whereas prior works approached the problem with fully neural implementations, it is shown that this neurosymbolic approach is able to create disentangled representations of data that produce qualitative and quantitative results that outperform many prior baselines. The empirical findings are complemented with an information-theoretic analysis of the proposed method. Additionally, it is shown that the process produces approximately independent symbol-value pairs (also called slots) and a per-slot capacity bound that quantifies how many distinct symbolic concepts the representation can reliably encode is also derived, providing a quantitative account of the inductive bias that leads to disentanglement. The disentangled representations produced in this process differ from other autoencoder based models in that the individual latent units are vectors themselves, which are summed together to form the whole representation, differing from the paradigm of latent units behaving as scalar dimensions of low dimensional vectors. It is shown that this distributed type of representation is more robust to noise than other disentangled representations and can maintain good reconstruction performance across a range of signal-to-noise rations (SNRs), while simultaneously gracefully degrading in the recoverable semantic content. These findings on robustness are also complemented with a quantification of how Gaussian noise affects the bit error rate of a separate family of symbolic vectors which use binary entries. In a nutshell, the results of this thesis show that the use of vector symbolic architectures (VSA), such as HRRs, may hold a promising potential for representation learning, paving the way toward exploring new ways in which the symbolic benefits of VSAs can be used to represent data with DNNs.