With the advent of ubiquitous computing and Internet of Things (IoT), potentially billions of devices will create a broad range of data services and applications, which will require the communication networks to efficiently manage the increasing complexity. Cognitive network has been envisioned as a new paradigm to address this challenge, which has the capability of reasoning, planning and learning by incorporating cutting edge technologies including knowledge representation, context awareness, network optimization and machine learning. Cognitive network spans over the entire communication system including the core network and wireless links across the entire protocol stack. Cognitive Radio Network (CRN) is a part of cognitive network over wireless links, which endeavors to better utilize the spectrum resources. Core network provides a reliable backend infrastructure to the entire communication system. However, the CR communication and core network infrastructure have attracted various security threats, which become increasingly severe in pace with the growing complexity and adversity of the modern Internet.

The focus of this dissertation is to exploit the security vulnerabilities of the state-of-the-art cognitive communication systems, and to provide detection, mitigation and protection mechanisms to allow security enhanced cognitive communications including wireless communications in CRNs and wired communications in core networks. In order to provide secure and reliable communications in CRNs: emph{first}, we incorporate security mechanisms into fundamental CRN functions, such as secure spectrum sensing techniques that will ensure trustworthy reporting of spectrum reading. emph{Second}, as no security mechanism can completely prevent all potential threats from entering CRNs, we design a systematic passive monitoring framework, emph{SpecMonitor}, based on unsupervised machine learning methods to strategically monitor the network traffic and operations in order to detect abnormal and malicious behaviors. emph{Third}, highly capable cognitive radios allow more sophisticated reactive jamming attack, which imposes a serious threat to CR communications. By exploiting MIMO interference cancellation techniques, we propose jamming resilient CR communication mechanisms to survive in the presence of reactive jammers. Finally, we focus on protecting the core network from botnet threats by applying cognitive technologies to detect network-wide Peer-to-Peer (P2P) botnets, which leads to the design of a data-driven botnet detection system, called emph{PeerClean}. In all the four research thrusts, we present thorough security analysis, extensive simulations and testbed evaluations based on real-world implementations. Our results demonstrate that the proposed defense mechanisms can effectively and efficiently counteract sophisticated yet powerful attacks.