Since its inception, cognitive radio (CR) has quickly been accepted as the enabling radio technology for next-generation wireless communications. A CR promises unprecedented flexibility in radio functionalities via programmability at the lowest layer, which was once done in hardware. Due to its spectrum sensing, learning, and adaptation capabilities, CR is able to address the heart of the problem associated with spectrum scarcity (via dynamic spectrum access (DSA)) and interoperability (via channel switching). It is envisioned that CR will be employed as a general radio platform upon which numerous wireless applications can be implemented.

For both theoretical and practical purposes, it is important for network researchers to model a cognitive radio ad hoc network (CRN) and optimize its performance. Such efforts are important not only for theoretical understanding, but also in that such results can be used as benchmarks for the design of distributed algorithms and protocols. However, due to some unique characteristics associated with CRNs, existing analytical techniques may not be applied directly. As a result, new theoretical results, along with new mathematical techniques, need to be developed. In this thesis, we focus on modeling and optimization of CRNs. In particular, we will study multicast communications in CRN and MIMO-empowered CRN, which we describe as follows.

An important service that must be supported by CRNs is multicast. Although there are a lot of research on multicast in ad hoc networks, those results cannot be applied to a CRN, because of the complexity associated with a CR node (e.g., multiple available frequency bands, difference in available bands from neighboring nodes). In addition, a single-layer approach (e.g., multicast routing) is overly simplistic when resource optimization (i.e., minimizing network resource) is the main objective. For this purpose, a cross-layer approach is usually necessary, which should include joint consideration of multiple lower layers, in addition to network layer. However, such a joint formulation is usually highly complex and difficult. In this thesis, we aim to develop some novel algorithms that provide near-optimal solutions. Our goal is to minimize the required network-wide resource to support a set of multicast sessions, with a certain bit rate for each multicast session. The unique characteristics associated with CR and distinguish this problem from existing multicast research for ad hoc networks. In this work, we formulate this problem via a cross-layer approach with joint consideration of scheduling and routing. Although the problem formulation is in the form of mixed integer linear program (MILP), we are successful in developing a polynomial time algorithm that offers highly competitive solution. The main ideas of the algorithm include identification of key integer variables, fixing these variables via a series of relaxed linear program (LP), and tying up such integer fixing with a bottom-up tree construction. By comparing with a lower bound, we find that the proposed algorithm can provide a solution that is very close to the optimum.

In parallel to the development of CR for DSA, multiple-input multiple-output (MIMO) has widely been accepted and now implemented in commercial wireless products to increase capacity. The goal of MIMO and how it operates are largely independent and orthogonal to CR. Instead of exploiting idle channels for wireless communications, MIMO attempts to increase capacity within the same channel via space-time processing. Assuming that CR and MIMO will ultimately marry each other and offer the ultimate flexibility in DSA and spectrum efficiency, we would like to inquire the potential capacity gain in this marriage. In particular, we are interested in how such marriage will affect the capacity of a user communication session in a multi-hop CRN. We explore MIMO-empowered CR network, which we call CRN^MIMO, to achieve ultimate flexibility in DSA and spectrum efficiency. Given that CR and MIMO handle interference at different levels (across channels vs. within a channel), we are interested in how joint optimization of both will maximize user capacity in a multi-hop network. To answer this question, we develop a tractable mathematical model for CRN^MIMO, which captures the essence of channel assignment (for CR) and degree-of-freedom (DoF) allocation (for MIMO). Based on this mathematical model, we use numerical results to show how channel assignment in CRN and DoF allocation in MIMO can be jointly optimized to maximize capacity. More important, for a CRN^MIMO with A_MIMO antennas at each node, we show that joint optimization of CR and MIMO offers more than A_MIMO-fold capacity increase than a CRN with only a single antenna at each node.