Recently, many new types of wireless networks have emerged for both civil and military applications, such as cognitive radio networks, MIMO networks. There is a strong interest in exploring the optimal performance of these new emerging networks, e.g., maximizing the network throughput, minimizing network energy consumption. Exploring the optimal performance objectives of these new types of wireless networks is both important and intellectual challenging. On one hand, it is important for a network researcher to understand the performance limits of these new wireless networks. Such performance limits are important not only for theoretical understanding, but also in that they can be used as benchmarks for the design of distributed algorithms and protocols. On the other hand, due to some unique characteristics associated with these networks, existing analytic techniques may not be applied directly to obtain the optimal performance. As a result, new theoretical results, along with new mathematical tools, need to be developed. The goal of this dissertation is to make a fundamental advance on network performance optimization via exploring a series of optimization problems. Based on the scale of the underlying wireless network, the works in this dissertation are divided into two parts. In the first part, we study the asymptotic capacity scaling laws of different types of wireless networks. By "asymptotic", we mean that the number of nodes in the network goes to infinity. Such asymptotic capacity scaling laws offer fundamental understandings on the trend of maximum user throughput behavior when the network size increases. In the second part of this dissertation, we study several optimization problems of finite-sized wireless networks. Under a given network size, we accurately characterize some performance limits (e.g., throughput, energy consumption) of wireless networks and provide solutions on how to achieve the optimal objectives. The main contributions of this dissertation can be summarized as follows, where the first three problems are on asymptotic capacity scaling laws and the last three problems are optimization problems of finite-sized wireless networks. 1. Capacity Scaling Laws of Cognitive Radio Ad Hoc Networks. We first study the capacity scaling laws for cognitive radio ad hoc networks (CRNs), i.e., how each individual node's maximum throughput scales as the number of nodes in the network increases. This effort is critical to the fundamental understanding of the scalability of such network. However, due to the heterogeneity in available frequency bands at each node, the asymptotic capacity is much more difficult to develop than prior efforts for other types of wireless networks. To overcome this difficulty, we introduce two auxiliary networks ζ and α to analyze the capacity upper and lower bounds. We derive the capacity results under both the protocol model and the physical model. Further, we show that the seminal results developed by Gupta and Kumar for the simple single-channel single-radio (SC-SR) networks are special cases under the results for CRNs. 2. Asymptotic Capacity of Multi-hop MIMO Ad Hoc Networks. Multi-input multi-output (MIMO) is a key technology to increase the capacity of wireless networks. Although there has been extensive work on MIMO at the physical and link layers, there has been limited work on MIMO at the network layer (i.e., multi-hop MIMO ad hoc network), particularly results on capacity scaling laws. In this work, we investigate capacity scaling laws for MIMO ad hoc networks. Our goal is to find the achievable throughput of each node as the number of nodes in the network increases. We employ a MIMO network model that captures spatial multiplexing (SM) and interference cancellation (IC). We show that for a MIMO network with n randomly located nodes, each equipped with γ antennas and a rate of W on each data stream, the achievable throughput of each node is Θ(γW/√ n ln n). 3. Toward Simple Criteria for Establishing Capacity Scaling Laws. Capacity scaling laws offer fundamental understanding on the trend of user throughput behavior when the network size increases. Since the seminal work of Gupta and Kumar, there have been tremendous efforts developing capacity scaling laws for ad hoc networks with various advanced physical layer technologies. These efforts led to different custom-designed approaches, most of which were intellectually challenging and lacked universal properties that can be extended to address scaling laws of ad hoc networks with a different physical layer technology. In this work, we present a set of simple yet powerful general criteria that one can apply to quickly determine the capacity scaling laws for various physical layer technologies under the protocol model. We prove the correctness of our proposed criteria and validate them through a number of case studies, such as ad hoc networks with directional antenna, MIMO, cognitive radio, multi-channel and multi-radio, and multiple packet reception. These simple criteria will serve as powerful tools to networking researchers to obtain throughput scaling laws of ad hoc networks under different physical layer technologies, particularly those to appear in the future. 4. Exploiting SIC forMulti-hopWireless Networks. There is a growing interest on exploiting interference (rather than avoiding it) to increase network throughput. In particular, the so-called successive interference cancellation (SIC) scheme appears very promising, due to its ability to enable concurrent receptions from multiple transmitters and interference rejection. However, due to some stringent constraints and limit, SIC alone is inadequate to handle all concurrent interference. We advocate a joint interference exploitation and avoidance approach, which combines the best of interference exploitation and interference avoidance, while avoiding each's pitfalls. We discuss the new challenges of such a new approach in a multi-hop wireless network and propose a formal optimization framework, with cross-layer formulation of physical, link, and network layers. This framework offers a rather complete design space for SIC to squeeze the most out of interference. The goal of this effort is to lay a mathematical foundation for modeling and analysis of a joint interference exploitation and avoidance scheme in a multi-hop wireless network. Through modeling and analysis, we develop a tractable model that is suitable for studying a broad class of network throughput optimization problems. To demonstrate the practical utility of our model, we conduct a case study. Our numerical results affirm the validity of our model and give insights on how SIC can optimally interact with an interference avoidance scheme. 5. Throughput Optimization with Network-wide Energy Constraint. Conserving network wide energy consumption is becoming an increasingly important concern for network operators. In this work, we study network-wide energy conservation problem which we hope will offer insights to both network operators and users. Specifically, we study how to maximize network throughput under a network-wide energy constraint for a general multi-hop wireless network. We formulate this problem as a mixed-integer nonlinear program (MINLP). We propose a novel piece-wise linear approximation to transform the nonlinear constraints into linear constraints. We prove that the solution developed under this approach is near optimal with guaranteed performance bound. 6. Bicriteria Optimization in Multi-hop Wireless Networks. Network throughput and energy consumption are two important performance metrics for a multi-hop wireless network. Current state-of-the-art is limited to either maximizing throughput under some energy constraint or minimizing energy consumption while satisfying some throughput requirement. However, the important problem of how to optimize both objectives simultaneously remains open. In this work, we take a multicriteria optimization approach to offer a systematic study on the relationship between the two performance objectives. We show that the solution to the multicriteria optimization problem characterizes the envelope of the entire throughput energy region, i.e., the so-called optimal throughput-energy curve. We prove some important properties of the optimal throughput-energy curve. For case study, we consider both linear and nonlinear throughput functions. For the linear case, we characterize the optimal throughput-energy curve precisely through parametric analysis, while for the nonlinear case, we use a piece-wise linear approximation to approximate the optimal throughput-energy curve with arbitrary accuracy. Our results offer important insights on exploiting the trade-off between the two performance metrics.