Many mobile applications in wireless networks such as military battlefield, emergency response, and mobile commerce are based on the notion of secure group communications. Unlike traditional security protocols which concern security properties only, in this dissertation research we design and analyze a class of QoS-aware protocols for secure group communications in wireless networks with the goal to satisfy not only security requirements in terms of secrecy, confidentiality, authentication, availability and data integrity, but also performance requirements in terms of latency, network traffic, response time, scalability and reconfigurability.

We consider two elements in the dissertation research: design and analysis. The dissertation research has three major contributions. First, we develop three "threshold-based" periodic batch rekeying protocols to reduce the network communication cost caused by rekeying operations to deal with outsider attacks. Instead of individual rekeying, i.e., performing a rekeying operation right after each group membership change event, these protocols perform batch rekeying periodically. We demonstrate that an optimal rekey interval exists that would satisfy an imposed security requirement while minimizing the network communication cost.

Second, we propose and analyze QoS-aware intrusion detection protocols for secure group communications in mobile ad hoc networks to deal with insider attacks. We consider a class of intrusion detection protocols including host-based and voting-based protocols for detecting and evicting compromised nodes and examine their effect on the mean time to security failure metric versus the response time metric. Our analysis reveals that there exists an optimal intrusion detection interval under which the system lifetime metric can be best traded off for the response time performance metric, or vice versa. Furthermore, the intrusion detection interval can be dynamically adjusted based on the attacker behaviors to maximize the system lifetime while satisfying a system-imposed response time or network traffic requirement.

Third, we propose and analyze a scalable and efficient region-based group key management protocol for managing mobile groups in mobile ad hoc networks. We take a region-based approach by which group members are broken into region-based subgroups, and leaders in subgroups securely communicate with each other to agree on a group key in response to membership change and member mobility events. We identify the optimal regional area size that minimizes the network communication cost while satisfying the application security requirements, allowing mobile groups to react to network partition/merge events for dynamic reconfigurability and survivability. We further investigate the effect of integrating QoS-aware intrusion detection with region-based group key management and identify combined optimal settings in terms of the optimal regional size and the optimal intrusion detection interval under which the security and performance properties of the system can be best optimized.

We evaluate the merits of our proposed QoS-aware security protocols for mobile group communications through model-based mathematical analyses with extensive simulation validation. We perform thorough comparative analyses against baseline secure group communication protocols which do not consider security versus performance tradeoffs, including those based on individual rekeying, no intrusion detection, and/or no-region designs. The results obtained show that our proposed QoS-aware security protocols outperform these baseline algorithms. â