A methodology for the measurement and evaluation of complex system designs

Most complex systems incorporate hardware, software and humanware elements operating synergistically under conflicting functional and nonfunctional objectives. These systems are usually embedded, mission-critical, performance-critical, real-time, distributed, highly integrated, heterogeneous, cost millions of dollars, and take many years to develop. Examples include space stations, combat vessels and aircraft, nuclear power stations, communication networks, and robotics-based manufacturing.

Early system design evaluation is essential to assess a design’s potential for satisfying operational and budgetary requirements, since a significant percentage of the system life cycle cost is committed by design decisions made early in the system life cycle. However, at the design decision point, knowledge of technology, the operational environment, the political climate, etc., on which to make technically effective and cost efficient decisions is incomplete. Consequently, early design evaluation approaches are needed which yield credible results in the presence of incomplete knowledge.

This dissertation describes a multifaceted methodology for complex system design measurement and evaluation which exploits experience, techniques, and heuristics of technical and operational domain experts. The methodology is computer and knowledge based, and includes indicator-based assessment, visual simulation, the analytic hierarchy process, and fuzzy mathematics. The use of this methodology enables qualitative and quantitative measurement and evaluation of system designs at any level of detail desired. An independent assessment of the methodology by researchers and systems engineering practitioners from the DOD, other federal government agencies, commercial industry, and academia affirmed the methodology to be a useful approach in the measurement and evaluation of complex system designs.