What to Buy: the Underexplored Dimension of the Smart-buyer Problem

Using one question of Donald Kettl's smart-buyer problem as the basis of investigation, this study empirically examines the relationship between five selected services contract characteristics related to requirements and evaluation ratings of 120 federal information technology investments. The five contract characteristics selected for investigation were: Contract Type, Extent Competed, Performance-Based Acquisition, Integrated Process Team, and Program Manager Qualification. Analysis of these characteristics is relevant because current federal acquisition policy advocates particular contract characteristics as the preferred methods of procurement and others as widely accepted best practices. The five selected contract characteristics were analyzed for over 200 information technology services contracts using two separate statistical tests and four variants of information technology investment ratings as the dependent variable. Empirical evidence failed to reject the null hypothesis that there is no statistically significant correlation between selected service contract characteristics related to contract requirements and investment ratings. This failure of rejection through multiple tests led to the conclusion that service contract requirements are not better defined in cases when they should be based on selected contract characteristics. In order to better inform and understand the quantitative findings, interviews were conducted with over 20 senior acquisition and information technology executives representing 11 different federal departments and industry. None of the senior executives interviewed disputed the null finding and nearly 70 percent of subject matter experts interviewed were unsurprised that the selected service contract characteristics did not correlate with investment ratings.

The lack of correlation between selected contract characteristics related to requirements and investment ratings indicates that service contract requirements definition is a significant problem for federal agencies. That conclusion was confirmed by interviews with senior subject matter experts who consistently stated that accurately defining and managing information technology service contract requirements is a genuine challenge facing the federal government today.

Although empirical evidence failed to reject the null hypothesis, subsequent interviews revealed other factors that may have greater bearing on requirements and acquisition program outcomes than the five selected service contract characteristics. They present promising topics that merit further research.