Distracted driving behaviors are closely related to crash risk, with the use of mobile phones during driving being one of the leading causes of accidents. This paper attempts to investigate the impact of cell phone use while driving on drivers’ control behaviors. Given the limitation of driving simulators in an unnatural setting, a sample of 134 cases related to cell phone use during driving were extracted from Shanghai naturalistic driving study data, which provided massive unobtrusive data to observe actual driving process. The process of using mobile phones was categorized into five operations, including dialing, answering, talking and listening, hanging up, and viewing information. Based on the concept of moving time window, the variation of the intensity of control activity, the sensitivity of control operation, and the stability of control state in each operation were analyzed. The empirical results show strong correlation between distracted operations and driving control behavior. The findings contribute to a better understanding of drivers’ natural behavior changes with using mobiles, and can provide useful information for transport safety management.