This paper discusses a methodology for diagnosing performance problems for parallel and distributed programs. The methodology is based on the formation and testing of hypotheses about the cause of performance bottlenecks. The process is illustrated with a case study of an actual problem arising in a parallel discrete event simulation program in which granularity is a primary bottleneck and barrier implementation is a secondary bottleneck. The paper also describes the evolution of Chitra, a software performance measurement and analysis tool whose objective is to automate certain steps in software performance diagnosis.