The quick growth of embedded systems and their increasing computing power has made them suitable for a wider range of applications. Despite the increasing performance of modern embedded processors, they are outpaced by computational demands of the growing number of modern applications. This trend has led to emergence of hardware accelerators in embedded systems. While the processing power of dedicated hardware modules seems appealing, they require significant effort of development and integration to gain performance benefit. Thus, it is prudent to investigate and estimate the integration overhead and consequently the hardware acceleration benefit before committing to implementation. In this work, we present FPGA-Roofline, a visual model that offers insights to designers and developers to have realistic expectations of their system and that enables them to do their design and analysis in a faster and more efficient fashion. FPGA-Roofline allows simultaneous analysis of communication and computation resources in FPGA-based hardware accelerators. To demonstrate the effectiveness of our model, we have implemented hardware accelerators in FPGA and used our model to analyze and optimize the overall system performance. We show how the same methodology can be applied to the design process of any FPGA-based hardware accelerator to increase productivity and give insights to improve performance and resource utilization by finding the optimal operating point of the system.