The luminosity function of active galactic nuclei (AGN) probes the history of supermassive black hole assembly and growth across cosmic time. To mitigate selection biases, we present a consistent analysis of the AGN luminosity functions (LFs) derived for both X-ray and mid-infrared (MIR) selected AGN in the XMM-Large Scale Structure field. There are 4268 AGN used to construct the MIR luminosity function (IRLF) and 3427 AGN used to construct the X-ray luminosity function (XLF), providing the largest census of the AGN population out to z = 4 in both bands with significant reduction in uncertainties. We are able for the first time to see the knee of the IRLF at z > 2 and observe a flattening of the faint-end slope as redshift increases. The bolometric luminosity density, a proxy for the cosmic black hole accretion history, computed from our LFs, shows a peak at z approximate to 2.25, consistent with recent estimates of the peak in the star formation rate density (SFRD). However, at earlier epochs, the AGN luminosity density is flatter than the SFRD. If confirmed, this result suggests that the build up of black hole mass outpaces the growth of stellar mass in high-mass systems at z greater than or similar to 2.5. This is consistent with observations of redshift z similar to 6 quasars that lie above the local M - sigma relationship. The luminosity density derived from the IRLF is higher than that from the XLF at all redshifts. This is consistent with the dominant role of obscured AGN activity in the cosmic growth of supermassive black holes.