Models of atmospheric composition rely on fire emissions inventories to reconstruct and project impacts of biomass burning on air quality, public health, climate, ecosystem dynamics, and land-atmosphere exchanges. Many such global inventories use satellite measurements of active fires and/or burned area from the Moderate Resolution Imaging Spectroradiometer (MODIS). However, differences across inventories in the interpretation of satellite imagery, the emissions factors assumed for different components of smoke, and the adjustments made for small and obscured fires can result in large regional differences in fire emissions estimates across inventories. Using Google Earth Engine, we leverage 15 years (2003-2017) of MODIS observations and 6 years (2012–2017) of observations from the higher spatial resolution Visible Imaging Infrared Radiometer Suite (VIIRS) sensor to develop metrics to quantify five major sources of spatial bias or uncertainty in the inventories: (1) primary reliance on active fires versus burned area, (2) cloud/haze burden on the ability of satellites to 'see' fires, (3) fragmentation of burned area, (4) roughness in topography, and (5) small fires, which are challenging to detect. Based on all these uncertainties, we devise comprehensive 'relative fire confidence scores,' mapped globally at 0.25° × 0.25° spatial resolution over 2003-2017. We then focus on fire activity in Indonesia as a case study to analyze how the choice of a fire emissions inventory affects model estimates of smoke-induced health impacts across Equatorial Asia. We use the adjoint of the GEOS-Chem chemical transport model and apply emissions of particulate organic carbon and black carbon (OC + BC smoke) from five global inventories: Global Fire Emissions Database (GFEDv4s), Fire Inventory from NCAR (FINNv1.5), Global Fire Assimilation System (GFASv1.2), Quick Fire Emissions Dataset (QFEDv2.5r1), and Fire Energetics and Emissions Research (FEERv1.0-G1.2). We find that modeled monthly smoke PM2.5 in Singapore from 2003 to 2016 correlates with observed smoke PM2.5, with r ranging from 0.64-0.84 depending on the inventory. However, during the burning season (July to October) of high fire intensity years (e.g., 2006 and 2015), the magnitude of mean Jul-Oct modeled smoke PM2.5 can differ across inventories by >20 μg m−3 (>500%). Using the relative fire confidence metrics, we deduce that uncertainties in this region arise primarily from the small, fragmented fire landscape and very poor satellite observing conditions due to clouds and thick haze at this time of year. Indeed, we find that modeled smoke PM2.5 using GFASv1.2, which adjusts for fires obscured by clouds and thick haze and accounts for peatland emissions, is most consistent with observations in Singapore, as well as in Malaysia and Indonesia. Finally, we develop an online app called FIRECAM for end-users of global fire emissions inventories. The app diagnoses differences in emissions among the five inventories and gauges the relative uncertainty associated with satellite-observed fires on a regional basis.