Fires and the aerosols that they emit impact air quality, health, and climate, but the abundance and properties of carbonaceous aerosol (both black carbon and organic carbon) from biomass burning (BB) remain uncertain and poorly constrained. We aim to explore the uncertainties associated with fire emissions and their air quality and radiative impacts from underlying dry matter consumed and emissions factors. To investigate this, we compare model simulations from a global chemical transport model, GEOS-Chem, driven by a variety of fire emission inventories with surface and airborne observations of black carbon (BC) and organic aerosol (OA) concentrations and satellite-derived aerosol optical depth (AOD). We focus on two fire-detection-based and/or burned-area-based (FD-BA) inventories using burned area and active fire counts, respectively, i.e., the Global Fire Emissions Database version 4 (GFED4s) with small fires and the Fire INventory from NCAR version 1.5 (FINN1.5), and two fire radiative power (FRP)-based approaches, i.e., the Quick Fire Emission Dataset version 2.4 (QFED2.4) and the Global Fire Assimilation System version 1.2 (GFAS1.2). We show that, across the inventories, emissions of BB aerosol (BBA) differ by a factor of 4 to 7 over North America and that dry matter differences, not emissions factors, drive this spread. We find that simulations driven by QFED2.4 generally overestimate BC and, to a lesser extent, OA concentrations observations from two fire-influenced aircraft campaigns in North America (ARCTAS and DC3) and from the Interagency Monitoring of Protected Visual Environments (IMPROVE) network, while simulations driven by FINN1.5 substantially underestimate concentrations. The GFED4s and GFAS1.2-driven simulations provide the best agreement with OA and BC mass concentrations at the surface (IMPROVE), BC observed aloft (DC3 and ARCTAS), and AOD observed by MODIS over North America. We also show that a sensitivity simulation including an enhanced source of secondary organic aerosol (SOA) from fires, based on the NOAA Fire Lab 2016 experiments, produces substantial additional OA; however, the spread in the primary emissions estimates implies that this magnitude of SOA can be neither confirmed nor ruled out when comparing the simulations against the observations explored here. Given the substantial uncertainty in fire emissions, as represented by these four emission inventories, we find a sizeable range in 2012 annual BBA PM2.5 population-weighted exposure over Canada and the contiguous US (0.5 to 1.6 µg m−3). We also show that the range in the estimated global direct radiative effect of carbonaceous aerosol from fires (−0.11 to −0.048 W m−2) is large and comparable to the direct radiative forcing of OA (−0.09 W m−2) estimated in the Fifth Assessment Report (AR5) of the Intergovernmental Panel on Climate Change (IPCC). Our analysis suggests that fire emissions uncertainty challenges our ability to accurately characterize the impact of smoke on air quality and climate.