TL;DR—The USEPA’s release of standardized testing protocols and performance targets for low-cost sensors in February 2021 provides a homogenous and comprehensive approach to low-cost sensor performance evaluation. As low-cost sensors become more widely available and are implemented across the world, having standardized ways to evaluate their performance helps air quality managers, organizations, and other low-cost sensor users to better understand their performance. While a multitude of performance metrics exist, the USEPA protocols set forth a straightforward approach to understanding sensor performance against specific performance targets.
How to conduct a sensor performance evaluation
Alongside the release of performance targets for low-cost sensors, the USEPA has provided guidance on recommended protocols for setting up and running your own evaluation testing, should you wish to do so for a certain make and model of air sensor. This can provide valuable information about how a sensor will perform given the climatic conditions and pollutant profile specific to your region—and also provides valuable inputs for a future, regionally-tailored calibration model!
While there are a number of testing centers that provide performance evaluations for low-cost sensors as covered in our previous blog, the protocols they use are heterogeneous and not as complete as the criteria recommended by the USEPA.
The USEPA protocol for sensor evaluation uses a variety of metrics to provide a more complete picture of sensor performance, compared to past approaches, many of which rely solely on using R². Using these metrics hand-in-hand helps to fill in the blind spots of using a single metric. For example, using MAE in conjunction with R² can help clarify whether a low R² value reflects inaccurate device measurements, or simply an insufficient pollutant concentration range in the testing environment.
To learn more about the role of metrics like R² and MAE in evaluating low-cost air sensor performance, check out our “How to assess sensor accuracy” blog series.
The EPA differentiates between two categories of testing: field testing (or “Base Testing”) and laboratory testing (or “Enhanced Testing”). While laboratory testing allows sensors to be evaluated across a wider range of conditions than might be possible in the field, we will focus on field testing here as it is more accessible for the majority of people that do not have access to a laboratory. Field testing also provides more relevant insight into how a sensor will perform under local conditions.
For both field and laboratory testing, the EPA recommends testing at least three identical sensors (of the same make, model, and firmware version) simultaneously. PM2.5 air sensors should be colocated with FRM/FEM monitors for a minimum of 30 days each, at a minimum of two testing sites in different climatic regions.
Equipment you’ll need the perform testing per the USEPA protocols
The USEPA provides the following information on best practices for siting and deploying sensors when establishing a colocation with an FRM or FEM monitor.
When it comes time to conduct a low-cost sensor evaluation, the following performance evaluation checklist helps ensure a well-planned and successful testing. You can download a PDF of this checklist here.
Once you complete the performance evaluation, you will have all of the data you need to quantitatively compare the measurements from your low-cost sensors to those derived from the reference instrument. You can see example results from one colocation that we ran in Los Angeles, California, below.
**EU and UK averages are based on 24-hour averages for PM10 or PM2.5 concentrations over a 40-day period.
Conducting an evaluation of low-cost sensor performance helps to ensure that accurate, reliable, and calibrated data form the basis for pollution abatement decision-making. By following the steps and considerations outlined above, you can determine the extent to which potential or already-implemented low-cost monitoring technologies meet the standards set forth by the USEPA and establish a sense of trust in the low-cost sensors you choose to utilize.
Conducting an evaluation of low-cost sensor performance helps to ensure that accurate, reliable, and calibrated data form the basis for pollution abatement decision-making. By following the steps and considerations outlined above, you can determine the extent to which potential or already-implemented low-cost monitoring technologies meet the standards set forth by the USEPA and establish a sense of trust in the low-cost sensors you choose to utilize.
For more information on the USEPA’s full set of standards for evaluating low-cost air sensor performance, read the Air Sensor Performance Targets and Testing Protocols for ozone and PM2.5. Check out our Guide to Accurate Particulate Matter Measurements for more tips and best practices for using low-cost sensors, and keep an eye out for the final blog in our series on the USEPA protocols in the coming weeks!