Monitoring & Evaluation
Evaluation and monitoring of wood stove change-out programs
Updated: November 12th, 2016
Performance evaluation is essential for all wood stove change-out programs. Change-out program managers should obtain feedback from all stakeholders and partners to ensure continued support for the program and to make adjustments. Program evaluations are also useful for securing funding for future change-out programs and highlighting success stories that might encourage more people to participate.
In this article, we highlight some of the techniques and methods that have been used for evaluating a change-out program's effectiveness in 1) reducing outdoor PM2.5 emissions (ambient air quality); 2) reducing indoor pollutants from wood smoke (indoor air quality); 3) achieving its objectives in a cost-effective manner (financial performance); and 4) monitoring satisfaction and obtaining feedback from stakeholders. The final section contains our suggestions for change-out program administrators seeking to conduct their own program evaluations.
1. Reducing outdoor PM2.5 emissions (ambient air quality)
.jpg)
Reducing particulate matter emissions is a primary objective of most wood stove change-out programs, particularly those that focus on in or close to non-attainment. However, many past program managers that are included in the EPA’s change-out case studies report that it is difficult to estimate how much of an impact the program had on improving ambient air quality. Most take fewer non-attainment days or an area moving out of PM2.5 non-attainment as a sign that the program was successful, but are cautious to attribute these results to the program and not to other factors. As ambient air quality varies seasonally, it is important to look at the long-term data before jumping to conclusions. In addition to comparing EPA PM2.5 measurements before and after the change-out program, some administrators and their partners use other models or techniques to estimate the program’s impact on ambient air quality.
Techniques: Estimating the PM2.5 emissions reductions achieved through a change-out program typically requires special knowledge, expertise, or equipment that goes beyond the skillset and budget of most program administrators. To help overcome these knowledge gaps, the EPA suggests that change-out programs partner with university or environmental research groups to help them conduct pre-and post-program air quality evaluations. Examples of past programs that have partnered with such groups include the Fairbanks, AK (University of Fairbanks) and Libby, MT (University of Montana) programs.
In addition, the EPA also provides estimation tools for Microsoft Excel and Access for program managers seeking to establish a baseline of wood smoke emissions and/or calculate the emissions reduced from a change-out program in this 2014 report. Required inputs include: percentage of people that burn wood in each appliance, number of occupied housing units, the amount of wood burned in each appliance, the density (moisture content) of the wood, the amount of emissions (per pollutant, in mass units) emitted per ton of wood burned, number of devices replaced or destroyed, and stove efficiency. A large list of residential wood burning surveys can be found here and an EPA sample can be found here.
Equipment:Tapered element oscillating microbalances (TEOMs) were used to record ambient PM2.5 data in a 2006-2009 study of the Nez Perce Reservation change-out. The Libby, MT researchers used a spiral ambient speciation sampler (SASS) to collected 24 hour integrated ambient air samplers for several air pollutants, as well as Federal Reference Method samples to test for ambient PM2.5 emissions.
Results:Academic journal articles that attempt to measure or model the effectiveness of change-out programs in terms of outdoor emissions reduction have been written for change-out programs in Fairbanks, Alaska (2008-2009) and Libby, Montana (2007-2008). One EPA report links stove change-out programs in Libby, MT (2005-2007) and Sacramento, CA (2008-2009) to reductions in fine particulate ambient pollution levels by 7 and 12 micrograms per cubic meter (µg/m³), respectively. Another study conducted in 2007-2008 found that the Libby’s PM2.5 emissions were 20-28% lower than levels detected before the change-out in 2003-2004 (Ward et al., 2010).
2. Reducing indoor pollution from wood smoke (indoor air quality)
.jpg)
Instrument used for evaluating indoor air quality.
Improving the indoor air quality of homes that heat with wood is often another goal and benefit of a wood stove change-out program, particularly those that focus on improving health for tribal, low-income, or other vulnerable residents. A program’s effectiveness in this area can be measured through a before/after comparison of wood smoke-related air pollution levels inside a sampling of homes. As with outdoor air quality evaluations, it may be beneficial for change-out program managers to partner with universities or research groups to conduct scientific indoor air quality assessments. For example, an indoor air quality analysis of the Libby, Montana change-out program were conducted by a team of scientists from the University of Montana, University of Washington, and Colorado School of Mines.
Claims that wood stove change-outs reduce indoor air quality by 70% are circulating with little substantiation. One area of future study is whether the age of the stove or the quality and type of the installation has more impact on indoor air quality. Poor draft is one of the biggest contributors to poor indoor air quality, as it can lead to back puffing in old and new stoves. Poor draft is common, for instance, in mobile home installations, where the stack may be only 10 feet tall, far below the height that the stove was designed for and tested with. Adding 5 feet of stovepipe could likely be a better indoor air mitigation measure than replacing the stove – if that is a main goal.
Techniques: We recommend that those who are interested in studying the effects of a change-out program on indoor air quality consult the existing literature (provided at the end of the article) for tips on study design. A few best practices we picked out include only using non-smoking homes in the sample and conducting 24 hour measures every three days for the length of the sampling program, as specified in the EPA ambient air monitoring guidelines. In addition, those who would like to compare their indoor PM2.5 concentration results with the National Ambient Air Quality Standards for PM2.5 should consider using a Federal Reference Method (FRM) sampler (Ward et al., 2015).
Educating homeowners on proper appliance use has been found to be critical for improving indoor air quality in homes that swap old, uncertified wood stoves for certified wood stoves (Ward et al., 2011). The 2006-2009 Nez Perce Reservation study found that 5 out of 16 homes tested had higher PM2.5 levels than they had prior to the change-out, some of which was attributed to residents burning wet wood, drying wood on top of the stove, and improper burning techniques (leaving the door cracked and opening the draft too much). The indoor air quality in these homes greatly improved when residents received additional training on proper stove use. The EPA recommends following-up with households two to three weeks after the stove replacement to see how well the residents are adjusting to their new device and to help answer questions about its proper use. Scientists should then return to the homes in a month to conduct a full indoor air quality assessment.
Equipment: There are multiple instruments and methods that can be used to measure indoor air quality. A 2006-2009 study of a change-out program on the Nez Perce Reservation in Idaho used portable DustTrack samplers and Leland pumps to measure indoor concentrations of PM2.5, organic carbon, elemental carbon, and chemical markets of wood smoke, while a 2007-2008 British Columbia study used single-stage Harvard Impactors and Leland pumps.
Results: Academic studies have shown that change-out programs have lowered indoor PM2.5 concentrations by an average of 71% in Libby, Montana (2005-2007) and 36% in the Nez Perce Reservation in Idaho (2006-2009), although no change was reported for a community in British Columbia, Canada (2007-2008). All three studies sampled a total of 17 homes, but the Nez Perce and had more compatible methodologies. A follow-up study of the Libby, Montana change-out, which included more post-change-out data, found the average indoor PM2.5 concentration reduction to be 53% for 21 homes, with five homes reporting no reduction in emissions.
3. Achieving program objectives cost-effectively (financial analysis)
Table comparing a sample of change-out programs with disclosed budgets.
However, it is also worth mentioning that a low cost to trade-out ratio is not the only marker of a ‘good’ or effective change-out program. Many programs are specifically aimed at helping their lower-income residents replace their uncertified at little or no cost to them; in this case, it would make sense that the program would spend more money per stove. Moreover, programs that offer very low incentives (e.g., $100-$250) may become inundated with applications of residents who ultimately do not complete the change-out, resulting in an inefficient use of administrator time and resources. One change-out program manager noted that the percentage of applicants who redeemed their rebate more than doubled when the program began to provide higher incentive amounts ($750-$2,000).
Libby, Montana is an example of a program that replaced uncertified stoves with EPA certified stoves at no cost to many of the residents, while others received $1,050 rebates. The 2007-2008 program had an overall budget of $2,683,700 and replaced 1,130 stoves for an average cost of $2,375 per stove. A 2007-2008 change-out program in the Makah Reservation in Washington State had a budget of $115,952 and replaced 45 stoves at full cost at an average of $2,576 per stove. In contrast, a 2009-2010 program in Keene, New Hampshire had a budget of $141,000 and replaced 87 stoves for an average cost of $1,621 per stove (the program offered participants $1,000 to $3,000 rebates).
Techniques: A 2015 report on the (2007-2013) British Columbia Wood Stove Exchange Program provides the most detailed and comprehensive stove change-out program evaluation we could find online. The report, which was written by a sustainability consultancy firm, includes a breakdown and analysis of the program’s allocation of funds, its efficiency in accomplishing its objectives, and its overall effectiveness. The report also contains a chart comparing the program’s performance with the Alliance for Green Heat’s list of best change-out practices and previous programs.
Results: An environmental consulting agency was hired by EPA to conduct a financial analysis on the best wood smoke reduction options for low-earning households in Pierce County, Washington. Part of the analysis involved assessing the effectiveness of the existing change-out program (2007-2013) in reaching this target group. The report revealed that program participation rates for this group were only 5% in 2013, and ultimately concluded, using their model, that a program which provided 100% of the cost of replacing an uncertified wood device would be the more cost-effective option for lowering wood smoke without burdening low-earning households.
4. Monitoring stakeholder and participant satisfaction
Surveys or interviews are often used to obtain stakeholder feedback on change-out programs.
Stakeholder surveys are low-cost and can be undertaken by even the smallest change-out program. Stakeholders should include a 360-degree perspective, and include funders, local experts, groups who may be vehemently opposed to wood burning, etc. Several programs, such as Keene, NH (2009-2010), have utilized surveys to gauge participant satisfaction with the program. The Keene program also partnered with students and faculty from an environmental health program at a local college to help conduct phone interviews with participants about their home heating use. Students and faculty also monitored the area’s air quality and conducted visual surveys of surrounding areas to help collect data useful for deciding whether to expand the change-out program.
A sample stakeholder survey can be found on pages 43-46 of the British Columbia Wood Stove Exchange Program Evaluation Report. The report groups survey questions into categories for local coordinators, local government staff, local government officials, participating retailers, industry, environmental NGOs, lung association staff, and air quality staff. The questions are open-ended and are tailored for gathering data on program effectiveness and future program recommendations.
5. Program evaluation recommendations for program administrators
Ideally, program monitoring should take place before, during, and after the stove change-out program, but much of this depends on the budget and scope of the change-out. The exact type of monitoring will be dependent on the program’s own specific objectives and the kind of resources it has or expects to have available. For example, if a program is not concerned with improving indoor air quality, there is little reason to conduct these assessments.
Before the program:Ambient air quality should be tested before the program begins to create a baseline for judging future performance. Local hearth retailers, chimney sweeps, and community members should additionally be consulted to ascertain the number and type of wood burning devices purchased in recent years, as well as how often people have their appliances cleaned, and how many cords of wood people tend to burn. This sort of information is also useful for designing targeted education and outreach campaigns.
During the program:As the program continues, administrators should consistently monitor the amount of applications they receive and/or the number of rebates vouchers they issue. Weekly follow-ups with partners are helpful for knowing how to adjust the program as it unfolds.
After the program: After the program (or a phase) ends, we recommend that program administrators conduct a financial analysis, post-air quality test, and stakeholder satisfaction survey. As a final step, the report that offers an evaluation of the program, recommendations for future programs, and any lessons learned should be posted and publicly available.
While it is preferable (for objectivity and other reasons) to leave the post-program evaluation to a third-party like a consulting firm, it is also possible for program administrators or their partners to conduct the assessment more cheaply in-house. For example, the final reports for the Libby, Montana (2007-2008), Keene, NH (2009-2010), and Makah, WA (2007-2008) change-out programs were conducted by program administrators or their partners (e.g., HBPA). The British Columbia post-program evaluation included the following tasks (page 4-5), which other programs can emulate:
-
Document the program by reviewing program related records
-
Design an evaluation framework to assess the program using qualitative and quantitative inputs
-
Conduct interviews and surveys with a wide range of stakeholders
-
Supplement with literature review and web research to provide context and comparisons
-
Summarize, analyze, develop recommendations, and report outcomes to improve future programs
Further examples of final program reports can be found in the references section below.
References
Outdoor air quality:
Noonan, C. W., Ward, T. J., Navidi, W., & Sheppard, L. (2012). A rural community intervention targeting biomass combustion sources: effects on air quality and reporting of children's respiratory outcomes. Occupational and environmental medicine. http://www.jstor.org/stable/pdf/23218153.pdf?_=1468611630686
Tran, H. N., & Mölders, N. (2012). Wood-Burning Device Changeout: Modeling the Impact on PM2. 5 Concentrations in a Remote Subarctic Urban Nonattainment Area. Advances in Meteorology, 2012. http://downloads.hindawi.com/journals/amete/2012/853405.pdf
Ward, T. J., Palmer, C. P., & Noonan, C. W. (2010). Fine particulate matter source apportionment following a large woodstove changeout program in Libby, Montana. Journal of the Air & Waste Management Association, 60(6), 688-693. http://www.tandfonline.com/doi/pdf/10.3155/1047-3289.60.6.688
Indoor air quality:
Allen, R. W., Leckie, S., Millar, G., & Brauer, M. (2009). The impact of wood stove technology upgrades on indoor residential air quality. Atmospheric Environment, 43(37), 5908-5915. http://www.sciencedirect.com/science/article/pii/S1352231009007389/pdfft?md5=e1134de0448cc55084095988ca1ab027&pid=1-s2.0-S1352231009007389-main.pdf
Ward, T., & Noonan, C. (2008). Results of a residential indoor PM2. 5 sampling program before and after a woodstove changeout. Indoor air, 18(5), 408-415. https://www.researchgate.net/profile/Christopher_Palmer/publication/23136325_Results_of_a_ residential_indoor_PM2.5_sampling_program_before_and_after_a_woodstove_changeout/links/ 004635214ebaf0f873000000.pdf
Ward, T., Boulafentis, J., Simpson, J., Hester, C., Moliga, T., Warden, K., & Noonan, C. (2011). Lessons learned from a woodstove changeout on the Nez Perce Reservation. Science of the Total Environment, 409(4), 664-670. http://www.sciencedirect.com/science/article/pii/S004896971001212X
Ward, T. J., Semmens, E. O., Weiler, E., Harrar, S., & Noonan, C. W. (2015). Efficacy of interventions targeting household air pollution from residential wood stoves. Journal of Exposure Science and Environmental Epidemiology. http://onlinelibrary.wiley.com/doi/10.1111/j.1600-0668.2012.00789.x/epdf
Financial reports and final program reports:
Boye, Hughes & Weiss (2013). Puget Sound Clean Air Agency Residential Wood Smoke Reduction Initiative: Burn Ban Financial Assistance Ex Ante Program Assessment. https://www.epa.gov/sites/production/files/2015-09/documents/pugetsoundfinalreport.pdf
Hearth, Patio & Barbecue Association (2008). Clearing the Smoke: The Wood Stove Changeout in Libby, Montana. http://www.hpba.org/government-affairs/woodstove-changeout-program/Libby_ReportFinal.pdf
New Hampshire Department of Environmental Services & City of Keene (2010). Keene Woodstove Changeout Campaign Final Report. https://www.epa.gov/sites/production/files/2015-09/documents/keenefinalreport2011.pdf
Makah Tribe (2010). Makah Clean Air-Healthy Homes Woodstove Change-Out Project Report. https://www.epa.gov/sites/production/files/2015-09/documents/makahtribereport2011.pdf
Measurable Outcomes of a Woodstove Changeout on the Nez Perce Reservation Final Performance Report (2009). https://www.epa.gov/sites/production/files/2015-09/documents/nezpercereport.pdf
Pinna Sustainability Inc. (2015). BC Wood Stove Exchange Program: Program Evaluation 2008-2014. http://www.bcairquality.ca/reports/pdfs/WSEP_evaluation.pdf
Other resources:
Environmental Protection Agency (2006). Guidance for Quantifying and Using Emission Reductions from Voluntary Woodstove Changeout Programs in State Implementation Plans. https://www.epa.gov/sites/production/files/2015-09/documents/guidance_quantfying_in_sips.pdf
Environmental Protection Agency (2007). Guidance for Using Supplemental Environmental Projects to Implement Wood Stove Changeout Programs. https://www.epa.gov/sites/production/files/2015-09/documents/guidance_for_using_state_supplementals_042807.pdf
Environmental Protection Agency (2009). Strategies for Reducing Residential Wood Smoke. http://www.cambridgeny.gov/forms/strategies-doc-8-11-09.pdf
Environmental Protection Agency, Burn Wide (2014). How to Implement a Wood-Burning Appliance Changeout Program. https://www.epa.gov/sites/production/files/2015-08/documents/howtoimplementawoodstovechangeout.pdf
Weiss (2014). EPA’s Guide to Financing Options for Wood-burning Appliance Changeouts. https://www.epa.gov/sites/production/files/2015-09/documents/guide_to_financing_options.pdf