A version of this article was originally published on November 29th, 2016 on Greentech Media.

By Mike O’Boyle

With future federal clean energy policies in doubt, proactive clean energy policy will likely be left largely to states in the next few years. Fortunately, a New York policy proposal could show the way forward on energy efficiency for utilities.

Though energy efficiency is the most cost-effective clean energy resource in America, existing policies and programs still leave significant value on the table for residences and businesses. One major barrier to more large-scale energy efficiency has been traditional evaluation, measurement and verification (EM&V) of utility savings, which can be slow to adapt to new technology and often encourages utilities to focus disproportionately on easy-to-obtain, shallow savings.

The New York Public Service Commission, seeking innovative solutions in its Reforming the Energy Vision proceeding, instead opted for an outcome-oriented approach to measuring and incentivizing efficiency performance. An outcome-oriented metric would focus on the policy goal of reduced energy use overall, putting a smaller emphasis on the administratively intensive business of attributing savings to specific actions.

New York’s Clean Energy Advisory Council added detail to the commission direction when it released its Energy Efficiency Metrics and Targets Options Report on implementing the New York Commission’s orders last month. The report represents a radical departure from program-based efficiency regulation, and highlights both opportunities and implementation challenges.

No state has ever before tried this kind of performance-based ratemaking for energy efficiency, and it could become a powerful new tool to expand investment. Ultimately, transparency and adaptability will be essential conditions to ensure an outcome-oriented approach achieves deep efficiency savings.

Why deviate from the bottom-up approach?

Thirty-six states measure utility performance on attributed “deemed savings” values for efficiency upgrades, which are average estimates for a given technology. For example, replacing an incandescent light bulb with a 15-watt CFL bulb is considered to save 71 kilowatt-hours per year in Vermont. But savings estimates are subject to change as technologies, standards and customer behaviors evolve — for example, switching to CFLs is estimated to save only 18 kilowatt-hours per year in California.

This “deemed savings” approach is attractive because of its relatively low administrative cost, but it sacrifices some accuracy in the process. In the end, utilities under deemed-savings approaches will prioritize activities producing cost-effective efficiency savings on average, but they are not incentivized to innovate and drive deep market transformation.

Take LED lighting: Typically, this is deemed among the most cost-effective measures a utility can undertake, even though real-world savings may be underwhelming depending on factors like how many hours per year those lights are turned on. Building retrofits are the opposite; they are often deemed less cost-effective on average, though some buildings have massive potential to achieve huge, cost-effective savings.

Using averages for deemed savings can stifle innovation in the market, resulting in over-rewarding some activities that deliver less ambitious outcomes, and conversely under-rewarding some of the best opportunities. Efficiency programs should encourage utilities to connect customers and innovative partners to maximize real-world savings, no matter the means.

To better align incentives, deemed savings could be adaptive, but building real-time updates for hundreds of thousands of efficiency investments into an incentive program has been resource-intensive and messy. Nowhere was this more apparent than in California’s old Risk-Reward Incentive Mechanism, which required utilities to measure and report each efficiency measure individually, rather than rely on preordained deemed savings estimates. In 2006, California spent 7.6 percent of efficiency program funding on EM&V — double the U.S. average of 3 percent to 5 percent. In addition, the lack of transparency meant utility incentives swung drastically due to disagreements on the savings achieved.

That doesn’t mean a measure-by-measure approach is unworkable. Deemed savings approaches have largely succeeded in driving meaningful, cost-effective efficiency savings. Independent stakeholder processes like the Northwest Regional Technical Forum build adaptation into deemed-savings approaches. Advanced data analytics mean real-time measurement is becoming more accurate, and deemed-savings approaches may become obsolete in the face of an emergence of a different kind of real-time, bottom-up approach like metered energy efficiency.

Still, focusing on outcomes rather than activity tracking has the potential to shift focus toward achieving the energy savings and market transformation policymakers seek.

How an outcome-oriented efficiency metric works

Ultimately, policymakers who care about efficiency want to reduce kilowatt-hours consumed, not increase utility-attributed savings. An outcome-oriented approach tracks total kilowatt-hours consumed, kilowatt-hours per customer, or kilowatt-hours per unit of GDP. By tying utility revenue to these outcomes, regulators can strengthen the link between utility profit and achievement of measured, real-world efficiency.

The real work comes into play in designing workable metrics and setting ambitious and realistic targets. For instance, New York’s Energy Efficiency Metrics and Targets Options Report adds detail to the PSC’s direction and highlights areas crucial to the success of an outcome-oriented approach.

The crucial question becomes this: How do regulators choose a metric for portfolio-wide efficiency that accounts for things outside of the utility’s control, avoids overpaying, prevents gaming, and properly incentivizes the utility?

Normalizing the metric to measure the outcome

Dozens of factors affect electricity use, but efficiency programs have a long history of normalization for these factors. Utilities’ annual electricity forecasts typically contain a 2 percent to 3 percent margin of error, with a tendency toward overestimation — not a small sum, but not insurmountable for purposes of setting an ambitious, realistic target for savings. And states already use revenue decoupling to close the utility revenue gap left by efficiency measures.

To get the outcome-oriented approach right, regulators must transparently adjust utility performance metrics like kilowatt-hour/customer for factors like economic growth and weather, so utilities don’t get unfairly over- or under-compensated. Over- or under-compensation may create more of a risk when using models to determine outcome-oriented metrics, but that risk should be weighed against the potential to drive innovative approaches to achieve a more efficient, affordable, clean electricity system.

Approaches and challenges of normalization

Modeling can rise to this task, but potential challenges exist with a top-down approach. Modeling savings and adjusting metrics to reflect outside factors will fall apart without transparency. Without agreeing upon a robust methodology first, utilities, regulators and stakeholders would fall into the same trap of fighting over assumptions and losing the benefits of an outcome-oriented approach.

Econometric regression models are able to capture effects with some accuracy, but may obscure the true cause of behavior that can better be captured in randomized control trials or metered efficiency. As Berkeley economist Max Auffhammer put it: “In practice, you are possibly attributing the effect of weather, for example, to programs. Cold winters make me want to be more energy-efficient. It’s the winter, not the rebate, that made me buy a more efficient furnace.” This begets some errors, but models can and should be adjusted over time, while error ranges should remain public information.

A pooled baseline prediction approach has promise — it models aggregate savings alongside the effects of weather and other factors to set the target, then allows the utility to use the same adjustment factors to measure performance against the target. This approach relies heavily on transparency for success, since undisclosed assumptions underlying the modeled savings would reopen the possibility to wrangle over counterfactuals anew.

New data analysis methods like metered energy efficiency can track the normalized energy savings at each customer meter. Metered efficiency calculates weather-normalized energy savings and realization rates based on meter data before and after installation. That data could add accuracy to metrics, feeding normalized individual customer data into a final aggregate metric.

Data availability is also a crucial challenge to ensure modeling rises to the task of accurate efficiency measurement. Weather and economic data are publicly available, but are not easily parsed into utility service territories, which can look more like Rorschach inkblots than simple geographic boundaries. Utilities may also lack data about whether appliances have been electrified, EVs bought and sold, and occupancies changed. To take advantage of third-party innovation, customer data must be standardized and made public easily through standards like Green Button that simultaneously balance privacy concerns.

Preventing gaming and manipulation

Any approach requiring setting a baseline will be subject to manipulation by the utility and all parties with a vested interest in the outcome of regulation. Ultimately, upfront transparency and certainty are the hallmarks of good metric and target design.

For example, the inputs and design of models normalizing consumption metrics must be vetted in a public, inclusive process. Data inputs must be made publicly available and subject to the scrutiny of regulators and intervenors. Data sources themselves must be opened to the public to the extent they are compatible with individual consumer privacy concerns. While adaptation should be built into any regulatory framework, such adaptations will have to abide by the same standards of upfront transparency and inclusivity.

So where does that leave us?

Efficiency programs will never be perfect, but the perfect should not be the enemy of the good. Bottom-up modeling can take the focus away from efficiency outcomes, so regulators may be willing to trade some precision for lower program costs and a greater ability to capture the value that rapidly evolving efficiency techniques present to the electricity system.

Outcome-oriented efficiency incentives will require thorough experimentation, but they have great potential. With improved data access, a transparent and adaptive approach, and new analytical techniques, outcome-oriented efficiency metrics can focus utility management on outcomes.

New York’s approach puts it on the cutting edge of state utility regulation and establishes a new business model to get the most out of efficiency investments. Regulators should watch New York closely and consider exploring this promising approach to utility efficiency compensation.

++++

Thanks to Devra Wang, Matt Golden, Robbie Orvis, and Eric Gimon for their input.