A Balanced Scorecard Approach to Measuring Lab-XL Environmental Performance

The Final Project Agreement (FPA) for the New England Laboratories XL Project designates nine Environmental Performance Indicators (EPIs). These indicators were selected to track the effects of the regulatory change granted by the project. These indicators were selected based on extensive discussions among the project stakeholders about regulatory goals (pollution prevention and compliance), the nature of laboratory chemical use and waste generation (irregular), and the potential value of Environmental Management Systems in improving the environmental performance of the schools. These issues were the key considerations when writing the Laboratory Environmental Management Standard (LEMS) being tested by the project.

 

One of the challenges of assessing the success of the Lab XL project is evaluating this number of EPIs in light of the lessons learned during the first four years of the project. This year’s report presents the EPIs in a format that connects the EPIs to the basic Environmental Management System (EMS) rubric of “Plan, Do, Check, Act”. This alternative format provides a perspective and a sense of prioritization that is not captured in the previous reports. In making this change, we are not making any changes in the methodology of measuring the EPIs

The Challenge of the EPIs

It has taken longer than expected for the implementation of the Environmental Management Plans (EMP) required by the LEMS to show clear trends in the EPIs selected. While some of the indicators demonstrated immediate progress, several of the indicators have only recently begun to demonstrate the anticipated benefits of the rule. Others have not demonstrated the expected benefits. We feel that this inconsistency is as much a result of the strengths and weaknesses of the indicators selected as a reflection of the overall value of the LEMS.

In reviewing the experience of the pilot schools, we have identified three key factors in the EPIs that contribute to this inconsistency:

• Normalization of some of the EPIs has proven more important and more problematic than anticipated. For example, research funding at Boston College and University of Vermont has increased substantially over the five year project, resulting in a significant, but indeterminate, increase in laboratory chemical use. In light of this increase, the goal of a decrease of 10% in the amount of chemical waste (EPI #5) from the baseline period is unlikely, even though the implementation of the EMP has been a success from a compliance point of view.
  
  Attempts to determine a simple correction factor for this increase in activity have proven unsuccessful. This is probably because confounding factors such as laboratory renovations and clean-outs accompany expansion of research activities. We believe that this is an issue that requires further study to make the waste generation EPI useful. Similarly, the raw number of people trained, as used in EPI #7, would be more helpful if it is viewed in the context of the number of new lab workers added in the same time period. Unfortunately, this second number has proven to be very difficult to establish within the decentralized structure of teaching and research labs, especially at the larger institutions.
• Some EPIs are problematic in the context of the overall project because they contradict goals associated with other indicators. For example, the goal of the Hazardous Chemicals of Concern indicators (EPIs #1 and #2) is to measure excess hazardous chemicals being stored in laboratories and the progress of efforts to stimulate their timely disposal. The disposal of these excess chemicals increases the amount of waste disposed of by the schools, while the goal set for the amount of hazardous waste generated by laboratories (EPI #5) was to decrease the amount by 10%. Similarly, the success of the most important pollution prevention technique for laboratory chemicals - improved chemical inventory control - decreases the opportunity for chemical recycling measured by EPI #4.
• At the time of the signing of the FPA in 1999, Environmental Management Systems were a new concept for environmental management in higher education. Therefore, much of the discussion about the implementation of an EMS-based rule was speculative. Experience at the three schools has shown that some of the EPIs chosen to measure this aspect of the project were not well defined. For example, EPI #8, which seeks to evaluate the school’s EMP effectiveness through its achievement of program objectives and targets has proven to be redundant with the other EPIs, because those EPIs serve as the program’s objectives and targets. Our experience suggests that the audit scores indicator, EPI #9, and the number of people trained, EPI #7, are better measures of the ongoing strength of the EMS because these EPIs demonstrate the relative success of the combination of institutional oversight and worker involvement in the program that an EMS relies upon.

Lessons Learned

The Lab XL project has been a valuable source of information about the nature of laboratory waste and pollution prevention. Three key lessons about the indicators used to improve the management of such waste can be identified:

The Nature of Laboratory Waste Indicators

One important lesson that the pilot schools have learned over the course of the project is that the EPIs should not be construed as specific measures of physical conditions that indicate the state of the laboratory waste process at a particular point in time. Laboratory work (whether teaching or research) is too complex and dynamic a process to identify specific data points that describe its environmental impact with respect to hazardous chemical use. The reason for this is that chemical purchases do not represent a significant portion of the laboratory budget (less than 20% in the case of chemistry teaching labs, less than 5% in the case of biomedical research labs at UVM). Therefore, chemicals are purchased and disposed of as the need arises, rather than regarded as a key part of the overall laboratory management process.

This factor does not prevent a commitment by laboratory workers to pollution prevention because the laboratory worker is the prime beneficiary of using less hazardous chemicals. But it does mean that the reduction in certain toxic chemicals in the laboratory is rarely addressed because the issue (e.g., costs of chemicals) is not systematically managed in laboratories. The consequent lack of data about laboratory chemicals makes identifying and assessing the effects of a regulatory change on pollution prevention activities through indicators a significant challenge. Therefore, we propose that several different EPIs be combined into a scorecard in order to provide a more complete and accurate picture of the effect of the LEMS than any single indicator would.

Timing

The second lesson involves the timing of the changes in the indicators. The relative significance of each Lab-XL indicator has varied over time as the EMP goes from the “implementing stage” to the “sustaining stage”. During the EMP implementation, many historical issues with laboratory hazardous chemical management practices were discovered and addressed. This process provides lessons that result in relatively quick improvements that are reflected in some EPIs. During the sustaining stage, the program maintains its focus on continuous improvement, but the results are seen more slowly, as practices change less often and affect different EPIs.

For example, all three pilot schools showed significant increases in the environmental awareness of laboratory workers in the first few years of the project as training and outreach to laboratories increased; after that period, this awareness has been maintained at a higher, but steady, level. This observation is the basis for our belief that the changes in the survey scores are more important as an EPI during the implementation of the EMP than during its sustaining period.

Conversely, other indicators provide better measures of the value of the EMP during its long-term sustaining phase than during its implementation. A good example of this is EPI #5, the amount of laboratory waste generated by the school. At each institution, the project’s emphasis on removal of excess chemicals from laboratories through the HCOC inventory and laboratory compliance audit processes resulted in increases in the amount of chemical waste during the initial stages of this pilot project. During the sustaining phase of the EMP, chemical clean-outs are less important in determining how much laboratory waste is generated. For purposes of the scorecard, those EPIs that identify and measure trends that relate to the organization’s long term environmental impact are given more weight during the second phase of the project.

Institutional Variation

One of the most important lessons learned over the first phase of the XL is how much variation there is between the chemical waste management programs at the three pilot schools, even though each serves a similar number of students. Because of the type and history of laboratory activities at each school, the best approach to managing hazardous waste and pollution prevention at each are significantly different. For example, Boston College’s chemical waste is generated almost entirely by its Chemistry Department, while UVM’s Chemistry Department produces only about 35% of that University’s waste and UMass Boston’s waste generation is distributed relatively evenly across several different science departments.

These differences explain why the best approach to maximizing environmental performance differs from institution to institution. This will be reflected in the development of different scorecards for each institution. For example, neither Boston College nor UMass Boston have the staff or facilities necessary to support a new chemical redistribution portion within their centralized chemical inventory support program as UVM’s ChemSource program does. Therefore, the ChemSource indicator proposed as UVM’s “Act” EPI is unlikely to work as well within the scorecards at the other campuses.

The regulatory flexibility in the LEMS that allows for these differences between schools to be reflected in their EMPs is probably the most important feature of the Lab-XL project. Because of this flexibility, the individual school’s scorecards can assign the various EPIs to different portions of the EMS cycle, while assuring that the EMP provide a comprehensive and effective approach to the management of this environmental aspect.

Moving Forward

Based on these considerations, BC, UMass Boston and UVM have organized the Lab-XL EPIs in a scorecard format, either matrix (BC) or in a chart (UMass Boston and UVM).

The scorecard places the EPIs in the context of the EMS “Plan, Do, Check, Act” paradigm, except for EPI #8, which is represented by the chart as a whole in the UMass Boston and UVM figures. We believe that the scorecard proposed provides the basis for continuous improvement of laboratory chemical waste program that not only meets the goals of compliance with RCRA at the institutional level but also provides the basis for an ongoing pollution prevention program within the laboratories based on a culture of hazardous chemical awareness.

Those EPIs that are most associated with the implementation phase of the EMP are given less emphasis. For example, these EPIs remain outside the scorecard chart for UMass Boston and UVM. The goal for these EPIs will be to maintain their current level, which we believe will be a natural consequence of the activities undertaken to sustain the effectiveness of the EMP.

By building on the EMS basis of the LEMS, the pilot schools are better able to put the results of the EPIs in context and prioritize program activities in order to maintain the continuous improvement shown during the first phase of the project. This scorecard approach also provides a level of reporting and transparency for regulatory oversight and enforcement at least as stringent as RCRA while providing flexibility and data necessary to support each school’s laboratory pollution prevention activities with respect to chemical waste generation from laboratories.

 

Ralph Stuart, University of Vermont

Gail Hall, Boston College

Zehra Schneider Graham, University of Massachusetts Boston