A Balanced Scorecard Approach to Measuring Lab-XL Environmental Performance
The Final Project Agreement (FPA) for the New
England Laboratories XL Project designates nine Environmental Performance
Indicators (EPIs). These indicators were selected to track the effects of the
regulatory change granted by the project. These indicators were selected based
on extensive discussions among the project stakeholders about regulatory goals
(pollution prevention and compliance), the nature of laboratory chemical use and
waste generation (irregular), and the potential value of Environmental
Management Systems in improving the environmental performance of the schools.
These issues were the key considerations when writing the Laboratory
Environmental Management Standard (LEMS) being tested by the project.
One of the challenges of assessing the success of the Lab XL project is evaluating this number of EPIs in light of the lessons learned during the first four years of the project. This year’s report presents the EPIs in a format that connects the EPIs to the basic Environmental Management System (EMS) rubric of “Plan, Do, Check, Act”. This alternative format provides a perspective and a sense of prioritization that is not captured in the previous reports. In making this change, we are not making any changes in the methodology of measuring the EPIs
The Challenge of the EPIs
It has taken longer than expected for the implementation of the Environmental Management Plans (EMP) required by the LEMS to show clear trends in the EPIs selected. While some of the indicators demonstrated immediate progress, several of the indicators have only recently begun to demonstrate the anticipated benefits of the rule. Others have not demonstrated the expected benefits. We feel that this inconsistency is as much a result of the strengths and weaknesses of the indicators selected as a reflection of the overall value of the LEMS.
In reviewing the experience of the pilot schools, we have identified three key factors in the EPIs that contribute to this inconsistency:
- Normalization
of some of the EPIs has proven more important and more problematic than
anticipated. For example, research funding at
Attempts to determine a simple correction factor for this increase in activity have proven unsuccessful. This is probably because confounding factors such as laboratory renovations and clean-outs accompany expansion of research activities. We believe that this is an issue that requires further study to make the waste generation EPI useful. Similarly, the raw number of people trained, as used in EPI #7, would be more helpful if it is viewed in the context of the number of new lab workers added in the same time period. Unfortunately, this second number has proven to be very difficult to establish within the decentralized structure of teaching and research labs, especially at the larger institutions. - Some EPIs are problematic in the context of the overall project because they contradict goals associated with other indicators. For example, the goal of the Hazardous Chemicals of Concern indicators (EPIs #1 and #2) is to measure excess hazardous chemicals being stored in laboratories and the progress of efforts to stimulate their timely disposal. The disposal of these excess chemicals increases the amount of waste disposed of by the schools, while the goal set for the amount of hazardous waste generated by laboratories (EPI #5) was to decrease the amount by 10%. Similarly, the success of the most important pollution prevention technique for laboratory chemicals - improved chemical inventory control - decreases the opportunity for chemical recycling measured by EPI #4.
- At the
time of the signing of the FPA in 1999, Environmental Management Systems
were a new concept for environmental management in higher education.
Therefore, much of the discussion about the implementation of an EMS-based
rule was speculative. Experience at the three schools has shown that some
of the EPIs chosen to measure this aspect of the project were not well
defined. For example, EPI #8, which seeks to evaluate the school’s EMP
effectiveness through its achievement of program objectives and targets
has proven to be redundant with the other EPIs, because those EPIs serve
as the program’s objectives and targets. Our experience suggests that the
audit scores indicator, EPI #9, and the number of people trained, EPI #7,
are better measures of the ongoing strength of the
Lessons Learned
The Lab XL project has been a valuable source of information
about the nature of laboratory waste and pollution prevention. Three key
lessons about the indicators used to improve the management of such waste can
be identified:
The Nature of Laboratory Waste Indicators
One important lesson that the pilot schools have learned over the course of the project is that the EPIs should not be construed as specific measures of physical conditions that indicate the state of the laboratory waste process at a particular point in time. Laboratory work (whether teaching or research) is too complex and dynamic a process to identify specific data points that describe its environmental impact with respect to hazardous chemical use. The reason for this is that chemical purchases do not represent a significant portion of the laboratory budget (less than 20% in the case of chemistry teaching labs, less than 5% in the case of biomedical research labs at UVM). Therefore, chemicals are purchased and disposed of as the need arises, rather than regarded as a key part of the overall laboratory management process.
This factor does not prevent a commitment by laboratory workers to pollution prevention because the laboratory worker is the prime beneficiary of using less hazardous chemicals. But it does mean that the reduction in certain toxic chemicals in the laboratory is rarely addressed because the issue (e.g., costs of chemicals) is not systematically managed in laboratories. The consequent lack of data about laboratory chemicals makes identifying and assessing the effects of a regulatory change on pollution prevention activities through indicators a significant challenge. Therefore, we propose that several different EPIs be combined into a scorecard in order to provide a more complete and accurate picture of the effect of the LEMS than any single indicator would.
Timing
The second lesson involves the timing of the changes in the indicators. The relative significance of each Lab-XL indicator has varied over time as the EMP goes from the “implementing stage” to the “sustaining stage”. During the EMP implementation, many historical issues with laboratory hazardous chemical management practices were discovered and addressed. This process provides lessons that result in relatively quick improvements that are reflected in some EPIs. During the sustaining stage, the program maintains its focus on continuous improvement, but the results are seen more slowly, as practices change less often and affect different EPIs.
For example, all three pilot schools showed significant increases in the environmental awareness of laboratory workers in the first few years of the project as training and outreach to laboratories increased; after that period, this awareness has been maintained at a higher, but steady, level. This observation is the basis for our belief that the changes in the survey scores are more important as an EPI during the implementation of the EMP than during its sustaining period.
Conversely, other indicators provide better measures of the value of the EMP during its long-term sustaining phase than during its implementation. A good example of this is EPI #5, the amount of laboratory waste generated by the school. At each institution, the project’s emphasis on removal of excess chemicals from laboratories through the HCOC inventory and laboratory compliance audit processes resulted in increases in the amount of chemical waste during the initial stages of this pilot project. During the sustaining phase of the EMP, chemical clean-outs are less important in determining how much laboratory waste is generated. For purposes of the scorecard, those EPIs that identify and measure trends that relate to the organization’s long term environmental impact are given more weight during the second phase of the project.
Institutional Variation
One of the most important lessons learned over the first
phase of the XL is how much variation there is between the chemical waste
management programs at the three pilot schools, even though each serves a
similar number of students. Because of the type and history of laboratory activities
at each school, the best approach to managing hazardous waste and pollution
prevention at each are significantly different. For example,
These differences explain why the best approach to maximizing environmental performance differs from institution to institution. This will be reflected in the development of different scorecards for each institution. For example, neither Boston College nor UMass Boston have the staff or facilities necessary to support a new chemical redistribution portion within their centralized chemical inventory support program as UVM’s ChemSource program does. Therefore, the ChemSource indicator proposed as UVM’s “Act” EPI is unlikely to work as well within the scorecards at the other campuses.
The regulatory flexibility in the LEMS that allows for these
differences between schools to be reflected in their EMPs is probably the most
important feature of the Lab-XL project. Because of this flexibility, the
individual school’s scorecards can assign the various EPIs to different
portions of the
Moving Forward
Based on these considerations, BC, UMass Boston and UVM have organized the Lab-XL EPIs in a scorecard format, either matrix (BC) or in a chart (UMass Boston and UVM).
The scorecard places the EPIs in the context of the EMS “Plan, Do, Check, Act” paradigm, except for EPI #8, which is represented by the chart as a whole in the UMass Boston and UVM figures. We believe that the scorecard proposed provides the basis for continuous improvement of laboratory chemical waste program that not only meets the goals of compliance with RCRA at the institutional level but also provides the basis for an ongoing pollution prevention program within the laboratories based on a culture of hazardous chemical awareness.
Those EPIs that are most associated with the implementation phase of the EMP are given less emphasis. For example, these EPIs remain outside the scorecard chart for UMass Boston and UVM. The goal for these EPIs will be to maintain their current level, which we believe will be a natural consequence of the activities undertaken to sustain the effectiveness of the EMP.
By building on the
Ralph Stuart,
Gail Hall,
Zehra Schneider Graham,