Conceptual model of comprehensive research metrics for improved human health and environment

Modelo conceitual de indicadores de desempenho abrangentes para a melhoria da saúde humana e do meio ambiente

REVISÃO REVIEW

Conceptual model of comprehensive research metrics for improved human health and environment^*

Modelo conceitual de indicadores de desempenho abrangentes para a melhoria da saúde humana e do meio ambiente

Jill Engel-Cox^I; Bennett Van Houten^II; Jerry Phelps^II; Shyanika Rose^III

^IBattelle Memorial Institute. 2101 Wilson Blvd., Suite 800, Arlington, VA 22201 USA. [e-mail]

^IIDivision of Extramural Research and Training, National Institute of Environmental Health Sciences, National Institutes of Health, Department of Health and Human Services, Research Triangle Park, North Carolina, USA

^IIIBattelle Memorial Institute, Durham, North Carolina, USA

ABSTRACT

Performance measurement predominantly consisted of near-term outputs measured through bibliometrics, but the recent focus is on accountability for investment based on long-term outcomes. Our objective is to build a logic model and associated metrics through which to measure the contribution of environmental health research programs to improvements in human health, the environment, and the economy. We developed a logic model that defines the components and linkages between extramural environmental health research grant programs and the outputs and outcomes related to health and social welfare, environmental quality and sustainability, economics, and quality of life, focusing on the environmental health research portfolio of the National Institute of Environmental Health Sciences (NIEHS) Division of Extramural Research and Training and delineates pathways for contributions by five types of institutional partners in the research process. The model is being applied to specific NIEHS research applications and the broader research community. We briefly discuss two examples and discuss the strengths and limits of outcome- based evaluation of research programs.

Key words: Conceptual model development, Environmental health research, Metrics development, Performance measurement, Research impact evaluation

RESUMO

A avaliação de desempenho compreendia predominantemente resultados de curto prazo avaliados através de bibliometria, mas recentemente a ênfase voltou-se à prestação de contas dos investimentos com base em resultados a longo prazo. Nosso objetivo é criar um modelo lógico e métricas associadas através dos quais possamos avaliar a contribuição de programas de pesquisa em saúde ambiental para melhorar a saúde humana, o meio ambiente e a economia. Desenvolvemos um modelo lógico que define os componentes e elos entre os programas de pesquisa em saúde ambiental extramuros subsidiados e os resultados relacionados à saúde e ao bem-estar social, qualidade ambiental e sustentabilidade, economia e qualidade de vida, com ênfase no portfólio de pesquisa em saúde ambiental do National Institute of Environmental Health Sciences (NIEHS), divisão de pesquisa e treinamento extramuros, delineando caminhos para as contribuições de cinco tipos de parceiros institucionais no processo de pesquisa. O modelo está sendo usado em aplicações específicas do NIEHS e na comunidade de pesquisa como um todo. Analisamos brevemente dois exemplos e os pontos fortes e limitações da avaliação baseada em resultados dos programas de pesquisa.

Palavras-chave: Desenvolvimento de modelo conceitual, Pesquisa em saúde ambiental, Desenvolvimento de métrica, Avaliação de desempenho, Avaliação de impacto de pesquisa

The mission of the National Institute of Environmental Health Sciences (NIEHS) is to reduce the burden of human illness and dysfunction from environmental causes. This mission is furthered partly through funding of extramural research in science that focuses on the cellular and molecular basis of environmentally induced disease. Other types of projects funded as part of the extramural research portfolio include epidemiologic and community-based participatory research, as well as worker training and education. NIEHS is achieving its mission by focusing on diseases for which there is a strong indication of an environmental component, and for which there is high or increasing prevalence in the U.S. population (e.g., asthma); by fostering integrated research teams testing complex hypotheses that address the interplay of environmental and other factors, such as genetics, sex or gender, age, and lifestyle; and by developing initiatives identifying the complex factors in the environment that can increase the risk of disease by supporting basic research that develops the scientific basis for health decisions, as well as applied research that fills gaps in understanding of environmental health risks¹.

Given the complexity and diversity of research, program evaluation is critical to understanding and documenting the effectiveness of funded research in illuminating the linkages between the environment and human health. Mandates such as the Government Performance and Results Act of 1993 have required research agencies to look beyond measures of output (e.g., publications produced) toward metrics related to longterm outcomes on public health. Guidance from the U.S. Office of Management and Budget Program Assessment Rating Tool (PART) requires that outcomes of a program (managed by a single entity) be linked to a clear set of program and agency goals, yet be external to the research program². When reviewing fundamental research programs using the PART guidance, managers of these programs face significant challenges in demonstrating a link between traditional research outputs and outcomes³. Health and environmental research organizations such as NIEHS have been challenged to define and measure outcomes distant in time and space from environmental health research⁴. Outcome-based measures of accountability for research grants are inherently difficult, because by definition in the Federal Grants and Cooperative Agreement Act of 1977, grants have indirect benefit to and little substantial involvement by federal agencies.

The objective of this study was to develop a conceptual framework to measure the impact of environmental health research programs on human health, the environment, and the economy, even when the impact may be indirect or diffuse.

Approach

Describing a research portfolio as comprehensive and multidisciplinary as that of NIEHS and measuring its effect on environmental health require a strategic approach that acknowledges all of the potential components of the research process and the application of that research to society in order to ultimately improve human health and quality of life. To design this approach, we developed a comprehensive logic model describing the agency's extramural research portfolio from grant award through ultimate outcomes. Logic models are graphic depictions of the relationship between a program's activities and its intended outcomes ^5,6 and help to explain a program's "theory" or the underlying structure of how the program is intended to work⁷. Besides being an evaluation tool, a logic model can also help program managers describe, and make explicit, how program "performance" is designed to achieve outcomes⁸. Research programs have extended traditional program logic to illustrate how research contributes to topics that inform federal decisions about protective health standards (e.g., National Research Council⁹. To broaden this conception and to incorporate requirements for outcome-based program evaluation, our logic model of a research program provides a visual and conceptual representation of what broad impacts the research program is likely to have and how the impacts are achieved. The simplest structure that defines the impact of research on society is a linear progression: inputs ' research activities ' outputs ' outcomes ' ultimate outcomes. We chose this format because much of the theoretical and methodological literature describing the research process either explicitly or implicitly provides information on the inputs, activities, outputs, or outcomes of research, as well as describing how these elements of the research process can be linked to one another¹⁰. Even though the process may not be linear, our focus is on the influence of specific research program inputs on a range of outcomes and does not attempt to evaluate all the influences of a particular outcome. Definitions of the logic model components are presented below. Inputs are resources that feed into the research program from NIEHS, other federal agencies, research institutions, and community and business partners (e.g., funding, staff qualifications, technical assistance, grantees, organizational resources, community resources). Activities are actions that describe how the inputs are used to carry out the research program or project (e.g., grant awarding, exposure/risk assessments). Outputs are the direct products of the research activities, such as publications, presentations, and new funding applications, as well as patents and products.

Outcomes are benefits or changes resulting from the use of the research outputs. Outcomes are defined further as short term, intermediate, long term, and ultimate. Assigning time frames to the four levels of outcomes is difficult, because the length of time taken is highly variable depending on the individual outcome and the many factors that may affect it. Short- to long-term outcomes may include:

. Translation into or adoption of policy or administrative decisions, clinical guidelines, improved allocation of resources, setting of health targets, development of criteria for evaluative and inspective bodies, commercial development and availability of products, behavioral change among practitioners, and the use of commercial products^11,12.

. New and improved products and processes; methods of organizing, managing, and evaluating products and environments; improved safety of products and work environments; and individual and sector productivity rates¹³

. The incidence, magnitude, and duration of social change¹³.

Ultimate outcomes of environmental health research may include:

. Health and social welfare gain and national economic benefit from commercial exploitation and a healthy workforce^11,12

. Environmental quality and sustainability, improved health care and healthy longevity, and provision of basic needs to the population¹³

. International balance of trade (i.e., the relation of exports to imports of various countries), energy independence, gross national product, and quality of life¹⁴.

Two other components of the logic model as they related to the NIEHS extramural research portfolio include contextual factors and reservoir of knowledge. Contextual factors could potentially affect the research environment through availability of resources or shifts in research or policy priorities that create constraints or opportunities for the research program. Examples include political or society interests, external triggers such as a disease outbreak, state of the economy, and other national and global socioeconomic influences.

Reservoir of knowledge represents the accumulation of understanding, knowledge, and previous research that may or may not be directly related to the NIEHS extramural research portfolio but contributes to the development of and, in turn, is contributed to as a result of the research activities described within the model. This "knowledge pool" is difficult to measure concretely, but encompasses both research and the interaction of individuals that "interact and produce innovation and discovery through unpredictable paths and at uneven intervals"³.

Conceptual logic model and submodels for research metrics

The logic model delineates separate pathways acknowledging contributions by the institutions partnering in the research process: NIEHS, other government (federal, state, and local) agencies, grantee institutions, business and industry, and community partners. Each institutional pathway contains specific logic model elements related to inputs, activities, outputs, and outcomes. Underlying each element are specific metrics (see Table 1 for examples).

Distinctions drawn between the institutional pathways are artificial to some degree, and there is considerable crossover between submodels. Generally, however, each pathway illustrates the research process that would be carried out most directly by a given institutional partner that is being evaluated. This should not be taken to imply that we consider the pathway shown to be the most influential on a particular outcome. In the following sections, we further describe the five institutional pathways and their components. Relationships between the institutions are represented by the arrows connecting components in different institutional pathways. However, relative strength and importance of these relationships cannot be determined from this model.

Government pathway: NIEHS and other agencies

This pathway describes the inputs, activities, outputs, and outcomes directly associated with the grant programs of both NIEHS and other government agencies (Table 1). Although this is a combined discussion of the two pathways, examples provided relate primarily to NIEHS.

Inputs

Inputs include funding and resources for NIEHS grant programs and programs of other related agencies such as the U.S. Environmental Protection Agency (EPA), Occupational Safety and Health Administration (OSHA), Food and Drug Administration (FDA), other members of the National Institutes of Health, and the Centers for Disease Control and Prevention. It also includes state and local government agencies that work to improve the environment and human health in their jurisdictions.

Activities

Activities include those by NIEHS in support of its mission and its extramural grant program, such as research grant awarding to external investigators; information transfer to a variety of audiences such as stakeholder outreach sessions, scientific panels, and information booths; and program formulation of new initiatives. Closely related to these activities is the use of grant funds by grantee institutions (shown in the grantee institution pathway).

Outputs

Outputs related to the NIEHS and government pathway include summary reports providing a synthesis of scientific information, press releases announcing research results or program activities, and information provided to legislative bodies as policy background. Related outputs are community outreach events conducted by NIEHS and other agencies (shown in the community pathway).

Outcomes

NIEHS and other government outcomes include those in the short, intermediate, and long term.

. Short term, NIEHS: monitoring and awareness of ongoing research. NIEHS staff maintain an awareness of ongoing environmental health research, whether NIEHS funded or not, to keep abreast of emerging science.

. Short term, government: policy assessments. Before the enactment of new laws and regulations, governmental agencies conduct reviews of research and review recommendations to determine the potential impact of an issue on the environment and human health.

. Short term, government: monitoring and surveillance systems. Monitoring and surveillance measures are put in place by federal, state, or local governments to measure levels of environmental exposures or human disease, sometimes partly in response to reports based on environmental health research. These may be new systems or adaptations of existing systems to measure emerging health hazards.

. Intermediate, NIEHS: identification of scientific needs and new science. Through ongoing monitoring and awareness of environmental health research, NIEHS is able to identify the scientific needs surrounding topics and emerging issues as well as the need for innovative science within the agency's mission.

. Intermediate, government: laws. Environmental and health-related laws develop from an improved understanding of the relationship between the environment and human health based partly on policy assessments made by NIEHS and other agencies. Major new legislation is relatively rare (compared with regulations) and develops from a combination of awareness of a problem and connection to a policy solution. NIEHS work would most likely contribute to the identification of human environmental health issues.

. Intermediate, government: regulations and standards. Regulatory agencies such as the EPA, FDA, and OSHA (as well as state and local regulatory agencies) promulgate and enforce environmental and health regulations and standards. Regulations and standards published in the Code of Federal Regulations are justified in publicly available staff reports, criteria documents, and technical support documents.

. Long term, NIEHS: new grant programs. Given identification of scientific need and new science, NIEHS formulates new initiatives and programs. This activity is similar to that cited under "Activities" and essentially begins the grant-making process anew, advancing scientific understanding by building on earlier research.

. Long term, government: improved environment. Changes in regulatory standards should improve the natural and built environment. There are many potential measures for intermediate outcomes, so for any particular NIEHS research area, specific physical environmental measures would be selected. In general, these would fall into three broad categories: ambient air pollutant concentrations, lake/river/ocean groundwater quality, and/or land use and soil contamination.

. Long term, government: reduced human exposure to environmental hazards. Reduced human exposure represents decreases in communities' or citizens' exposure to environmental hazards that result from regulations and standards. Measures for reduced human exposure are drawn from measures of improvements in the built and natural environments with addition of the number of individuals located within a specified location.

Grantee institution pathway

This pathway describes the inputs, activities, outputs, and outcomes associated with grantee institutions and the research conducted by those institutions.

Inputs

The inputs describe the staff, financial, and organizational resources of the grantee institution receiving NIEHS funding. The resources are available to the grantee investigators to support the institutions' research program.

Activities

These describe the use of the grant funds provided by NIEHS by the grantee institutions. The activities include specific types of research projects that are funded through grants as well as the development of interventions, tools and methods, and other products. Types of activities include basic, epidemiologic, and clinical research; intervention research and development; technology transfer/innovation research; exposure assessments; and training. Related to the activities of universities and other research institutions are the research and development activities of business and industry (shown in the business and industry pathway) and the summary dissemination of results by NIEHS (shown in the NIEHS pathway).

Outputs

The outputs are the direct products of the grantee institution's use of NIEHS grant funds. They include tangible products such as presentations, publications, curricula, intervention, and certifications. They also include less tangible products such as knowledge gained from research, new tools and methodologies, and the career development of investigators such as new funding applications, promotions, and membership in committees or working groups that may result from affiliation with NIEHS-funded research. Related to the outputs associated with grantee institutions and investigators is the public awareness of research activities and research results that affect their health and communities (shown in the community pathway), as well as the awareness of NIEHS staff of ongoing research (shown in the NIEHS pathway).

Outcomes

The grantee outcomes in the model include the following:

. Short term: communities of science. Communities of science are created when investigators working in the same or related areas develop relationships and research networks that contribute to the advancement of knowledge.

. Short term: replication and new research. The use of research findings typically depends upon compilation of evidence from multiple research studies. These can include replication of an initial study and new research that extends earlier studies.

. Intermediate: clinical guidelines and recommendations. Health institutes and professional societies publish clinical guidelines and recommendations related to practice, treatments, and drug use. These are developed based on research and clinical trials that may be funded by NIEHS.

. Intermediate: accumulation of knowledge. Replication and related new research, along with information drawn from other sources, contribute to the accumulation of knowledge and understanding about environmental health. It is the weight of evidence that drives changes in behavior as well as changes in funding priorities.

. Long term: clinical practice changes. As a result of research dissemination, along with the development of laws, policies, and guidelines, health care providers change their practice and treatment behaviors. These changes may be voluntary or regulated, but they are based on the knowledge accumulated through research conducted by NIEHS-funded investigators and others.

Business and industry pathway

This part of the logic model describes the inputs, activities, outputs, and outcomes directly associated with business and industry. It includes research and development activities leading to new commercial products and drugs, as well as the operational and infrastructure changes that industry makes in response to environmental and health hazards.

Inputs

The inputs describe the major relevant research areas of business and industry that may benefit from NIEHS-funded research, through product development or the use of results to adjust their operations. Industries included are a) health care and pharmaceutical companies, b) environmental science companies that prevent or reduce pollution and other environmental hazards, and c) regulated industries that may produce waste or by-products that are pollutants, or d) other environmental hazards.

Activities

The activities in this submodel include the cooperative research conducted by business and industry with research partners; the development of health and environmental products and services such as drugs, medical devices, and monitors; and the use of research results by business and industry. Cooperative research with universities may contribute to investigator career development (in the grantee institution pathway).

Outputs

Intellectual property developed by industry is protected by patents. As the result of research and the development of intellectual property, business and industry develop commercial products related to environmental health. These include drugs and medical products to address health issues, and sales of environmental controls and services.

Outcomes

The business and industry outcomes in the model include the following:

. Short term: commercial products and drugs. The new commercial products developed by business and industry are then sold in the marketplace. Sales represent a short-term outcome because they reflect the amount of influence on health and the environment, as well as measuring benefit to the economy.

. Short term: awareness of environmental health impacts and proposed regulations. Business and industries that are subject to environmental regulations or that produce byproducts that are potential health hazards become aware of the accumulation of research results indicating their potential involvement in environmental health hazards. Outcomes related to this awareness are preregulatory and may include voluntary actions undertaken by companies to avoid legal action or community censure. Although voluntary actions by industry have been found to be limited in reducing emissions compared with mandatory approaches¹⁵, we include them in the model as potential short-term pathway to change, given the political and administrative constraints to fashioning regulations.

. Intermediate: change in operations to reduce environmental hazards. Specific regulations and sometimes awareness of the environmental impacts of their products and actions encourage businesses and industries to reduce the hazards caused by their operations. These are intermediate outcomes because they are most often in response to laws, standards, and regulations.

. Long term: reduced environmental emissions. As in the government pathway, changes in regulatory standards should improve the natural environment. As the primary polluters or the manufacturers of consumer products that release pollutants, business and industry are the main actors in reducing emissions. There are many potential measures for long-term outcomes, including air pollutant emissions, hazardous waste land disposal, chemical discharges into water bodies, and releases of toxics to all media.

Community pathway

This pathway describes the inputs, activities, outputs, and outcomes associated with the community, the general public, that is influenced by or associated with NIEHS extramural funding. The community is also in itself a driver of environmental health impacts in that community activities apart from environmental health research can be a strong influence on broader public policies or research agendas, and promote actions by governmental agencies and business and industry. However, the goal of this model is to show possible mechanisms by which research can influence outcomes, rather than to depict a comprehensive view of how such outcomes may occur.

Inputs

The inputs describe the staff, financial, and organizational resources of the community and the public partners of NIEHS. In addition to individuals making up the general public, the community includes nongovernmental agencies addressing environmental health or environmental justice, community hospitals and clinics providing health care to the public, and schools.

Activities

Activities in this pathway are undertaken by the community and public as a result of NIEHS-funded research. The activities include participating in and/or facilitating community-based participatory research; outreach and education such as health fairs, information sessions, and educational forums; and training on environmental hazards to community members or groups such as first responders, teachers, industrial workers, and children/families.

Outputs

Community outreach including the wide dissemination of environmental health information to the general public, as well as development of publicprivate partnerships and community technology centers for the advancement of environmental health awareness, is the main output of this pathway.

Outcomes

The community outcomes in the model include the following:

. Short term: public awareness. Public awareness is an immediate outcome influenced in part by dissemination of NIEHS-funded research through accessible media.

. Intermediate: change in knowledge and attitudes. As a result of awareness raised by NIEHS-funded research, the public's knowledge and attitudes about the environment, environmental justice, and environmental health issues may be positively influenced.

. Long term: public behavior change and advocacy. Behavior change occurs as a result of changes in knowledge and attitudes about environmental health issues. It includes increased worker protection from environmental hazards; decreased use of toxics and hazardous materials at home, work, and school; decreased consumption of food and water with significant pollutant concentrations; decreased exposure to air pollutants; increased use of public transportation, car pools, and bicycles; and increased access to and awareness of relevant health care. It can also influence business and industry to change practices in response to consumer demand for less toxic and hazardous products.

Ultimate outcomes and contextual conditions

The connection of research to the ultimate outcomes of improved human health involves multiple steps and actors. Typically, these outcomes would appear 1050 years after the initial research, as new clinical practices, laws and regulations, and public behavioral changes are implemented and have an effect. The ultimate outcomes are related to the intermediate outcomes of all institutional pathways and fall into two categories: improved human health and well-being and benefit to the economy. Examples of ultimate outcomes related to improvement of human health include decreases in disease and injuries associated with exposures to adverse environmental health agents. Those associated with benefit to the economy include decreases in health care use, increases in worker productivity, and decreases in worker and school absenteeism due to symptoms and diseases associated with exposures to adverse environmental health agents. Less tangible are increases in value of natural resource goods, services, amenities, and intrinsic value from improved environment.

Discussion

The value of the logic model lies in its utility in developing pathways by which to link NIEHS funded research to ultimate outcomes. In addition, metrics associated with each component document the contribution. To illustrate the potential application of the model, we present two brief examples for discussion. These examples demonstrate a simplified approach of how to trace "forward" the influence that research may have on outcomes, even when that influence may be indirect, diffuse, or delayed. This approach does not attempt to identify all of the possible contributing factors to the noted impact.

Knowledge of the human health effects of ambient airborne pollutants has increased over the last several decades, from an initial focus on ozone and pulmonary diseases such as asthma, to a growing scientific understanding of the effects of fine airborne particulate matter (PM) on cardiovascular disease (e.g., Dockery¹⁶; Donaldson et al.¹⁷; Pope et al.¹⁸). For example, NIEHS-funded researchers at Johns Hopkins University published results of mortality from fine PM in major U.S. cities¹⁹ and on hospital admissions related to fine PM²⁰. Subsequently, research results from these studies and others were disseminated by NIEHS and the institutions themselves through press releases (e.g., Johns Hopkins University²¹; NIEHS²²).

During the last decade, the U.S. EPA shifted its monitoring network to measure finer PM, specifically, PM with diameter less than 2.5 mm (PM2.5). The U.S. EPA revised its regulations to include an annual ambient standard for PM2.5, conducting multiple stages of staff and public review of the new standard from the mid-1990s through 2006 (e.g., U.S. EPA²³). NIEHS-funded research was cited in the regulatory docket (www.regulations.gov) of the later revisions as key evidence for the health effects of PM2.5 (e.g., McConnell et al.²⁴; Raizenne et al.²⁵; Schwartz et al.²⁶). States are required to submit implementation plans to achieve compliance with the new ambient standards; as part of these plans, state and local governments pass rules and regulations requiring industry and consumers to change their operations and reduce emissions. Reduced emissions required by the state implementation plans will improve air quality to the new U.S. EPA standard by 2010.

In response to research documenting cardiovascular and other health effects, the U.S. EPA added fine PM to its air quality index reporting²⁷ and specifically included cardiovascular effects in its public health messages²⁸. Better knowledge of daily air pollution levels and the fact that those with heart disease are also at risk results in behavior modification by the public to reduce activity during pollution events²⁹ and to advocate to reduce local emissions, thus resulting in reduced human exposure and mortality on high-pollutant days. Multiple studies (including some funded by NIEHS) over the last few decades contributed to and were cited by the EPA when setting and modifying the PM2.5 standards.

As the influence is traced through the logic model, it becomes more diffuse and suffers from time discontinuities and lack of documentation. This example illustrates that, with a full evaluation and expert elicitation, it is possible to more specifically identify and semiquantify the impact of NIEHS research, starting with this overview of potential influence. The case of lowered blood lead levels through phase-out of leaded gasoline and other lead-containing products demonstrates the influence of a pathway through the logic model related to the impact of government policy changes. The NIEHS has sponsored research on the health effects of lead for more than 20 years⁶. Beginning in the 1970s, research funds from federal sources (vs. industry-sponsored studies) were allocated to the study of such health effects particularly in children [reviewed by Needleman³⁰]; NIEHS was a large supporter of these studies. Early studies showed that exposure to low levels of lead during early childhood can lead to delays in cognitive and behavioral development, such as lower IQ levels. Dissemination of these results was accomplished through early conferences and publications on low-lead toxicity; for example, an NIEHS-sponsored conference in 1974, the proceedings of which were published that same year in Environmental Health Perspectives. Information from studies like these added to the justification of the need to remove lead from gasoline starting in the 1970s. A criteria document, Air Quality Criteria for Lead³¹, assessed the scientific basis for regulation, and a standard of 1.5 ìg/m3 (maximum quarterly calendar average) lead was set in 1978³². These air pollution regulations have removed significant amounts of lead from the environment^33,34. Data from the second and third National Health and Nutrition Examination Studies show that between 1976 and 1980, there was an average drop in blood lead levels of 30%, concurrent with a 50% reduction in use of leaded gasoline³⁵. These trends have continued with an 86% drop in lead poisoning of children in the United States since the late 1970s and other improvements in health^36,37. In this example, the NIEHS- funded research on blood lead levels supported ongoing regulations and contributed to the documentation of positive health effects.

The challenge of identifying a specific impact from a research program illustrated in these examples arises from how grant-funded research has an indirect benefit to and little substantial involvement by federal agencies³⁸. Although fundamental research on both fine PM and blood lead levels contributed to awareness and monitoring of their relevant environmental health issue, the studies were not designed to set standards or to be used in policy decision making, except as an indirect contribution as aggregate knowledge. The 2004 National Research Council report on airborne PM identifies the synthesis of multiple research studies as a requirement for gauging research progress. Although independent research studies may be ideal process for scientific discovery, structured logic models are needed to trace the diffuse yet important role of specific research programs.

Conclusions

The conceptual logic model for research metrics focuses on NIEHS-funded research programs to measure the contribution of environmental health research to improvements in human health, the environment, and the economy. The model is successfully illustrated here with two brief case examples: effects of PM and blood lead levels. In addition, this logic model approach has been applied to two full case studies—asthma and endocrine disruptors — as part of the larger study, the results of which are to be published separately. Furthermore, a database has been created that maps the logic model components and specific indicators to known published information, online databases, and document repositories that serve as sources of information for measuring outcomes for each logic model component. Although the main application of the logic model presented here was the environmental health research portfolio of NIEHS, its basic elements are applicable to other environmental or health research programs. The institutions that are part of the research process—government agencies, grantee institutions, business and industry, and community partners—are key players in nearly all environment and health programs. Despite the strengths of this approach, persistent challenges still remain. These include the lack of direct attribution of NIEHS-supported work to many of the outcome measures and the lack of robust electronic databases that can be easily searched to help establish these linkages. Mitigation of these problems will require a stronger effort to include better linkages to the primary literature/ grant support and organization of electronic information, particularly policy and/or health guidelines, in an easy format for indexing and searching. This can be achieved only by greater communication among all the stakeholders described in this logic model. We hope that such dialogue will be stimulated by the present study. Finally, this logic model narrows the focus to only one type of input — researchand — its potential contribution to impacts. Therefore, it does not attempt to demonstrate all of the many factors that may have contributed to a given impact. It is therefore important for the analyst using this model to not overstate the contribution of research to the impact versus other types of competing influences. The logic model has been developed to apply to diverse programs within NIEHS and will be used as an ongoing program analysis tool. An area of further research is to apply the model to environment and health research programs at other government agencies, universities and research institutions, and private industry.

Acknowledments

We acknowledge A Powers for her extensive work at the beginning of this project. We gratefully acknowledge the assistance of T Bernichon, M Sill, K Versendaal, and M Wooton, all of Battelle.

Received 26 September 2007

Accepted 11 February 2008

• ¹
  National Institute of Environmental Health Sciences. New Frontiers in Environmental Health Sciences and Human Health. NIEHS 20062011 Strategic Plan. 2006. [accessed 1 March 2007]. Available from: http://www.niehs.nih.gov/external/plan2006/Strategic Plan2006final.pdf
• ²
  Office of Management and Budget. Program Assessment Rating Tool Guidance No. 2006-02. 10 March 2006. [accessed 12 December 2007]. Available from: http://www.whitehouse.gov/omb/part/2006_ part_ guidance.pdf
• 3. Cozzens S. The knowledge pool: measurement challenges in evaluating fundamental research programs. Eval Program Plann 1997; 1:7789.
• 4. Van Houten B, Phelps J, Barnes M, Suk W. Evaluating scientific impact. Environ Health Perspect 2000; 108: A392A393.
• 5. Centers for Disease Control and Prevention. Introduction to Program Evaluation for Public Health Programs: A Self- Study Guide Atlanta, GA: U.S. Centers for Disease Control and Prevention; 2005.
• ⁶
  Department of Health and Human Services. HHS helps in efforts to eliminate childhood lead poisoning. U.S. Department of Health and Human Services. 2002 [accessed 4 April 2007]. Available from: http://www.hhs.gov/news/press/2002pres/lead.html
• 7. Chen H. Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness Thousand Oaks, CA: Sage Publications; 2005.
• 8. McLaughlin J, Jordan B. Logic models: a tool for telling your program's performance story. Eval Program Plann 1999; 22:65-72.
• ⁹
  National Research Council. Research Priorities for Airborne Particulate Matter. IV: Continuing Research Progress. Committee on Research Priorities for Airborne Particulate Matter, Board on Environmental Studies and Toxicology, Division on Earth and Life Studies, National Research Council of the National Academies. Washington, D.C.: National Academies Press;2004.
• 10. Powers A, Rose S, Bernichon T, Engel-Cox J, Sill M, Versendaal K. Literature Review on Research Impact Assessment Methods and Concepts. Work Assignment No. 1, Program Assessments and Evaluations for NIEHS. Contract HHSP23320045006XI, Task Order HHSP 233000015T. Arlington, VA: Battelle; 2006.
• 11. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy 1996; 1:3543.
• 12. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003; 1:2.
• 13. Bozeman B. Public value mapping of science outcomes: theory and method. In: Bozeman B, Sarewitz D, Feinson S, Foladori G, Gaughan M, Gopta A, editors. Knowledge Flows and Knowledge Collectives: Understanding the Role of Science and Technology Policies in Development Vol. 2. Washington, D.C.: Center for Science Policy and Outcomes; 2003. p. 348.
• 14. Rubenstein AH, Geisler E. The use of indicators and measures of the R & D process in evaluating science and technology programmes. In: Roessner JD, editor. Government Innovation Policy Design, Implementation, Evaluation New York: St. Martins Press; 1988. p. 185203.
• 15. Morgenstern RD, Pizer WA, editors. Reality Check: The Nature and Performance of Voluntary Environmental Programs in the United States, Europe, and Japan Washington, D.C.: RFF Press; 2007.
• 16. Dockery D. Epidemiologic evidence of cardiovascular effects of particulate air pollution. Environ Health Perspect 2001; 109(Suppl):483486.
• 17. Donaldson K, Stone V, Seaton A, MacNee W. Ambient particle inhalation and the cardiovascular system: potential mechanisms. Environ Health Perspect 2001; 109 (Suppl 4):523527.
• 18. Pope CA III, Burnett R, Thurston G, Thun M, Calle E, Krewski D.et, Godleski JJ. Cardiovascular mortality and long-term exposure to particulate air pollution: epidemiological evidence of general pathophysiological pathways of disease. Circulation 2004; 109:7177.
• 19. Samet J, Dominici F, Curriero F, Coursac I, Zeger S. Fine particulate air pollution and mortality in 20 U.S. cities, 19871994. N Engl J Med 2000; 343:17421749.
• 20. Dominici F, Peng D, Bell M, Pham L, McDermott A, Zeger S, Samet JM. Fine particulate air pollution and hospital admissions for cardiovascular and respiratory diseases. JAMA 2006; 295:11271134.
• ²¹
  Johns Hopkins University. Particulate Air Pollution Needs to Be Controlled to Protect Public Health. [Press release]. 2000 [accessed 17 May 2007]. Available from: http://www.jhsph.edu/publichealthnews/press_releases /PR_2000/air_pollution_cities.html
• ²²
  National Institute of Environmental Health Sciences. Elderly Have Higher Risk for Cardiovascular, Respiratory Disease from Fine Particle Pollution. PR 06- 063/8/06. 2006 [accessed 17 May 2007]. Available from: http://www.niehs.nih.gov/oc/news/particle.htm
• ²³
  U.S. Environmental Protection Agency. Air Quality Criteria for Particulate Matter. EPA 600/P-99/002aF-bF. Washington, D.C.: U.S. Environmental Protection Agency; 2004.
• 24. McConnell R, Berhane K, Gilliland F, London S, Vora H, Avol E, Gauderman WJ, Margolis HG, Lurmann F, Thomas DC, Peters JM. Air pollution and bronchitic symptoms in Southern California children with asthma. Environ Health Perspect 1999; 107:757760.
• 25. Raizenne M, Neas L, Damokosh A, Dockery D, Spengler J, Koutrakis P, Ware JH, Speizer FE. Health effects of acid aerosols on North American children: pulmonary function. Environ Health Perspect 1996; 104:506514.
• 26. Schwartz J, Norris G, Larson T, Sheppard L, Claiborne C, Koenig J. Episodes of high coarse particle concentrations are not associated with increased mortality. Environ Health Perspect 1999; 107:339342.
• ²⁷
  U.S. Environmental Protection Agency. Air quality index reporting. Final rule. Fed Reg 40 CFR Part 58:42530–42549. Washington, D.C.: U.S. Environmental Protection Agency; 1999.
• ²⁸
  U.S. Environmental Protection Agency. Air Quality Index: A Guide to Air Quality and Your Health. EPA-454/K-03-002. Washington, D.C.: U.S. Environmental Protection Agency; 2003.
• 29. Bresnahan B, Dickie M, Gerking S. Averting behavior and urban air pollution. Land Econ 1997; 73:340357.
• 30. Needleman HL. The removal of lead from gasoline: historical and personal reflections. Environ Res 2000; 84:2035.
• ³¹
  U.S. Environmental Protection Agency. Air Quality Criteria for Lead. EPA-600/8-77-017. Research Triangle Park, NC:Health Effects Research Laboratory, Criteria and Special Studies Office, U.S. Environmental Protection Agency. Washington, D.C.: U.S. Environmental Protection Agency; 1977.
• ³²
  U.S. Environmental Protection Agency. National primary and secondary ambient air quality standards for lead. Fed Reg 40 CFR Part 50:46246–46277. Washington, D.C.: U.S. Environmental Protection Agency; 1978.
• ³³
  U.S. Environmental Protection Agency. EPA Takes Final Step in Phaseout of Leaded Gasoline. Washington, DC:U.S. Environmental Protection Agency; 1996. [accessed 31 March 2007]. Available from: http://www.epa.gov/history/topics/lead/02.htm
• ³⁴
  U.S. Environmental Protection Agency. Latest Findings on National Air Quality: 2001 Status and Trends. U.S. Environmental Protection Agency. 2002 [accessed 7 May 2007]. Available from: http://www.epa.gov/air/airtrends/aqtrnd02/2002_airtrends_final.pdf
• 35. Annest J, Pirkle J, Makuc D, Neese J, Bayse D, Kovar M. Chronological trend in blood lead levels between 1976 and 1980. N Engl J Med 1983; 308:13731377.
• 36. Meyer P, Pivetz T, Dignam T, Homa D, Schoonover J, Brody D. Surveillance for elevated blood lead levels among childrenUnited States, 19972001. MMWR Surveill Summ 2003; 52:121.
• 37. National Institute of Environmental Health Sciences. Low Levels of Lead and IQ Deficits. 2003 [accessed 4 April 2007]. Available from: http://www.niehs.nih.gov/dert/profiles/hilites/2003/lead-IQ.htm
• 38. Federal Grants and Cooperative Agreement Act of 1977. Public Law 95-224.

*

This article was originally published by

Environ Health Perspect 116:583–592 (2008). doi:10.1289/ehp.10925 available via

http://dx.doi.org/ [Online 12 February 2008] and is part of the scientific collaboration between

Cien Saude Colet and EHP. This work was conducted under contract HHSP- 23320045006XI, Task Order HHSP233000015T, "Program Assessments and Evaluations for NIEHS." The authors declare they have no competing financial interests.

Publication Dates

• Publication in this collection
  03 Feb 2009
• Date of issue
  Apr 2009

History

• Received
  26 Sept 2007
• Accepted
  11 Feb 2008