Logic models and organizational training evaluations

Abstracts

A área de treinamento tem produzido pouco sobre análises organizacionais, de forma que avaliações de treinamento em níveis abrangentes tendem a ser escassas na literatura científica. Este estudo teve por objetivo propor um modelo lógico de avaliação de treinamento sobre o desempenho de uma organização pública, bem como testar sua avaliabilidade. Para tanto, foi realizada entrevista coletiva com 11 funcionários para identificação de treinamento capaz de alterar o desempenho da organização, entrevista com um funcionário para escolha de indicadores de desempenho organizacional sensíveis ao treinamento, entrevistas com sete funcionários com vistas à seleção de material para proposição do modelo, grupo focal com 28 funcionários para validação do modelo lógico. Ainda que a avaliabilidade do curso não tenha sido plenamente constatada, o modelo lógico permitiu a clarificação das relações entre o treinamento e o desempenho organizacional, principal entrave em processos avaliativos desta natureza.

treinamento de pessoal; avaliação de programa; avaliação de necessidades; avaliação de recursos humanos

ARTICLE

Logic Models and Organizational Training Evaluations

Modelos Lógicos y Evaluaciones de Entrenamientos Organizacionales

Sônia Maria Souza Damasceno; Gardênia Abbad; Pedro Paulo Murce Meneses

Universidade de Brasília, Brasília-DF, Brasil

^{Correspondence to}

ABSTRACT

The area of personnel training has produced little about organizational analysis, so that evaluations at broader levels of analysis tend to be scarce in the literature. This research aimed to propose a logic model for training evaluation on the performance of a public organization, as well as test their evaluability through: group interview with 11 employees to identify a training capable of impacting the organization's performance, interview with a qualified employee to choose organizational performance indicators sensitive to the selected training, interviews with seven employees to select material for the proposition of the model, and focus group with 28 staff members to validate the logic model. Although the evaluability of the training model has not been fully established, the logical model has served to clarify the possible relationship between training and organizational performance, the main obstacle in evaluation processes of this nature.

Keywords: personnel training, program evaluation, needs assessment, personnel evaluation

RESUMEN

El área de entrenamiento ha producido pocos análisis organizacionales, por lo que evaluaciones de la formación en niveles más amplios tienden a ser poco frecuentes en la literatura científica. Esta investigación objetivó proponer un modelo lógico para evaluar el impacto de una capacitación en el desempeño de una organización pública, así como poner a prueba su posibilidad de evaluación. Para ese fin, fue llevada a cabo una entrevista colectiva con 11 empleados para identificar una formación con la posibilidad de modificar el desempeño de la organización, entrevista con un trabajador para elección de los indicadores de desempeño de la organización sensibles a la formación, entrevistas con siete empleados para elección de los materiales para la composición del modelo, grupo focal con 28 funcionarios para validar el modelo lógico. Aunque la evaluabilidad del curso no fue totalmente verificada, el modelo sirvió para aclarar las relaciones entre la formación y el desempeño de la organización, principal obstáculo en ese tipo de evaluaciones.

Palabras clave: entrenador personal, evaluación de programa, evaluación de necesidades, evaluación de recursos humanos

Since the 1970's, emerging technological innovations and new workforce management forms have profoundly affected relations between companies and their main stakeholders, so that, today, the Brazilian consumption market, especially with regard to public services, increasingly demands high-quality services. In that context, the acknowledged importance of an updated workforce that is able to adapt to constant variations in the socio-environmental context results in a large flow of financial resources to personnel training. Despite high investments, mainly in companies that chose the form of Corporate Education Centers, most of these tend to add little value to organizational business (Eboli, 2004).

In the attempt to explain these investments' small rates of return, training evaluation has stood out among different study areas in organizational and work psychology, training psychology and even personnel management. But most evaluation studies concentrate on effects at the individual analysis level. What training efficacy is concerned, a dimension that covers measures and indicators at higher analytic levels, the difficulty to article individual performance aims and targets with organizational results has been a constant challenge.

Thus, this study discusses the construction of an evaluation proposal for a training program on the performance of a federal public organization, with a view to guiding researchers and professionals in the attempt to overcome the challenges the personnel training, development and education area is confronted with. Academically, the aim was to assess the evaluability or pertinence of evaluating the impact of the proposed training on the performance of a public entity, using the tool called the Logic Model. Therefore, the following specific aims were proposed: (a) identify a training action that is potentially capable of influencing organizational performance, (b) select indicators to evaluate training efficacy, (c) develop a training efficacy evaluation model, and (d) identify evidences of training efficacy.

Theoretical Framework

In a highly competitive context, personnel training, development and education (TD&E) actions gain fundamental importance, currently empowered by the technological enhancements that have so strongly contributed to the transformation of traditional training centers into so-called corporate education centers. According to Abbad and Borges-Andrade (2004), the outstanding role of learning has stimulated companies to shape training systems whose opportunities guarantee higher individual and organization performance levels.

This training system includes three components - needs assessment, planning and execution and evaluation of effects - that articulate with the personnel management functions and other organizational subsystems. This research concentrates on the latter subsystem, which examines the results the promoted actions entail for individuals, teams, organizations and even for society. In this field, great advances have been made, to the extent that, as a result of the availability of well-designed theoretical-methodological reference objects and frameworks, the area recently gained the condition of scientific research area (Salas & Cannon-Bowers, 2001).

Most of these advances concentrated on evaluation criteria that were centered on an individual analytic perspective though, typically put in practice through satisfaction and utility measures of training, learning and post-training performance, so that higher-level effects are not guaranteed for the work units and the organization (Mourão, 2004). According to Meneses (2007), little is known about how training programs, commonly designed to respond to individual needs only, produce results at higher analytic levels.

Only some studies have attempted to scientifically overcome this limitation, including the Brazilian studies by Borges-Andrade, Pereira, Puentes-Palácios and Morandini (2002), Freitas (2005), Mourão (2004), Mourão, Borges-Andrade and Salles (2006) and Meneses (2007). According to a review published in the Annual Review of Psychology (Aguinis & Kraiger, 2009), a renowned journal that, since the 1970's, has periodically assembled international production on TD&E, the major part refers to opinion surveys that are hardly strict in academic terms.

According to Freitas, Borges-Andrade, Abbad and Pilati (2006), the small number of studies is due to the fact that: (a) organizational control systems rarely address efficacy indicators that are sensitive to training actions, (b) the design of training programs tends to be detached from the desired organization effects, and (c) in most cases, evaluation practices only cover measures of satisfaction with the course, learning and individual performance post-training. To that list, Mourão et al. (2006) add the fact that not all TD&E events aim to reach results at more comprehensive analytic levels.

Therefore, the assessment of training programs' efficacy is an urgent concern in TD&E. According to Pilati (2006), as from the 1990's, different integrated instructional program efficacy assessment models have been developed and tested, with sufficient results to describe a general evaluation model, adaptable to any organizational reality. In view of the appointed limitations, however, the question remains how they can be adapted to a training context that is commonly marked by the systematic absence of training needs assessments and effective instructional planning and execution processes.

But the adaptation of a general evaluation model to specific training contexts is only effect to be overcome by dealing with its original cause, which is the scarce production on organizational analyses in needs assessment, whose main function is to articulate instructional objectives with individual, group and organizational performances (Ferreira, Abbad, Pagotto, & Meneses, 2009). This stagnation, also emphasized in Pilati (2006), probably derives exactly from the intended study problem in more comprehensive TD&E outcome assessments - organizational effectiveness.

According to contemporary research currents (e.g., Cameron, 1981; Connolly, Conlon, & Deutsch, 1980; Selden & Sowa, 2005; Zammuto, 1984), in which efficacy varies in function of dominant coalitions and their key interests in organizational activities, one of the main difficulties in research that works with this construct is exactly to characterize the phenomenon. Identifying these coalitions and their main interests does not seem to be an easy task, to the extent that other bottlenecks emerge, even when it is properly executed. Among these, the need for convergence among these coalitions' interests is highlighted, and mainly among their beliefs as to how a given TD&E program aligns with specific organizational performance aspects.

Therefore, in line with Meneses (2007), literature on social program evaluation was consulted, which has been confronted for a long time with the difficulties faced today in training evaluation. More specifically, the tool called logic model was used for support, whose central aim, according to McLaughlin and Jordan (2004), is to clarify causal relations among the main elements that, in this case, are part of a TD&E program - (human, financial, material, among other) resources, activities (instructional and operational programming and training execution), products (satisfaction levels with the course and learning), short (individual performance), medium (group and team performance) and long-term results (organization performance) and interfering variables (factors, whether previewed or not, that influence training-associated results). These relations, neglected in training needs assessment processes, are part of efficacy evaluations at more comprehensive analytic levels.

To clarify and approximate different influential coalitions' perceptions on how a certain training program affects an organization's performance is hence the main contribution of the tool used in the development of contextualized TD&E efficacy evaluation models. Despite their complex composition in hardly systematic training contexts, the logic models represents of what is called the evaluability of training effectiveness assessment.

In that sense, before advancing on the description of the research methods, procedures and techniques used here, it is important to remember the main study aim, which is to identify the evaluability of assessing the impact of a given training on the performance of a public entity. In addition, the following specific aims were proposed: (a) identify a training action that is potentially capable of influencing organizational performance, (b) select indicators to assess training efficacy, (c) develop a training efficacy evaluation model, and (d) identify evidences of training efficacy.

Method

Participants and Place

This research was developed at a public electric energy service concessionaire in the Federal District, whose aim is to accomplish studies, projects, construct and operate electric energy production plants and transmission and distribution systems, as well as commercial acts deriving from these activities. Developed based on a qualitative opinion survey, this study was subdivided in four complementary phases, each of which was executed through the application of a specific set of techniques and procedures to research a given specific objective: (a) interviews to identify training that can affect organizational performance (phase 1), (b) interview to select organizational performance indicators sensitive to the operational results the selected program is supposed to affect (phase 2), (c) interviews and documentary research to elaborate the Logic Model of the training program (phase 3), and (d) focus group to validate the Logic Model (phase 4).

Eleven persons participated in the first phase, responsible for monitoring the organization's performance indicators related to the institution's TD&E programs: one director, one superintendent, three division managers and six process leaders. Participants are male regular staff members, whose ages range between 44 and 58 years, and mostly engineers with between 17 and 27 years of time on the job. In the second phase, as only one employee was available to provide the requested information, a single interview took place with a male 49-year-old professional with 21 years of professional accounting experience, who is responsible for monitoring and analyzing the company's financial indicators. In the phase focused on the elaboration of the logic model (phase 3), seven employees from the organization's human resource unit participated: one superintendent, one manager, two course coordinators, one training evaluation coordinator and two trainees responsible for feeding the course evaluation database. The participants' profile is predominantly female, age range between 25 and 59 years and between eight and 27 years of time on the job. In the final phase, nine direct managers of the trained employees participated: eight male; all regular staff members; between 41 and 51 years of age; industrial technicians and four engineers. Hence, 28 people, including the managers whose profile was just described, participated in the final validation phase of the Model.

Instruments

As appointed earlier, the study design comprised four phases, each of which based on the application of one or more data collection instruments: (a) structured interview script (phases 1, 2 and 3), and (b) focus group script (phase 4). The documentary research in phase 3 did not follow the previously established script; in this case, data were collected from the documents included in the study, merely aiming to identify some elements needed to compose the Logic Model, but not expressed in the preliminary interviews. Details on each instrument used are presented further ahead.

Interview script phase 1. To permit the identification of a training program that would be able to affect organizational performance, the script included the following items and questions: (a) organizational processes - "In view of the processes accomplished, indicate which process reflects the most important operational results in your company, as you perceive it"; (b) process indicators - "Identify the indicators that measure the processes appointed in response to the previous question"; (c) educational action: "Identify, in the list of education actions the company offers, which action enhances performance improvements in process indicators and the employees who accomplish these processes".

Interview script phase 2. To identify the organizational performance indicators that were most associated with the operational outcomes selected in the first phase, in this case, the interview was based on a single question: considering the educative action and operational indicators identified in the previous phase, which organizational performance indicators can be associated with the action? Hence, the idea was to confirm the indicators identified in the previous phase, but based on an inverse, bottom-up research logic, which departed from the selected training instead of organizational performance.

Interview script phase 3. To elaborate the preliminary version of the previously selected Logic organizational training Model, the instrument aimed to identify coordinators, instructors, facilitators and evaluators of the educative action; select reports and other documents issued in the planning, execution and evaluation phases of the selected program; and identify conditions that could affect the training outcomes. Again, it is highlighted that the documentary research, accomplished in this phase as well, did not follow the previously elaborated instrument.

Interview script phase 4. To validate the established Logic Model, the script included the following items and questions: (a) indicators - "Do the efficacy indicators adjust to the intended training targets and aims?", "Is there any indicator the training affected?"; (b) resources - "Were the financial, material and human resources made available for the course appropriate to the needs?"; (c) planning - "Were the educational objectives adequate and are they aligned with the proposed results?"; (d) evaluation - "Did the instructional strategies help to reach the instructional objectives?", "Did the learning assessment influence the contents gained during the course?"; (e) processes - "Did the assessment of students' satisfaction with the course capture their perception about the conditions in which the training took place (instructors and program)?"; (f) impacts - "Was the individual performance assessment post-training able to translate the program's effects on the work and support conditions to apply new knowledge?", "Are the course effects influencing other processes, besides the transmission (pre-operation, real time and post-operation)?", (g) proposal of goals - "Are other outcomes expected as a consequence of the course's effects?".

Procedure

Data collection. To accomplish phase 1, the interview script was applied to the group of 11 persons involved in monitoring the institution's performance indicators. The interview took approximately two hours and was conducted in group in function of the need for swift data collection. Individual interviews could repeatedly have culminated in the identification of training actions with little chances of affecting organizational performance, and therefore improper for the research aims. Therefore, the identification of organizational performance indicators preceded the location of a training program with the ability to affect it.

In the second and third phases, the interviews were held individually with the employee responsible for monitoring and analyzing the company's financial indicators and with the seven employees from the institution's human resource unit, respectively. In both cases, the interviews took an average 30 minutes and the researchers properly registered the verbal reports, which also happened in the first and final study phases. Concerning the documentary research, it should be appointed that, based on phase one and two participants' indications, pertinent information and data were obtained from documents that were physically and electronically available at the research institution: Management Report, Strategic Planning, Development and Education Plan, Personnel Management Report, Training Evaluation System Report and Operational Performance Report.

Finally, phase 4, the validation of the established Logic Model, was put in practice through a focus group, which involved nine direct managers of training participants. Differently from the above, this phase took three hours and 30 minutes and aimed to collect consensus views on the questions in the script presented above. Then, the participants were confronted with the questions, who were supposed to discuss them and present the researchers with arguments to reflect the view of the entire group, and not of one or another participant.

Data analysis. Although the qualitative approach guided this study, the analytic treatment of the answers obtained from the application of the interview scripts, documentary research and focus groups is compatible with the way the component questions were structured. Hence, in view of the limited opening in the questions participants were confronted with, as the scripts were structured, no appropriate analytic techniques were applied to understand more open verbal reports, like content or discourse analysis for example. In all analytic phases, the researchers merely attempted to identify, in the participant groups' reports, the necessary elements to compose and validate the previously selected Logic Model of the organizational training program.

Ethical Considerations

The study complied with the ethical principles for research involving human beings. Although no organizational psychology research efforts were involved, whose research routines required evaluation by appropriately established ethics committees, in this study, all participants freely and formally agreed to submit to the application of the described instruments and to allow the researchers to register and disseminate their verbal reports, provided that any attempt to identify the participants would be impossible.

Results and Discussion

In the first phase, aiming to identify an organizational training action that could affect organizational performance, in function of the first question they were confronted with, the 11 participants considered the Systems Operation Process most relevant, which is necessary to make available equipment to continue maintaining the contracts signed with suppliers. After selecting this process, the participants identified the following indicators: Availability Index of Transmission Lines and Equipment - probability that, at a given time, the electric energy transmission lines or equipment are operating satisfactorily, or are ready to be operated if necessary; and the Individual Performance Index - composed based on employees and managers' perceptions about personnel management and workers' holistic health, the quality of work processes and the conditions to put this work in practice.

The highlights given to these processes and indicators converges with the organization's activities, whose efforts, concentrated on social accountability actions, are operationally directed at process optimization, cost reduction and enhanced client and collaborator satisfaction. These efforts, in turn, directly result from the main organizational stakeholders' interests, in four large coalitions - society, stockholders, clients and persons.

In this study, at the end of the first phase, the sole focus on the clients' perspective was evidenced, whose main aim, according to information taken from documents analyzed in subsequent phases, is to guarantee the quality and availability of electric energy to clients, to guarantee the feasibility of enterprises, partnerships, authorizations and concessions in the electric energy market and to consolidate the company's participation in telecommunication business.

After identifying the most relevant processes for the organization's performance and the indicators associated with these processes, a training action had to be selected with the ability to affect them. Based on the institution's Educational Master Plan, the participants considered that the System Operator Recycling (ROSI) course was appropriate to the research goals. This professional recycling program, whose clients are workers classified in the company's Career Plan as System Operators, relates operational aspects, technical standards, current network procedures, regulatory and supervisory entities' rules and the supply policy the company has defined.

The group of instructors, employees from the same organization, was defined based on the specific condition of the transmission process, knowledge on the region the company attends and the knowledge transfer needs of workers with a profound view on activities in the electric energy transmission process. The method used in ROSI included presentations, guided study, individual and group work and group dynamics. Fully in-class, the total hour load was 341 hours.

Aware of the selected training action, in the second research phase, the organizational performance indicators were identified that were better articulated with the recycling course's instructional objectives. In the interviewee's opinion, these indicators were present in the Transmission System Performance Index, in the Client Satisfaction Index and in the Organizational Climate Satisfaction Index. In Table 1, the link among some ROSI objectives, outcome indicators and organizational performance dimensions identified in phases 1 and 2 is clarified.

With this information at hand, the third phase involved the elaboration of a graphic representation, called Logic Model, of the ROSI Program Theory. Program theory is considered as the supposition and description of a range of associative or causal relationships between variables or components in an in this case educative program. Based on documentary analyses and interviews with seven employees, the research aimed to collect information on the main components in the Logic Model, adapted to TD&E situations in accordance with Meneses (2007).

Table 2 presents the characteristics of each of these components, as a result of the analyses used in phase 3, as well as the hypothetic causal relations. In view of the linear configuration of these relations, a top-down Logic Model was chosen, in line with McLaughlin and Jordan (2004). In this modality, the primary variables are placed at the top of the representation and the final variables at the bottom.

After graphically representing the ROSI training Program theory, validation was needed. In the focus group, some manifestations culminated in the incorporation of new elements in the presented graphic representation, but only related to medium-term results, which incorporated expectations on reduced system restoration and maneuvering times for line and equipment maintenance. As a result, the course's Program Theory was considered validated.

Thus, the training program was prepared with a view to more comprehensive analyses that went beyond the traditional individual focus, as operated in the TD&E area. In other words, the evaluability of the ROSI efficacy evaluation was ready for testing. Similarly to the acceptance of the term evaluability in Social Program Evaluation literature, in TD&E situations, evaluability can be understood as the degree of proximity between the rhetorical and real model of a given training program. Rhetorical model is considered a set of suppositions on how the program will affect the needs that justified it. The real model, in turn, indicates what is actually done to satisfy those needs.

In TD&E situations, evaluability gains fundamental importance as, generally, systematic training needs assessments do not precede the implementation of training programs. In fact, ever since the area has gained scientific status, with the publication by Campbell (1971), until the most recent international review (Aguinis & Kraiger, 2009) published in the Annual Review of Psychology, specialized literature has advanced little with regard to needs assessment. Orientation is lacking to articulate needs idealized at the organizational analysis level - which then guide training efficacy evaluations - with the needs observed at the group and individual levels. Until data, the model by McGehee and Thayer (1961) is one of the most important needs assessment guides.

This baseline model orients needs assessment based on three questions: where does training have to be implanted (organizational analyses), what should be trained (task analysis) and who should be trained (individual analyses)? Although they better direct the process under discussion, these three questions do not permit articulating a given training program with an organization's performance. In the best of hypotheses, if the first question is answered, it becomes possible to link the training action with a sector, area or department's performance, among others.

It should be highlighted, however, that McGehee and Thayer's model was formulated before the period when General Systems Theory starts to be applied (Mattessich, 1978) in organizational research. Thus, at the time of its proposition, the need for organizations to have business strategies that would allow them to rationalize external aspects and compete in turbulent scenarios was not discussed that incisively yet. It was enough for people to be trained on stable and predictable routines to maintain organizational performance at acceptable levels.

In this context, discussing the need to align TD&E programs with performance and organizational efficacy did not make sense, a discourse that is strongly emphasized in the training area today, but without following scientific and professional practices. Consequently, needs assessment initiatives at more comprehensive analysis levels are rare. Until today, the individual continues as the main yardstick in decision making on training needs, as if individual training outcomes like satisfaction, learning and performance would invariably culminate in group and even organizational performance improvements. This relates to the belief, already highlighted in Alliger and Janak (1989) and prominent until today, that results observed at the individual level, independently of the performance type the trainingaction focuses on and the context, contribute to produce changes at higher levels.

In this scenario, there is an urgent need to reduce the discrepancy between rhetorical and real models on how a given training program affects the results organizations establish. In this study, the use of Logic Models promoted this reduction. This remedial alternativewas applied in the evaluation phase of the ROSI training's effects. Anyway, it was useful to collect information on the pertinence of training evaluation at more comprehensive analysis levels, discussed here.

As informed, the aim of the four research phases described earlier was to produce sufficient information to understand the mechanisms through which the ROSI program affected the organization's performance. Before that, essentially the first three phases provided sufficient data to identify the elements the program covered and which were necessary to elaborate the Logic Model.

As program evaluation experts discuss (Fitzpatrick, Sanders & Worthen, 2004; McLaughlin & Jordan, 2004; Posavac & Carey, 2003), combining and closing agreements on program information, generally distributed among different actors, each of whom has a distinct impression about the reality faced, represents an initial condition for assessments. Without reliable data, no advances are possible. With regard to the ROSI program, which was not preceded by a needs assessment to indicate the organizational results it needed to influence, this information was collected based on interviews, documentary research and a focus group.

These data collection techniques were useful in the order they were applied in. Even if training needs assessments do not actually take place, if the training is strategic to the organization, data on the program's supposed articulation with organizational performance will probably exist. As for ROSI, these data were available. That was so because the organization has a corporate education center, whose aim is exactly to align professional qualification actions with performance needs.

Although available, the data hardly reflected the connections between individual and organizational training needs or, in the evaluation logic, between individual results and training efficacy. At most, it was known that, according to the dimensions the company's corporate education center emphasized, that the training program fit into the client's perspective and that, in this perspective, it was more closely related with operational process performance. There was no information whatsoever, however, on the relation among these more comprehensive aspects as learning and individual post-training performance indicators. At that moment, the interviews and focus group were very valuable.

That is another point to be highlighted. According to Freitas and Borges-Andrade (2004), training efficacy evaluation becomes very comprehensive as it requires measures of changes in work processes, productivity, organizational climate and culture, among other possibilities. This requirement reinforces the idea that, through the use of more participatory methods and analytic techniques, training professionals share the responsibility for planning and executing the assessment with stakeholders who are more familiar with the measures and indicators for future use. In this research, interviews and a focus group were held with experts on the organization's performance dimensions that are more related to the ROSI program. Sharing responsibilities is needed, not only in the training efficacy evaluation phase, but mainly during needs assessment.

After collecting this information, the need emerged to locate the indicators that would be of use to the assessment that was being prepared. In line with Fitzpatrick et al. (2004), the availability of data on the outlined outcome indicators is another fundamental point when determining the evaluability of a program assessment. When this kind of information is absent, evaluations cannot be put in practice. Partial availability, then, limits alternative research designs that are more robust to threats to the research's internal and external validity.

Concerning ROSI, the organization was systematically monitoring these data, even before putting in practice the educational solution. Once again, what explains this is the fact that the training program analyzed integrates an extremely relevant organizational performance perspective with a view to the maintenance and expansion of its position in the market. Cases like the one reported in Meneses (2007), in which data on the selected indicators are unavailable, can represent considerable bottlenecks to the evaluability of training efficacy.

Training evaluation studies are relevant not only because they permit observing these kinds of outcomes, but mainly because they permit attributing the perceived outcomes to the training program. As some professionals and experts claim, the investment logic in TD&E actions demands concrete demonstrations that training programs truly affect organizational performance. And that demands the use of quasi-experimental designs that isolate a range of variables, capable of explaining, even without training programs, the desired organizational results.

The use of this kind of research designs marked the consolidation of the program evaluation area in the mid 1950's, as a scientific research area, and now also plays a central role in TD&E research. According to Shadish, Cook and Campbell (2002), different quasi-experimental designs can be used, some more fragile and others more solid to interferences from variables that, although known, are external to the research context. What determines the choice of one or the other design, among so many aspects, is data availability.

As the authors highlight, studies that more strongly resist the influence of interfering variables comprise different evaluation parameters, such as pretests and different groups for comparison. More fragile studies use a smaller number of parameters (like a single group and test for example), resulting in more doubtful conclusions on the relations among the study variables. In the present study, the data were systematically monitored, even before the training action was projected, so that further evaluation studies could very well use more robust research designs.

But when this is not the case, efforts to prepare the assessment system of the training program's impact on organizational performance are put at risk. That may be the main limitation of using the Logic Models in the evaluation phase of TD&E effects only. Elaborating them demands considerable effort and the pertinence of the evaluation is not always guaranteed. As James (1992) and Shelton and Aligger (1993) affirm, organizational training effectiveness evaluations are not always appropriate.

Again, the use of quasi-experimental designs in evaluation research is highlighted, in view of the need to securely attribute the results observed in organizational performance to the training. And exactly because these designs permit greater control of so many other variables, like policies, programs and parallel organization actions, which run for the same results the training action aims for.

In these case studied here, there were signs that actual evaluation studies would have to consider other variables as, according to the results of phase 3, although the participants assessed the training program well in terms of didactic performance, their own performance and the course program (4.37 < Mean < 4.67), post-training individual performance rates were relatively low (M = 2.93 for self-assessment and M = 2.95 for hetero-assessment as perceived by the managers).

In other words, it can be affirmed that the participants liked the course, but did not use it that intensely to perform their activities and tasks. If utilization rates were low, attributing organizational performance changes to the training seemed improbable. This does not mean that changes could not be verified, they could, but this does not determine that the organizational outcomes solely derived from the ROSI training program.

This fact remits to a final but very important aspect, as it permits conclusions about the evaluability of the ROSI program's assessment at the organizational outcomes levels - the behavior of the other Logic Model components, mainly regarding short-term products and outcomes, equivalent to the learning and individual post-training performance levels. As informed, the results at the latter level - it should be reminded that systematic data on learning evaluations were not available - remained below expectations, making it improper to assess the course's impact on the medium and long-term results included in the model.

At least from the professional viewpoint, efforts to keep up this evaluation were not justified as, if the results were verified, they could not be fully credited to the ROSI program. Anyway, the organization could actually use them to compose some remedial strategy to enhance individual post-training performance levels. From the scientific viewpoint, then, the scarce theoretical-methodological development in the area about the efficacy of TD&E evidences the relevance of continued research, mainly if a research model could be constructed that included organizational context variables already acknowledged in international literature as determinants of effects at the individual analysis level. It is supposed that these variables also determine more comprehensive results, making tests in that sense of great value to the area.

Final Considerations

The principal research aim was reached - to identify the evaluability of a training program assessment in terms of organizational results. As discussed by Meneses (2007), the Program Theory, put in practice through the elaboration of Logic Models, was of great value to understand the ROSI training program's potential to positively influence performance levels in the organization under analysis. Important information could be collected, revealing the low probability of holding the training responsible for any changes observed in organizational performance.

In practical terms, as a remedial solution in the training evaluation phase, the tool was useful for the organization to propose a context that is more favorable to post-training performance, no matter the intended analysis level. For scientific ends, the use of the tools permits recovering productivity rates in the field of TD&E efficacy assessment, which became stagnated when higher analysis levels are considered. Mainly because the tool requires the sharing of data and impressions that, at these higher levels, are beyond the reach of the professionals traditionally responsible for evaluation research.

It is only recently that training science became more systematically concerned with the planning and delivery of TD&E actions to work groups and teams, so that tools that permit a better understanding of these levels have a lot to contribute to the area. That is the case of the Logic Models which, within a decision sharing logic, permit better manipulation of variables at these analysis levels without training professionals having to master related theories and concepts.

The latter are only expected to be able to articulate between the rhetoric and real models the different organizational stakeholders interested in the training program conceived. Although the aim was reached, this research comes with one important limitation. As discussed earlier, learning data were not considered in the composition of the Logic Model because of the ROSI program. Hence, the rupture in the Model links may have occurred in the component called Short-term Result, while it actually could have occurred at a higher analysis level - Results. Another weakness is the mere speculation, without first obtaining reliable data on the issue, on variables interfering in the causality relations that were supposed in the Model prepared for the ROSI program.

When verifying discrepancies between the ideal and real model of the evaluated program, studies using the tool presented here should incorporate questions on individual, group, organizational and socio-environmental variables that are capable of affecting the supposed causal relations. In this study, questions were only raised about other outcomes deriving from the course's effects, but not about intervening variables. Anyway, an important remedial tool is provided for non-systematic training needs assessment processes, which, without publications on the theme, and at least from the professional viewpoint, may even turn into a new qualitative organizational analysis method.

References

Freitas, I. A. (2005). Impacto de treinamento nos desempenhos do indivíduo e do grupo de trabalho: Suas relações com crenças sobre o sistema de treinamento e suporte à aprendizagem contínua. Tese de doutorado não publicada, Universidade de Brasília, Brasília, DF.

Meneses, P. P. M. (2007). Avaliação de um curso de desenvolvimento regional sustentável no nível de resultados: A contribuição dos modelos lógicos e do método quase-experimental. Tese de doutorado não publicada, Universidade de Brasília, Brasília, DF.

Mourão, L. (2004). Avaliação de programas públicos de treinamento: Um estudo sobre o impacto no trabalho e na geração de emprego. Tese de doutorado não publicada, Universidade de Brasília, Brasília, DF.

Received: 10/07/2010

1st revision: 04/03/2011

Final acceptance: 10/12/2011

Sônia Maria Souza Damasceno, M.Sc. in Business Administration, Graduate Program in Business Administration, Universidade de Brasília, campus Darcy Ribeiro.

Gardênia Abbad, Adjunct Professor, Institute of Psychology, Universidade de Brasília, campus Darcy Ribeiro.

Pedro Paulo Murce Meneses, Adjunct Professor, School of Business Administration, Accountancy and Economics, Universidade de Brasília, campus Darcy Ribeiro.

• Abbad, G. S., & Borges-Andrade, J. E. (2004). Aprendizagem humana em organizações de trabalho. In J. C. Zanelli, J. E. Borges-Andrade, & A. V. B. Bastos (Orgs.), Psicologia, organizações e trabalho no Brasil (pp. 237-275). Porto Alegre: Artmed.
• Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel Psychology, 42(2), 331-342. doi: 10.1111/j.1744-6570.1989.tb00661.x
• Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals teams, organizations e society. Annual Review of Psychology, 60, 451-474. doi: 10.1146/annurev.psych.60.110707.163505
• Borges-Andrade, J. E., Pereira, M. H. G., Puentes-Palácios, K. E., & Morandini, D. C. (2002). Impactos individual e organizacional de treinamento: Uma análise com base num modelo de avaliação institucional e na teoria multinível. Revista Psicologia, Organizações e Trabalho, 2(1), 117-146.
• Cameron, K. S. (1981). Domains of organizational effectiveness in colleges and universities. Academy of Management Journal, 24(1), 25-47. doi: 10.2307/255822
• Campbell, J. P. (1971). Personnel training and development. Annual Review of Psychology, 22, 565-602. doi: 10.1146/annurev.ps.22.020171.003025
• Connolly, T., Conlon, E. J., & Deutsch, S. J. (1980). Organizational effectiveness: A multiple-constituency approach. The Academy of Management Review, 5(1), 211-217.
• Eboli, M. P. (2004). Educação corporativa no Brasil: Mitos e verdades. São Paulo: Gente.
• Ferreira, R. R., Abbad, G. S., Pagotto, C. P., & Meneses, P. P. M. (2009). Avaliação de necessidades organizacionais de treinamento: O caso de uma empresa latino-americana de administração aeroportuária. REAd - Revista Eletrônica de Administração, 15(2), 1-26.
• Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston, MA: Pearson Education.
• Freitas, I. A., & Borges-Andrade, J. E. (2004). Efeitos de treinamentos nos desempenhos individual e organizacional. Revista de Administração de Empresas, 44(3), 44-56. doi: 10.1590/S0034-75902004000300005
• Freitas, I. A., Borges-Andrade, J. E., Abbad, G. S., & Pilati, R. (2006). Medidas de impacto de TD&E no trabalho e nas organizações. In J. E. Borges-Andrade, G. S. Abbad, & L. Mourão (Orgs.), Treinamento, desenvolvimento e educação em organizações e trabalho: Fundamentos para a gestão de pessoas (pp. 489-504). Porto Alegre: Artmed.
• James, H. (1992). Simplifying ROI. Training, 29(9), 53-58.
• Mattessich, R. (1978). Accounting and analytical methods: Measurement and projection of income and wealth in the micro and macro economy. Homewood, IL: R. D. Irwin.
• McGehee, W., & Thayer, P. W. (1961). Training in business and industry. New York: John Wiley & Sons.
• McLaughlin, J. A., & Jordan, G. B. (2004). Using logic models. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 7-32). San Francisco, CA: Jossey-Bass.
• Mourão, L., Borges-Andrade, J. E., & Salles, T. J. (2006). Medidas de valor final e retorno de investimento em avaliação de TD&E. In J. E. Borges-Andrade, G. S. Abbad, & L. Mourão (Orgs.), Treinamento, desenvolvimento e educação em organizações e trabalho: Fundamentos para a gestão de pessoas (pp. 505-513). Porto Alegre: Artmed.
• Pilati, R. (2006). História e importância de TD&E. In J. E. Borges-Andrade, G. S. Abbad, & L. Mourão (Orgs.), Treinamento, desenvolvimento e educação em organizações e trabalho: Fundamentos para a gestão de pessoas (pp. 159-176). Porto Alegre: Artmed.
• Posavac, E. J., & Carey, R. G. (2003). Program evaluation: Methods and case studies (6th ed.). Boston, MA: Pearson Education.
• Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52, 471-499. doi: 10.1146/annurev.psych.52.1.471
• Selden, S. C., & Sowa, J. E. (2005). Testing a multi-dimensional model of organizational performance: Prospects and problems. Journal of Public Administration Research and Theory, 14(3), 395-416. doi: 10.1093/jopart/muh025
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
• Shelton, S., & Aligger, G. (1993). Who's afraid of level 4 evaluation? Training & Development, 47(6), 43-48.
• Zammuto, R. (1984). A comparison of multiple constituency models of organizational effectiveness. The Academy of Management Review, 9(4), 606-616. doi: 10.2307/258484

Publication Dates

• Publication in this collection
  10 Dec 2012
• Date of issue
  Aug 2012

History

• Received
  07 Oct 2010
• Accepted
  12 Oct 2011