Evaluation of a Training Program for Credit Union Counselors

Avaliação de um Programa de Treinamento para Gestores de Cooperativas de Crédito

Evaluación del Programa de Capacitación para Consejeros de Cooperativas de Crédito

Abstract

Training, Development, and Education assessments have intensified in recent decades, and empirical evidence proves their effectiveness in organizational outcomes. However, the literature shows little research on the results of training in cooperative organizations. This study aimed to evaluate a training program for credit union counselors based on Training, Development, and Education evaluation models associated with logical models. This is an empirical study with a qualitative design. Data collection was performed through an analysis of 229 documents, three semi-structured individual interviews, and a focus group. The results of the research corroborate findings related to the usefulness of the integrated use of systemic evaluation approaches and indicate that these models apply to the context of cooperative training. This study fills a methodological gap in the field of instructional psychology in third sector organizations.

Keywords:
educational program evaluation; transfer; third sector; learning

Resumo

As avaliações de Treinamento, Desenvolvimento e Educação têm se intensificado nas últimas décadas, e evidências empíricas comprovam a eficácia delas nos resultados organizacionais. No entanto, a literatura evidencia poucas pesquisas sobre os resultados de treinamentos em organizações cooperativas. Este estudo teve por objetivo avaliar um programa de treinamento para conselheiros de cooperativas de crédito a partir de modelos de avaliação em Treinamento, Desenvolvimento e Educação associados a modelos lógicos. Trata-se de um estudo empírico com delineamento qualitativo. A coleta de dados foi realizada por meio de análise de 229 documentos, três entrevistas individuais semiestruturadas e grupo focal. Os resultados da pesquisa corroboram achados relacionados à utilidade do uso integrado de abordagens sistêmicas de avaliação e indicam que esses modelos são aplicáveis ao contexto de treinamentos de cooperativas. Uma das contribuições deste estudo foi suprir uma lacuna metodológica no campo da psicologia instrucional em organizações do terceiro setor.

Palavras-chave:
avaliação de programa educacional; transferência; terceiro setor; aprendizagem

Resumen

Durante las últimas décadas, las evaluaciones de capacitación, desarrollo y educación se han intensificado, y la evidencia empírica demuestra su eficacia en los resultados de la organización. Sin embargo, la literatura muestra poca investigación sobre los resultados de la capacitación en cooperativas. Este estudio tuvo como objetivo evaluar un programa de capacitación para consejeros de cooperativas de crédito con base en modelos de evaluación asociados a modelos lógicos. Se trata de un estudio empírico cualitativo. La recopilación de datos se realizó mediante el análisis de 229 documentos, tres entrevistas semiestructuradas y un grupo focal. Los resultados de la investigación corroboran los hallazgos relacionados con la utilidad del uso integrado de enfoques de evaluación sistémica e indican que estos modelos son aplicables al contexto de cooperativa. Una de las contribuciones de este estudio ha sido la de llenar la laguna metodológica en el campo de la psicología instructiva en organizaciones del tercer sector.

Palabras clave:
evaluación de programa educacional; transferencia; tercer sector; aprendizaje

The deep and constant changes and innovations that characterize the world of work today have required an increasingly qualified workforce to meet the challenges related to the employability and sustainability of work organizations. To meet this qualification demand, programs that provide continuing and lifelong education are necessary. Considering this background, the Training, Development, and Education System (TD&E) has adopted a strategic role in organizations that intensified their financial and human investments for the training of their professionals, with results that can be proven at different levels of analysis. This explains the growing investment in TD&E actions (Ford, Baldwin, & Prasad, 2018).

When we analyzed the national and international literature, we observed that although several empirical studies prove the relevance of training for workers, organizations, and society, we found few empirical studies on learning, training, training assessment and its influence on the third sector organizations (Cassiano & Borges-Andrade, 2017; Nascimento, Borges-Andrade, & Porto, 2016).

In the literature review conducted by Nascimento et al. (2016), among the 629 studies found, only 31 (5%) were conducted in third sector organizations, demonstrating how little this sector of the economy is represented in national surveys on learning and training. The lack of research that identifies learning demands and evaluates training programs for those working in the third sector results in a poor offer of TD&E actions to continuously prepare and train managers in the cooperative governance field, affecting the consolidation and continuity of these enterprises in the labor market. However, despite the few studies on evaluation in third sector organizations, Mourão and Borges-Andrade (2005, 2013) have stood out in the last decade for evaluating a public training program in the behavior in office, organizational change, and final value levels in an associative organization.

When analyzing the theoretical framework of instructional psychology, we observed that, over the years, several traditional, summative, and integrated models of TD&E assessment were developed to facilitate the process of evaluating training programs in organizations, as well as methodological tools to support this process, such as logical models. Traditional evaluation models are based on results, among them, the Kirkpatrick model (1976), with four levels of assessment (reaction to training, learning, behavior in the position, and results), is the most used. The summative and integrated evaluation models in TD&E are based on the approach of instructional systems and were developed to evaluate not only the results but all variables that could interfere and predict the expected training effects (Borges-Andrade, 2002; Richter, 2018).

Among the integrated models, the Modelo Integrado de Avaliação do Impacto do Treinamento no Trabalho (IMPACT) proposes to investigate the relationship among variables related to the individual, training, organizational context, immediate training results (reaction and learning), and the criterion-variable impact of training at work, consisting of seven assessment components: (A) Perception of Organizational support; (B) Training Characteristics; (C) Characteristics of the Clients; (D) Reaction; (E) Learning; (F) Training influence at work; and (G) Transfer support (Richter, 2018; Seidl, Leandro-France, & Myrtle, 2018).

In addition to the traditional and integrated models, the logical models have been used as a methodological assessment tool, which allows verifying if the training program was designed and structured in line with the results expected by the organization (Damasceno, Abbad, & Meneses, 2012; McLaughlin & Jordan, 2010; Souza, Abbad, & Gondim,2017). The logical model tool is based on the system approach and enables the construction of the program theory according to the perception of the stakeholders, who elucidate the hypotheses of the causal relationship between the program’s activities and their results, as well as the identification of alternative explanations to the results obtained (Damasceno et al., 2012; Souza et al., 2017; McLaughlin & Jordan, 2010). The logical models enable the collection and organization of a wide set of descriptive and prescriptive information about a program regarding its conception, implementation, and the requirements to achieve the desired effects, considering the influence of externalities (Damasceno et al., 2012; Souza et al., 2017; McLaughlin & Jordan, 2010).

The logical models associated with the summative and integrated evaluation models have been successfully adopted in several TD&E assessment studies in Brazil, and their association with a theoretical model is necessary for the survey, organization, and interpretation of data and their relationships (Damasceno et al., 2012; Souza et al., 2017). Figure 1 summarizes the theoretical model used in this research, presenting the components of the Logical Model and its interfaces with the IMPACT Model and the TD&E System for the assessment of the Training Program for Credit Union Counselors.

Figure 1
Logical Model and its interfaces with the Impact Model and the TD&E System.

The TD&E System consists of applying evidence-based principles that guide the design of training and helps individuals learn and develop skills that influence work performance and organizational effectiveness, and it is divided into three subsystems: Training Needs Assessment (TNA) (I), planning and implementation (II), and assessment (III) (Borges-Andrade, 2002; Damasceno et al., 2012; Khalil & Elkhider, 2016; Kraiger & Ford, 2020). The conception of the program needs to be based on a systematic analysis of the learning demands of the individual, team, and/or organization (I), in a way that allows for the development of learning and teaching solutions (II) and evaluation strategies (III), considering the contextual variables that can interfere in the relationships between demand, teaching solutions, and results (Holloway, Arcus, & Orsborn, 2018; Khalil & Elkhider, 2016; Kraiger & Ford, 2020; Lacerenza, Reyes, Marlow, Joseph, & Salas, 2017)

The TNA (I) must contain the context variables (1) with the detailed description of the problem (1.1) that the organization must solve, the characteristics of the target audience (C), and the competencies gaps identified in the work that will give rise (1.2) to the training using taxonomies of learning (Ford et al., 2018; Holloway et al., 2018; Khalil & Elkhider, 2016). In this phase, the hypotheses/assumptions (1.3) about the possible causes that led to the problem are also investigated, as well as actions and strategies that will solve this problem.

The data obtained from TNA will provide support to the planning and implementation (II) of the educational solution, which should be very well aligned with the expected learning outcomes (Abbad, Zerbini, Carvalho, & Meneses, 2006; Ford et al., 2018; Khalil & Elkhider, 2016; Lacerenza et al., 2017). This second phase (II) requires the use of inputs (2) - human sources (2.1), financial resources (2.2), and materials (2.3) - for the creation and implementation of the training program. Human sources involve TNA, development and evaluation professionals, as well as stakeholders and the target audience, who must work together for the success of the program. Financial resources consist of the amount invested to hire instructors, developers, printing materials, etc. Material resources comprise the educational and pedagogical materials developed to achieve the results (Abbad et al., 2006; McLaughlin &Jordan, 2010).

The development of activities (3) should focus on solving the problems identified in the TNA, focusing on the quality of instructional planning (3.1), which should meet the specificities of the target audience and the characteristics of the training (C); these, in turn, should be based on theories of instructional design regarding nature, the level of complexity, the modality, the training method, the feedback, the type of participation, the spacing, among others (Abbad et al., 2006; Khalil & Elkhider, 2016; Kraiger & Ford, 2020; Lacerenza et al., 2017).

In the instructional design, the instructional objectives should be described in terms of observable performances (Abbad et al., 2006; Bloom, Engelhart, Furst, Hill, & Krathwohl,1956); the mode of delivery (face-to-face, semi-face-to-face, and distance) must be aligned with (C) characteristics of the clients (demographic, functional, professional, and physiognomic) and the degree of complexity of the objectives (Abbad et al., 2006; Khalil & Elkhider, 2016; Kraiger & Ford, 2020). In turn, the sequence of objectives and content should follow a categorization according to the systems for classifying learning outcomes, and it is important to adjust strategies and procedures to their respective fields of learning and to the context in which the instruction will take place in order to facilitate learning acquisition, retention and sharing processes (Abbad et al., 2006; Khalil & Elkhider, 2016). Finally, the criteria and instruments for evaluate reactions (D) and learning (E) should be directly related to the nature and complexity of instructional objectives (Abbad et al., 2006; Martins & Zerbini, 2015; Khalil & Elkhider, 2016).

Since the conception of the training, it is necessary to know who the target audience is and which/how many people the organization intends to train so as to classify which/how many results (4) will be generated by the instructional action. This result can be measured by the number of training sessions performed, the number of trained people, or another related indicator. Finally, the Assessment (III), the last phase of the TD&E System, evaluates the short-term, intermediate, and long-term effects exercised by the training program on beneficiaries, organizations, or society, depending on the achievement of the program objectives (Kraiger & Ford, 2020).

Short-term outcomes (5) include reaction (D) and learning (E) evaluation. The reaction evaluation should aim at identifying the attitudes and opinions of the students about the several aspects of training and/or their satisfaction. On the other hand, the learning evaluation should refer to the degree of assimilation and retention of content taught in the course (Borges-Andrade, 2002). The intermediate outcomes (6) consist of the effects of training on training performance and can be measured through the influence evaluation of the training at work (F) (Richter, 2018; Seidl et al., 2018).

Long-term outcomes of a TD&E program (7) generally consist of the influence and benefits generated by the program at the individual, teams, organization, and/or society levels (7.1). The training effects assessment on organizational results is carried out by financial or social returns indicators (Mourão & Borges-Andrade, 2005,2013). The outcomes of the Assessment in TD&E (III) are adopted for scientific feedback of instructional systems, aiming to improve the subsystems and increase the effectiveness of the programs (Abbad et al., 2006; Borges-Andrade, 2002).

As can be seen in Figure 1, other variables contained in this type of model refer to the context, such as Perception of organizational support (A) and Support for training transfer (G). These variables, which can affect the relationships between the program components and their results from conception to the training implementation, can be measured through the Training Transfer Support instrument (Martins, Zerbini, & Medina, 2018; Seidl et al., 2018). Transfer support is an important predictor of the influence of the training on the student’s work (Balsan, Lopes, Alves, Vizzotto, & Costa, 2016; Ford et al., 2018; Martins et al., 2018; Seidl et al., 2018).

This theoretical model also includes (8) external contextual factors, understood as variables external to the program, which can interfere in the relations between training and results and, for this reason, may constitute alternative explanations to the results expected by the program (Souza et al., 2017). In short, the adoption of system perspectives in the designing, development, and assessment of training enables more comprehensive and improved visualization of the phenomena studied, significantly affecting the effectiveness of training (Ford et al., 2018; Khalil & Elkhider, 2016).

In this context and based on this theoretical basis, the assessment of TD&E actions is a great contribution to the effectiveness of cooperative management, and this study aimed to evaluate a training program for credit union counselors based on assessment models in Training, Development, and Education associated with logical models. This study was conducted in an organization that offers training for professionals working in Brazilian credit unions based in the five geographic regions of Brazil, and its specific objectives are to describe the training program and the relationship between its components using logical models and to assess the instructional quality of the training program based on instructional design theories.

Method

This study is qualitative, more specifically a case study, and the choice of this methodological strategy was due to the need to deeply explore the variables that interfere in the effects of assessment in a training program. The design adopted is descriptive and exploratory due to the lack of production of systematic knowledge about leadership training in the third sector. Primary (human - instructors and managers) and secondary (documentary) sources of information (Creswell & Creswell, 2021) were used to evaluate face-to-face training, which targeted credit union counselors and a total workload of 96 class hours. The training was chosen for availability and because it has the potential to positively influence the behavior of these leaders in organizations.

Participants

Ten people participated in this study, seven (70%) women and three (30%) men. Regarding the capacity, four (40%) were representatives of the National Unity and six (60%) representatives from the following states: Federal District, Rio Grande do Sul, Mato Grosso, Bahia, Minas Gerais, and Pernambuco. Concerning the position, four (40%) were analysts, one (10%) manager, one (10%) director, two (20%) superintendents, one (10%) educational coordinator and one (10%) content developer.

Instruments

The instrument used to describe the training program was the Theoretical Model presented in Figure 1, which allowed the investigation of issues related to context, inputs, activities, products, and results. This theoretical model was elaborated based on the Logical Model (Damasceno et al., 2012; McLaughlin & Jordan, 2010; Souza et al., 2017) associated with theories of instructional design (Abbad et al., 2006; Borges-Andrade, 2002) and summative evaluation models, especially, the IMPACT Evaluation Model (Abbad et al., 2006; Richter, 2018).

We used an evaluation script of instructional materials developed based on instructional planning theories to evaluate the instructional quality (Abbad et al., 2006; Bloom et al., 1956; Lacerenza et al., 2017). It is composed of 48 items and subdivided into seven categories of analysis: (1) Instructional Objectives; (2) Instructional Strategies; (3) Planning activities; (4) Teaching sequence; (5) Content; (6) Exercises; and (7) Evaluation.

To conduct the interviews, a semi-structured questionnaire was elaborated focusing on the validation of the Logical Model, created after documentary analysis, and on the filling in of the gaps found. For this purpose, the instrument contained the following questions: (a) What problems or demands originated the Training Program?; (b) Could you make a history of the Program considering from its creation to the moment?; (c) What resources (financial, material and human) were invested for the development of the Program?; (d) What is the profile of the instructional instructors/designers and developers who participated in the process of creating/building the Program? How were they chosen?; (e) What were the teaching strategies adopted in the training? Why?; (f) What were the practical exercises done by the student during the training?; (g) What short-term, intermediate and long-term results did the “Institution Name” expect to achieve with the Program?; (h) What social and economic return does the “Institution Name” expect to achieve with the Program?; and (i) Which variables external to training can negatively and/or positively influence the results of the Program?

Procedures

Data collection. Data collection was performed in a professional training organization for cooperative organizations, in which the face-to-face course of Training Program for Credit Union Counselors with 96 hours-class was evaluated. Data from secondary (documentary and archival data) and primary sources (interviews and focus group) of information were used, selected for convenience. The organization was asked for documents related to the demands that gave rise to training, didactic-pedagogical materials, and reports. A total of 229 documents related to training were obtained, which were submitted to qualitative analyses for the elaboration of the evaluation model and the evaluation of the instructional quality of the training, namely: 5 lesson plans; 34 learning and reaction evaluation instruments, 170 evaluation questionnaires filled out by the participants, 19 reports of the training carried out and 1 manual with the Guidelines for Good Practices of Financial Cooperative Governance developed by the Central Bank of Brazil (BACEN). After the elaboration of the first version of the theoretical model, interviews were conducted with professionals linked to corporate education in order to eliminate the gaps in the information about the program. Then, a focus group was also conducted with state training managers to validate the descriptions of the components and expected results of the program.

Data analysis. The qualitative analysis of data, obtained through the analysis of documents, interviews, and focus groups, was performed based on categories extracted from the theoretical and methodological references of the TD&E field (TNA, instructional design, implementation, and evaluation).

Data analysis was performed in two stages. The first stage consisted of the documental analysis and the description of the training program with the support of the Theoretical Model presented in Figure 1, which allowed not only to describe the relationship between the components and the identification of causal hypotheses, expected effects, and variables that act as an alternative explanation for the training results but also to analyze it. Three interviews were conducted to understand the logic of the program and eliminate the gaps found in the theoretical model. Finally, the final version of the Logical Model was validated through a focus group.

After the logical model elaboration, each of the components and subcomponents of the model can be analyzed separately, without disassociating it from the context. Thus, figure 1 shows that the second stage addressed the evaluation of the instructional quality of training based on instructional planning theories (3.1). The data were organized and consolidated through the instructional quality evaluation script, in an Excel spreadsheet, and the convergent and divergent information found was localized by the authors and analyzed by expert judges of the Research Group formed by a Ph.D., Ph.D. students, master’s and undergraduate students at a public university. The assessment of the instructional quality of the training comprised an individual and independent analysis of these experts in TD&E, and disagreements were discussed in groups until consensus was reached among the evaluators.

The results achieved in the two stages were obtained from the synthesis, triangulation, and interpretation of the data obtained from different sources (human and documentary).

Ethical Considerations

This research was submitted and approved by the Research Ethics Committee of the Institute of Human and Social Sciences of the Universidade de Brasília, CAAE no. 75189317.7.0000.5540, Opinion No. 2,388,614.

Results

The Logical Model associated with the TD&E System and the IMPACT Model, elaborated through the triangulation of methods and results, allowed us to describe the training program for Training for Credit Union Counselors, to build the graphic organizer presented in Figure 2, to analyze the relationship between the context (origin and problem) and its components (inputs, activities, products, and results) and evaluate the subcomponent of the (3.1) Instructional Planning and its influence on the results of the program.

Figure 2
Logical Model of the Training Course of Credit Cooperative Counselors.

Figure 2 shows that the Training Program for Credit Union Counselors was (1.1) originated in 2010, on the demand of the stakeholders who envisioned the growth and solidity of credit unions, due to the following (1.2) problem: lack of preparation of administrative and fiscal counselors to take office, whose mandates have a maximum term of 4 years and 3 years, respectively and turnover.

Based on this (1) background, (2.1) human sources, (2.2) financial resources, and (2.3) material resources for the creation and development of a face-to-face training program were mobilized with the (1.3) assumptions that all credit unions counselors participate in the training program and that the contents would be able to train the counselors for the exercise of the position in a transparent manner, aiming at maintaining the cooperative identity, sustainability and the growth of this niche market in Brazil. For this purpose, three content developers were hired separately to build the face-to-face training program (3), structured in two stages: general, with 3 modules (behavioral, legal, and organizational); and specific, with 2 modules (legal and organizational), totaling a workload of 96 hours.

The mobilization of (2) inputs and instructional planning (3.1) aimed at obtaining the following outputs (4): train administrative and fiscal counselors of 976 credit unions in Brazil. However, the modules were individually elaborated by their respective content developer, without connection with the training demands, negatively influencing the development of content, which were not aligned with the results desired by stakeholders.

As a short-term outcomes (5), we expected that the participants would be satisfied with the training, reaction evaluation (D), and acquire the knowledge conveyed to the program, evaluation of pre- and post-training learning (E). We did not measure the influence of the training on the performance of the students in cooperatives (F), intermediate (7), and long-term (8) outcomes.

The reaction evaluations (D) were not applied in all states, and the data were not systematized, making the evaluation of the results impossible. Similar pre- and post-training tests were applied to verify learning (E). The lack of standardization of the evaluations made it impossible to compare the results due to the absence of valid pieces of evidence of the instruments, preventing a conclusive evaluation on the effectiveness of training at these evaluated levels.

Among the (8) external factors that may affect the verification of training outcomes, we identify: (a) legal issues, due to the several standards established by the BACEN, as well as their updates, which may make the content of the training outdated; (b) the turnover of counselors, who, after training, may fail to exercise the position, making impossible to apply the training at work; (c) the education level, which may impair the learning of the contents and its due application in the credit union; (d) the work experience in the role because counselors with a lot of experience have previous knowledge of the higher contents in relation to counselors with little experience; and (e) counselors who took other courses with similar content, making the verification of the training effects at work difficult.

The theoretical model adopted allowed us to identify that, despite the origin of the Training from external demand to meet a specific audience, a TNA that could support the content developers financially in the planning of the instructional action was not performed. This gap seems to have directly influenced the choice and allocation of (2) inputs, the design, the activities implementation (3), the outputs (4), and, consequently, the results (5). The understanding of the descriptive and prescriptive elements of the program enabled an in-depth analysis of the instructional quality based on material resources (2.3) and instructional planning (3.1).

The evaluation of the instructional quality of the Training Program carried out based on the instructional quality assessment script showed that the instructional objectives were not defined in terms of observable performances, as suggested by the instructional design approaches (Abbad et al., 2006; Bloom et al., 1956; Khalil & Elkhider, 2016). Moreover, discrepancies in the level of complexity between the content taught in the classroom and the activities that should be performed by the participants were verified, that is, contents addressed only at the level of knowledge and activities that required more complex skills related to problem-solving at the analysis or synthesis levels from the participant (Bloom et al., 1956) were identified.

Regarding the teaching modality, we did not identify which criteria were used for the choice of the face-to-face course, considering that no TNA was performed and no characteristics related to the profile of the clientele were found in the documental analysis. Concerning the planning of the activities, a didactic-pedagogical plan was provided for each module to be made available to the instructor. However, discrepancies of workloads were identified in relation to that proposed in the activities and the course, and some contents were not included in the didactic material.

The teaching sequence was not structured by level of complexity, and how the themes/contents are arranged in the modules, in some cases, makes them not related to each other. Since instructional objectives were not defined for each module, analyzing if the instructional strategies used were adequate to the nature of the objectives, as well as to the level of complexity of them, was not possible.

Regarding the content, the Portuguese language is used correctly in terms of spelling, concordance, punctuation, vocabulary adequacy, cohesion, and coherence. The bibliography used in almost all modules was made available, but in some modules, it is quite outdated, with references before the 2000s. No indication of alternative sources of information on the topics addressed was observed. The paragraphs are well written, with an idea per paragraph, but do not simulate dialogue between student and material. Regarding the exercises, analyzing if they are compatible with the nature and level of complexity of the instructional objectives was not possible to analyze. Case studies and drills do not always simulate the actual work situation, and identifying the occurrence or not of feedback is not possible.

In the reaction evaluations, the instructions and items of the instrument were not harmoniously related to the concordance scale. Readiness and learning assessments were standardized in terms of content, but no alignment was identified between the items and objectives of each module. Moreover, the application of the tests in all states before and after training did not occur, making the analysis of the short-term results impossible.

Discussion

The proposed theoretical model allowed us to organize and analyze the training program from its conception to the evaluation process, allowing the identification of the relationship between the components of the TD&E System, the Logical Model, and the IMPACT Model, as well as the gaps and proposals to improve the training for feedback of the TD&E System, as proposed by Damasceno et al. (2012) and Souza et al. (2017).

The results also corroborate the findings of Damasceno et al. (2012) and Souza et al. (2017) on the need to associate the logical model with an instructional theory so that evaluators can lead themselves in the process of data collection, organization and interpretation, at the same time that the logical model, as a methodological tool, allows not only to interconnect the components (context, inputs, activities, products, and results) with each other and to measure results, but also allows stakeholders to understand what relevant questions would need to have been asked at each stage of the process of designing, developing and assessing training, in order to identify opportunities for improvement of instructional activities and, thus, increase the positive influence of the course on the behavior of the students and the results of teams and credit unions, generating greater value for the organization.

TNA, the first step to be performed in the development of an instructional action, is crucial for the success of the training. It comprises a prior analysis of the organizational context, which, from the identification of learning demands, the origin of the problem or gap, allows us to choose an instructional design appropriate to the profile of the target audience and the results expected by the TD&E program. The lack of a systematic TNA, such as that occurred in this case study, negatively affected the other subsystems: planning and assessment, proving the findings by Ford et al. (2018), Holloway et al. (2018), and Khalil and Elkhider (2016).

The second stage, planning, and implementation comprises inputs, activities, and products. Regarding the inputs, we observed that the stakeholders did not work together, so there was no alignment between demand and the implemented activities, which interfered with the expected results (McLaughlin & Jordan, 2010). Connecting these points is important since stakeholders (target audience, content developers, instructional designers, and clients) have different roles, responsibilities, and opportunities to influence the results of the training (Abbad et al., 2006; Khalil & Elkhider, 2016; Kraiger & Ford, 2020; McLaughlin & Jordan, 2010). Moreover, the lack of diversification and communication between objectives, strategies, methods, materials, instructional resources, and expected results could have been solved with the use of theories, instructional design, and taxonomies of learning (Abbad et al., 2006; Bloom et al., 1956; Khalil & Elkhider, 2016).

Also at this stage, the evaluation of instructional design and teaching materials, with the support of the logical model as a graphic organizer, facilitated the chaining of information, allowing the identification of possible hypotheses of relationships between the components of the program (Damasceno et al., 2012; McLaughlin & Jordan, 2010; Souza et al., 2017). Therefore, we verified the importance of the instructional objectives being described in terms of observable performance, condition, and criterion, since these gaps make it impossible to choose the appropriate delivery modality, to hinder the planning of activities, the establishment of teaching sequences, the definition of instructional strategies, and the creation of evaluation measures (Abbad et al., 2006; Bloom et al., 1956; Khalil & Elkhider, 2016; Martins & Zerbini, 2015).

In the third stage, evaluation of the training program, in relation to short-term outcomes, focusing on the importance of the quality of the items of reaction assessments, considering that, for this evaluation to be an indicator of training effectiveness, the instructions need to be self-explanatory, the items must be harmoniously related to the scale and the application should not be attributed to the instructor, in order to minimize misunderstanding (Abbadet al., 2006; Martins & Zerbini, 2015. Given the gaps found in the instructional materials, we assume that the training will not produce effects expected by the organization in the performance of the students, since the learning objectives and exercises do not correspond to the levels of complexity of the functions of the counselors at work (Abbad et al., 2006).

The Logical Model, associated with the IMPACT Model and the TD&E System, corroborates the findings of Damasceno et al. (2012) and Souza et al. (2017) because it presents advantages associated with the use of systemic models in the evaluation process that allow: the understanding of the internal logic of the program and how the components - inputs, activities, outputs, and outcomes - relate to the context; the graphical visualization of the components, which enables the identification of causal relationships between components; the identification of contextual variables, which act as an alternative explanation of training outcomes; the identification of gaps found in the process of designing and developing the training program, facilitating improvement interventions; the establishment and identification of outcome indicators for evaluation; and the evaluation of training at more than one level of analysis.

The use of systemic and integrated evaluation models associated with logical models, in this study, facilitated the documental analysis of the program using theoretical references of instructional psychology and dialogue with stakeholders through multiple sources of information, favoring the evaluation process and allowing qualitative advances in the psychology field, since most studies focus on quantitative studies, which makes it impossible to further deepen the studied phenomenon (Damasceno et al., 2012; McLaughlin & Jordan, 2010; Souza et al., 2017). Moreover, this study fills a gap in research in cooperative organizations, a context little explored (Nascimento et al., 2016).

Among the theoretical-methodological contributions, the following stand out: the theoretical model, with a qualitative approach, to direct the process of conception, planning, implementation, and assessment, as well as for the interpretative synthesis of the results; and a proposal to assess the quality of the instructional design of the training through the instructional planning approach. Concerning practical contributions, the following stand out: the identification of weaknesses in the TNA, in the instructional design and the evaluations of reaction and learning, as well as the lack of influence assessment on the behavior of the students; the construction of the theory of the training program, with an indication of the main aspects that need improvement: instructional objectives; delivery modality and teaching strategies; and the evaluation of the pedagogical didactic material of the program with proposals for improvements for the improvement of the lesson plan, contents, teaching strategies and evaluations of reaction and learning so that the training influences the students and the results of credit unions.

The lack of psychometric quality of the evaluations of reactions and learning adopted by the organization, which prevented the pairing of information from the immediate results of the training and, consequently, hindered the analysis of the relationship between training and other levels of effect; the focus group elaboration and interview without the participation of the students of the training, which made it impossible to detect any differences of opinion about the program, especially about the reactions to learning and the other expected effects of the program are the limitations and challenges of this study. As a research agenda, we suggest the adoption of the logical models integrated with models of evaluation of training in research on the effectiveness of instructional systems and the interdependence relationships between subsystems; use instructional design theory to link descriptions of training needs to learning objectives and expected results after training; and investigation of the use of mixed methods (qualitative and quantitative) in the assessment of educational programs that allow investigating the relationship of the components and their prediction in the three levels of analysis: individual, group, and organization.

References

• Abbad, G. S., Zerbini, T., Carvalho, R. S., & Meneses, P. P. M. (2006). Planejamento instrucional em TD&E [Instructional desing in TD&E]. In J. E. Borges-Andrade, G. S. Abbad, L. Mourão, & Colaboradores, Treinamento, desenvolvimento e educação em organizações e trabalho: Fundamentos para a gestão de pessoas (pp. 289-321). Porto Alegre, RS: Artmed.
• Balsan, L. A. G., Lopes, L. F. D., Alves, J. N., Vizzotto, F. B., & Costa, V. M. F. (2016). Impacto do treinamento, comprometimento e entrincheiramento organizacionais em servidores de uma universidade pública [Impact of training, organizacional commitment, and entrenchment of public servants in a public university]. Revista Gestão Universitária na América Latina-GUAL, 9(1), 143-164. doi:10.5007/1983-4535.2016v9n1p143
  » https://doi.org/10.5007/1983-4535.2016v9n1p143
• Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals: Handbook 1: Cognitive domain. London, United Kingdom: Longmans.
• Borges-Andrade, J. E. (2002). Desenvolvimento de medidas em avaliação de treinamento [Development of measures in training evaluation]. Estudos de Psicologia (Natal), 7(spe), 31-43. doi:10.1590/S1413-294X2002000300005
  » https://doi.org/10.1590/S1413-294X2002000300005
• Cassiano, S. K., & Borges-Andrade, J. E. (2017). Methodological delimitations of the Brazilian research on effects of training. Revista Psicologia, Organizações e Trabalho, 17(3), 141-149. doi:10.17652/rpot/2017.3.13053
  » https://doi.org/10.17652/rpot/2017.3.13053
• Creswell, J. W., & Creswell, J. D. (2021). Projeto de pesquisa: Métodos qualitativo, quantitativo e misto [Qualitative, quantitative, and mixed methods approaches] (M. F. Lopes, Trans., 5th ed.). Porto Alegre, RS: Penso.
• Damasceno, S. M. S., Abbad, G. S., & Meneses, P. P. M. (2012). Logic models and organizational training evaluations. Paidéia (Ribeirão Preto), 22(52), 217-227. doi:10.1590/S0103-863X2012000200008
  » https://doi.org/10.1590/S0103-863X2012000200008
• Ford, J. K., Baldwin, T. T., & Prasad, J. (2018). Transfer of training: The known and the unknown. Annual Review of Organizational Psychology and Organizational Behavior, 5, 201-205. doi:10.1146/annurev-orgpsych-032117-104443
  » https://doi.org/10.1146/annurev-orgpsych-032117-104443
• Holloway, K., Arcus, K., & Orsborn, G. (2018). Training needs analysis - The essential first step for continuing professional development design. Nurse Education in Practice, 28, 7-12. doi:10.1016/j.nepr.2017.09.001
  » https://doi.org/10.1016/j.nepr.2017.09.001
• Khalil, M. K., & Elkhider, I. A. (2016). Applying learning theories and instructional design models for effective instruction. Advances in Physiology Education, 40(2), 147-156. doi:10.1152/advan.00138.2015
  » https://doi.org/10.1152/advan.00138.2015
• Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig & L. R. Bittel (Eds.), Training and development handbook: A guide to human resource development (pp. 87-112). New York, NY: McGraw-Hill.
• Kraiger, K., & Ford, J. K. (2020). The science of workplace instruction: Learning and development applied to work. Annual Review of Organizational Psychology and Organizational Behavior , 8, 45-72. doi:10.1146/annurev-orgpsych-012420-060109
  » https://doi.org/10.1146/annurev-orgpsych-012420-060109
• Lacerenza, C. N., Reyes, D. L., Marlow, S. L., Joseph, D. L., & Salas, E. (2017). Leadership training design, delivery, and implementation: A meta-analysis. Journal of Applied Psychology, 102(12), 1686-1718. doi:10.1037/apl0000241
  » https://doi.org/10.1037/apl0000241
• Martins, L. B., & Zerbini, T. (2015). Evidências de validade de instrumentos de reações no ensino superior à distância [Validity evidences of instruments of reaction in distance higher education]. Estudos e Pesquisas em Psicologia, 15(1), 116-134. doi:10.12957/epp.2015.16064
  » https://doi.org/10.12957/epp.2015.16064
• Martins, L. B., Zerbini, T., & Medina, F. J. (2018). Estrutura fatorial e propriedades psicométricas da Escala de Suporte à Transferência de Treinamento [Factor Structure and Psychometric Properties of the Scale of Support for Training Transfer]. Psicologia: Teoria e Pesquisa, 34, e3452. doi:10.1590/0102.3772e3452
  » https://doi.org/10.1590/0102.3772e3452
• McLaughlin, J. A., & Jordan, G. B. (2010). Using logic models. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (3rd ed., pp. 55-80). San Francisco, CA: Jossey-Bass.
• Mourão, L., & Borges-Andrade, J. E. (2005). Avaliação de programas públicos de treinamento: Um estudo sobre o impacto no trabalho e na geração de emprego [Evaluation of public training programs: A study on the impact on work and job creation]. Organizações & Sociedade, 12(33), 13-38. doi:10.1590/S1984-92302005000200001
  » https://doi.org/10.1590/S1984-92302005000200001
• Mourão, L., & Borges-Andrade, J. E. (2013). Impact evaluation of T&D at the societal level. Journal of Workplace Learning, 25(80), 505-520. doi:10.1108/JWL-12-2012-0081
  » https://doi.org/10.1108/JWL-12-2012-0081
• Nascimento, T. T., Borges-Andrade, J. E., & Porto, J. B. (2016). Produção científica brasileira sobre comportamento organizacional no terceiro setor [Brazilian scientific production on organizational behaviour in the nonprofit sector]. Estudos de Psicologia (Campinas), 33(2), 367-374. doi:10.1590/1982-02752016000200018
  » https://doi.org/10.1590/1982-02752016000200018
• Richter, L. D. D. (2018). Auto evaluations of distance management training in a federal autarchy. Revista de Administração Mackenzie, 19(2), 1-29. doi:10.1590/1678-6971/eramg180100
  » https://doi.org/10.1590/1678-6971/eramg180100
• Seidl, J., Leandro-França, C., & Murta, S. G. (2018). Avaliação de impacto e suporte de um curso de preparação para aposentadoria [Impact and support evaluation of a retirement planning course]. Revista Psicologia Organizações e Trabalho, 18(4), 494-502. doi:10.17652/rpot/2018.4.14732
  » https://doi.org/10.17652/rpot/2018.4.14732
• Souza, D. B. L., Abbad, G. S., & Gondim, S. M. G. (2017). Modelos lógicos na avaliação de um mestrado profissional: Um exemplo de aplicação [Logical models in the evaluation of the professional masters: A practical example]. Revista Brasileira de Pós-Graduação, 14, 1-17. doi:10.21713/2358-2332.2017.v14.1429
  » https://doi.org/10.21713/2358-2332.2017.v14.1429

• 1
  Article derived from the master’s thesis of the first author under the supervision of the second, defended in 2018, in the Postgraduate Program in Social Psychology of Work and Organizacionas of Universidade de Brasília.

Edited by

Associate Editor:

Fabio Scorsolini-Comin

Publication Dates

• Publication in this collection
  01 Nov 2021
• Date of issue
  2021

History

• Received
  14 Dec 2020
• Reviewed
  13 May 2021
• Accepted
  21 June 2021