Procedural model in the evaluation of public policy through gamification

Abstract

This article proposes a procedural model designed to optimize the experience of citizens in decision making of public policies through gamification. The study assumes that gamification increases motivation and contributes to increase involvement within a specific area. In this case, gamification is used to encourage citizens’ engagement in decision-making, based on three models of evaluation of public policies that incorporate civil participation in three different roles: auditor, passive and active. After applying the Delphi methodology, the results of reliability and internal consistency show that experts attribute a higher value to the model of active participation using gamification. In conclusion, the model designed opens a new area of knowledge within the evaluation of public policies, embedded in the interactivity of the media context.

Keywords:
public policies; citizen participation; gamification; evaluation model; user experience.

Resumen

Este artículo tiene por objeto proponer un modelo procedimental diseñado para optimizar la experiencia de la ciudadanía en la toma de decisiones de las políticas públicas a través de la ludificación. Se parte de la consideración que la ludificación promueve la motivación hacia una actividad prolongando su participación dentro de un ámbito específico. En este caso, se encamina su aplicación para motivar la integración de los ciudadanos en la toma de decisiones. Para ello se contrastan tres modelos de evaluación de políticas públicas que incorporan la participación civil en tres roles diferentes: auditor, pasivo y activo. Al aplicar la metodología Delphi, los resultados de fiabilidad y consistencia interna demuestran que el modelo de participación activa tiene mayor valoración por parte de los expertos respecto a la ludificación, Como conclusión, el modelo generado abre una nueva área de conocimiento dentro de la evaluación de políticas públicas imbuida en la interactividad del contexto mediático.

Palabras clave:
políticas públicas; participación ciudadana; ludificación; modelo de evaluación; experiencia del usuario.

Resumo

Este artigo tem como objetivo propor um modelo procedimental concebido para otimizar a experiência dos cidadãos na tomada de decisões de políticas públicas por meio da gamificação. Parte-se do pressuposto de que a gamificação promove a motivação para uma atividade, aumentando a participação dentro de um âmbito específico. Nesse caso, a gamificação é utilizada para incentivar o envolvimento dos cidadãos na tomada de decisões. Com esse objetivo, comparam-se três modelos de avaliação de políticas públicas que incorporam a participação civil em três papéis diferentes: auditor, passiva e ativa. Ao aplicar a metodologia Delphi, os resultados de fidelidade e consistência interna revelam que o modelo de participação ativa é alvo de uma valoração superior por parte dos especialistas, no que se refere à gamificação. Em conclusão, o modelo gerado abre uma nova área de conhecimento no âmbito da avaliação de políticas públicas, integrada na interatividade do contexto midiático

Palavras-chave:
políticas públicas; participação cidadã; gamificação; modelo de avaliação; experiência do usuário.

1. Introduction

At the present time, technological innovation is presented as a resource used to see to the needs of the citizens and to motivate participation through inclusive dynamics, and is raised as a banner in the processes of political representation and democratization (Zurbriggen and Lago, 2014). Therefore, the integration of techniques, as in the case of gamification, serves as an alternative to involve civil society in processes that are exclusively granted to the public administration’s civil servants.

The term gamification can be defined as the use of game design elements in non-game contexts (Deterding et al., 2011:2). In general terms, its objective is focused on people’s behavior, appealing to persuasion for prolonging their participation within a specific area (Scoltock, 2016; Werbach and Hunter, 2012). This means that it focuses on increasing people’s commitment and motivation towards an activity, making it more fun, with the aim of stimulating learning (Landers and Callan, 2011; Simões, Redondo and Fernández-Vilas, 2013). The link between gamification and public policies can be mainly viewed as a use of gamification as a resource for optimizing behavioral changes, favoring participation (Bista et al., 2014) and strengthening community values (Dirienzo, Krishnan and Santos, 2014; Martí et al., 2012; Sox, Kline y Crews, 2014).

The analysis of the public policy field can be defined as a prolific field comprised at the same time of an analytical framework for politics and for policies. Public policies have made available, in agreement with the theoretical frameworks of the social sciences, a wide terrain for exploring political behavior and the institutions.

The goals and the relation of the analysis with public policies are diverse. In many cases, the analysis is a process of determining the problems and solutions that could be considered a range of activities. This range is given from two perspectives: knowledge of public policies and knowledge for public policies (Parsons, 1999).

The process of analysis of public policies include two stages: their establishment and their content. The first refers to how, why, when and for whom this policy is created. The second is the specific description of the policy and its background, and it can also be comprised of a theoretical or analytical framework.

The monitoring and evaluation process does not belong to any of these two categories, and is specifically responsible for measuring the performance with respect to the policy’s specific objectives. It is important that the impact on the problem is considered, which merits a new examination of the target community or society. The results can provide innovation and better practices that can be implemented in the future in order to offer a solution.

The analysis for the policies has two stages. First, the support of the policies involves modifying the agenda within and outside of the government. This explicitly refers to arguments and research on the policies in order to elicit changes within them. Second, the information for the policies tries to provide detailed research, advice or judgement in order to help generate better options for the specific solutions that the policies put forward.

In this line of study, the inclusion of civil society is evaluated in the understanding of the knowledge of public policy, in the monitoring and evaluation, and lastly, in the transmission of knowledge for public policies. As there are specific references that back the usefulness of using game elements (Asquer, 2013; Hu and Chen, 2015; Mahnic, 2014; Nacke and Deterding, 2017; Thiel and Fröhlich, 2017), this research tries to take a step further into the academic world by creating a procedural model that is supported by the use of gamification.

In line with the theory reviewed, the adaptation of gamification is evaluated within three generic models of evaluation of public policies that integrate civil society into the process. Therefore, the general objective of this research is based on proposing a procedural model for the establishing of public policies that is associated to gamification. The specific objectives of this research are: (1) to contrast the models of evaluation of policies based on the categorization of the dimensions associated to gamification; and (2) to apply the indicators of gamification to the model of evaluation of public policies that has the greatest acceptance among the experts.

2. Background and theoretical considerations

The interdisciplinary character of the aspects from public policies, new technologies and psychology shape this unprecedented research study, particularly when the concept of gamification is incorporated as an innovative factor of the approach.

Within the field of public policies, no similar studies have been found. However, the Modeling and Simulation of the Impact of Public Policies on SMEs (MOSIPS) analyzes the public policies without delving into the citizen’s education, applying game elements but without specifying gamification (Pablo-Martí et al., 2012). In fact, Koch (2014) explains that it is about a process of execution of public policy that considers the need to include the participation of multiple social actors, strengthening the feedback through the social networks, and empowering the citizens by granting them the opportunity to gather information and discuss public policies.

Another initiative is the one promoted by He and Chen (2015) that creates an online forum for the development of public policies by applying a gamified design, demonstrating the impact of mechanisms such as the medals and the leaderboard in the exchange of information between the participants and the transfer of the motivational aspects from business sciences to social sciences. However, in this case, the success of gamification is observed through the creation of an attractive experience among the users, a competitive atmosphere in the forum, and creating changes within a system that was considered as being inefficient (Bremner and Zastrow, 2008) and destructive (Zastrow, 2014).

However, the field of new technologies within the European stage is present in this subject matter through the research and innovation program Horizon 2020, in which a space is proposed for projects related to Advanced digital gaming/gamification technologies with the aim of developing new methodologies, tools and scientific evidence in non-game digital contexts for the benefit of governments, businesses and individuals. Among the 28 projects selected, none of them list the evaluation of public policies within their objectives.

Accordingly, the theoretical basis that allows for connecting game elements with the non-game context is developed according to principles of interactions of the User Experience (Deterding et al., 2013; Kumar, 2013). This means that in order to apply gamification, it is necessary that the mechanisms are embedded within the principles accepted by the User Experience, highlighting its subjective character. This is because it depends on the resulting emotional aspects of each user (Hassenzahl, 2003; Battarbee and Koskinen, 2005; Law et al., 2009) and its holistic character, as the interaction produced is observed as an all (Battarbee and Koskinen, 2005; Hassenzahl and Tractinsky, 2006; Bargas-Avila and Hornbæk, 2011) and its dynamic character, since that the experience can change during the process of interaction (Law et al., 2009; Karapanos et al., 2009; Bargas-Avila y Hornbæk, 2011). Ultimately, the user’s experience can be defined as the subjective, holistic and dynamic evaluation of the interaction process of the citizens in the evaluation of public policies.

Although there are multiple categorizations with respect to the elements that comprise the user’s experience (Castañeda, Muñoz-Leiva and Luque, 2007; Law et al., 2009; Garrett, 2010; Park et al., 2013; Jang et al., 2016), the proposal by Jang and partners (2016) exclusively evidences its association with the design of game elements in a non-game context. Therefore, the model designed towards the public policies is prescribed according to the following dimensions:

• Hedonic goals - This refers to internal experiences of user enjoyment.

• Serendipity - Unexpected positive experience. This is related to unpredictability, creativity and the facilitation of the user’s experience.

• Playability - Enjoyable internal experiences of the users with respect to a service in particular within a specific context.

• Utilitarian goal - Practical purpose of the service. This refers to factors of performance, providing of useful experiences, instrumental benefits and the performing of tasks.

• Compatibility - Degree in which technology fits with existing values and experiences with the aim of increasing the perception of efficacy and to reduce the perception of risk due to its use.

• Consistency - Transfer of abilities of the experience to other situations, increasing confidence and the utilitarian value of the service.

• Degree of autonomy goal - This refers to the degree of control or independence of the user with respect to the service the user interacts with.

• Personalization - Based on the manner in which the users can change the system according to their preferences and circumstances. It is focused on recognizing the flexibility of the process and on providing the user with the freedom of management in specific circumstances.

• Adaptability - It is focused on evaluating how the system is organized according to the user’s wishes. The users judge the importance and the value of using the information gathered.

As for the evaluation of public policy, Johnson (1975), Epstein and partners (1977), Peters (2012) and Villoria (2014), believe that it is a coordinated set of processes whose aim is to measure, analyze, plan and interpret the fulfillment of the objectives, evaluate the policy’s results, receive feedback, and control the distribution of resources of the public administration. More specifically, the need to evaluate public policies has three purposes: legitimize the public action of the institutions, account for the management of public resources (accountability), and lastly, to strengthen democratic values. Emery, Mulder and Frewer (2015) have stated the problems with respect to its evaluation, more specifically on the monitoring of the impact, the understanding and the supervision that are already formalized as policy.

When focusing on the approach of the evaluation of public policy in current models that allow for the interposing of gamification techniques without producing axiomatic changes within the evaluation, three models are prioritized, which have the feature of incorporating the citizens in the process of understanding of the knowledge of the public policies, the monitoring and evaluation, and the transmission of knowledge for public policies. The models are: client oriented (Vedung, 2009), evaluation of user’s satisfaction (Vargas, 2009) and participative evaluation (Vargas, 2009).

The client oriented model is defined by Vedung (2009) as the value expressed by the wishes, expectations, assumptions and issues of the clients. The idea of considering the relationship State-citizens is a reaction to the client relationship from the capitalist system’s political ideology, as in the free market it is believed that the consumer’s pressure towards a service can generate changes in their provision, leading to its improvement and an increase in satisfaction.

This model is specifically applied to the evaluation of the provision of local services (medical services, crime prevention, services for the disabled and the young), cultural programs (libraries, museums, theaters), recreation programs (national parks, swimming pools, football fields), and other programs (cleaning and maintenance, urban traffic). The essence of the evaluation of this model is focused on the citizen’s own criteria for the evaluation. According to Schmidt (1977), these criteria can be determined according to the following dimensions: quality of service, satisfaction with the service, understandability of the procedures, client-centered orientation, adaptability of rules to client needs, and coordination of the personnel.

On its part, the achievement of goals criteria limits the client dimensions to the objectives of the level of government that is evaluated. This means that the assessment of the priorities from the national, state or local levels will never be the same, although this model will preferentially be used at the local level (Stufflebeam, 1983:34). The six dimensions referred by Schmidt (1977) are the only ones from which evidence exists on the application of the “client-focused model” in the public sector:

• Quality of service: degree of fulfillment of the citizens’ expectations in the long term. Overall attitude towards a service.

• Satisfaction with the service: transitory judgement of the citizens based on a specific experience with respect to a service.

• Understandability of the procedure: The procedure required for the obtaining of services should be easy to understand. It depends on the experience and knowledge of the citizens on the processing of applications.

• Client-centered orientation: It is the degree in which the citizens perceive that the public administration is looking out for their interests.

• Adaptability of the rules to the client’s specific needs: Personalization of the interaction and flexibility of the norms for satisfying the specific needs of the citizens.

• Coordination of the personnel: Perspective of the citizens on the synchronization of the civil servants when working together in the activities for satisfying their needs.

In second place, the model of evaluation of the user’s satisfaction has been precisely steered towards the evaluation of health policies (Crow et al., 2002; Ruggeri, Farrington and Brayne, 2013). Likewise, the advances in information and the new technologies have facilitated their implementation, resulting in new experiences that are able to identify society’s requirements (Alawneh, Al-Refai, and Batiha, 2013). In general terms, this type of model, which was previously designed to measure private services, has been adapted to the perception of quality of public service according to two paradigms: expectations-dissatisfaction of the citizens and the performing of the tasks by the civil servants of the public administration (Carvalho and Brito, 2012). Although the evaluation of performance has its own prominent system of measurement named Servperf (Cronin and Taylor, 1992), this same is constructed based on a previous system of expectations-dissatisfactions named Servqual, which currently has three versions: 1^st: Original version (Parasuraman, Zeithaml and Berry, 1988); 2^nd: Modifications and theoretical basis that is posterior to the creation of the measuring system of quality performance (Parasuraman, Zeithaml and Berry, 1994); and 3^rd: Electronic version (Parasuraman, Zeithaml and Malhotra, 2005).

In the three versions, the application of five dimensions proposed in the original version persists (Parasuraman, Zeithaml and Berry, 1988): (1) reliability: the ability to fulfill the objectives with the least number of possible errors; (2) responsiveness: willingness to help users and provide immediate service; (3) assurance: professionalism, ability to inspire confidence in the service provided; (4) empathy: individualized and personalized attention to the needs of the citizens; and (5) tangibles: appearance and usefulness of equipment, infrastructure, installations and communication materials.

Lastly, the participative evaluation model, defined as “expert or non-expert intervention or research on a specific subject matter working together at the same level in a specific project” (Úcar, 2014:15). Indeed, it is defined as an endogenous model formed on the values and aspirations of the participants of a pluralist society through the long process of evaluation. In agreement with Izquierdo (2008) and Dos Santos (2016), it implicates the citizens in the collaboration and participation of the process, magnifying the sense of belonging in the design of policies and in the results, creating actions that motivate collective learning.

Cousins and Whitmore (1998) proposed three dimensions where the transformative (Pursley, 1996) and practical (Owen and Lambert, 1995; Torres, Preskill and Piontek, 1997) participative evaluations converge: control of the process of evaluation, choosing of participants, and depth of participation. Two decades later, Daigneault and Jacob (2014:16) in the review of the measurement instruments of the validation and review of participative evaluation, confirm the current nature of these dimensions stating that “the general level of participation in public policies is derived from the size of the participant’s diversity, degree of participation and control”. Control in the making of decisions means that this is prioritized on the basis of equilibrium in the decisions that underpin the program (balanced, the participants predominate, the technicians predominate, the evaluator predominate). Participant selection refers to the groups that are legitimate, who benefit and who are authorized to make decisions and implement the program (primary users, legitimate users, non-specified). Finally, depth of participation means the number of phases which the citizens can participate in (limited, moderate and extensive).

In conclusion, the client-oriented evaluation model (Vedung, 2009) considers the citizen as an active auditor of the activities and services related to public service, explicitly guided by their own wishes, values, aims, worries, expectations and assumptions; the user satisfaction model (Vargas, 2009) considers the citizens as passive actors, who are not involved in the public sector, trying to delegate the decision making to their representatives, only participating in specific situations; and the participative evaluation model (Vargas, 2009) considers the notion of the active citizen who participates in the activities of the public sector, focused on fulfilling the common needs in a society.

3. Methodology

The present research adopts a non-experimental design, with a systematic approach using the Delphi methodology, ensuring the validation of the process after the identification of the dimensions that illustrate the standardization of the decision making of the main model that related to the experience of the user and the evaluation of public policy. The questionnaire is a common instrument in the Delphy methodology, as it systematizes and shows the convergence of opinions from a group of experts (Astigarraga, 2003; García-Valdés and Suárez-Marín, 2013). With respect to the size of the panel of experts that comprise the Delphi, there is little empirical evidence on the effect of the number of participants in the processes of consensus oriented to the reliability and validity that vary according to the resources and the focus of the problem (Keeney, Hasson and McKenna, 2001). Ultimately, Powell (2003) believes that for the size of the panel, the expert’s qualification has a greater repercussion in the impartial view of the findings and the fulfillment of the objectives. Therefore, in this study, the experts were divided into two groups, the first formed by experts in Communication Technologies, Orientation and Socio-cultural Intervention, who supervised the questionnaire in content editing as well as statistical inferences, providing the corrections needed to create the final version.

In agreement with the changes proposed by the experts, the final version of the questionnaire was composed by three models of evaluation of the public policies. Each model had three dimensions where the 71 items were distributed: the client-oriented model had 30 items — 17 for hedonic goals, six for the utilitarian goals, and seven for the degree of autonomy goal; user satisfaction was comprised by 24 items — 12 belonging to the hedonic goals, five to utilitarian goals, and seven to the degree of autonomy goal; and the third model contained 17 items — seven for hedonic goals, six for utilitarian goals, and four for the degree of autonomy goal. The three models were presented under the same conditions, maintaining an acceptable internal consistency and a positive value between the elements of each assignment. Tables 1, 2 and 3 presents this results.

Table 1
Cronbach’s coefficient from model 1

Model 1: Client-oriented model
Hedonic goals	Utilitarian goals	Degree of autonomy
Cronbach’s Alpha: .943	Cronbach’s Alpha: ,767	Cronbach’s Alpha: ,738
Number of indicators: 17	Number of indicators: 6	Number of indicators: 7

• Source: Elaborated by the authors.

Table 2
Cronbach’s coefficient from model 2

Model 2: User satisfaction evaluation model
Hedonic goals	Utilitarian goals	Degree of autonomy
Cronbach’s Alpha: .957	Cronbach’s Alpha: .931	Cronbach’s Alpha: .897
Number of indicators: 12	Number of indicators: 5	Number of indicators: 7

• Source: Elaborated by the authors.

Table 3
Cronbach’s coefficient from model 3

Model 3: Participative evaluation model
Hedonic goals	Utilitarian goals	Degree of autonomy
Cronbach’s Alpha: .717	Cronbach’s Alpha: .800	Cronbach’s Alpha: .719
Number of indicators: 7	Number of indicators: 6	Number of indicators: 4

• Source: Elaborated by the authors.

The second group who was willing to complete the questionnaire was comprised of representatives from the University of Gothenburg (Sweden) from the Department of Applied Technology and from the Department of Education, Communication and Learning, who due to their background, have been involved in multiple projects oriented towards games in public administration.

4. Results

After obtaining responses from the second group, the degree of acceptance was calculated taking into account the means from each model, observing that the model of participative evaluation had the highest degree of acceptance. However, to make the results more rigorous, an Anova was also performed. According to Gueorguieva and Krystal (2004), the Anova is a type of parametric test that is applied when there are more than two independent samples to contrast the null hypothesis. The statistical significance (Sig.) was also calculated, which gives information on the data’s Normality (>0.05). Their compliance for all three models are shown in table 4.

Table 4
Normality tests of the three models

Normality test
	Kolmogorov-Smirnova
	Statistics	gl	Sig.
Model 1	.255	10	.064
Model 2	.171	10	.200
Model 3	.149	10	.200

• Source: Elaborated by the authors.

The statistical tests show that the model of participative evaluation (Vargas, 2009) and the degree of autonomy had the greatest degree of agreement among the experts with respect to the evaluation of the user’s experience. The most important differences were found between model 2 and model 3, with these being the ones that had, respectively the least and greater degree of agreement, and a greater and lesser standard deviation with respect to the responses obtained.

The third model, the model of participative evaluation (Vargas, 2009), is characterized by the intervention of civil society actors who represent the plurality of value systems in the management, design and implementation of public policies. This type of evaluation is discerned as an especially-efficient method for the formulation of recommendations for the appropriate authorities in the program or the policy (Horta and Ysa, 2007).

The user’s experience, from the model of Jang and partners (2016), takes into account the quality of the experience and judgement criteria according to the user’s perception, validating the interaction of the factors that comprise it. More specifically, its association with game design, digital platforms and with dimensions related to behavior formalize its framing within the model of participative evaluation.

When combining the criteria of the model of participative evaluation (Vargas, 2009) with the three goals and the six constructs of the user’s experience (Jang et al., 2016) in the creation of the questionnaire, the following is shown:

Item 1: The unexpected positive experiences of the users affect the control of the making of decisions (hedonic).

Item 5: The fun experiences of the users affect the control in the making of decisions (hedonic).

Item 8: The use of technology in the user’s experience affects the making of decisions (utilitarian)

Item 10: The use of technology in the user’s experience affects the depth of participation in the making of decisions (utilitarian).

Item 13: The adaptation of the system to the wishes of the users affects the depth of participation in the making of decisions (utilitarian).

The high degree of acceptance of the results obtained with respect to the model of participative evaluation, combined with the goals of experience of the user, grants it a theoretical and methodological consistency which set the basis before the addition of gamification elements.

Therefore, the last objective, the application of gamification indicators to the model of evaluation of public policy with the greatest acceptance among the experts, implies that the acceptance of the third model provides the opportunity for adding gamification elements, thereby providing elements for each goal of the user’s experience.

Serendipity and playability, both derived from the hedonic goal, commend the use of game design elements. Serendipity is understood as an unexpected positive experience, and therefore, random game elements depict their impact when they generate surprise, delight and feelings, fomenting changes in behavior (Liang, 2012). Independent of the tool applied, chance should be promoted as the objective within the game activity. On its part, playability refers to the pleasant internal experiences of the users with respect to a service in particular, albeit it will depend on the service provided. The study by Fitz-Walter and partners (2013) assesses the insertion of a system of achievements, tutorials and competitive elements.

Compatibility and consistency, indicators from the utilitarian goal, also prescribe the use of game elements. Compatibility is the degree in which technology allows for the fitting of existing values and experiences. In this case, technology should not become a limitation when applying game elements, but the use of interfaces should produce a sense of assurance in the users and facilitate interaction. The more compatible the platform, the greater feedback received from the user. Consistency in the user’s experience is the transfer of abilities from the experience to other situations. The challenges and levels are game elements applied to evaluate the acquisition of abilities, operating as a system of classification on the progress accomplished (Silpasuwanchai, Ma, Shigemasu and Ren, 2016).

Personalization and adaptability form part of the degree of autonomy goal. Personalization is the manner in which the users can change the system according to their preferences and circumstances, their implication in the game design depends on the objectives planned. Thus, the inclusion of a section that is under the exclusive control of the users is indispensable so that the users can establish their interests (Chauhan, Taneja y Goel, 2015). Adaptability is focused on evaluating how the system fits with the wishes of the users. In this occasion, the game elements should allow the users to share their results, give advice on alternative uses and promote feedback from their peers, thereby facilitating data gathering with the aim of improving interaction (Scoltock, 2016).

Ultimately, the design of gamification for the evaluation of public policies from the perspective of the user’s experience is composed of random elements, systems of achievements, tutorials, competitive elements, feedback, challenges, levels and personalization of the user’s profile. Likewise, the platform should utilize terminology that is adapted to the user, language, expressions and discursive resources with the objective of creating a feeling of belonging.

The random elements promote uncertainty and diversion, breaking the linearity of an activity. The most common system of randomness are dice, cards and random number generators, among others. Randomness affects decision making and, therefore, an equilibrium should be created between the user’s sense of mastery of the platform and the randomly-placed elements. In this case, randomness should be oriented towards extra benefits for the user as unexpected rewards.

System of achievements means that a reward is given when a goal is achieved. There are multiple elements, among which we find historical information, progress, points, medals, etc. In this case, the implementation of the PBL (points, badges and leaderboards) mechanics will be opted for. These are the points granted continuously when completing an activity, the medals when accumulating a series of activities performed, and a visible and public leaderboard.

The tutorials element means that at the start, the presence of a didactic explanation on the use of the platform is needed. Mission, vision and values are anachronistic. In this case, a clear and precise goal should be promoted, allowing the fomenting of a common objective in order to be able to explain the game dynamics that comprise it.

Elements of competition is used to bolster intrinsic motivation. It is more feasible that people internalize the objectives when they understand that they possess abilities that are relevant for achieving them. Ultimately, the obstacles and challenges stimulate a person to move forward. The challenges and the levels, as well as providing consistency to the system, also foster playability.

Feedback is the most important mechanism in the gamification process, as it re-enforces behaviors that must be repeated. It is presented as the information that is given back to the player after performing an action, and due to this, it is considered to be intimately related to points and progress. It allows players to recognize their performance in the activities and evaluate the opportunity of continuing participating. In the public policy sphere, a certain priority is placed on the use of interpretative feedback that is focused on the perception of the citizens on the aspects of the political program that exert influence on their lifestyles. However, the recursive feedback is also incorporated in a specified manner in order to shape the patterns of behavior that determine the civil capacity and participation of the citizens.

Personalization of the user’s profile is the recognition of multiple tactics of disinformation, so the platform seeks to integrate transparency as a basic premise. The user should be able to access the platform with real data that nourish their working, closely with the public administration. Mainly, the user must present their name, date of birth and interests. The interests are subordinate to the level of government. In the Central administration, they correspond to policies in the following sectors: Economic — fiscal, monetary and controls; Political — institutional and participative; Social — educational, social security and public health; Cultural — communicative and values. At the local level: agricultural, well-being, culture, sports, employment, cultural board, tourism, urbanism and youth.

Figure 1
Final model of evaluation of public policies

Figure 1 presents the final model of evaluation of public policies. In the model, the user is viewed as the main focus of the process. To access it, the user identifies him or herself with his or her name and last name, email (another method of contact), date of birth (+16 or +18 years of age allows the user to participate) and interests, which depends on the governmental level to which they will be applied. To start, a tutorial that explains the use of the platform and its challenges is shown.

The system is organized in real time to the program of public policy. However, the user decides when he or she joins the participation, and the only condition is that this should be done during the period of discussion or after it. This model is composed of three levels, and although the user can foray in all of them in an unspecified order, it is understood that the challenges in level 3 are more difficult that the challenges in level 1.

Level 1 is based on the control of the decisions, and in this case it is reduced to knowing the opinions of the authorized representatives expressed anonymously in the interface, accepting and rejecting each argument, with acceptance implying freedom, and rejection limitation. Each intervention has a random number of points, and the greater the number of interventions that are interacted with, the greater the opportunity to gain points. On their part, the medals are given with respect to the accumulation of a specific number of points.

Level 2, named “selection of the participants”, equally gathers the intervention of the participants. However, in this case, besides only accepting or rejecting a proposal, the user must select the level of education and professional experience in the area of the person who expressed the argument, who remains anonymous. The closer the user is to the right answer, the greater will be the number of points, and a medal will also be given for fulfilling objectives.

In level 3, depth of participation, the user must broadcast his or her opinion on the degree of depth of the arguments given in the discussions on the specific piece of public policy. Just as in level 1, each argument has a randomly assigned score, and the more answered questions, the greater the score obtained and the closer to the medals the user is. Although it may seem that the level of difficulty is the same as in level 1, this level merits a series of previous knowledge on the terminology before making the posting of an answer available. Likewise, completing the third step implies the anonymous transmission of the data from the third phase to the public administration.

In the lower section, all the users registered at the start of each session randomly uncover three cards. The image of the user’s Town Hall or Council, if at the local level, or the Parliament if at the central level, will appear, and an email will be sent automatically asking for a date and time for a meeting with the council of a specific area. This will therefore facilitate the personal interaction of civil society with specific representatives of public policy.

In this last instance, this model of evaluation seeks that the user evaluate the decisions of the public administration, make decisions and become actively involved in the programs of public policy, especially the youth, who will face the direction of future public policy.

5. Conclusions

In conjunction with the objectives presented, the main goal was to propose a procedural model for the establishment of public policy associated to gamification, which was shaped based on the model of participative evaluation (Vargas, 2009) through the application of the following gamification components: random elements, system of achievements, tutorials, elements of competition, feedback, challenges, levels and personalization of the user profile. The specific objectives of contrasting of the models of evaluation of policies based on the categorization of the dimensions of the user’s experience and of applying gamification indicators to the model of evaluation of public policies that had the greatest acceptance among the experts were, therefore, satisfied.

According to the first objective, the Model of Participative Evaluation (Vargas, 2009) and the degree of autonomy were the ones that obtained the greatest degree of agreement among the experts as far the evaluation of the user’s experience. This means that the model selected for the incorporation of the gamification elements was the one that envisaged the active citizen who participates in the activities of the public sector, focused on fulfilling the common needs of a society.

Although at the theoretical level the participative evaluation stood out as the method of evaluation of the public administration, the data gathered in this work showed that this model of evaluation of public policy was adapted to the factors of influence of the user’s experience. The model of participative evaluation, besides being efficient, was also compatible within the area of new technologies. Also, the perspective of the user’s experience was considered to drive the capacitation and influence of the active citizen in the evaluation of public policy.

In order to fulfill the second objective, gamification elements that were selected specifically for the attribution of the user’s experience were provided: random elements, the system of successes, tutorials, elements of competition, feedback, challenges, levels and personalization of the user’s profile, which were added to the final design, maintaining the user as the center of the model. Ultimately, this is a first draft that allows for the opening of a new area of knowledge within the new technologies and innovation, formally showing and following methodological parameters from the social sciences, and the possibility of adding game elements in a model of evaluation of public policy.

The major contribution is related to gamification. The technique of gamification has turned from being an emergent technology to a fixed technology within digital marketing. However, in the theoretical arena, beyond behaviorism, gamification has been subjected to empirical studies within the business sphere. This study tries to take a step forward in the acceptance of gamification within the academic environment, within which its substitution as a multidisciplinary technique that tries to make use of the addition of points, medals and leaderboards to find short-term benefits is sought for a process derived from the perspective of the user’s experience and at the same time generate changes in behavior in any area of study.

The innovative approach of this research has invited the conducting of a tour through the possibilities that the new technologies and games offer. Although there is still a domain of games from the trivial perspective, the portability, the advanced multimedia capacity, the cloud services, the three-dimensional graphics and the touchscreens have facilitated their incursion into the public sector as an applied technique in specific programs of public policies, among which the education and health areas are highlighted, setting aside their application in the evaluation of public policies.

Ultimately, it is understood that this procedural model seeks to make society evaluate, in an interactive and entertaining way, the civil servants during the making of public decisions processes, giving enough importance to the discourse they construct and the ideas they express with respect to the common needs of the population, thus activating popular participation.

At present, the credibility and legitimacy granted by the formalization and evaluation of the present research work, sequentially supported by a research group comprised of experts from diverse academic areas, strengthens the potential incorporation of the model of public administration, guaranteeing the application of the procedural model within the empirical plan of the public administration at the central and local levels, formalizing the results in future publications.

REFERENCIAS

• ALAWNEH, Ali; AL-REFAI, Hasan; BATIHA, Khaldoun. Measuring user satisfaction from e-Government services: Llessons from Jordan. Government Information Quarterly, v. 30, n. 3, p. 277-288, 2013.
• ASQUER, Alberto. Not just videogames: gamification and its potential application to public services. En: DIGITAL PUBLIC ADMINISTRATION AND E-GOVERNMENT IN DEVELOPING NATIONS: POLICY AND PRACTICE, 2013, IGI Global.
• ASTIGARRAGA, Eneko. El método Delphi San Sebastián: Universidad de Deusto, 2003.
• BARGAS-AVILA, Javier A.; HORNBÆK, Kasper. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. En: PROCEEDINGS OF THE SIGCHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2011, ACM.
• BATTARBEE, Katja; KOSKINEN, Ilpo. Co-experience: user experience as interaction. CoDesign, v. 1, n. 1, p. 5-18, 2005.
• BISTA, Sanat Kumar et al. Gamification for online communities: a case study for delivering government services. International Journal of Cooperative Information Systems, v. 23, n. 2, p. 201-225, 2014.
• BREMNER, Judith; ZASTROW, Charles. Advanced standing revisited: Current status, structure, and issues. Journal of Teaching in Social Work, v. 28, n. 1-2, p. 101-116, 2008.
• CARVALHO, Cláudia; BRITO, Carlos. Assessing users’ perceptions on how to Improve public services quality. Public Management Review, v. 14, n. 4, p. 451-472, 2012.
• CASTAÑEDA, J. Alberto; MUÑOZ-LEIVA, Francisco; LUQUE, Teodoro. Web Acceptance Model (WAM): moderating effects of user experience. Information & Management, v. 44, n. 4, p. 384-396, 2007.
• CHAUHAN, Jyoti; TANEJA, Shilpi; GOEL, Anita. Enhancing MOOC with augmented reality, adaptive learning and gamification. En: MOOCS, INNOVATION AND TECHNOLOGY IN EDUCATION (MITE), IEEE 3RD INTERNATIONAL CONFERENCE, 2015, IEEE.
• COUSINS, J. Bradley; WHITMORE, Elizabeth. Framing participatory evaluation. New Directions for Evaluation, v. 19, n. 80, p. 5-23, 1998.
• CRONIN JR, J. Joseph; TAYLOR, Steven A. Measuring service quality: a reexamination and extension. The Journal of Marketing, v. 56, n. 3, p. 55-68, 1992.
• CROW, Rosemarie et al. The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature. Health Technol Assess, v. 6, n. 32, p. 1-244, 2002.
• DAIGNEAULT, Pierre-Marc; JACOB, Steve. Unexpected but most welcome mixed methods for the validation and revision of the participatory evaluation measurement instrument. Journal of Mixed Methods Research, v. 8, n. 1, p. 6-24, 2014.
• DETERDING, Sebastian et al. From game design elements to gamefulness: defining gamification. En: PROCEEDINGS OF THE 15TH INTERNATIONAL ACADEMIC MINDTREK CONFERENCE: ENVISIONING FUTURE MEDIA ENVIRONMENTS, 2011, ACM.
• DETERDING, Sebastian et al. Designing gamification: creating gameful and playful experiences. En: CHI’13 EXTENDED ABSTRACTS ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2013, ACM.
• DIRIENZO, Thomas; KRISHNAN, Nathan; SANTOS, Joost. Effects of smart appliances on residential consumption patterns. En: SYSTEMS AND INFORMATION ENGINEERING DESIGN SYMPOSIUM (SIEDS), 2014, IEEE.
• DOS SANTOS, Priscilla R. Inovações participativas, diálogo social e construção de consensos. Rev. Adm. Pública, v. 50, n. 3, p. 501-511, maio/jun. 2016.
• EMERY, Steven; MULDER, Henk; FREWER, Lynn. Maximizing the policy impacts of public engagement: a European study. Science, Technology, & Human Values, v. 40, n. 3, p. 421-444, 2015.
• EPSTEIN, Irwin et al. Research techniques for program planning, monitoring, and evaluation New York: Columbia University Press, 1977.
• FITZ-WALTER, Zachary et al. Driven to drive: designing gamification for a learner logbook smartphone application. En: PROCEEDINGS OF THE FIRST INTERNATIONAL CONFERENCE ON GAMEFUL DESIGN, RESEARCH, AND APPLICATIONS, 2013, ACM.
• GARCÍA VALDÉS, Margarita; SUÁREZ MARÍN, Mario. El método Delphi para la consulta a expertos en la investigación científica. Revista Cubana de Salud Pública, v. 39, n. 2, p. 253-267, 2013.
• GARRETT, Jesse James. Elements of user experience: the user-centered design for the web and beyond. Nueva Jersey: Pearson Education, 2010.
• GUEORGUIEVA, Ralitza; KRYSTAL, John H. Move over anova: Progress in analyzing repeated-measures data andBits reflection in papers published in the archives of general psychiatry. Archives of General Psychiatry, v. 61, n. 3, p. 310-317, 2004.
• HASSENZAHL, Marc. The thing and I: understanding the relationship between user and product. En: BLYTHE, Mark et al. (Ed.). Funology Houten: Springer Netherlands, 2003. p. 31-42.
• HASSENZAHL, Marc; TRACTINSKY, Noam. User experience-a research agenda. Behaviour & Information Technology, v. 25, n. 2, p. 91-97, 2006.
• HORTA, Xavier Bertrana; YSA, Tamyko. Los retos de la evaluación de políticas públicas: entre la participación y la responsabilidad. Nuevas Políticas Públicas: Anuario Multidisciplinar para la Modernización de las Administraciones Públicas, n. 3, p. 129-142, 2007.
• HU, Shueh-Cheng; CHEN, I-Ching. A gamified online forum inspiring group intelligence distillation for policy making. En: INTERNATIONAL CONFERENCE IN SWARM INTELLIGENCE, 2015, Springer International Publishing.
• IZQUIERDO, Beatriz. De la evaluación clásica a la evaluación pluralista: criterios para clasificar los distintos tipos de evaluación. Empiria: Revista de Metodología de Ciencias Sociales, n. 16, p. 115-134, 2008.
• JANG, Jinkyu et al. Application of experiential locus of control to understand users’ judgments toward useful experience. Computers in Human Behavior, v. 54, p. 326-340, 2016.
• JOHNSON, Ronald W. Research objectives for policy analysis. Public Policy Evaluation, v. 2, p. 76-83, 1975.
• KARAPANOS, Evangelos et al. Measuring the dynamics of remembered experience over time. Interacting with Computers, v. 22, n. 5, p. 328-335, 2010.
• KEENEY, Sinead; HASSON, Felicity; MCKENNA, Hugh. A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, v. 38, n. 2, p.195-200, 2001.
• KOCH, Andreas, Strengthening intangible infrastructures: a commitment to communities. Newcastle: Cambridge Scholars Publishing, 2014.
• KUMAR, Janaki. Design gamification at work: Designing engaging business software. User experience, and usability. Health, learning, playing, cultural, and cross-cultural user experience. Lecture Notes Computer Sci, v. 8013, p. 528-537, 2013.
• LANDERS, Richard N.; CALLAN, Rachel C. Casual social games as serious games: The psychology of gamification in undergraduate education and employee training. En: MA, Minhua; OIKONOMOU, Andreas; JAIN, Lakhmi (Ed.). Serious games and edutainment applications Londres: Springer, 2011. p. 399-423.
• LAW, Effie Lai-Chong et al. Understanding, scoping and defining user experience: a survey approach. En: PROCEEDINGS OF THE SIGCHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2009, ACM.
• LIANG, Rung-Huei. Designing for unexpected encounters with digital products: Case studies of serendipity as felt experience. International Journal of Design, v. 6, n. 1, p. 41-58, 2012.
• MAHNIC, Nika. Gamification of politics: start a new game! Teorija in Praksa, v. 51, n. 1, p. 143-161, 2014.
• MARTÍ, Irene et al. Mobile application for noise pollution monitoring through gamification techniques. En: ENTERTAINMENT COMPUTING-ICEC, 2012, Springer-Verlag.
• NACKE, Lennart; DETERDING, Sebastian. The maturing of gamification research. Computers in Human Behavior, v. 71, p. 450-454, 2017.
• OWEN, John M.; LAMBERT, Faye C. Roles for evaluation in learning organizations. Evaluation, v. 1, n. 2, p.237-250, 1995.
• PABLO MARTÍ, Federico et al. Mosips Agent-based model for predicting and simulating the impact of public policies on SMEs. En: PROCEEDINGS OF THE EUROPEAN CONFERENCE ON COMPLEX SYSTEMS, 2012, Springer International Publishing.
• PARASURAMAN, Arun; ZEITHAML, Valarie A.; BERRY, Leonard L. Servqual. Journal of Retailing, v. 64, n. 1, p. 12-40, 1988.
• PARASURAMAN, Ananthanarayanan; ZEITHAML, Valarie A.; BERRY, Leonard L. Reassessment of expectations as a comparison standard in measuring service quality: implications for further research. The Journal of Marketing, v. 58, n. 1, p. 111-124, 1994.
• PARASURAMAN, Ananthanarayanan; ZEITHAML, Valarie A.; MALHOTRA, Arvind. ES-QUAL a multiple-item scale for assessing electronic service quality. Journal of Service Research, v. 7, n. 3, p. 213-233, 2005.
• PARK, Jaehyun et al. Developing elements of user experience for mobile phones and services: survey, interview, and observation approaches. Human Factors and Ergonomics in Manufacturing & Service Industries, v. 23, n. 4, p. 279-293, 2013.
• PARSONS, Wayne. Public policy: an introduction to the theory and practice of policy analysis. Northampton: Edward Elgar, 1999.
• PETERS, Guy. American public policy: Promise and performance. Washington: Cq Press, 2012.
• POWELL, Catherine. The Delphi technique: myths and realities. Journal of Advanced Nursing, v. 41, n. 4, p. 376-382, 2003.
• PURSLEY, Linda Arlene. Empowerment and utilization through participatory evaluation. Nueva York: Cornell University, 1996.
• RUGGERI, Kai; FARRINGTON, Conor; BRAYNE, Carol. A global model for effective use and evaluation of e-learning in health. Telemedicine and e-Health, v. 19, n. 4, p. 312-321, 2013.
• SCHMIDT, Stuart M. Client-oriented evaluation of public agency effectiveness. Administration & Society, v. 8, n. 4, p. 403-422, 1977.
• SCOLTOCK, Judith. Adaptive design: how to build an effective learning experience. En: CONFERENCE PROCEEDINGS. THE FUTURE OF EDUCATION, 2016, libreriauniversitaria. it Edizioni.
• SILPASUWANCHAI, Chaklam; MA, Xiaojuan; SHIGEMASU Hiroaki; REN, Xiangshi. Developing a comprehensive engagement framework of gamification for reflective learning. En: PROCEEDINGS OF THE 2016 ACM CONFERENCE ON DESIGNING INTERACTIVE SYSTEMS, 2016, ACM.
• SIMÕES, Jorge; REDONDO, Rebeca; FERNÁNDEZ-VILAS, Ana. A social gamification framework for a K-6 learning platform. Computers in Human Behavior, v. 29, n. 2, p. 345-353, 2013.
• SOX, Carole B.; KLINE, Sheryl; CREWS Tena. Identifying best practices, opportunities and barriers in meeting planning for Generation Y. International Journal of Hospitality Management, n. 36, p. 244-254, 2014.
• STUFFLEBEAM, Daniel L. The CIPP model for program evaluation. En: EVALUATION MODELS, pp. 117-141. Springer Netherlands, 1983.
• THIEL, Sarah-Kristin; FRÖHLICH, Peter. Gamification as motivation to engage in location-based public participation? En: PROGRESS IN LOCATION-BASED SERVICES, 2017, Springer International Publishing.
• TORRES, Rosalie T.; PRESKILL, Hallie S.; PIONTEK, Mary E. Communicating and reporting: Practices and concerns of internal and external evaluators. American Journal of Evaluation, v. 18, n. 2, p. 105-125, 1997.
• ÚCAR, Xavier. Evaluación participativa y empoderamiento. Pedagogia Social, n. 24, p. 13-19, 2014.
• VARGAS, Carlos S. La evaluación y el análisis de políticas públicas. Opera, v. 9, n. 9, p. 23-51, 2009.
• VEDUNG, Evert. Public policy and program evaluation New Brunswick: Transaction Publishers, 2009.
• VILLORIA, Manuel. Integridad. Eunomía Revista en Cultura de la Legalidad, v. 2, n. 1, p. 107-113, 2014.
• WERBACH, Kevin; HUNTER, Dan. For the win: How game thinking can revolutionize your business. Pennsylvania: Wharton Digital Press, 2012.
• ZASTROW, Charles. Social work with groups: a comprehensive worktext. Boston: Congage Learning, 2014.
• ZURBRIGGEN, Cristina; LAGO, Mariana G. Innovación y co-creación. Nuevos desafíos para las políticas públicas. Revista de Gestión Pública, v. 3, n. 2, p. 329-361, 2014.

Publication Dates

• Publication in this collection
  Sep-Oct 2017

History

• Received
  02 Oct 2016
• Accepted
  27 June 2017