The Smart Cities MethodoLogy Based on Public Value: The First Evaluation Cycle

Abstract

Several smart cities initiatives are currently underway around the world, and this trend is expected to grow in the coming years. Such initiatives should be designed according to the needs and conditions of each municipality, with the aim of expanding or generating public value for stakeholders. This paper analyzes the results obtained from a first evaluation cycle of the Smart Cities MethodoLogy and its components (a reference model and an assessment method), developed based on a Public Value perspective, in a study conducted within the Design Science epistemological paradigm and Design Science Research method. The artifacts developed to solve the identified problem were evaluated, based on the perceptions reported by representatives of the Quadruple Helix (government, industry, academia and citizens) from different knowledge areas. The results of this first evaluation cycle allowed for improvements to be made to the initial versions of the three artifacts and the perceptions obtained indicated they can aid in the design and evaluation of smart cities by bringing together supposed intelligent best practices that generate public value, thus constituting a scientific contribution of a prescriptive nature.

Keywords:
design science research; smart cities; public value; quadruple helix; smart cities methodology

Introduction

The world’s urban population is expected to increase 75% by 2050 (Anthopoulos, 2015). In Latin America, in that same period, that figure is predicted to exceed 85%, while in Brazil the projected growth is 91% (Cunha, Przeybilovscz, Macaya, & Santos, 2016). In this scenario, it is important that cities are prepared to meet the needs of citizens in a sustainable way (Macaya, 2017).

With the growth of the urban population come management-related challenges including traffic congestion, increased crime, and so on (Alawadhi et al., 2012). Against this background, the smart city concept seeks to explore innovative solutions to such problems and challenges through the use of technologies that facilitate the innovation process (Gil-Garcia, Helbig, & Ojo, 2014), improve quality of life and transport, among other aspects (Ismagilova, Hughes, Dwivedi, & Raman, 2019).

It is estimated the global market for additional solutions and services to introduce smart cities will reach US$ 408 billion by 2020 (Department for Business, Innovation and Skills [BIS], 2013). In Brazil, recent investments and events point to a favorable scenario in the coming years (ABDI & INMETRO, 2017; Banco Nacional de Desenvolvimento Econômico e Social [BNDES], 2018; Ministério da Ciência, Tecnologia, Inovações e Comunicações [MCTIC], 2018). In turn, implementing smart cities may require high-visibility projects and substantial, exceptional public investments (Van den Bergh, Dootson, Kowalkiewicz, & Viaene, 2018), requiring technologies, new skills, and the upgrading of the current urban infrastructure. Public administrators may start simultaneous projects and programs (Ojo, Dzhusupova, & Curry, 2016), without adequately expanding current capacity and resources. Failings in such projects may have consequences, such as financial and reputational losses, diminished trust in government (Lee & Kwak, 2012) and the inability to generate public value.

Therefore, to minimize any such negative impacts, it is important that such projects evolve gradually and are able to ensure the achievement of public value. However, there is a scarcity of studies that share best practices regarding smart cities, investigate their impact on society (Oliveira & Campolargo, 2015) and assess their ability to deliver public value (Castelnovo, Misuraca, & Savoldelli, 2013). How to plan new smart cities and how to make existing cities smarter are topics insufficiently addressed in the literature (Ismagilova et al., 2019).

The projects undertaken by national and international organizations have not adopted a gradual evolutionary approach or a managerial view of the smart city that incorporates intelligent best practices aimed at generating public value. It is understood that “the task of the public administrator is to create public value” (Moore, 2014, p. 296). This theoretical lens has been employed to develop the conceptual model and to develop the present initial version of the Smart Cities MethodoLogy (SCML).

This methodology seeks to guide the implementation and identify the stages of intelligence according to the conditions of each city, in a step-by-step way, while aiming to generate public value for society. It combines top-down and bottom-up approaches in the planning of what to do or build, according to the particular characteristics of a city, aligned with potential directions for future research (Ismagilova et al., 2019).

The SCML is relevant because, while public administrators around the world have little doubt regarding the importance of investing in this area, there are still questions about how to make a city smart (Ruiz & Tigre, 2014). This answer depends on the characteristics of each city (Ruiz & Tigre, 2014) and the conditions of each locality, region or country (Macaya, 2017), which thus constitute the expected functional SCML requirements.

This paper analyzes the results of a first SCML evaluation cycle. It is being carried out as part of a study within the context of smart cities, in which the Design Science (DS) paradigm (March & Smith, 1995) and Design Science Research (DSR) method are applied (De Sordi, Azevedo, & Meirles, 2015; Dresch, Lacerda, & Antunes, 2015; Hevner, March, Park, & Ram, 2004; Vaishnavi, Kuechler, & Petter, 2004). It can be represented by the following question: Are the architecture and components of the SCML developed applying DS suitable and able to generate public value in smart cities?

By means of the analysis procedures employed, this paper presents the actual results obtained during the first DSR evaluation cycle, which involves representatives of the Quadruple Helix (Schuurman, Baccarne, & De Marez, 2012; Veeckman & van der Graaf, 2015). This preliminary evaluation based on the perception of those representatives led to the identification of a set of adjustments to the SCML aimed at improving its design and the initial version of each artifact, prior to checking the performance in practice, in the environment for which they were designed. It also enables the maintenance of items in the adopted DSR protocol, keeping it updated with due rigor.

The application of DS in the area of management and in the context of public organizations, as proposed in this study, has been adopted by a few recent publications (Bergvall-Kareborn, Eriksson, & Ståhlbröst, 2015). This demonstrates the relevance of the research considering it characterizes a differentiated, novel methodological approach towards investigations in the field of the smart city (De Sordi et al., 2015).

After this introductory section, the paper is organized as follows. Second section introduces the concept of smart cities from the Public Value perspective. Third section describes the methodological aspects adopted in this paper. Fourth section provides a brief overview of the SCML and its artifacts. Fifth section analyzes the results obtained in the first DSR evaluation cycle. Finally, sixth section offers some concluding remarks.

Smart Cities from a Public Value Perspective

There is no consensus in the scientific community regarding the definition of the concept of smart cities (Ismagilova et al., 2019), with different nomenclatures, contexts and meanings being adopted within different areas of knowledge. Originally, the term was applied in the limited context of technological infrastructure, but has evolved in recent years to cover a more systemic view, which considers the parties involved and their relationships, focused on sustainability and the improvement of quality of life.

In the present study, it is understood that smart cities should have the generation of public value as guiding principle and objective (Moore, 1995). Public value is considered an emerging and developing theory (Pang, Lee, & Delone, 2014). It suggests that the qualities of public organizations be assessed based on their ability to deliver the value expected by citizens (Cordella & Bonina, 2012).

In order to analyze the public value, first a conceptual framework was developed, characterizing the public return in six types of value, from investments in Information and Communication Technology (ICT) and identifying specific mechanisms that facilitate the generation of public value in ICT (Cresswell, Burke, & Pardo, 2006). Then, the framework was applied to the Open Government and expanded to incorporate quality of life as a seventh type of public value (Harrison et al., 2012; Harrison, Pardo, Cresswell, & Cook, 2011).

In this study, the framework was adapted to smart cities, as they are intended to address public issues, using ICT solutions, based on multiple stakeholders, in a partnership within the municipality (Manville et al., 2014). Smart Cities adopt ICTs to improve the efficiency and effectiveness of useful and necessary processes, typical activities and services in the city, in conjunction with various elements and actors, in an intelligent interactive system (Manville et al., 2014). This system can be part of the Systems of Systems (SoS), the development of which is challenging due to complex aspects such as infrastructure, integration and data management in decision support systems for all stakeholders (Bernardini et al., 2017).

Smart cities are understood to be able to accumulate public value, and as such provide specific benefits directly to individuals, groups or organizations (Harrison et al., 2012). Public value can be described in several types of impacts that capture possible outcomes from government actions (Harrison et al., 2011), and may be considered a multidimensional concept, because it deals with different dimensions of value, in measured monetary or non-monetary terms (Castelnovo, 2013).

In order to create public value and achieve these various types of impacts and outcomes, there are value-generating mechanisms that represent types of change that each stakeholder can experience (Harrison et al., 2011): efficiency, effectiveness, intrinsic improvements, transparency, participation, collaboration. The connection of at least one type of value with one of the value-generation mechanisms communicates and makes clear how a government action is expected to produce one or more public values (Harrison et al., 2012).

Therefore, it is possible to identify how each stakeholder perceives the changes which generate the public value. Determining the public value of smart cities requires careful analysis of multiple-stakeholder perspectives so that positive and negative impacts are identified and understood by administrators (Harrison et al., 2011; Harrison et al., 2012).

As in public administration in general, from the perspective of public value, the role of public participation is crucial and multifaceted in smart cities (Kelly, Mulgan, & Muers, 2002). This approach is adopted as a way of delivering and describing the value produced, when certain smart elements are achieved in smart cities. According to the adopted the theoretical lens and conceptual model (Porto & Macadar, 2017) that supported the initial version of SCML, smart cities can expand the capacity of value-generating mechanisms that create the various types of public value (economic, political, social, strategic, quality of life, ideological and stewardship) for multiple stakeholders (government, industry, academia, citizens and other stakeholders) who, in this context, are representatives of the Quadruple Helix (Schuurman et al., 2012; Veeckman & van der Graaf, 2015).

Method

The DS epistemological paradigm is adopted in the present study because the intention is to solve a real problem and to make a relevant scientific contribution of prescriptive nature in the area of management (Hevner et al., 2004), combining theoretical-methodological rigor and practical utility for society (Dresch et al., 2015; Vaishnavi et al., 2004). It has two fundamental macro steps (March & Smith, 1995): (a) To construct artifacts for the specific purpose of this research; (b) To evaluate and measure the performance of the artifacts in the environment for which they were designed to perform.

The paper focuses on the results obtained from an assessment of these artifacts in a first evaluation cycle (step b), being limited to the macro-level architecture of the SCML, which corresponds to its structural, conceptual and aggregator components, previously constructed in the study (step a). DSR is used to accurately design, develop and assess the SCML, which consists of the SCRM and the SCAM. These artifacts represent a possible solution to the problem of what to implement and how to evaluate smart cities, by which public value can be generated.

DSR is applied to operationalize the DS concepts and objectives (Dresch et al., 2015; Vaishnavi et al., 2004). Epistemologically, DSR incorporates characteristics of pragmatic, problem-centered, outcome-oriented and practice-oriented paradigms (De Sordi et al., 2015). The abductive scientific method is also employed within the DS paradigm. In the cognitive process, abduction is an intrinsic creative process, which allows new ideas to be introduced and suggests a possible solution to a practical problem or class of problems (Dresch et al., 2015).

Table 1 presents a summary of the steps in this cognitive process, the DSR method and this research, the relationships between the steps and the different procedures employed, based on Dresch, Lacerda and Antunes (2015) and Vaishnavi, Kuechler and Petter (2004). In the table, the steps shaded grey represent those that have yet to be completed. The DSR phases were formalized in a research protocol, based on Dresch et al. (2015), which has been updated as progress has been made and was omitted herein due to limitations of space. It consists of a methodological contribution that serves as a reference for the maintenance of the present study’s artifacts and, when completed, for the design and development of similar future artifacts in different contexts.

Table 1
Study design

Cognitive process	DSR	Research
	Awareness	Systematic literature review
		Identification of the problem
		Defining the scope
Abduction	Suggestion	Artifact 1: a methodology for smart cities
		Artifact 2: a reference model for smart cities
		Artifact 3: an assessment method for smart cities
Deduction	Development	Proposing the methodology
		Proposing the reference model
		Proposing the assessment method
	Evaluation	Conducting interviews with representatives of the Quadruple Helix Reviewing and refining the artifacts
		Conducting pilot assessments
		Reviewing and refining the artifacts in each pilot assessment
Reflection and	Conclusion	Discussing and divulging the partial results
Abstraction		Discussing and divulging the results
		Publication of the artifacts on a hotsite

In Table 1, the Awareness phase involves focusing on the problem (Vaishnavi et al., 2004). For this, a systematic review of the literature was carried out on the concept of the smart city, knowledge and artifacts existing in previous studies. The topic, class of problems, problematic situation, guiding question, and justifications regarding the importance and relevance of the research were defined based on the findings. The functionalities and expected performance of the artifacts, the requirements to be considered in the following phases and contingency heuristics of the problem were also described (Dresch et al., 2015).

Based on the analysis of the results of the previous phase, the knowledge and previous experiences, the artifacts were proposed in the Suggestion phase, for the purpose of improving the current situation and solving the identified specific problem, through a process that is creative by nature, a fruit of reasoning (Dresch et al., 2015) and the abductive scientific method (Vaishnavi et al., 2004), according to Table 1.

The Development phase deals with the deductive construction of the artifacts proposed for the solution of the problem (Vaishnavi et al., 2004). In this phase, the expected functionality and performance, specified requirements, design information and artifact architecture were considered. Internal environment characteristics and construction heuristics were constructed (Dresch et al., 2015). In summary, the artifacts were developed based on the literature, supported by a systematic review protocol, to establish the design as a whole, identify current evaluation methods and best practices in smart cities published in the literature, which have significantly contributed to the generation of public value and the transformation towards a smarter city.

Thus, the first versions of the SCML, the SCRM and the SCAM were built, which made it essential to ensure the validity of the research, through assessment and field-testing procedures. Construction heuristics were formalized from the design and development of the first version of the artifacts, which represents one of the contributions of the DS, as it generates a specific knowledge to be used in future evolutions of the artifacts themselves and/or in the design of new artifacts (Dresch et al., 2015).

The Evaluation phase involves evaluating the artifacts, while seeking to demonstrate that they can fulfill their function and solve the problem (Dresch et al., 2015). This phase employs the deductive scientific method in the cognitive process (Vaishnavi et al., 2004) and seeks pragmatic validity from the application of the constructed artifact in the context for which it was designed, ensuring a satisfactory solution to the problem (Van Aken, 2011). In this study, the forms of assessment were chosen according to the characteristics, performance requirements and applicability of the developed artifacts (Dresch et al., 2015), and this phase has not yet been completed, as shown in Table 1.

Semi-structured interviews were conducted, with representatives from the Quadruple Helix, different stakeholders and areas of knowledge. The choice of the interviewees was based on the foundations and the multidisciplinary nature of the Quadruple Helix, which is a model of the relationship between academia, government, industry and citizens. It is a way of generating knowledge and innovation for cities (Veeckman & van der Graaf, 2015). In this model, a fourth helix, consisting of citizens, is added to the traditional triple helix model. In this study, citizens were treated as end-users because they are an interested party involved in smart cities (Schuurman et al., 2012).

In the selection of the interviewees, convenience and balance between the numbers of representatives of the quadruple helix were adopted. During the survey, respondents’ suggestions were added and then the criterion of theoretical saturation was adopted, through the absence of new elements or information in new interviews (Eisenhardt, 1989). The search focused on criteria of previous experience in: (a) definition and implementation of reference models; (b) implementation and/or use of smart city solutions; (c) diagnoses, formal or informal assessments of previous technical and/or theoretical references; (d) teaching, research and/or project development in smart city or e-Gov; (e) managerial or governmental activities, at the municipal, state or federal level; (f) development and/or marketing of solutions for smart city projects; and (g) being a citizen, resident of, or visitor to a Brazilian city.

Each interview began with a presentation of the methodology and its components. Afterwards, once permission was obtained, the script-based interview that lasted approximately 30 minutes was recorded. Of the total of 23 interviews conducted from July 10 to 27, 2018, 17 were made face-to-face, at the interviewee’s preferred location. The rest were conducted remotely, via Skype and its screen sharing feature.

Table 2 shows a summary of the profile of the interviewees, with identification and information on the level and area of academic formation, professional field, time in the professional field, city, country acronym, age and gender. In the identification, the letter U stands for University representatives, I for Industry, G for Government and C for Citizens.

Table 2
Characterization of the interviewees

U01. Doctorate degree; Architecture and urbanism; Professor, Researcher, and Consultant; 14 years' experience; Canoas/BR; 36 years; Female. G02. Master's degree; Administration - systems analysis; Customer Relationship Manager and Teacher; 31 years' experience; Porto Alegre/BR; 51 years; Female. U03. Doctorate degree; Administration; Professor, Researcher, and Project Office Coordinator; 15 years' experience; Canoas/BR; 35 years; Male. C04. Master's degree; Informatics - systems analysis; Information Technology Researcher and Manager (IT); 15 years' experience; Porto Alegre/BR; 42 years; Male. U05. Master's degree; Architecture and urbanism; Professor, Researcher, Undergraduate and Postgraduate Coordinator; 40 years' experience; Porto Alegre/BR; 63 years; Male. G06. Specialization; IT management; Head of IT Dep.; 22 years' experience; Porto Alegre/BR; 42 years; Male. G07. Specialization; Computer networks; Director of ICT Department; Chairman of the ICT Governance Committee; 15 years' experience; Porto Alegre/BR; 34 years; Male. U08. Doctorate degree; Production engineering; Researcher, Professor, and Research Group Coordinator; 24 years' experience; Porto Alegre/BR; 39 years; Male. U09. Postdoctoral degree; Administration; Teacher and Researcher; 25 years' experience; Porto Alegre/BR; 47 years; Female. C10. Master's degree; Administration and law; Retired; 38 years' experience; Canoas/BR; 60 years; Male. U11. Master's degree; Law; Teacher and Researcher; 10 years' experience; Braga/PT; 36 years; Female. U12. Postdoctoral degree; Computer science; Teacher and Researcher; 28 years' experience; Porto Alegre/BR; 45 years; Female. G13. University graduate; Law; Coordinator of ICT Governance; 15 years' experience; Porto Alegre/BR; 34 years; Male. I14. University graduate; Administration; Consultant; 11 years' experience; São Leopoldo/BR; 31 years; Male. I15. Master's degree; Data processing; Consultant; 41 years' experience; Porto Alegre/BR; 62 years; Male. I16. Specialization; Electrical engineering; CEO; 30 years' experience; Porto Alegre/BR; 60 years; Male. I17. Specialization; Administration - systems analysis; Coordinator of Best Practices; 30 years' experience; Porto Alegre/BR; 57 years; Female. G18. Doctorate degree; Administration; Administrator and Quality Auditor; 35 years' experience; Porto Alegre/BR; 57 years; Male. U19. Master's degree; Production engineering; Professor, Researcher, and Undergraduate Coordinator; 18 years' experience; Portão/BR; 35 years; Female. U20. Master's degree; Administration; Professor and Researcher; 10 years' experience; Curitiba/BR; 29 years of professional experience; Male. C21. University graduate; IT management; Systems Support Consultant; 18 years' experience; Nova Santa Rita/ BR; 36 years; Male. U22. Doctorate degree; Informatics and administration; Teacher and Researcher; 39 years' experience; Curitiba /BR; 55 years; Female. U23. Master's degree; Forestry engineering; Professor, Researcher, and Postgraduate Coordinator; 21 years' experience; São Leopoldo/BR; 48 years; Female.

The first objective of the interviews was to discuss and validate the reference model’s architecture and components, whether the maturity levels and areas of competence are suitable, if they represented a logical sequence for the implementation of smart cities and if they were able to generate any type of public value.

The second objective concerned the evaluation method and process, its activities, required and generated products, roles, rules for characterizing the results and attributing the maturity level, in addition to its alignment with the reference model. This part of the script was only applied to those subjects experienced in diagnostics, formal or informal assessments, based on some other technical reference. Applying sociodemographic questions, the third objective of the script was to characterize the interviewees and the organizations they represented.

Subsequently, using content analysis, the interview recordings were analyzed individually (Dresch et al., 2015) in order to understand the subjects’ perceptions and the interpretation of the data obtained, in accordance with the study’s general objective. The results of this analysis provided useful insights that were applied in designing a new version of the evaluated artifacts. Due to its qualitative nature, the data analysis was made according to the artifacts designed and developed in the research and was carried out following to the dimensions and categories associated with the questions in the script, as shown in Table 3. Codes and sub-codes emerged through the application of coding techniques and described in detail in a memo (Dresch et al., 2015), which was refined and adjusted in each interview record analyzed.

Table 3
Analytical structure of the evaluation by representatives of the Quadruple Helix

Dimensions	Categories	Scripted questions
Reference model (SCRM)	Architecture	1.a In your opinion, are the model's architecture and components suitable?
		1.e Would you alter the model in any way?
	Maturity levels	1.b In your opinion, are the levels of maturity and the areas of competence suitable and do they represent a logical sequence for the implementation of smart cities?
	Areas of competence	1.c In your opinion, what other areas of competence are recommended to make a city smart?
	Public value	1.d In your opinion, can these areas of competence in the model generate public value? If so, what types of public value?
Assessment method (SCAM)	Assessment process	3.a In your opinion, are the steps in the assessment method suitable?
		3.f Would you change the assessment method in any way?
	Required products	3.b In your opinion, are the products required by the assessment method suitable?
	Generated products	3.b In your opinion, are the products generated by the assessment method suitable?
	Roles	3.c In your opinion, are the roles in the assessment method suitable?
	Scoring rules	3.d In your opinion, are the rules for characterizing the results and attributing the maturity level suitable?
Methodology (SCML)	Architecture	3.e In your opinion, are the assessment method and the reference model suitably aligned?

In DSR, if the artifacts are assessed to be unsatisfactory in the Evaluation phase, it is possible to return to the Awareness phase and repeat the subsequent phase (Vaishnavi et al., 2004). If necessary, further interviews and complementary analyzes can be made, to consider and treat the identified adjustments and confirm the practical usefulness.

When the results of the evaluations are satisfactory, the adjusted version of the SCML and its components will be made available to the general public. The contingency heuristics will be reviewed, thus enabling the description of the limits of the artifacts and their conditions of use and utility (Dresch et al., 2015). In this way, the external environment of the artifacts is characterized, the knowledge of which is useful for the design and development of new artifacts or redesign of the artifacts, as required due to changes in environmental contingencies.

In the Conclusion phase, the obtained results are presented and communicated. If there is a failure in the development of the artifacts, the method permits one to return to the first phase, since it may reveal an incomplete or insufficient level of awareness. In addition, the findings and lessons learned from the study may also reveal new research gaps and, if appropriate, a new cycle can be restarted (Vaishnavi et al., 2004).

This last phase is important because it is linked with reflection and abstraction in the cognitive process. It requires reflection on the problem and the proposed solution, in order to generalize the knowledge acquired while carrying out the research, to define the design proposition for a class of problems and to generate knowledge in the practical and theoretical fields (Dresch et al., 2015). They must only be formalized after several application tests and therefore will not be presented in this article. This phase emerges from the saturation of construction and contingency heuristics, in the Development and Evaluation phases of the DSR, respectively, and from the generalization of those heuristics, to a class of problems (Dresch et al., 2015).

According to DS, the knowledge produced by the research must be recognized by the academic community, as well as be useful and disseminated in the context in which it is to be applied (Dresch et al., 2015). Thus, the application of the DSR guidelines (Hevner et al., 2004), the conceptual model and the preliminary results were discussed in previous publications and gathered in the SCML interim hotsite (Porto, 2019). Therefore, they are not explored in this article, which aims to follow up and focus on the results of the following stages.

It should be emphasized that this strategy of communicating partial results made it possible to gather feedback and criticism from peers and specialists, based on reviews in the submission process and questions and/or comments received during scientific events in the area. The results obtained from the publications served as intermediary evaluations of the artifacts and constituted a rich source for improvements, while still in the Development phase.

The Smart Cities MethodoLogy (SCML)

The SCML is being developed within the DS paradigm, to be applicable to different city profiles, with the aim of providing a foundation for a gradual evolution in public administration practices in the context of smart cities. It brings together best practices and provides guidance on what to implement in an evolutionary and continuous way, potentiating the generation of public value, in accordance with the theoretical lens adopted. Additionally, it allows the stage of intelligence to be determined, according to the reality of each city.

The approach is pragmatic, conceived as a practical tool, to help government administrators understand and measure the level of intelligence and public value of smart cities. The details of the SCML and its architectural components are contained technical documents, work in an integrated fashion and consist of: a Smart Cities Reference Model (SCRM) and a Smart Cities Assessment Method (SCAM).

According to the conceptual model and technical basis used for the definition of SCML and the components of its architecture (Porto & Macadar, 2017), as a fundamental technical base, CERTICS was adapted to the context of smart cities. This methodology has not been previously applied in the software sector and represents an inspiring technical reference for the creation of similar instruments in other contexts and sectors (Alves, Salviano, & Stefanuto, 2015). Like CERTICS, SCML adopts the concept of competencies as a unit in the assessment of smart cities.

This approach is justified because the smart city is an evolutionary construction. Developing projects in this area and incorporating innovative solutions in a city requires a set of diverse capacities that must be wielded to achieve the desired objectives. A city is unlikely to be transformed within the period of a single term in political office, therefore it is necessary to seek a consensus regarding the smart city, based on a long-term view (Cunha et al., 2016).

In smart cities, it is necessary to maintain and expand existing competencies, while mobilizing them to reach an objective and, if necessary, disseminating and transferring them (Alves et al., 2015). This competency-based approach is justified because, in addition to considering the priorities defined in the strategic planning of each city, smart city projects generally intersect several dimensions (economy, environment, people, among others), such as a “public transportation solutions based on low carbon emissions, impacting mobility and the intelligent environment” (Bernardini et al., 2017, p. 20).

The main technical base of the SCAM is formed from the set of ISO/IEC 33000 standards (International Organization for Standardization/International Electrotechnical Commission [ISO/IEC], 2015), which provides a framework for measuring the capability of processes and requirements, guidelines and examples of reference models used by current models and methods, across different objectives and domains. They enable objective, unbiased, repeatable and representative assessment results (Alves et al., 2015). In the SCAM, the objective is to identify the level of intelligence of a given city, based on its capacity to meet certain predetermined results and areas of competence that are considered to be intelligent, which facilitate the generation of public value.

Thus, the organizational unit to be evaluated must present verifiable results that demonstrate the implementation of certain competencies, according to the level of maturity sought by the SCRM. To do so, the organizational unit must maintain an appropriate and dynamic cycle of use of its capabilities (Alves et al., 2015), for purposes of generating public value and innovative solutions for the city, according to its needs and conditions. In the SCRM, the public value perspective was primarily adopted to aid the identification and mapping of areas of competence and their respective expected results.

Technically, the SCML is based on aspects of the assessment method (SOFTEX, 2017) and the MPS maturity scale (SOFTEX, 2016a). The foundations of this model and its division into more mature levels makes it possible to meet the reality of the numerous small cities, ensuring visibility regarding the progress and results of improvements, in shorter time frames.

Architecturally, the SCML is structured in hierarchical layers, adapted from CERTICS (Alves et al., 2015). Following a logical structure, the SCML’s top-down approach is guided by the fundamental concept driving the development of the SCRM, which is that a smart city should enable the generation of public value. Whereas, in the bottom-up view, the evidence suggests the use and implementation of this structure in each organizational unit to be evaluated using the SCAM. Therefore, each city can be viewed in terms of its stage of evolution, depending on its current situation, its demands and needs. This section offered an overview of the SCML, its architectural components and main technical base. The next section details the first evaluation cycle.

Results

The study contributes towards theory in the field of smart cities by mapping good practices from the perspective of public value, using a reference model, the SCRM. It also provides a method that allows the level of maturity of smart cities to be assessed in terms of the attendance of certain intelligent elements, the SCAM. Together, these components constitute the SCML architecture, as they say how to do something, being classified as “Type V - Theory for Design and Action” (Gregor, 2006, p. 628).

The first version of each artifact was constructed following the DSR phases, in a creative process, based on: (a) analysis of the results of the systematic literature review; (b) the researchers’ prior knowledge and experience. The requirements were specified based on the technical references obtained in the review and the ISO/IEC 25010 (ISO/IEC, 2011) quality model, adapted to the context of smart cities and the objectives of this study.

These requirements and the expected performance of the artifacts were considered in the Development phase. Then, the first evaluation cycle contemplated a macro-level investigation of the SCML, its structural, conceptual and aggregating components, the results of which are the focus of this article. For this purpose, in the Evaluation phase, the perceptions of representatives from the Quadruple Helix were surveyed and the results are analyzed.

Analysis of the reference model (SCRM) dimension

Table 4 shows a summary of the improvements considered pertinent and relevant based on the analysis in the Reference Model dimension. The first column shows the categories of the analytic structure, followed by the codes and sub-codes emerging from the data.

Table 4
Results of the assessment of the SCRM by representatives of the Quadruple Helix

Categories	Codes	Sub-codes
Architecture	Figure	Legend
		Position of the maturity levels
Maturity levels	Nomenclature	Ubiquitous city
		City in optimization
		Human city
	Sequence	Sustainable city
		Knowledge-based city
		Sequential model
	Quantity	Maturity levels
Areas of competence	Nomenclature	Results
		Architecture and urbanism
		Electronic governance
	Sequence	Capturing financial resources
		Strategies and plans
		Technological infrastructure
	Description	Management knowledge
		Development of service and products
		Continuous improvement
	Alteration	Sustainable development
		Electronic governance

The Architecture category was considered suitable by all the interviewees and no adjustments were deemed necessary in this first evaluation cycle, only the inclusion of legends and repositioning the maturity levels in the Figure code, which represents the visual schema of the SCRM architecture, for the purpose of making it self-explanatory.

Organizing the structure into maturity levels was perceived as a systematic and orderly way of implementing the areas of competence. The areas consisting of expected results and those including guidelines and examples were considered suitable architectural components that aided the interpretation of the SCRM. For C10 “it establishes a relationship, which for me at least is very clear, understood, it makes sense ..., there is a logical and coherent relationship.” G06 says it is “appropriate because it establishes various levels and, according to the area of competence, produces results that generate the perception of public value.”

This is because the SCRM architecture and components are based on models that include recognized stages (CMMI Institute, n.d.; Lee & Kwak, 2012; SOFTEX, 2016b) and the competency approach of CERTICS (Alves et al., 2015). I16 even emphasized its innovative character:

“The reference model was built based on other models currently in the market ... so, I even see this as a form, an unprecedented application, right? One thinks about improving the maturity of a ‘company’ organization. Applying this concept to the maturity of a city, as far as the concept of smart city is concerned, is really an innovation.”

In Table 4, in the Maturity Levels category, pertinent and relevant adjustments were identified in the Nomenclature, Sequence and Quantity codes. Among the most significant modifications is the Nomenclature of the maturity levels, which is given importance in the implementation and responsibility logic in the grouping of the areas of competence. According to U12 there is a symbolic link with the name of the level: “the descriptions are ok, which is the most important thing ... but the names are very symbolically charged.” In short, we chose to make the following nomenclature modifications: (a) City in Optimization to Innovative City, aligning it with the notion of development and continuous improvement of competencies and not of an optimized process; (b) Ubiquitous City to Creative City to better reflect the purpose of the level; (c) Human City was changed to Governed City, in order to better align with the meaning, description and areas of competence of that level.

In the Sequence code a change was made in the order of 3 levels, considering the new nomenclatures and the analysis of the interviewees’ perception. The assumption of Sequential Model and Quantity of Maturity Levels was maintained. For confirmation purposes, future research will consist of empirical assessments, considering different maturity levels and contexts.

Regarding the Areas of Competence, adjustments were required in the codes and sub-codes in Table 4. Figure 1 illustrates the revised maturity levels and the areas of competence.

Figure 1
The SCRM – maturity levels and areas of competence

The detailing is being done in the SCRM artifact itself, based on new evaluation cycles, at the micro level. According to all those interviewed, in the Public Value category, within the Reference Model dimension, the areas of competence in Figure 1 can broaden the perception and enable the generation of different types of public value, for the different stakeholders, once implemented and developed in the municipalities. As is shown in the following examples: U19: “I believe that yes, furthermore, I think that from the most basic levels of maturity, I think it is already heading towards this, right?”; G02: “Here in this model I can see more than the perception of value, I can also see collaboration in the co-creation of value, value not being created only on the side of the government but involving society.”

It is recognized that the management approach based on Public Value corresponds to a paradigm that contrasts traditional public administration practices with the New Public Management paradigm (Stoker, 2006), and therefore may face resistance, especially in non-citizen-centered governments. Such an approach proved to be an acceptable perspective in the view of the interviewees and could become one possible way to improve municipal public management. This theoretical lens will be maintained in the research, in its conceptual model, in the fundamental concept of smart cities and as an inclusion criterion of the areas of competence.

Analysis of the assessment method (SCAM) dimension

Table 5 synthesizes the improvements considered pertinent and relevant, according to the perception of 17 representatives of the Quadruple Helix in the dimension Assessment Method, given the criterion of previous experience in diagnostics, formal or informal assessments based on some reference technique. The first column contains the analytical categories, followed by columns of codes and sub-codes, emerging from empirical evidence.

Table 5
Results of the assessment of the SCAM by representatives of the Quadruple Helix

Categories	Codes	Sub-codes
Assessment process	Inclusion	Organizational unity
		Initial training
Required products	Inclusion	Evidence
		Previous assessment report
Generated products	-	-
roles	Inclusion	Representative of the society
	Amendment	Assessment team
Scoring rules	Amendment	Type of scale

In Table 5, Assessment Process, the adjustments and improvements considered pertinent and relevant focused on the code Inclusion. In the case of Initial training an activity was incorporated to the SCAM: U01: “at some point I think there is a need for training …”; U08: “training in the model and method. I think it’s very important because the individual can already go mapping where they can get the data.”

In the Organizational Unit, the situation highlighted by I15 is to be investigated in larger municipalities: “When a large city wants to make an assessment ... involving the city as a whole, the complexity of this assessment is another concern that must be included in the method, how this type of assessment would be feasible.” I15 suggests “benchmarking in practical cases, procuring experience from those with experience of complex assessments, to learn how to deal with this level of complexity, due to the population size of larger cities, how an entire city would be assessed.”

In this category, most interviewees considered the process to be suitable, as U22 stated: “I think this is very traditional and I think your method is the coolest part.” This result is due to the SCAM being based on adaptations of established assessment methods, with evolutions arising from a significant amount of empirical experience. In addition, they pointed to “Validate assessment” step as a differential feature: U19: “I liked the assessment validation part you included, which ... is precisely to check the adherence between what the assessment team saw and what is, in fact, the observed reality ...”; G07: “like any assessment it is based upon perceptions, so if you do not have this time for joint reflection, you generate inadequate results ...” Being an external and independent assessment, this step was designed to verify whether the results reflect the reality, since the assessor does not experience the everyday life, and there may be an incorrect or inappropriate interpretation.

In the category Required Products, code Inclusion, sub-code Evidence, the definition of clear criteria for resolving possible impasses during the assessment process between assessment team and members of the organizational unit involved, regarding the evidence for expected results. According to I17: “I don’t know what the impasses would be, right, how they would be treated in some way ... I think it has to be very clear, because we are talking about a city, involving a community.” Further still, U08 commented on supplementing the evidence with what content published in the press, while mentioning in the SCAM: “such indirect evidence from the city’s newspapers, the news could be a preliminary source ...”

The assessment can be performed in a previously assessed organizational unit. Therefore, if there is any, Prior Assessment Report it can be included in the Required Products. Thus, the results and evolution of the municipality since the last evaluation will be taken into account in the assessment. For I17, “innovation does not happen overnight ... there will be things that are not fully addressed, because it is a process, it is not a fact ... we see evidence that it is ongoing with actions ... it is a process.”

In the Generated Products category, there were no adjustments to the first version of the SCAM. In Roles, code Inclusion, the role Representative of Society was suggested, thus differentiating the SCAM, in relation to the methods upon which it was based. It makes sense to include someone from outside the municipal public administration, particularly given the Public Value perspective and because the object is to assess public organizations. For G18, someone “who is not connected, who shares the view of the society, an external view of those who receive the services ...”

In the Amendment code, criteria were identified in the definition of the Assessment Team due to the responsibilities of the role, the effect of the judgments on results of the assessment and their recognition by agents from public organizations. For I17, “I think that assessment team should always have more than one member ... so the assessors can discuss the points .... A leader and a deputy, at least, otherwise you leave the leader in a very uncomfortable position, alone.” And for I15, “the question of prerequisites for training and qualifications …. Who would be competent to judge all this.” These criteria were clearly incorporated into the SCAM, seeking to avoid problems in the assessments.

In Scoring Rules and Amendment code, practically all respondents considered the rules and the scoring scale to be suitable. One suggestion is a limitation observed in the assessment methods used as bases, since they follow the same type of scale. According to U08, “I have a doubt regarding the four-point scale, but even so I think it is an assumption of the model the depends on the appraiser .... It assumes the assessor is highly qualified and a form of training that standardizes the process of observation.” It was decided to invest in training and not to deal with that alteration, because until now it fulfills the expected functionality. However, such a decision may be reviewed following further investigation.

Analysis of the methodology (SCML) dimension

The interviewees involved in assessing the SCAM also assessed the SCLM. From the analysis of the data in the Architecture category, the representatives of the Quadruple Helix perceived an alignment between the SCRM and the SCAM, which are the architectural components of the SCML that can respectively guide the implementation and assessment of smart cities. Regarding the integrated operation of the components, G02 pointed out: “for me there is complete alignment ... every time I say that it is on one level, I’m going to look for the expected results that are there in the model .... So, I found it logical, consistent, well-suited.” For G02, performing the assessment according to the SCAM and the SCRM makes it possible to “know where the level of maturity is in the city, but it also be used to consult regarding improvements in the maturity level ... I think it has these two objectives.”

In other words, the SCML and its architectural components, when used in an integrated way, enable the improvement of practices and competences, as an assessment according to the SCAM will reveal gaps in relation to the SCRM, offering recommendations in searching for the desired situation, besides the level of maturity reached in the assessment.

Improvements were considered pertinent and relevant in the Architecture category, with the emerging Amendment code. However, they will be dealt with later, in continuity with and based on the results of this study, as it is understood to be constitute a new component to be incorporated into the architecture. It is a platform to support the assessment and related administrative activities. According to I15, the SCML “needs to have a tool to make the assessment feasible. I think that, depending on the level of complexity ... it will become virtually unfeasible ... I think it would be a very important thing to have.” The platform can also publish current assessments, as suggested by G02: “this can be published transparently, wherever the city is, the people can also have that reference ..., one can also compare one city with another.”

In general, the first evaluation cycle revealed divergences and convergences between the results. Among the convergences is the possibility of generating public value for the different stakeholders in the Quadruple Helix, since smart cities are designed by the municipal public administration, considering the development of the areas of competence gathered in the SCRM. Further examples of convergence are the assessment team being formed by the Lead Appraiser and at least one more Adjunct Appraiser, as well as the general perception that the Beta version of the SCAM is consistent and robust for pilot evaluations.

The main divergences were found to be related to the SCRM’s structure and mode of organization. Two representatives of the Quadruple Helix questioned the assumption that the SCRM is a sequential model, providing another point of view regarding the problem under study. It was suggested that the model be continuous or multidimensional, given the different characteristics of the municipalities, which should be considered in future evaluation cycles.

Moreover, the results of the analysis procedures from the data obtained in this first evaluation cycle contributed and generated useful knowledge for the development of a better version of the evaluated artifacts, as can be seen in Figure 1, for example. This shows that the chosen form of evaluation was adequate and allowed the achievement of the intended objectives with the interviews. They also reinforce the importance of performing various cycles in the Evaluation phase, seeking to demonstrate that the artifact can fulfill its function and that such artifacts developed from design-based research are proof of their validity after the conclusion of this phase of DSR (Dresch et al., 2015).

Conclusion

The adoption of DS as an epistemological paradigm facilitates the advancement of knowledge and the rigorous development of useful artifacts, aimed at the resolution of the real research problem. Its products can help the academic community and professionals in public organizations gain greater understanding of what to implement and how to run assessments using a methodology regarding smart cities that can generate public value to society.

To date, DSR has been found to be efficient in the operationalization of this study. Its procedures enabled the refinement of the artifacts, the updating of the research protocol, future investigations and assessment cycles. The results of this article and previous related publications have contributed to the maturation of the artifacts and the conceptual model. In general, the goal of establishing a set of artifacts, providing a prescriptive scientific contribution, from a public value perspective, and solving a real problem in the public management of the cities is being achieved at each completed step of the DSR.

This article contributes to the analysis of the results of a first DSR evaluation cycle. Therefore, we recognize that the artifacts are in the developmental stage and future investigations are required, such as the execution of more DSR evaluation cycles, by conducting pilot assessments in the context of different municipalities. Consequently, access to the study’s artifacts will be controlled until adjustments are finalized and it is considered appropriate to release the 1.0 version.

However, even the intermediate version of each artifact is considered proof of the validity of this design-based study (Dresch et al., 2015), in which procedures that are aimed at ensuring their validity and completion are being adopted. In this article, these procedures enabled the identification of various adjustments and improvements, considered pertinent and relevant to the refinement of the artifacts.

An important lesson arising from research of this nature is that more time may be required for completion, depending on the complexity of the design and development of the artifacts. Accordingly, if necessary, with DSR it is possible to return to the initial step and repeat the subsequent ones to ensure a better understanding, conduct complementary research and make any necessary adjustments (Vaishnavi et al., 2004).

Regarding the present study, there are still limitations in the current versions of the artifacts that need to be treated in a timely manner, with caution and scientific rigor, while observing the construction heuristics and research protocol. These artifacts can be considered initial prescriptive tools, serving to designate a level of maturity to the SCRM in each organizational unit. In addition to being perceived as promoting innovation culture, which is essential in improving practices in municipal public administration, the artifacts facilitate the generation of public value for the stakeholders, when the areas of competence of SCRM are discussed and implemented in this context.

The effort involved in implementing the SCRM and performing the assessment according to the SCAM leads to a continuous process of improvement, when conducted with enthusiasm and commitment by the municipal public administration, due to the recommendations and possible referrals contained in the formal or informal assessment report or, through self-assessment. In addition, improving cities from the perspective of public value represents an opportunity for the development of the country and its population, as well as a possible response to management problems and a theoretical lens through which the scientific community can investigate research gaps.

Finally, among the methodological contributions to be obtained with the continuance of this research will be the research protocol and the heuristics, which detail how these artifacts were rigorously designed and developed, to be used for future evolutions of the artifacts themselves and/or the design of new artifacts, in different contexts and classes of problems.

References

• ABDI & INMETRO. (2017). Documento de referência: Ambiente de demonstração de tecnologias para cidades inteligentes Retrieved from https://docplayer.com.br/88471841-Documento-de-referencia-ambiente-de-demonstracao-detecnologias-para-cidades-inteligentes.html
  » https://docplayer.com.br/88471841-Documento-de-referencia-ambiente-de-demonstracao-detecnologias-para-cidades-inteligentes.html
• Alawadhi, S., Aldama-Nalda, A., Chourabi, H., Gil-Garcia, R. J., Leung, S., Mellouli, S., Nam, T., Pardo, T. A., Scholl, H. J., & Walker, S. (2012). Building understanding of smart city initiatives. In H. J. Scholl, M. Janssen, M. Wimmer, C. Moe, & L. Flak (Eds.), Electronic government (Vol. 7443, pp. 40-53). Berlin/Heidelberg, Germany: Springer.
• Alves, A. M., Salviano, C. F., & Stefanuto, G. N. (Orgs.). (Eds.). (2015). Certificação CERTICS: Um instrumento de política pública para inovação tecnológica em software Campinas, SP: CTI Renato Archer.
• Anthopoulos, L. (2015). Defining smart city architecture for sustainability. In E. Tambouris, H. J. Scholl, M. Janssen, M. A. Wimmer, K. Tarabanis, M. Gascó, B. Klievink, I. Lindgren, M. Milano, P. Panagiotopoulos, T. A. Pardo, P. Parycek, & Ø. Sæbø (Eds.), Electronic government and electronic participation (pp. 140-147). https://doi.org/10.3233/978-1-61499-570-8-140
  » https://doi.org/10.3233/978-1-61499-570-8-140
• Banco Nacional de Desenvolvimento Econômico e Social. (2018, March 2). BNDES lança cartilha sobre uso da Internet das coisas na criação de cidades inteligentes Retrieved August 1, 2018, from https://www.bndes.gov.br/wps/portal/site/home/imprensa/noticias/conteudo/bndes-lanca-cartilha-sobre-uso-da-internet-das-coisas-na-criacao-de-cidades-inteligentes
  » https://www.bndes.gov.br/wps/portal/site/home/imprensa/noticias/conteudo/bndes-lanca-cartilha-sobre-uso-da-internet-das-coisas-na-criacao-de-cidades-inteligentes
• Bergvall-Kareborn, B., Eriksson, C. I., & Ståhlbröst, A. (2015, August). Stakeholders in smart city living lab processes. Proceedings of the Americas Conference on Information Systems, Fajardo, Puerto Rico, 21.
• Bernardini, F. C., Viterbo, J., Vianna, D. S., Martins, C. B., Medeiros, A. P., Meza, E. B. M., Moratori, P. B., & Bastos, C. A. M. (2017). General features of smart city approaches from information systems perspective and its challenges. In C. Boscarioli, R. M. Araujo, & R. S. P. Maciel (Orgs.), I GranDSI-BR: Grand research challenges in information systems in Brazil 2016 - 2026 (pp. 25-40). Porto Alegre, Brazil: CE-SI - SBC. Retrieved from http://www2.sbc.org.br/ce-si/arquivos/GranDSI-BR_Ebook-Final.pdf
  » http://www2.sbc.org.br/ce-si/arquivos/GranDSI-BR_Ebook-Final.pdf
• Castelnovo, W. (2013). A stakeholder based approach to public value. Proceedings of European Conference on EGovernment, Como, Italy, 13.
• Castelnovo, W., Misuraca, G., & Savoldelli, A. (2013). Citizen’ s engagement and value co-production in smart and sustainable cities Paper presented at the International Conference on Policy Public, Grenoble, France. Retrieved from https://www.ippapublicpolicy.org/file/paper/1433973333.pdf
  » https://www.ippapublicpolicy.org/file/paper/1433973333.pdf
• CMMI Institute. (n.d.). CMMI development Retrieved from https://cmmiinstitute.com/cmmi/dev
  » https://cmmiinstitute.com/cmmi/dev
• Cordella, A., & Bonina, C. M. (2012). A public value perspective for ICT enabled public sector reforms: A theoretical reflection. Government Information Quarterly, 29(4), 512-520. https://doi.org/10.1016/j.giq.2012.03.004
  » https://doi.org/10.1016/j.giq.2012.03.004
• Cresswell, A. M., Burke, G. B., & Pardo, T. A. (2006). Advancing return on investment analysis for government IT: A public value framework Albany, NY: Center for Technology in Government, University at Albany.
• Cunha, M. A., Przeybilovscz, E., Macaya, J. F. M., & Santos, F. B. P. dos. (2016). Smart cities: Transformação digital de cidades São Paulo, Brazil: Programa Gestão Pública e Cidadania - PGPC.
• Department for Business, Innovation and Skills. (2013, October). The smart city market: Opportunities for the UK [BIS Research Paper No.136]. Retrieved from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/249423/bis-13-1217-smart-city-market-opportunties-uk.pdf
  » https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/249423/bis-13-1217-smart-city-market-opportunties-uk.pdf
• De Sordi, J. O., Azevedo, M. C., & Meirles, M. (2015). A pesquisa design science no Brasil segundo as publicações em administração da informação. Journal of Information Systems and Technology Management, 12(1), 165-186. https://doi.org/10.4301/S1807-17752015000100009
  » https://doi.org/10.4301/S1807-17752015000100009
• Dresch, A., Lacerda, D. P., & Antunes, J. A. V. J. (2015). Design science research: Método de pesquisa para avanço da ciência e tecnologia Porto Alegre, Brazil: Bookman.
• Eisenhardt, K. M. (1989). Building theories from case study research. The Academy of Management Review, 14(4), 532-550. https://doi.org/10.2307/258557
  » https://doi.org/10.2307/258557
• Gil-Garcia, J. R., Helbig, N., & Ojo, A. (2014). Being smart: Emerging technologies and innovation in the public sector. Government Information Quarterly, 31(S1), I1-I8. https://doi.org/10.1016/j.giq.2014.09.001
  » https://doi.org/10.1016/j.giq.2014.09.001
• Gregor, S. (2006). The nature of theory in information systems. MIS Quarterly, 30(3), 611-642. https://doi.org/10.2307/25148742
  » https://doi.org/10.2307/25148742
• Harrison, T. M., Guerrero, S., Burke, G. B., Cook, M., Cresswell, A., Helbig, N., Hrdinova, J., & Pardo, T. (2012). Open government and e-government: Democratic challenges from a public value perspective. Information Polity, 17(2), 83-97. https://doi.org/10.3233/IP-2012-0269
  » https://doi.org/10.3233/IP-2012-0269
• Harrison, T., Pardo, T., Cresswell, A., & Cook, M. (2011). Delivering public value through open government Retrieved from https://www.ctg.albany.edu/media/pubs/pdfs/opengov_pubvalue.pdf
  » https://www.ctg.albany.edu/media/pubs/pdfs/opengov_pubvalue.pdf
• Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75-105. https://doi.org/10.2307/25148625
  » https://doi.org/10.2307/25148625
• Ismagilova, E., Hughes, L., Dwivedi, Y. K., & Raman, K. R. (2019). Smart cities: Advances in research - An information systems perspective. International Journal of Information Management, 47, 88-100. https://doi.org/10.1016/j.ijinfomgt.2019.01.004
  » https://doi.org/10.1016/j.ijinfomgt.2019.01.004
• International Organization for Standardization/International Electrotechnical Commission. (2011). ISO/IEC 25010: Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models Retrieved from https://www.iso.org/standard/35733.html
  » https://www.iso.org/standard/35733.html
• International Organization for Standardization/International Electrotechnical Commission. (2015). ISO/IEC 33001: Information technology - Process assessment - Concepts and terminology Retrieved from https://www.iso.org/standard/54175.html
  » https://www.iso.org/standard/54175.html
• Kelly, G., Mulgan, G., & Muers, S. (2002). Creating public value: An analytical framework for public service reform Retrieved from http://webarchive.nationalarchives.gov.uk/20100416132449/http://www.cabinetoffice.gov.uk/media/cabinetoffice/strategy/assets/public_value2.pdf
  » http://webarchive.nationalarchives.gov.uk/20100416132449/http://www.cabinetoffice.gov.uk/media/cabinetoffice/strategy/assets/public_value2.pdf
• Lee, G., & Kwak, Y. H. (2012). An open government maturity model for social media-based public engagement. Government Information Quarterly, 29(4), 492-503. https://doi.org/10.1016/j.giq.2012.06.001
  » https://doi.org/10.1016/j.giq.2012.06.001
• Macaya, J. F. M. (2017). Smart cities: Tecnologias de informação e comunicação e o desenvolvimento de cidades mais sustentáveis e resilientes. Panorama Setorial da Internet, Ano 9(2), 1-16. Retrieved from http://cetic.br/publicacao/ano-ix-n-2-smart-cities/
  » http://cetic.br/publicacao/ano-ix-n-2-smart-cities/
• Manville, C., Cochrane, G., Cave, J., Millard, J., Pederson, J. K., Thaarup, R. K., Liebe, A., Wissner, M., Massink, R., & Kotterink, B. (2014). Mapping smart cities in the EU (PE 507.480) Brussels, Belgium: European Parliament's Committee on Industry, Research and Energy.
• March, S. T., & Smith, G. F. (1995). Design and natural science research on information technology. Decision Support Systems, 15(4), 251-266. https://doi.org/10.1016/0167-9236(94)00041-2
  » https://doi.org/10.1016/0167-9236(94)00041-2
• Ministério da Ciência, Tecnologia, Inovações e Comunicações. (2018). Estratégia brasileira para a transformação digital - E-Digital Retrieved from http://www.mctic.gov.br/mctic/export/sites/institucional/estrategiadigital.pdf
  » http://www.mctic.gov.br/mctic/export/sites/institucional/estrategiadigital.pdf
• Moore, M. H. (1995). Creating public value: Strategic management in government Cambridge, MA: Harvard University Press.
• Moore, M. H. (2014). Public value accounting: Establishing the philosophical basis. Public Administration Review, 74(4), 465-477. https://doi.org/10.1111/puar.12198
  » https://doi.org/10.1111/puar.12198
• Ojo, A., Dzhusupova, Z., & Curry, E. (2016). Exploring the nature of the smart cities research landscape. In J. R. Gil-García, T. A. Pardo, & T. Nam (Eds.), Smarter as the new urban agenda: A comprehensive view of the 21st century city (pp. 23-47). https://doi.org/10.1007/978-3-319-17620-8_2
  » https://doi.org/10.1007/978-3-319-17620-8_2
• Oliveira, A., & Campolargo, M. (2015, January). From smart cities to human smart cities. Proceedings of the Hawaii International Conference on System Sciences, Kauai, Hawaii, 48.
• Pang, M.-S., Lee, G., & Delone, W. H. (2014). IT resources, organizational capabilities, and value creation in public-sector organizations: A public-value management perspective. Journal of Information Technology, 29(3), 187-205. https://doi.org/10.1057/jit.2014.2
  » https://doi.org/10.1057/jit.2014.2
• Porto, J., & Macadar, M. (2017, June). Assessment methodology in smart cities based on public value. Proceedings of the Annual International Conference on Digital Government Research, New York, NY, USA, 18. https://doi.org/10.1145/3085228.3085276
  » https://doi.org/10.1145/3085228.3085276
• Porto, J. B. (2019). Smart cities methodology (SCML) Retrieved from https://sites.google.com/view/smart-cities-methodology
  » https://sites.google.com/view/smart-cities-methodology
• Ruiz, I., & Tigre, A. (2014). Smart cities além da tecnologia: Gestão e planejamento para a inovação urbana. Cadernos FGV Projetos, Ano 9(24), 86-95. Retrieved from http://hdl.handle.net/10438/11878
  » http://hdl.handle.net/10438/11878
• Schuurman, D., Baccarne, B., & De Marez, L. (2012). Smart ideas for smart cities: investigating crowdsourcing for generating and selecting ideas for ICT innovation in a city context. Journal of Theoretical and Applied Electronic Commerce Research, 7(3), 11-12. https://doi.org/10.4067/S0718-18762012000300006
  » https://doi.org/10.4067/S0718-18762012000300006
• SOFTEX. (2016a). MPS.BR-Guia geral MPS de gestão de pessoas Retrieved from https://www.softex.br/wp-content/uploads/2018/11/MPS.BR_Guia_Geral_RH_2016.pdf
  » https://www.softex.br/wp-content/uploads/2018/11/MPS.BR_Guia_Geral_RH_2016.pdf
• SOFTEX. (2016b). MPS.BR-Guia geral MPS de software Retrieved from https://www.softex.br/wp-content/uploads/2018/11/MPS.BR_Guia_Geral_Software_2016-com-ISBN.pdf
  » https://www.softex.br/wp-content/uploads/2018/11/MPS.BR_Guia_Geral_Software_2016-com-ISBN.pdf
• SOFTEX. (2017). Guia de avaliação - Parte I: Processo e método de avaliação MA-MPS Retrieved from https://www.softex.br/wp-content/uploads/2018/11/Guia-de-Avalia%C3%A7%C3%A3o-Parte-1.pdf
  » https://www.softex.br/wp-content/uploads/2018/11/Guia-de-Avalia%C3%A7%C3%A3o-Parte-1.pdf
• Stoker, G. (2006). Public value management: A new narrative for networked governance? American Review of Public Administration, 36(1), 41-57. https://doi.org/10.1177/0275074005282583
  » https://doi.org/10.1177/0275074005282583
• Vaishnavi, V., Kuechler, W., & Petter, S. (Eds.). (2004). Design research in information systems Retrieved February 6, 2016, from http://desrist.org/design-research-in-information-systems/
  » http://desrist.org/design-research-in-information-systems/
• Van Aken, J. E. (2011). The research design for design science research in management Eindhoven: (n.p.).
• Van den Bergh, J., Dootson, P., Kowalkiewicz, M., & Viaene, S. (2018, May/June). Smart city initiatives: Designing a project-level smart value assessment instrument. Proceedings of the Annual International Conference on Digital Government Research: Governance in the Data Age, Delft, The Netherlands, 19. https://doi.org/10.1145/3209281.3209288
  » https://doi.org/10.1145/3209281.3209288
• Veeckman, C., & van der Graaf, S. (2015). The city as living laboratory: Empowering citizens with the citadel toolkit. Technology Innovation Management Review, 5(3), 6-17. https://doi.org/10.22215/timreview/877
  » https://doi.org/10.22215/timreview/877

Guest editors: Aline Dresch, Daniel Pacheco Lacerda, Paulo Augusto Cauchick-Miguel, Joan van Aken, and Raymond Opdenakker

Editor-in-chief: Carlo Gabriel Porto Bellini

Editorial assistant: Luciane Kato Kiwara

Publication Dates

• Publication in this collection
  06 Apr 2020
• Date of issue
  2020

History

• Received
  15 Apr 2019
• Accepted
  21 Oct 2019
• Published
  10 Mar 2020