A novel technological performance measurement indicator: a smart manufacturing approach

Abstract The implementation of digital manufacturing technologies (DMTs) represents the beginning of transforming a manufacturing system towards a smart manufacturing system (SMS). Assessing the performance of the DMTs implemented is essential to meet the objectives in a SMS and allows identifying their usefulness. However, estimating this performance is a challenging task due to the heterogeneous characteristics of the DMTs, such as the origin of information, capacity, connectivity, etc. Although some SMS performance measurement metrics are known, none are intended to identify the performance of DMTs. This article follows a methodology for the construction of technological performance indicators and proposes a novel indicator based on the individual characteristics of the DMTs and the smart factory concept of interoperability. The proposed indicator allows approaching the behavior of one or multiple DMTs implemented simultaneously and introduces a quantifiable measurement that can be applied to any industrial process. It is noteworthy, that such an indicator is not present in the literature and may be of great interest to enterprises currently implementing DMTs related to SMS. The applicability of the indicator considering multiple DMTs is validated through an illustrative test case.


Introduction
Currently, many industries are transforming into a smart manufacturing system (SMS) by adopting advanced Information and Communication Technologies (ICT) to increase the level of automation and digitization of production, manufacturing, and industrial processes.The ICTs involved in this transformation are known as technological enablers, which are pillars of the industry 4.0 (I4.0) concept, such as emerging digital technologies, digital manufacturing technologies (DMTs), etc.The DMTs mainly mentioned in the literature include the Internet of things (IoT), cloud computing, integration systems, industrial robots, simulation, virtual reality, big data, cyber security, additive manufacturing, and various analytics techniques (Kamble et al., 2020).
From the academic point of view, some of the technologies examined are still rapidly transforming (Lu, 2017).This is due to the unprecedented increase in the number of technology sources and the heterogeneity in their characteristics (e.g., the origin of information, capacity, and connectivity).Integrating of the operation of the DTMs and their effective use allows achieving the SMS objectives (Wu, 2009).Therefore, analyzing the characteristics of DTMs is essential to discover their performance (Wu, 2009).Furthermore, DMTs can be implemented individually or in combinations, which reinforces the importance of exploring the interoperability feature.
Performance is an expression that compares the achievement of a process, device, or product to a given reference level and indicates a deficiency that can then be acted upon (Kamble et al., 2020).An exponential increase in the literature on DMTs can be seen in this direction.For example, the authors of Büchi et al. (2020) identify the main characteristics of I4.0 and its DMTs and verify the causal relationship between the degree of openness to I4.0 and performance.In Nara et al. (2021), the authors propose a model to analyze the impact of I4.0 technologies on several key performance indicators related to sustainable development.In Dalenogare et al. (2018), the authors discuss work contributes by examining the real expectations on the industry's future performance when implementing DMTs, providing a background to advance in the research on tangible benefits of I4.0.These studies mention the characteristics of DMTs, their positive relationship between them, their impact, and possible benefits for SMS (Büchi et al., 2020;Cugno et al., 2021).The literature reports performance measures of some DMTs, such as big data, evaluating characteristics such as volume of information, variety, and veracity (Ferrari et al., 2017).Furthermore, IoT performance by assessing features such as latency and unlimited throughput (Cappa et al., 2021).However, the literature lacks a measurement indicator where the performance is considered according to the interaction between different DMTs.
Therefore, this article proposes a performance indicator for DMTs to ascertain their usefulness, considering the individual characteristics of each DMT and the connection between them under the concept of interoperability.Compared to the available performance measurements of DMTs, the proposed indicator considers the state of operation from a technical point of view with a quantitative perspective.This paper uses the performance indicator design methodology proposed in Ibarguen-Valverde et al. (2017) as a guide, starting with a diagnostic to identify the testing features.Next, it compares of how interoperability between DMTs has been measured in the literature, followed by the proposal of the mathematical model.The development of the mathematical model was inspired by a structure presented in the Society of Automotive Engineers (SAE) J4000 standard developed by the SAE, establishing a numerical measure of the expected level and the actual value of a specific element (SAE, 1999).Finally, the proposed indicator is tested in an application case.It is noteworthy, that such an indicator is not present in the literature and may be of great interest to enterprises currently implementing DMTs related to SMS.
The rest of the paper will be structured is as fallows.In Section 2, an overview of the existing literature is presented, where we investigate available approaches to measure the performance of DMTs.In Section 3, we present the methodology for the construction of the technological performance indicators.Furthermore, we introduce the proposed novel indicator based on the individual characteristics of the DMTs and the smart factory concept of interoperability.

Theoretical background
The expected effects of the adoption of DMTs in SMS have been a topic of interest during the last years.The positive effects of adopting the DMTs in SMS are associated with their correct implementation, prepared human resources, the functionality of the technologies, among others (Dalenogare et al., 2018;Nara et al., 2021;Büchi et al., 2020;Zeid et al., 2019;Silva et al., 2022;Cugno et al., 2021).
The performance measures have been considered according to the characteristics of the DMTs.Some of these characteristics are analyzed from a qualitative point of view and others from a quantitative point of view.Quantitatively, the performance of DMT Big data has been addressed in (Ferrari et al., 2017), analyzing these three characteristics: volume, variety, and veracity.Table 1 shows the proposed measure for each feature.Similarly, DMT IoT performance has been addressed in Cappa et al. (2021) through latency estimation using the initiator-to-partner round-trip latency indicator (RTL IP) and the time offset indicator between Initiator and Partner (OFF IP).The performance of DMT Virtual reality has been analyzed based on quantitative characteristics such as projection efficiency, which considers the relationship between the spherical surface area and the calibrated projection area.Also, qualitative characteristics such as subjective quality have been examined (Hu et al., 2021).
In the vision of DMT functionality, one of the main requirements is to achieve interoperability across diverse technologies (Zeid et al., 2019;Frederico et al., 2021).
Interoperability is "the ability for two systems to understand one another and to use the functionality of one another" (Chen et al., 2008, p. 648).In other words, interoperability is the ability of two systems to interchange data, information, and knowledge (Lu, 2017), (Guédria et al., 2015).The Institute of Electrical and Electronic Engineers (IEEE) defines interoperability as "The ability of two or more systems or components to exchange information and use the information that has been exchanged" (Geraci, 1991, p. 42.The stated role of interoperability within the needs of SMS is to synthesize software components, business processes and application solutions through a diversified heterogeneous and autonomous process.To achieve a level of interoperability, establishing global standards and architecture guidelines is necessary for the development of SMS (Burns et al., 2019;Pedone & Mezgár, 2018).For example: Reference Architecture Model for Industry 4.0 (RAMI 4.0), Standards Landscape for Smart Manufacturing Systems, and National Smart Manufacturing Standards Architecture Construction Guidance.In Saturno et al. (2017) an analysis of the level of interoperability between systems within an existing automation platform (ISA-95) is presented.This work uses the AHP method for this evaluation, extracting criteria from the literature and the experience of experts.
Interoperability measures can be approached from both qualitative and quantitative perspectives.Qualitative measures use a rating scale made up of linguistic variables (for example, "Good," "Optimized," and "Adaptive") to rate a system.Qualitative measures are commonly used in maturity models.Quantitative measures define numerical values to characterize interoperability.In general, the rating scale is from 0 to 100%.For example, some approaches use equations to determine interoperability based on the "actual/expected" relationship (Pedone & Mezgár, 2018).Another approach is the evaluation of interoperability from two criteria: independent and dependent.Independent criteria include cost, time, and quality.Dependent criteria include the degree of coupling or compatibility, as well as who evaluate the system (Neghab et al., 2015).Formal interoperability measures based on the semantic relations between two information systems are presented in Table 2 (Yahia et al., 2012).

Maximal potential interoperability
, where   2 is the semantic relationships and is the total number of the expected semantic relationships to fully interoperate.

Minimal effective interoperability
, where   2e is the returned effective semantic relationships and   2 is the total number of the expected semantic relationships to fully interoperate.
Although the studies discussed above established the relevance of the functionality of the DMTs, the measure of this functionality, i.e., performance, for each DMTs is not mentioned.Moreover, the measure of interoperability between them is disregarded.The cited studies can be seen as a first approach to the study of the performance of DMTs.While studies have extensively examined measures of functionality of individual DMTs, there is a significant research gap in understanding the overall functionality of DMTs through interoperability.Figure 1 illustrates this research gap, highlighting the need for more research in this area.5/15

Performance indicator of digital manufacturing technologies: design and evaluation
The methodology proposed in Ibarguen-Valverde et al. ( 2017) for the construction of energy performance indicators was adapted for the design of the technological performance indicator, as shown in Figure 2. The methodology begins with a diagnostic phase that identifies the characteristics of the DMTs with which self-performance will be evaluated.In addition to identifying the interoperability characteristics with which mutual-performance will be evaluated.This step is followed by establishing the mathematical model that quantifies the performance of DMTs against their expected performance.The final phase involves the analysis and monitoring of the proposed indicator; here the numerical result and its significance is established.

Diagnostic phase
A performance indicator is a quantitative expression that compares the achievement of a process, device, or product to a given reference level and indicates a deficiency that can then be acted upon (Klovienė & Uosytė, 2019).That is why a performance indicator is proposed to measure DMTs to identify functionality.The performance indicator considers the individual characteristics (self-performance) and the connection between them (mutual-performance) under the concept of interoperability.
In studies such as Büchi et al. (2020), Bigliardi et al. (2020), Ardito et al. (2018) and Mabkhot et al. (2021), the list of DMTs is defined as: advanced manufacturing; additive manufacturing; augmented reality; simulation; cloud computing; industrial internet of things; cyber security; and big data analysis.These studies also define the characteristics of each DMT.A summary of the findings on DMTs is show in Table 3, where the characteristic identified in each DMT is evaluated according to the performance.Integration in two dimensions: internal versus external.The first (horizontal integration) concerns the integration and exchange of information among the different areas in the company.The second (vertical integration) concerns the company's relationships with its suppliers and customers Reference Architecture From Zeid et al. (2019), Sun et al. (2020) and Burns et al. (2019) it is evident that the concept of interoperability is highly related to the performance of DMTs.Although some tools measure the interoperability, none of them is directly designed for its use with DTMs from a technical point of view.Instead, they focus on a semantic vision of communication between systems, which results in qualitative characterizations.However, in Saturno et al. (2017), an analysis is carried out under the concept of interoperability of I4.0 in the areas shown in Table 4.Each area is assigned a value of three levels according to its degree of maturity.

Area Definition
Infrastructure Define a single standard for networks and communication protocols between the interoperable systems facilitating the communication between different suppliers for the same solution

Architecture standard
An architecture that meets the international standards, with the application of open protocols and ease of access to decrease the complexity and barriers in the integration of this architecture Software platform Flexible software platforms with easy remote access and availability of access by Web Services

Technology update
Define the potential of future integration with other systems.Software updates facilitated and hardware components exchange occurs in a modular way.
Hence, there is a gap in the literature for a quantitative indicator measuring the performance of DTMs based on the concept of interoperability.

Mathematical model
The proposed indicator is inspired by a structure presented in the Society of Automotive Engineers (SAE) J4000 standard developed by the SAE, which was developed to identify and measure companies' best lean manufacturing practices (SAE, 1999).The standard establishes a numerical measure of the expected level and the actual value of a specific element.In Lucato et al. (2019), the same standard was used to define the degree of general readiness for the adoption of I4.0 by a company considering all DMTs.However, neither SAE (1999) nor Lucato et al. ( 2019) measure performance directly.This paper proposes an indicator based on the parameter introduced in the SAE J4000 standard, that provides a performance rate of each DMTs and interaction performance rate between pairs of DTMs.
The proposed indicator, here referred to as , includes a contribution from the arithmetic mean of the performance evaluation of self-performance and the mutualperformance evaluation of all pairs of technologies, Equation 1: where, - is the number of technologies to analyze and ,  ∈ {1, 2, … , }, -  is the rate of performance of the DMT  relative to DMT , -and ℎ is the number of elements in the sum.
It should be noted that the rate of performance can be expressed as the matrix  consisting of all possible values of   , Equation 2: The sum of all the elements  is given by ∑ ∑    =1  =1 ; the condition  ≤  in Equation 1 removes the elements located above the main diagonal from the sum; this is done because the matrix is symmetric (  =   ).The number of elements in the summation is given by ℎ = (+1) . Note that the elements that correspond to  =  describes the self-performance of each technology.
Formally,   is represented as the ratio between the length of the projection of the vector   and the length of   ´, Equation 3. The vector   is referred to as the result vector and the vector   ´ is termed the standard vector, as shown in Figure 3.The vector   is composed of the measurements related to each characteristic evaluating the performance between DMT  and DMT , Equation 4. This evaluation is formally defined as   , with  representing the number of characteristics considered, A characteristic (capacity) has been defined based on the properties of DMTs (Büchi et al., 2020;Bigliardi et al., 2020;Ardito et al., 2018;Mabkhot et al., 2021;Rubio et al., 2018;Rajan, 2013) for assessing the self-performance of a technology (corresponding to the elements   for  = ): a. Capacity: This measure establishes the degree of fulfilment of the principal characteristic of the given technology.These characteristics are shown in Table 5 related to various DTMs.There are four possible responses to each statement that have been defined for measuring the characteristics described above.Each response is associated with a certain number of points indicating the observed degree of compliance.
Similarly, we proposed the analysis of the mutual-performance of DMTs based on the definition of interoperability considering four defining characteristics (i.e.,  = 4).These four characteristics have been previously used in Saturno et al. (2017) for analyzing interoperability; however, it must be noted that it is a general approach and can be tailored according to specific requirements: Four possible responses to each statement can be defined to measure each of these characteristics ( = 4).Each response is associated with a certain number of points: -Level 0 (L0): 0 points, DMT  does not share the characteristic  with DMT j.
The standard vector,   ´, is made up of the values that describe the optimal performance of the process under evaluation, Equation 5.This vector represents the maximum possible number of points that can be obtained.In this case, the maximum is 3, thus, and   ´ = √3 2 , we see that Equation 3 can be rewritten as Equation 6,

Indicator analysis and monitoring phase
The values that the elements of  can take vary between 0 and 1, where zero denotes the lowest level of performance and one the highest. depends on the range of values that   can achieve, which determines the possible vectors of   .
The mutual-performance monitoring of interoperability offered by the performance indicator indicates the least robust relationships between DMTs, presenting an opportunity to improve technologies efficiency.Similarly, individual DMT evaluation enables the identification of strategic actions to enhance their performance levels.
The  indicator has been designed to measure the performance of a set of technologies and identify areas for improvement in a production system, which can ultimately lead to greater efficiency and productivity.This type of indicator is a crucial tool for making data-driven decisions and improving overall performance in any industry.

Test case
In this section, we demonstrate how the proposed indicator is implemented.Considering a small and medium-sized enterprise manufacturing system for which the ISA95 standard has been identified as architecture; a DeviceNet network used to connect devices at the plant level; a monitoring and simulation process with online services that have the option to connect to the DeviceNet network.In addition to standard manufacturing devices such as milling machines, jointers, conveyors, and a manual station.The manufacturing system has an industrial robot in charge of highprecision tasks, adjustable production speed, and a DeviceNet communication module.
We identify the DMTs currently available in the process: Industrial Robots and Simulation.Therefore,  = 2.Each one of the evaluation characteristics selfperformance are identified.First, we identify the following evaluation characteristics: short cycle time for Industrial Robots, and representation level for Simulation.
The characteristics mentioned above are then graded using the four-point evaluation procedure defined in Section 3.2.Here we focus on the process used to obtain  11 related to the Industrial Robots performance.
1 (L3) Short cycle time is very high in the Industrial Robots DMT due to the characteristics of the installed robot.
Similarly, we focus on the process used to obtain  22 associated with the Simulation performance.
1 (L1) Representation level is average in the Simulation DMT due to simulation being only of the process.
Then, each one of the evaluation characteristics mutual-performance are identified: DeviceNet communication network for Infrastructure, ISA 95 for Standard architecture, and Platform online supervision for Software platform.
The characteristics mentioned above can then be graded using the four-point evaluation procedure, as defined in Section 3.2.Particularly, the procedure used to obtain the value of the parameter  21 , which is related to the mutual-performance between Industrial Robots and Simulation, is shown below: When all the elements   have been evaluated, the values of   can be obtained using Equation 6.Table 6 illustrates this process.Finally, using Equation 1 the total performance value can be obtained as in Equation 7 This means that the performance of the DMTs in this manufacturing system is 69%.

Discussion
It has been found in the literature that performance can be evaluated from differing perspectives using various metrics.One of these perspectives concerns the concept of interoperability of systems; we focused on this characteristic in this paper due to the diversity of the DTMs analyzed and the impossibility of using other performance metrics.
When compared to other indicators such as maximal potential interoperability and minimal effective interoperability, the proposed indicator was found to exhibit similar behavior to that observed when measuring interoperability between two elements.However, the known indicators related to interoperability measure the interoperability of systems in computational terms, whereas in the proposed parameterization, the interoperability between two technologies is measured in terms of compatibility.It is worth noting that the indicator proposed in this work measures the interoperability between all possible pairs of DTMs that are present in the process rather than individually, as it is done in other methodologies.
One possible deficiency of the proposed parameterization is that the evaluation of the defined characteristics (capacity, digitization, and technological upgradability) may be subjective when assessing the self-performance of each individual technology.
We note that the proposed methodology evaluates the overall performance of the DTMs in a SMS and makes it possible to separate the   data group; this permits the evaluation of individual metrics related to the process.This kind of indicator is intended for use in strategies related to continuous improvement and the adoption of DTMs in a SMS.

Conclusion
Monitoring systems and processes using well-defined metrics can reveals the gap between reality and expectations; this is how performance metrics can serve as a feedback tool, helping identify and correct potential problems.In this work, performance measures were investigated within the field of SMS; we identified performance measures of technologies from different perspectives; as a business model, the semantic vision of communication and value chain.An indicator was defined as a measurement pattern resulting from the observations undertaken in this methodology.
This work presented a practical approach to the development of a quantifiable indicator that can be adjustable to a given environment and manufacturing process; this work is based on the premise that the performance of DTMs can be characterized by the interoperability of the various technologies present within the process.
From a theoretical approach, this indicator starts with the identification and analysis of existing indicators and measures to estimate the functionality of DMTs.In this analysis, the heterogeneous capacity characteristics for each DMT are highlighted, such as latency for IoT, volume for big data, and storage capacity for cloud computing.
Measuring the performance of DMTs through a quantifiable indicator can have a significant impact on understanding how the potential of these technological tools is being used and identifying ways to increase their usage.The information obtained through the indicator can guide future DMT implementation processes in production systems, leading to system improvement and innovation.
In practical terms, the proposed indicator could aid in the decision-making process regarding which DMTs to implement in a production system.This would be done by evaluating the capacity characteristics of each DMT and the interoperability characteristics of the DMTs as a whole, such as communication architecture and standard platform.Furthermore, using the indicator could improve cost evaluation during the DMT implementation process by analyzing the interoperability characteristics of different devices in the market, enabling the identification of compatible devices before installation and potentially reducing technological misusage.
In future research, the proposed performance indicator could be incorporated into a decision-making model for the implementation of DMT in manufacturing systems.Evaluating the performance of a DMT assembly can determine its suitability in a specific manufacturing system.In addition, it is possible to extend the use in case studies with data from real manufacturing systems, performing simulations of the manufacturing system to identify the behavior of the process and how it is affected by the implementation of DMTs.

Figure 2 .
Figure 2. Methodology for the construction of technological performance indicators.

Figure 3 .
Figure 3. Ratio vector result vs vector standard.
a. Infrastructure: Network infrastructure level.For example: connection modules, smart network.b.Standard architecture: Architecture implementation level.For example: Industrial Internet Reference Architecture (IIRA), Reference Architecture Model for Industry 4.0 (RAMI 4.0), ISA 95, etc. c.Software platform: Software implementation level.For example: SCADA, Communication between architectures levels, Remote communication between architecture levels.d.Technological upgradability: Update level.For example: update between current devices, update adding new devices.

𝑒𝑒 1 (
L3) Simulation DMT fully shares the DeviceNet network infrastructure with the Industrial Robots DMT.Both the simulation software and the industrial robot have a DeviceNet network communication module. 2 (L2) Simulation DMT almost entirely shares the ISA 95 standard architecture with the Industrial Robots DMT.The architecture defined by the ISA95 standard does not directly mention any level of simulation.It can thus be integrated at the planning level. 3 (L3) Simulation DMT fully shares a software platform with the Industrial Robots DMT.Both are compatible with this platform, and it is possible to communicate through the initially defined network infrastructure. 4 (L1) Simulation DMT partially shares Technological upgradability with the Industrial Robots DMT.Due to the simulation tool having a finite number of inputs to simulate.

Table 6 .
Obtained values of   and   .