A value based method for measuring performance on Forensic Science service

Rodrigues, Claudio Vilela; Toledo, José Carlos de

doi:10.1590/0104-530X2137-16

Claudio Vilela Rodrigues

Posto de Perícia Integrada – SPTC, Polícia Civil de Minas Gerais – PCMG, Av. Deputado Renato Azeredo, 1342, CEP 37410-000, Três Corações, MG, Brazil, e-mail: claudiovilela@hotmail.com

Faculdade Cenecista de Varginha, Rua Professor Felipe Tiago Gomes, 173, Vila Bueno, CEP 37006-020, Varginha, MG, Brazil

José Carlos de Toledo

Departamento de Engenharia de Produção, Universidade Federal de São Carlos – UFSCar, Rod. Washington Luiz, Km 235, SP-310, CEP 13565-905, São Carlos, SP, Brazil, e-mail: toledo@dep.ufscar.br

	Details
Title
Purpose
Refers to
Goal
Formula
Frequency of measurement
Frequency of review
Who measures?
What do they do?
Data source
Who acts on the data?
What do they do?
What are the possible actions?
Notes and comments
Date/Document number

1. Performance Indicators for crime scene processing and evidence storage	Ability to find, secure, and process crime scenes
	Ability to locate and package properly physical evidence
	Ability to document crime scenes (sketches, notes, photographs, etc.)
	Ability to properly physical evidence for analysis or storage
	Ability to properly store and secure evidence (prevent alteration, destruction, or theft of evidence while stored)
	Ability to dispose of/destroy physical evidence when appropriate
	Proper use of forensics processing by investigators and police
2. Performance Indicators for Analyzing Evidence	Speed of analysis
	Size and age of backlogs
	Accuracy of analysis
	Workable ways to expedite and triage cases and analyses
	Ability to properly store and secure evidence (prevent alteration, destruction, or theft of evidence while stored)
	Ability to dispose of/destroy physical evidence when appropriate
3. Performance Indicators for Information Dissemination, Usage, and Utility	Dissemination of information from forensic analyses to investigators and prosecutors
	Understandability of information for investigators and prosecutors
	Utility of forensics information for cases, prosecutions, and clearances
	Availability of forensics information for investigators and prosecutors
	Users overall satisfaction (investigators, prosecutors, judges) with the forensics information they receive

Stage	Description	Procedures adopted in this research
1) Problem Identification	The problem and the value of the solution are specified, mainly due to the importance of the problem	The problem was identified both based on the gap in the literature and in meetings with the members of the researched unit
2) Set the solution goals	Seek a solution to the problem that is possible and feasible	Goal was set to design a customized solution to measure and evaluate the performance of the Forensic Science service and its professionals and promote strategic organizational alignment based on the value of the service
3) Project and development	Creation of the artifact (constructs, models, methods and instantiations)	Based on the problem, the literature review, the search for the existing artifacts, the dimensions of value that the service must deliver and in meetings with members of the unit and managers of the Forensic Science service researched, to develop a method to measure performance
4) Demonstration	Consists of demonstrating the use of the artifact. It may be by experimentation, simulation, or other appropriate activity	The method was demonstrated through experimentation in a Forensic Science researched unit, where it was partially implemented
5) Evaluation	Consists of any empirical evidence or logical proof that the artifact has contributed to the solution of the problem, which can be done in a variety of ways, such as the evaluation of practitioners and demonstration	The method was evaluated by the members of the researched unit and later by the top managers of the Forensic Science Service agency researched, according to criteria of utility, validity, quality and effectiveness, as suggested by Gregor & Hevner (2013)Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. Management Information Systems Quarterly, 37(2), 337-355..
6) Communication	Communicate the importance of the problem and the utility of the artifact and its relevance to the academy and practitioners through a scientific paper, for example.	The communication is being conducted through this paper, following the structure for scientific papers, whose method of research is DSR, as proposed by Gregor & Hevner (2013)Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. Management Information Systems Quarterly, 37(2), 337-355..

Criterion	Grade (0 to 10)	Weighted Average	Score (Grade × Weighted Average)
1 – Hierarchy	10	0.6	6.00
2 – Discipline	10	0.6	6.00
3 – Work quality	10	1.2	12.00
4 – Productivity at work	10	0.8	8.00
5 – Initiative	9	0.7	6.30
6 – Promptness	10	0.4	4.00
7 – Interest in participating in a training program	10	0.3	3.00
8 – Use in training program^* * It is not evaluated in all cases. Source: Research Data.	-	0.0	-
9 – Attendance	9	0.2	1.80
10 – Punctuality	9	0.2	1.80
11 – Time management and timing	10	0.4	4.00
12 – Rationality	10	0.4	4.00
13 – Communication	8	0.5	4.00
14 – Professional Ethics	10	1.6	16.00
15 – Adaptability	9	1.3	11.70
16 – Ability to work in team	9	0.8	7.20
Total Score			95.80

CRITERION	Dimension	Grade (0 to 10)	Weighted Average	Score (Grade × Weighted Average)
1 – Management Skills	Delegation of functions	10.00	0.8	48.00
	Staff developing	10.00
	Flexibility	10.00
	Participative Management	10.00
	Planning	10.00
	Decision Making	10.00
2 - Technical Competence	Work knowledge	10.00	0.6	28.80
	Effectiveness	8.00
	Efficiency	10.00
	Quality and productivity	10.00
	Professional qualification	10.00
3 – Interpersonal Competence	Communication	10.00	0.5	10.00
3 – Interpersonal Competence	Initiative	10.00	0.5	10.00
4 - Discipline	Attendance	10.00	0.4	12.00
	Professional ethics	10.00
	Proper use of equipment and facilities	10.00
Total Individual Performance Score				98.80

	Details
Title	Contribution to definition of authorship
Purpose	To measure how much the Forensic Science report contributed to link the suspect to the crime scene (or to innoccent someone wrongly accused) or to define the crime autorship
Refers to	Critical utility of the Forensic Science service
Goal	100% of the forensic reports must contain some traces that allow to identify the true author of an offence or to innocent someone wrongly accused
Formula	Does the Forensic Science report link the suspect to the crime scene (or innocent someone wrongly accused) or define the authorship? Likert nominal scale response
Frequency of measurement	Each report, or those considered as priorities by the Government and / or Forensic Science managers
Frequency of review	Annual
Who measures?	Clients (judge, prosecutor, police investigator and defense) and forensic scientists
What do they do?	Measure how much the Forensic Science service contributes to the identification of the perpetrator of a crime (or to innocent someone wrongly accused), using the scientific knowledge and technologies applied to Forensic Science
Data source	Forensic Science report
Who acts on the data?	Forensic scientist responsible for the report
What do they do?	They collect the traces, perform complementary and laboratory tests and prepare the final Forensic Science report
What are the possible actions?	Constant staff training and investment in technological innovations
Notes and comments	This is a critical KPI for Forensic Science service

Who called Forensic Science service?
(___) CIAD^* * Note: CIAD = Integrated Center of Service and Dispatch or similar. Source: Authors. (__) Patrol Police officer (__) Police Investigator (___) Other: __________

1) The information contained in the request was sufficient.
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree

2) How long after the call did the forensic scientist arrive at the crime scene and how far has he traveled?
____________hours/minutes and the distance traveled was _____________km. (___) Not applicable

3) Who was present at the crime scene? You can pick more than one.
(__) Patrol Police (__) Investigation Police (__) Highway Patrol (__) Firefighter (___) Not applicable

4) The crime scene was perfectly isolated.
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree (___) Not applicable

5)The crime scene was prefectly preserved.
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree (___) Not applicable

6) The material was properly conditioned (for complementary and / or specialized internal examinations)
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree (__) Not applicable

7) The forensic scientist had all the necessary resources to conduct an examination of excellence.
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree
If lacked all, what resources were missing? ______________________________________________.

8) The Forensic Science service followed SENASP SOPs, or documented SOPs.
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree
( ) There is no SOP for this forensic exam ( ) There is SOP, but it is not implemented

9) The chain of custody (traceability of evidence from the crime scene to custody) was followed
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree

10) The Forensic report allows to link the suspect to the crime scene (or to innocent someone wrongly accused), or elucidate the event
( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree

11) The report was provided in ______________days.

12)Insert comment that you consider relevant: _______________________________________________
____________________________________________________________________________________

1) The Forensic Science service has implemented SENASP / MJ SOPs (or their own) in which % of the number of units?
( 0 ) 0% ( 1 ) 1 to 10% ( 2 ) 11 to 20% ( 3 ) 21 to 30% ... ( 10 ) 90 to 100%

2) Level of autonomy of the Forensic Science service: Organizational structure. To whom is the service subordinated?
(0) To the Investigation Police (4) To the Investigation Police with management autonomy (6) Subordinated to the Department of Public Safety (8)To the Department of Public Safety, with budgetary and management autonomy (9) To the State Governor (10) It has financial, administrative, budgetary and functional independence

3) Percentage of institutionalization of the Chain of Custody management
( 0 ) 0% ( 1 ) 1 to 10% ( 2 ) 11 to 20% ( 3 ) 21 to 30% ... ( 10 ) 91 to 100%

4) Percentage of units with the minimum number of forensic scientists to operate 24 hours a day 7 days a week
( 0 ) 0% ( 1 ) 1 to 10% ( 2 ) 11 to 20% ( 3 ) 21 to 30% ... ( 10 ) 90 to 100%

5) Percentage of clerical workers in relation to the number of forensic scientists:
( 0 ) 0% ( 1 ) 1 to 10% ( 2 ) 11 to 20% ( 3 ) 21 to 30% ... ( 10 ) 90 to 100%

6) Percentage of technologies available by Forensic Science units or of the agency
( 0 ) 0% ( 1 ) 1 to 10% ( 2 ) 11 to 20% ( 3 ) 21 to 30% ... ( 10 ) 90 to 100%

7) Service efficiency: Average cost per unit and forensic report
e = Ʃ cost ($) dos inputs (installations + staff)/ ∑of the reports issued => e = Average cost ($)/report

8) Productivity by forensic scientist, per unit and overall service
Productivity = (Ʃ of issued reports/Ʃ required exams) x 100%

9) Percentage of backlog reports by forensic scientist, per unit and overall service _____________%

10) Number of complaints per year, per forensic scientist unit and overall service ___________________________________.

Dimension	Indicators	Grade (0 to 10)	Weighted Average	Score (Grade × WeightedAverage)
Internal processes	% of reports that used SOPs (for existing cases)	4	1.25 (25%)	5.00
Internal processes	Chain of Custody (traceability of evidence)	2	1.25 (25%)	2.50
Clients	Number of complaints / year (target: maximum 3 complaints / year)	10	1.25 (50%)	12.5
	Contribution to elucidation of the event	9		11.25
	Contribution to define authorship (or to innocent someone wrongly accused)	8		10.00
	Provision of forensic science reports timely	9		11.25
Efficiency	Annual productivity ((Ʃ of issued reports/Ʃ required exams) x 100%)	8.88	1.25 (25%)	11.10
Efficiency	% of reports issued after the deadline / year	9.28	1.25 (25%)	12.50
TOTAL				76.10

Dimension	Indicators	Grade (0 to 10)	Weighted Average	Score (Grade × Weighted Average)
Internal processes	Percentage of SOPs implementation	4	1.25 (25%)	5.00
Internal processes	Chain of Custody (traceability of evidence)	4	1.25 (25%)	5.00
Clients	Number of complaints / year (target: maximum 24/ year)	10	1.25 (50%)	12.50
	Contribution to elucidation of the event	8		10.00
	Contribution to define authorship (or to innocent someone wrongly accused)	8		10.00
	Provision of forensic reports timely	9		11.25
Effiency	Annual productivity ((Ʃ of issued reports/Ʃ required exams) x 100%)	8.72	1.25 (25%)	10.90
Effiency	% of delayed reports / year	8.98	1.25 (25%)	11.23
TOTAL				75.88

Dimension	Indicators	^{^} Assigned grades are a simulation, to demonstrate the use of the method. Source: Authors. Grade (0 to10)	Weighted Average	Score (Grade × Weighted Average)
Internal processes	Percentage of SOPs implementation	2	0.75 (15%)	1.50
Internal processes	Chain of Custody implementation percentage	3	0.75 (15%)	2.25
Clients	Number of complaints / year / agency	8	1.0 (40%)	8.00
	Contribution to elucidation of the event	8		8.00
	Contribution to define authorship (or to innocent someone wrongly accused)	7		7.00
	Provision of forensic science reports timely	7		7.00
Efficiency	Annual productivity / agency ((Ʃ of issued reports/Ʃ required exams) x 100%)	7.5	0.75 (15%)	5.63
Efficiency	Percentage of delayed reports / year / agency	6	0.75 (15%)	4.50
Exemption	Organizational Structure: subordination and budget	4	1.5 (15%)	6.00
Universalization	Percentage of units with the minimum number of forensic scientists	10	0.75 (15%)	7.50
Universalization	Percentage of minimum equipment required (based on a list obtained by consensus for the agency)	2	0.75 (15%)	1.50
TOTAL				58.88

*Ranking*	Forensic Scientist	Total of Exams	Concluded Reports	% of Concluded Reports	% of Delayed Reports	No of complaints / year
1^st	Forensic Scientist H	815	797	97.79	0.00	0
2^nd	Forensic Scientist A	304	270	88.82	7.24	2
3^rd	Forensic Scientist B	393	349	88.80	7.89	1
4^th	Forensic Scientist C	305	276	90.49	5.90	4
5^th	Forensic Scientist G	372	329	88.44	11.29	0
6^th	Forensic Scientist D	333	239	71.77	26.73	4
7^th	Forensic Scientist E	302	203	67.22	28.48	8
8^th	Forensic Scientist F^* * F is the manager. In 2014, the unit received two forensic scientists (G and H). Source: Research data.	8	7	87.50	12.50	0
TOTAL/MANAGEMENT		2832	2470	87.22	10.20	19

Year	Evaluated	Total of Exams	Concluded Reports	% of Concluded Reports	% of Delayed Reports	No of complaints/year
2012 ^* * Data for 2012 were recovered for comparison purposes. Source: Research data.	Manager/Unit	2100	1194	56.86	43.14	Not avaiable
2013	Manager/Unit	2123	1648	77.63	20.91	62
2014	Manager/Unit	2184	1803	82.55	15.93	46
2015	Manager/Unit	2832	2470	87.22	10.20	19

A v e r a g e \cos t / C a s e = \frac{(A v e r a g e c o m p e n s a t i o n \times t e s t int e n s i t y)}{(l a b o r p r o d u c t i v i t y \times l a b o r \cos t s h a r i n g)}

I_{\exp e r t e f f e c t i v e n e s s} = n l [(\sum_{i = 1}^{n} I_{c e l e r i t y} / n) . \sum_{i = 1}^{n} T_{e s t i m a t e d} . Q_{d e a d l i n e} / Q_{t o t a l}]

[1] Source: Neely et al. (1997Neely, A., Richards, H., Mills, J., Platts, K., & Bourne, M. (1997). Designing performance measures: a structured approach. International Journal of Operations & Production Management, 17(11), 1131-1152. http://dx.doi.org/10.1108/01443579710177888.
http://dx.doi.org/10.1108/01443579710177... , 2002Neely, A., Bourne, M., Mills, J., Platts, K., & Richards, H. (2002). Getting the measure of your business: strategy and performance. Cambridge: Cambridge University Press.).

Forensic Science Report No ____________	Forensic Science Unit: __________________
Respondent: ( ) Court Judge ( ) Prosecutor ( ) Defender ( ) Police Investigator ( ) Forensic Scientist
1. The Forensic Science report contributed in some way to the elucidation of the event. ( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree
2. The Forensic Science report linked the author to the crime scene (or innocented someone wrongly accused), or provided sufficient evidence to establish the authorship of the crime. ( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree ( ) Not applicable
3. The Forensic Science report was made available in the time required for the exercise of your activity. ( 0 ) Strongly disagree (4) Disagree (6) Partially agree (8) Agree ( 10) Totally agree
4. If you like, insert any comments you consider relevant to improve the service delivery: _____________ ____________________________________________________________________________________

Brasil

Brasil

A value based method for measuring performance on Forensic Science service

Abstract: