Creation , validation , and reliability of a shooting simulator instrument for reaction time evaluation

"Creation, validation, and reliability of a shooting simulator instrument for reaction time evaluation." The aim of this study was to develop, validate, and verify the reliability of a shooting simulator instrument for reaction time evaluation. 90 Santa Maria Air Base military personnel participated on the study. Software was developed for use with an electronic gun where participants performed two shooting task tests: simple reaction time and choice reaction time. The results of concurrent validity were satisfactory, no significant differences were found between the two instruments and good agreement was observed. The reliability results were significant in both tests. The instrument can be used for research purposes, or for the purposes of military training as a simple, low-cost tool involving speed, accuracy, and decision-making shooting tasks.


Introduction
Motor reaction time is a measure widely used in research as an indicator of sensorimotor performance by analyzing the time spent by subjects when performing tasks 1 .Researchers have investigated the reaction time in different populations, such as children 2 , adults 3,4 , the elderly 5,6 , and special populations 7 .Reaction Time (RT) is the time interval between the presentation of a stimulus and the beginning of an answer 8 .Accordingly, it can be simple reaction time (SRT), when there is a single stimulus to be answered with only one possible answer, or choice reaction time (CRT), when there are two or more stimuli to be answered with the most adequate response among several possible alternatives 8 .
In broad terms, the TR varies according to the sex, age, and task complexity.Men in general are faster than women 9 , with maximum performance in this variable being reached between 20 and 30 years old and later declining 2 .
When the reaction time is extremely fast, conducting research with this variable requires the use of specific software to register the time between the presentation of stimuli and the early movements 3 .Over time, different tools have been developed for the RT assessment using different stimuli, especially visual and auditory, in simple and complex tasks, among which manual tasks such as pressing a mouse button or keys of a keyboard on a computer are the most common [10][11][12] .
Manual tasks have been used in different contexts, however, to evaluate the reaction rate in more specific situations, so it is necessary to create tools that enable the reliable execution of the appropriate tasks in the context in which individuals are embedded.Thus, in situations where specific skills are required, such as in occupations involving the use of firearms, it is appropriate that the evaluation is as close as possible to reality.In the case of military professionals, for example, the skill required to achieve shooting efficiency is acquired by means of learning the fundamentals of shooting in activities where directed attention to important events or objects within a scene in real time is required 13,14 .
In this regard, it is important that the assessment tool simulate, even in a simplified way, the variable component of reality, and reproduce the real system's main features, which will make the assessment more accurate and reliable.Therefore, the objective of this study was to develop, validate, and establish the reliability of a shooting simulator instrument for the evaluation of reaction time.

Method
The study complied with the standards and criteria required by Law 196/96 of the National Health Council (CNS).The research project was submitted to the Ethics Committee on Human Research of the Federal University of Santa Maria (UFSM) and approved under 16764113.5.0000.5346protocol.All participants were informed about the research purpose and signed an Informed Consent Form (ICF).

Participants and Procedures
The group was intentionally selected, and was composed of male military personnel (n = 90), between 18 and 50 years old, belonging to Infantry Battalions and Air Police from Santa Maria Air Base in (BASM).The factors adopted as inclusion criteria were as follows: being in the pre-determined age group, accomplishing the same shot tasks related to frequency and type of training, and having not reported diagnoses of disease, or physical or mental limitations that prevented testing.
All military participating in the study underwent the same training developed by BASM every six months.Training activities involved free shot, in which the individual is positioned at a certain distance from a central target with a total of ten weapon shots; and the instant defense shot, in which targets are presented in black, representing the enemy and in white, representing the absence of danger, in which the individual must fire shots only at black targets.The military positioning for these activities can be standing, sitting or squatting.
Assessments occurred on the premises of BASM during working hours.Before starting data collection all participants were briefed and informed about the objectives and procedures adopted in the research, and they completed an interview with identification information.Then an interview about possible variables that could influence the testing, such as sleepiness or quality of sleep at the night before, tiredness, alcohol or caffeine intake, and physical activity practice was performed (Figure 1).After the tests TRS and TRE were performed.These procedures were performed in two assessment days (test and retest), within an interval of 24 hours.
The subjects were divided into three groups (n=30 in each) according to age: G1 (between 18 and 29 years old); G2 (between 30 and 39 years old) and G3 (between 40 and 50 years old).

Instrument construction
The reaction time in shooting task test (RTST) consisted of specific software using an RF Top Gun electronic wireless pistol compatible with Windows Vista (32bit), XP (32bit), and all types of screen monitors (CRT/LCD/Plasma/Projectors).The software, built on the Java SE 7 platform, regardless of a computer platform, is composed of two tasks (SRT and CRT) using images that are projected on a computer screen as stimuli in the individual evaluation.The main tasks are to shoot (pulling the trigger) as quickly as possible and select the most appropriate response according to the presented stimulus.For the elaboration of the images used in the software, actors were hired and image rights assigned by contract.In to the test, the stimulus is represented by the image of a subject characterized as a criminal, dressed in black, or the image of a victim, dressed in white (Figure 2).The stimuli are presented twenty times for each task, randomly, with a minimum time of 100 milliseconds.
In order to accomplish the tests, a closed room is needed so that the lighting conditions will not interfere with the action of the gun.The test begins with the individual in a shooting position (both arms forward and gripping the gun with both hands), with the gun pointing toward the target located on the computer screen (Figure 2).Minimum body movement is required, since the reaction time does not involve performing movements, but only the onset of response.At the end of each task the software generates the mean time, in milliseconds, from the median 10 attempts (excluding the 5 best and worst times) and further, the number of errors in advance or shots off target, as well as response errors.After calculating the mean value, a time of 84 milliseconds is deducted from mean value due to the transmission error between the output timing of the stimulus of the gun and the arrival of the response on the computer screen, considering the wireless condition of the electronic gun.
Both tasks are described below: Task 1: Simple reaction time In this task only the gun trigger is used, being pulled with the index finger of the hand of choice.A single stimulus appears on the screen represented by the image of a criminal, so that, when viewing the stimulus, the individual should pull the trigger as fast as possible.The SRT mean and the number of anticipation errors is generated.

Task 2: Choice reaction time
In this task two buttons are used (the trigger and a button below the trigger of the gun).The image of the criminal or the victim was randomly presented on the screen.When viewing the image of the criminal, the subject should pull the trigger, whereas when seeing the image of the victim, the subject must press the button that is below the trigger of the gun.To develop this task the subject is instructed to place the index finger of one hand (of his choice) on the trigger and the other hand on the button below the trigger.The software calculates the CRT and generates the number of response errors and errors in advance or shots off target.

Statistical Analysis
Data were analyzed using SPSS version 14.0 for Windows with significance level of 5%.For a description of the sample, we used descriptive statistics (mean, standard deviation, and percentage frequency).The normality of variables was verified by the Shapiro-Wilk test.To analyze the validity of the instrument a t test was used for paired samples and the Bland-Altman diagrams for the application of the gold standard method 15 .We used the intraclass correlation coefficient to test the reliability of the instrument, considering ICC > 0.70 as appropriate values 16 .

Analysis of Concurrent Validity and Reliability
To establish the validity of RTST the gold standard method was used 15 .The level of concurrent validity was determined by comparing the measurements obtained in the reaction time test that were created and validated in a previous study, and used as a benchmark to measure the proposed new instrument 11 .A paired samples t test was initially performed to compare the mean values between the two instruments, and finally the Bland-Altman diagrams were produced for the analysis of agreement between the two measurements 16 .
The reliability was investigated by means of temporal measures stability (reproducibility test, and retest).Testing for both tasks SRT and CRT were performed in two days, with an interval of 24 hours between each evaluation.For the test and retest procedures the environmental conditions were controlled, with both assessments done in the morning at the same place, a closed room with a temperature of 28 ° C.
To analyze the reproducibility of the instrument the intraclass correlation coefficient (ICC) was used based on the parameters of the repeated measures ANOVA 16 , as shown below: ICC = (MSs-MSe) MSs in which MSs = mean square for subjects MSe = mean square for error

Results
The 90 subjects were divided into three groups for analyzation (n=30 in each group) according to age: G1 (between 18 and 29 years old); G2 (between 30 and 39 years old) and G3 (between 40 and 50 years old).The data distribution analysis using the Shapiro-Wilk test showed normality in all groups (p>.05).
Regarding concurrent validity, as shown in Table 1, for the three analyzed groups, no significant differences (p <.05) between the two tests in the tasks of SRT and CRT were found.
The results of the analysis of agreement between subjects for the three groups are shown in figures 3 and 4. It is possible to observe that there was agreement between both instruments in the task of SRT as in the CRT, which demonstrates the validity of RTST in all groups evaluated.
The results concerning the reliability of the instrument by means of test-retest procedure are shown in Table 2.
As shown in table 2, the results for the reliability of RTST in the three groups were significant (p < 0.05) within the reference values considered adequate (ICC > 0.70).ANOVA for repeated measures found no significant differences between test and retest ratings (p > 0.05), confirming the reproducibility of the instrument.It is also possible to observe that the values concerning the number of errors in both tests, and the average error by anticipation (SRT) were lower than the average errors of response choice (CRT).

Discussion
The RTST was designed based on the use of electronic technologies for shooting tasks simulation.The proposed tests allow the evaluation of reaction times in simple situations where a single answer is possible, and in more complex situations, when it is necessary to choose the most appropriate answer for making the right decision according to the presented situation.The instruments developed for the assessment of RT usually consist of software using the computer keyboard or mouse to perform manual tasks.These tasks are often used for the evaluation of RT in different contexts and situations.Studies with swimmers 3 , volleyball players 17 , soccer players 18 , military pilots 19 , among other contexts, were found.However, the tasks using the computer keyboard or mouse are not always suited to situations experienced by individuals under evaluation, disregarding the specificity of the activities developed in the context of the studied population.
The construction of a measuring instrument requires the observance of certain fundamental criteria that confirm the results safely, among which we emphasize the procedures reliability and validity 20 .A test validity begins at the moment someone considers to build it, and subsists throughout the development, implementation, correction, and results interpretation process 21 .As to reliability, it is an integral part of validity and refers to the consistency or repeatability of a measurement.Whether a test is not consistent, i.e., if its repetition does not produce the same results, it will not be reliable 22 .
In this study the procedures for validity and reliability of the proposed instrument were performed by following the steps according to previous studies for the development of measuring steps instruments 16,20,22 .Concurrent validity analyses showed agreement between the reference test and the RTST, and the paired t test confirmed no significant difference between the two measures.Regarding the analysis of reliability, appropriate values were found related to internal consistency of the instrument, verified by the test-retest procedure for the three groups.Although a value of consensual reference has not been found, in general, ICC values ≥ .70 are considered satisfactory 16 .These results are in agreement with a previous study involving the creation procedures, development, and reproducibility of an RT test 11 .The authors also found satisfactory reproducibility values for the SRT (ICC = .805)and for the CRT (ICC = .838).
Assessing the differences in the reaction time variable between different age groups was not the aim in this study, but rather to create a valid and reliable instrument to be used by people of different age groups that perform specific tasks related to shooting situations.Accordingly, no instruments that fulfill these particular requirements were found.A study developed by the Research Laboratory of the United States Army evaluated the performance of military in their simple and procedure (choice) reaction time.For this purpose, a battery for computerized tests (The Automated Neuropsychological Assessment Metric -ANAM) characterized by the presentation of visual stimuli on the computer screen and sequential manual responses by pressing keys on a keyboard was used.Although it is used for RT evaluation of the active military army personnel, the battery does not correspond to more specific situations of those professionals' activities, which mainly involves the firing of weapons 12 .Another study performed with the Rio de Janeiro military's police of the Special Operations Battalion (BOPE) evaluated the RT through the reaction oculomotor (ruler test).In this test, the individual must catch the ruler, which is released by the evaluator, in the shortest possible time, and the distance the ruler falls is measured.In this case, the evaluation may not reveal accurate data, since it disregards the specific daily demands of professional military police 23 .
It is important to highlight that, although a simulation does not have all the components of a real variable environmental situation, one should seek to reproduce the main features of its demands 24 .Thus, creating this instrument is justified by scientific research purposes seeking to assess individuals in their most specific contexts so that the results are as close to reality as possible.

Conclusion
Based on the results presented in this study it can be inferred that the software created to evaluate the reaction time in the shooting task is valid and reliable for use in research involving a specific instrument with weapon firing activity contexts.
Further, the instrument created can be used for military training as a simple, low cost and efficient tool that involves speed, accuracy and decision-making task shooting.

Figure 2 .
Figure 2. Positioning for the test realization and representation of images used as stimuli in the tasks of SRT and CRT.

Figure 3 .
Figure 3. Validity of RTST compared to the reference test for SRT.

Figure 4 .
Figure 4. Validity of RTST compared to the reference test for CRT.

Table 1 :
Analysis of concurrent validity (t test for paired samples) and descriptive figures for the SRT and CRT (milliseconds) for the reference test and the RTST Note: 1: reference test and 2: RTST; SD: standard deviation.