Lesions of the orbitofrontal cortex do not affect the reinforcement omission effect in rats

The reinforcement omission effect (ROE), reflected by response rates that are higher after reinforcement omission than after reinforcement delivery, has been attributed to both motivational and attentional consequences of the surprising reinforcement omission. These processes depend on the operation of separate amygdala areas and their connections with other brain systems. The interaction between the amygdala and orbitofrontal cortex has been suggested to be important in the modulation of motivational processes. The present study sought to verify whether the mechanisms involved in the ROE depend on the integrity of the orbitofrontal cortex. Prior to acquisition training, rats received bilateral excitotoxic lesions of the orbitofrontal cortex or sham lesions. Following postoperative recovery, the rats were trained on a fixed-interval 12 s limited-hold 6 s signaled schedule of reinforcement. After the acquisition of stable performance, the training was changed from a 100% to 50% schedule of reinforcement. The results showed that rats in both groups exhibited the ROE, with no differences in performance between groups following nonreinforcement. These data do not support the hypothesis that the orbitofrontal cortex is included in the neural substrates related to ROE modulation. The results also showed no difference in response rates between groups in the periods that preceded and followed nonreinforcement. These findings confirm previous studies that showed that the ROE is not related to the facilitation of behavior induced by nonreinforcement.


Introduction
In animals that respond on intermittent schedules of reinforcement, response rates are often higher after reinforcement omission than after reinforcement.This reinforcement omission effect (ROE) has been attributed to both motivational and attentional consequences of the surprising omission of reinforcement.For example, Amsel, & Roussel (1952) reported that the introduction of partial reinforcement in the first goal of a double runway paradigm led to the greatest response on the second runway immediately after omission than after reinforcement delivery.This effect was explained by increments in the drive induced by primary frustration (Amsel, & Roussel, 1952;Papini, & Dudley, 1997;Papini, 2003).However, the ROE can be interpreted in terms of multiple processes that involve behavioral facilitation after nonreinforcement, and transient behavioral inhibition after reinforcement induced either demotivation or a resetting of the internal clock (Seward, Pereboom, Butler, & Jones, 1957;Staddon, 1974;Judice-Daher, Tavares, & Bueno, 2011;Bueno, Judice-Daher, & Tavares, 2013).According to Staddon (1974), in intermittent schedules of reinforcement, each reinforcement delivery acts as a time marker that resets the animal's internal clock.After reinforcement omission, the absence of this resetting would not inhibit responses rates that remain higher after reinforcement compared with nonreinforcement.From this perspective, attentional processes are involved in the modulation of the ROE.
Thus, the term "reinforcement omission effect" can be considered empirically and from a theoretical point of view.The ROE is empirically defined as the difference in responding after nonreinforcement relative to responding following reinforcement.However, theoretically considering that the difference between performance after nonreinforcement and after reinforcement conditions can be driven by multiple processes is necessary.One interpretation may be made in terms of frustration, demonstrated by an increase in responding after nonreinforcement (Amsel, & Roussel, 1952).Another interpretation may be made in terms of behavioral transient inhibition after reinforcement delivery induced by demotivation or the resetting of the internal clock (Seward et al., 1957;Staddon, 1974).This latter view is supported by data from previous studies (Judice-Daher et al., 2011;Judice-Daher, Tavares, & Bueno, 2012;Judice-Daher, & Bueno, 2013), suggesting that the ROE cannot be explained only in terms of the frustration effect theory proposed by Amsel, & Roussel (1952).
From this perspective, Judice-Daher, & Bueno (2013) showed that the NAC and AMY are part of the neural substrate involved in the modulation of the ROE.The authors trained rats with NAC lesions on a fixed-interval 12 s limited-hold 6 s signaled schedule of reinforcement.After the acquisition of stable performance, the training was changed from a 100% to 50% schedule of reinforcement.The results showed that NAC lesions did not eliminate the ROE but interfered with this effect.After nonreinforcement, the response rates of the rats with NAC lesions were lower than those of sham-lesioned rats.
The data from animal research that show that the AMY and NAC are involved in the modulation of the ROE support recent findings from human neuroimaging studies.These studies revealed that reward receipt enhances the activity of subcortical structures (NAC and AMY) and cortical regions (mPFC and OFC), whereas reward omission tends to reduce the activity of these structures (Delgado, Nystrom, Fissell, Noll, & Fiez, 2000;Breiter, Aharon, Kahneman, Dale, & Shizgal, 2001;Knutson, Fong, Adams, Varner, & Hommer, 2001;O'Doherty, Kringelbach, Rolls, Hornak, & Andrews, 2001).
Despite evidence that suggests the involvement of the OFC in processes related to violations of expected reward outcomes, its role in the modulation of the ROE remains unexplored.Thus, the present study sought to clarify whether the mechanisms that underlie the ROE depend on OFC integrity.If functional impairment linked to OFC activation in lesioned animals interferes with the ROE, then the processes related to these effects could be better understood.Prior to acquisition training, the rats received bilateral excitotoxic lesions of the OFC.After postoperative recovery, the rats were trained on a fixed-interval 12 s limited-hold 6 s signaled schedule of reinforcement (acquisition training).On a fixed-interval schedule with a limited-hold contingency (FI LH), reinforcement is available for only a specific period of time after the FI terminates.Consequently, on an FI 12 s LH 6 s schedule, all responses that occur between 0 and 12 s after the start of the FI have no effect on reinforcement, but the first response that occurs between 12 and 18 s is followed by reinforcement (Black, Walters, & Webster, 1972).After the acquisition of stable performance, the training was changed from a 100% to 50% reinforcement schedule (Testing: Partial Reinforcement).The role of the OFC in the ROE is examined by comparing performance between rats with OFC lesions and sham-lesioned rats after reinforcement and after nonreinforcement.

Methods
The study was approved by the Ethics Committee on Animal Use of the Faculty of Philosophy, Sciences and Letters of Ribeirão Preto, University of São Paulo at Ribeirão Preto, Brazil (protocol no. 11.1.1208.53.8).

Subjects
Thirty-two experimentally naive male Wistar rats, 90 days old at the beginning of the experiments and weighing 416-433 g, were used in the study.Throughout the experiments, the animals were singly housed in steel cages in the laboratory colony room on a 12 h/12 h light/ dark schedule (lights on 8:00 AM to 8:00 PM).The rats were maintained on a water deprivation schedule at 85% of their ad libitum body weight by limiting access to water.Food was freely available in their cages.

Apparatus
The experiment was conducted in operant chambers (Lafayette model 80201) equipped with a speaker that delivered a 1000 Hz, 30 dB tone, a 5 W house-light lamp, and a retractable 5 cm lever.Each chamber was in a soundproof wooden box with a transparent acrylic window.These chambers were located in soundproof experimental rooms.An electrical interface (MRA-Electronic Equipment, Ribeirão Preto, Brazil) connected the experimental chambers to a computer.This system used a program prepared with Microsoft QuickBasic 4.0 designed for this experiment, which controlled the reinforcement schedules and recorded lever presses.

Surgical procedures
The animals were anesthetized with an intraperitoneal injection of a mixture that contained 0.8 ml ketamine hydrochloride (0.028 mg/ml) and 0.7 ml xylazine (3.33 mg/ml).Each rat received 0.1 ml of anesthetic for each 100 g body mass.Excitotoxic lesions of the OFC (n = 20) were made using 0.09 M quinolinic acid dissolved in 0.1 M phosphate buffer (composition: 0.07 M Na 2 HPO 4 and 0.028 M NaH 2 PO 4 in double-distilled water, sterilized by filtration) and adjusted with NaOH to a final pH of 7.2-7.4.Toxin infusions were made using a 5 µl Hamilton syringe according to the following coordinates: site 1 (anterior/posterior [AP], +3.7 mm; lateral [L], ±0.9 mm; dorsal/ventral [DV], -4.5 mm; 0.2 µl infused over 2 min), site 2 (AP, +3.0 mm; L, ±2.1 mm; DV, -4.9 mm; 0.3 µl infused over 3 min), and site 3 (AP, +2.7 mm; L, ±2.6 mm; DV, -5.4 mm; 0.3 µl infused over 3 min).The AP coordinate was taken from bregma.The L coordinate was taken from the midline.The DV coordinate was taken from dura.Sham lesions (n = 12) were made in the same manner, with the exception that that no solution was infused (Berlau, & McGaugh, 2003;Judice-Daher et al., 2012).At the end of the operation, the animals were given a single subcutaneous injection of Banamine (2.15 mg/ ml; 0.1 ml for each 100 g body mass) the amelioration of pain.They were given 1 week to recover, with free access to water and food.

Behavioral training procedures
Preoperative training.To avoid the potential effects of the lesions on learning, the rats were trained preoperatively (Cardinal, Pennicott, Sugathapala, Robbins, & Everitt, 2001).Preoperative training was performed over two sessions.In the first session, each rat was placed into the operant chamber and trained to lever press for one 0.05 ml drop of water.The following session consisted of continuous reinforcement (CRF training, with a single water drop [0.05 ml] delivered with each lever press), for a total of 100 lever presses.Each session lasted a maximum of 1 h.
Acquisition training.After recovery from surgery (approximately 7 days), the rats were trained on a FI 12 s LH 6 s schedule that was signaled by an auditory stimulus for 18 s.The first lever press that occurred between 12 and 18 s was always followed by water delivery (0.05 ml).Each session consisted of 20 trials that were interpolated with variable intertrial intervals (mean, 75 s).Each acquisition training session lasted 30 min.All of the rats received a single training session per day for 24 days.At the end of each session, the rats were returned to their home cages and given access to water that was sufficient to maintain them within the planned body weight schedule.Thus, the rats were waterdeprived for approximately 23 h before the beginning of each session.
The rats underwent 10 sessions to attain stable baseline performance, indicated by the following criteria: (1) average response percentage that is higher during the signaled schedule than during intertrial intervals and (2) 80% accuracy in the session for three consecutive days.
Testing: partial reinforcement.The rats underwent four sessions (test phase), in which partial reinforcement was introduced (50% reinforcement schedule).The rats were subjected to the same conditions as acquisition training, but the reinforcement was not delivered after the correct response in half of the trials.

Histological procedures
After completion of the behavioral training procedures, the rats were sacrificed by carbon dioxide asphyxiation.Their brains were removed and postfixed in paraformaldehyde (10%) before being dehydrated in 30% sucrose for cryoprotection (24-48 h).The brains were sectioned coronally at a thickness of 60 µm on a freezing microtome.Every section was saved and stained with Cresyl violet.Slides were examined under a microscope, and neural structures were identified with reference to the stereotaxic atlas of Paxinos, & Watson (2007).Lesions were detectable as the absence of visible neurons, often associated with a degree of tissue collapse and gliosis (visible as the presence of smaller, densely stained cells).

Statistical analysis
For statistical analysis, the average percentage of responses during the FI 12 s LH 6 s was grouped into 3 s periods (3 s FI, 6 s FI, 9 s, FI 12 s, 3 s LH, 6 s LH).These data were analyzed using two-way analysis of variance (ANOVA), with group as an inter-group factor (OFC and Sham groups) and period as an intra-group factor (3 s FI, 6 s FI, 9 s, FI 12 s, 3 s LH , 6 s LH periods).Significant effects in the ANOVA were followed by the Newman-Keuls post hoc test.Values of p ≤ .05were considered statistically significant.

Histological results
Reliability criteria were established for the analysis of the lesions based on the atlas of Paxinos, & Watson (2007) by qualitatively comparing the borders, extent, and homogeneity of both intact and damaged structures.In all cases, the lesion sites were marked by heavy gliosis.At higher magnification, intact neurons were clearly visible at the borders of the lesions (Figure 1).
One of the 12 operated sham-lesioned OFC rats did not survive surgery.Data from the remaining sham-lesioned OFC rats were included in the statistical analysis (n = 11).Four animals were excluded from the OFC lesion group because the lesions were incomplete, unilateral, or extended into the prelimbic cortex.Lesions of the OFC that destroyed neurons in a target area that included medial, ventral, and lateral orbital regions were included in data analyses.Only experimental data from lesions correctly located in the OFC were used for the statistical analysis (n = 16).

Behavioral results
A two-way ANOVA was used to analyze the average response percentages in both groups of rats (OFC and Sham), grouped into 3 s periods, during the presentation of FI 12 s LH 6 s signaled schedules in the last three sessions of acquisition training (Figure 2).The ANOVA revealed no significant difference between groups (F 1,150 = 0.00, p = 1.0) but revealed a significant effect of period (F 5,150 = 219.85,p < .001)and a group × period interaction (F 5,150 = 2.43, p = .038).The Newman-Keuls post hoc test showed significant differences between all of the periods, with the exception of the comparison between the 9 s FI and 12 s FI periods.The Newman-Keuls post hoc test also showed that the OFC and Sham groups were different from each other in the 3 s LH period.These data confirmed task acquisition because of the increase in responding during the signaled schedule.
Figure 3 shows average response percentages in the four sessions of testing, grouped into 3 s periods, during the periods that preceded the omission or delivery of reinforcement (3 s FI, 6 s FI, 9 s FI, 12 s FI), during the periods that followed delivery reinforcement (R) (3 s LH R, 6 s LH R), and following the omission of reinforcement (N) (3 s LH N, 6 s LH N)..The ANOVA revealed a significant effect of period (F 7,200 = 149.15,p < .001)but no effect of group (F 1,200 = 0.00, p = 1.0) and no group × period interaction (F 7,200 = 1.72, p = .10).The Newman-Keuls post hoc test showed that performance in both the OFC and Sham groups during the periods that followed reinforcement delivery (3 s LH R, 6 s LH R) were different from the periods that followed reinforcement omission (3 s LH N, 6 s LH N).These data indicate the ROEs were exhibited by both the OFC and Sham groups.No difference was found in performance between the OFC and Sham groups during the periods that followed reinforcement omission (3 s LH N and 6 s LH N), suggesting that OFC lesions did not interfere with the ROE.
The Newman-Keuls post hoc test also showed that the period that preceded the omission or delivery of reinforcement (12 s FI) was not different from the periods that followed the omission (3 s LH N, 6 s LH N).Thus, no increase in responding was observed after nonreinforcement, but a decrease in responding was found after reinforcement.

Discussion
The acquisition training data suggest discriminative control during the signal in both the OFC and Sham groups, which produced different response distributions that depended on timing.Responding during the fixedinterval schedule was higher when recorded in the last seconds of the fixed interval, but responding was less in the first seconds.These findings suggest that OFC lesions did not interfere with performance acquisition under the FI 12 s LH 6 s signaled schedule.Similarly, Ramirez, & Savage (2007) suggested that the OFC is not essential for establishing the predictive relationship between the specific reward and stimulus conditions.Other studies have shown that the cortex is not critical to acquisition tasks that involve instrumental conditioning (Schoenbaum, Nugent, Saddoris, & Setlow, 2002).
When partial reinforcement was introduced, performance in the OFC lesion group was not different from the Sham group.The rats in both groups exhibited response rates that were higher following omission than following reinforcement delivery.Moreover, no difference in performance was found between lesioned rats and sham-lesioned rats in the periods that followed reinforcement omission (3 s LH N and 6 s LH N).These data indicate that the OFC lesions did not interfere with the ROE.
Additionally, the test phase data showed no behavioral transient facilitation after nonreinforcement (3 s LH N, 6 s LH N periods) because the increase in rats that responded after nonreinforcement was not found.In both the OFC and Sham groups, behavioral transient inhibition was observed after reinforcement (3 s LH R, 6 s LH R periods), reflected by the suppression of responding after reinforcement.The results also showed that the rats in the OFC and Sham groups maintained their performance in the periods that followed reinforcement omission (3 s LH N, 6 s LH N) at the same level as during the period that preceded reinforcement omission or reinforcement (12 s FI).
These data support previous studies (Judice-Daher et al., 2011, 2012), suggesting that the ROE cannot be explained only in terms of the frustration effect theory proposed by Amsel (1992).From the theoretical perspective, the response rate differences between the after-nonreinforcement and after-reinforcement conditions can be driven by multiple processes.One interpretation may be made in terms of frustration, demonstrated by an increase in responding after nonreinforcement (Amsel, & Roussel, 1952).Another interpretation may be made in terms of behavioral transient inhibition after reinforcement induced by demotivation or the resetting of the internal clock (Seward et al., 1957;Staddon, 1974).
The findings that showed that OFC lesions did not interfere with the modulation of the ROE do not support evidence from neuroimaging studies in humans.These studies have suggested that reward omission tends to reduce the activity of subcortical structures (NAC and AMY) and cortical regions (mPFC and OFC; Delgado et al., 2000;Breiter et al., 2001;Knutson et al., 2001;O'Doherty et al., 2001).From this perspective, structures such as the AMY, NAC, mPFC, and OFC are involved in the modulation of the ROE.In fact, recent animal studies have shown that lesions of either the AMY or NAC affect the expression of the ROE (Bueno et al., 2012;Judice-Daher et al., 2012;Judice-Daher, & Bueno, 2013).
These results are similar to those reported by Spicer, Galvan, Hare, Henning, Glover, & Casey (2007).These authors examined whether ventral frontostriatal regions differentially code expected and unexpected reward outcomes.The results showed that both the NAC and OFC had greater activation in rewarded trials relative to nonrewarded trials, but the NAC appeared to be most sensitive to violations in expected reward outcomes.Lesions of the OFC did not interfere with the ROE, and the integrity of the AMY and NAC was sufficient for rats to exhibit the ROE.
The OFC is a cortical region with extensive connections to the basolateral complex of the amygdala (BLA) and other cortical and subcortical regions that may act directing the behavior according to reward expectancy in the presence of a discriminative stimulus (Holland, & Gallagher, 2004;McDannald, Saddoris, Gallagher, & Holland, 2005).However, the BLA and OFC appear to process different aspects of reward information.The BLA may be responsible for the neutral stimulus' gaining of associative strength with the incentive value of the reward but not for the actual use of these expectancies in shaping future behavior.The OFC seems to guide responses based on BLAmediated expectancies (Setlow, Gallagher, & Holland, 2002;Pickens, Saddoris, Gallagher, & Holland, 2005).Moreover, the BLA and OFC send strong and direct projections to the NAC (McDonald, 1991;Haber, Kunishio, Mizobuchi, & Lynd-Balta, 1995), which is implicated in a wide range of emotional and motivational processing, including the acquisition and expression of the motivational properties of predictive cues (Everitt, Cardinal, Hall, Parkinson, & Robbins, 2000;Schoenbaum, & Setlow, 2001).
Although extensive connections exist between the OFC and BLA, the present results suggest that the OFC is not part of the brain circuitry involved in the modulation of the ROE.Other studies have shown that either NAC or BLA lesions interfere with the ROE (Bueno et al., 2012;Judice-Daher et al., 2012;Judice-Daher, & Bueno, 2013), and the neural substrates involved in the processes related to the ROE may include subcortical rather than cortical structures.

Fig. 1 .
Fig. 1.Histological results.Photomicrographs of the regions of orbitofrontal cortex damage in lesioned rats.Each photomicrograph shows the following: (1) the vertical arrow indicates the dorsoventral range along the syringe in the lesioned area; (2) the drawn line indicates the lesioned area.The location of the sections in the anteroposterior plane (mm anterior to bregma) is indicated below each section.Notice the sparse cells and gliosis, especially along the syringe tracks in the lesioned areas.

Fig. 3 .
Fig. 3. Testing: partial reinforcement.The graph shows average response percentages in the four sessions of testing, grouped into 3 s periods, during the periods that preceded the omission or delivery of reinforcement (3 s FI, 6 s FI, 9 s FI, 12 s FI), during the periods that followed delivery reinforcement (3 s LH R, 6 s LH R), and following the omission of reinforcement (3 s LH N, 6 s LH N).The black bars indicate the average response percentage in the Sham group.The white bars indicate the average response percentage in the OFC group.FI, fixed-interval; LH, limited hold; R, reinforcement; N, nonreinforcement; OFC, orbitofrontal cortex.Error bars indicate ± SEM.

Fig. 2 .
Fig. 2. Acquisition training.The graph shows average response percentages in the last three sessions of acquisition training, grouped into 3 s periods, during the periods that preceded reinforcement delivery (3 s FI, 6 s FI, 9 s FI, 12 s FI) and during the periods that followed reinforcement delivery (3 s LH, 6 s LH).The black bars indicate the average response percentage in the Sham group.The white bars indicate the average response percentage in the OFC group.FI, fixedinterval; LH, limited hold; OFC, orbitofrontal cortex.Error bars indicate ± SEM.