A Biological Neural Network of Visual Cell Responses: Static and Motion Processing

A Biological Neural Network of Visual Cell Responses: Static and Motion Processing

Luiz Pessoa Programa de Engenharia de Sistemas e Computacao - COPPE Sistemas Universidade Federal do Rio de Janeiro Ilha do Fundao, Rio de Janeiro, RJ 21945-970, Brazil email: [e-mail]

Alexander Grunewald Division of Neurobiology California Institute of Technology Pasadena, CA, USA email: [e-mail]

Heiko Neumann and Enno Littmann Fakultaet fur Informatik - Abteilung Neuroinformatik Universitaet Ulm Oberer Eselsberg, D-89069 Ulm, Germany email: [e-mail] email: [e-mail]

Abstract

This paper integrates knowledge from physiology and psychophysics (i.e., visual perception) to propose a biological neural network model of cortical visual cell responses. We attempt to provide a model of how retinal and cortical cell interactions are able to detect static image luminance discontinuities -- such as at edges --, as well as moving luminance discontinuities -- i.e., motion stimuli. We address how important cortical cells known as simple cells combine retinal and thalamic signals to produce an effective contrast detection mechanism. An extension of the static model is then discussed in light of both psychophysical and physiological data on motion processing. The motion extension suggests a role for another important class of cortical cells known as complex cells . The static model is evaluated through a series of computer simulations that probe its capabilities with natural images, synthetic images (to assess noise tolerance), as well as images that allow us to compare the model's behavior with physiological results. The motion processing capabilities of the extended scheme are also evaluated through computer simulations. We suggest that this type of investigation can be used to attempt to advance our understanding of brain function, as well as devise powerful computational schemes that can be incorporated into artificial vision systems . Keywords : edge detection, motion detection, neural networks, vision

1 Introduction

The past 10 years have witnessed an explosion in neural network research. The development of powerful learning algorithms, such as backpropagation, has allowed artificial neural networks to gain wide applicability. A perhaps less well-known fact is that the concomitant research in biological neural networks has proven equally fruitful and have helped advance our knowledge of how the brain functions -- as well as having inspired artificial neural networks and engineering work (e.g., Kohonen networks [19]). One particular area that has gained a lot of attention from the biological neural network and neural modeling communities is vision research. Models of cellular properties, such as orientation, motion, and wavelength selectivity have been proposed, as well as models of higher-level processes such as attention and object recognition. One line of research that has proven fruitful is that advanced by Stephen Grossberg and colleagues (see, e.g., [9,11,12]).

For the past several years we have followed a similar line of research and investigated several aspects of vision, including brightness [25,26,28], lightness [27], motion [14], grouping [20,24], and object perception [21], as well as cortical oscillations [13]. In this paper we present our recent work on modeling visual cortical responses to static and motion stimuli. We address the problem of detecting image luminance discontinuities -- both static (such as at image edges) and moving, as it constitutes a fundamental process in vision that starts early on with retinal processing, and continues at the level of the thalamus and striate cortex (and higher). It is from these early contrast-detection mechanisms that the rest of visual processing is built. As expanded below, it is our hope that this type of research can contribute both to the understanding of brain visual mechanisms, and to the development of artificial vision systems. Finally, this paper also attempts to illustrate how biological mechanisms can be formalized through biological neural networks.

One of the key concepts in visual physiology is the receptive field (RF), which is defined as the region of space within which a visual stimulus enhances or suppresses neural activity of a cell under study ( Figure 1 ). The RF can be viewed as that region of space that a cell monitors (which can vary from less than a degree for cells at early stages of the visual system to most of the visual field for cells at later stages). At the level of the retina ( Figure 2A ), RFs typically display a concentric, center-surround, antagonistic structure. ON cells are excited by light at the center of the RF and inhibited by light at the surround. They are thus sensitive to light increments. OFF cells are excited by light in the surround of the RF and inhibited by light at the center. They are sensitive to light decrements. ON and OFF cells actually comprise two independent parallel systems capable of signaling light increments and decrements [31]. At the level of the striate cortex (also called visual area V1 in primates), ON and OFF cells converge onto the same target cells. Moreover, cells display an elongated structure that allows them to be sensitive to edge orientation ( Figure 2B ) -- and display orientation selectivity. Only stimuli whose orientation matches that of the RF elongation will effectively drive the cell (other stimuli will not overlap the RF as much and hence do not affect the cell's response). Selectivity to orientation is believed to be one of the important precursors of form and object perception.

Figure 1 : Receptive fields (RFs) of visual cells correspond to the region of space within which a visual stimulus enhances or suppresses neural activity of a cell. Here we schematically show a concentric RF of a retinal ganglion cell.

Figure 2 : Receptive fields (RFs) of visual cells. (A) At the retina, RFs are typically concentric, with center-surround organization. Light regions are excited by light and dark regions inhibited by light. An ON cell is shown on the left and an OFF cell on the right. (B) At the cortex, RFs are typically elongated such that cells exhibit orientation selectivity. RFs are also sensitive to the direction-of-contrast of an edge, such as for cortical simple cells. The cell on the left is sensitive to dark-light transitions and the one on the right to light-dark transitions. (C) Other cortical elongated RFs, such as those of complex cells, are not sensitive to the direction-of-contrast of an edge and respond to both dark-light and light-dark transitions.

Traditionally, physiologists have studied how the RFs of distinct cells differ, but the question of how the properties of a given RF emerge have received comparatively little attention. In other words, physiologists have concentrated on investigating the input-output relationships of cells in the visual system. Typical stimulus dimensions probed include orientation, direction of motion, chromaticity, and spatial frequency.

In computer vision and neural modeling research (including biological neural networks), on the other hand, a lot of effort has focussed on designing optimal filters to detect edges within a visual scene. Image processing filters built from Gabor or Gaussian functions are commonly employed. While these serve as convenient descriptors of cell behavior -- sometimes capable of matching cell responses quite closely -- they do not address the chief issue of how the cells obtain their RF properties in the first place. Moreover, for the most part, physiological plausibility has played only a minor role in such investigations.

Finally, psychophysicists study the relationship between a visual stimulus and the ensuing visual perception in an attempt to assess perceptual capabilities of the visual system, for example edge localization. While these studies have inspired many theories of visual perception, their impact on models of edge and motion detection has been small.

Here we describe a neural model that combines the above approaches -- physiology, computer and neural modeling, and psychophysics. From a physiological point of view, it is a neural model of important cell types found at early visual areas, namely, cortical simple and complex cells. From a computational point of view, it is a very robust edge detector. Lastly, from a psychophysical point of view, it is a model of first- and second-order motion detection -- as well as having been used to model brightness [25,26,28] and lightness [27] perception.

Our approach advances the notion that powerful computational schemes can be devised by combining insights from the above three disciplines. Constraints observed from each of them help shape our model. Physiology provides evidence that effective computations can be built to solve vision problems -- in this case provided by nature. Computer and neural modeling provide a formal vocabulary with which to describe, simulate, and understand such computations. Finally, psychophysics helps constrain the set of possible computations that can be devised. This threesome combination provides a promising approach to study biological neural networks.

It has been very fruitful to look at the linear properties of visual perception. Linearity makes it very easy to decompose visual processing tasks into subcomponents that then process only smaller features. For example, global edge localization can be accomplished through local linear edge localization. Global edge localization is then just the sum of the local components. However, recent data indicates that this method of studying visual perception leads to problems very early on. Data by Hammond & MacKay [15] (see below) argues strongly against spatial linearity even at early stages of visual processing. One resolution to this seeming nonlinearity might be a spatio-temporal interaction. Thus, if space is embedded in space-time, nonlinearities at the spatial level might disappear. However, recent data on second-order motion processing (see below) suggest that the visual system processes space-time energy in a highly non-linear way [4]. This means that in fact the non-linearities observed by Hammond & MacKay are true spatial non-linearities. In our modeling study we set out to understand what kind of non-linearities might be able to account for both types of data using the same basic unit.

In order to provide the proper context for the model, we start by reviewing cortical simple cell physiology (Section 2). Unlike many other cortical simple cell models that suggest that simple cells behave as linear devices, our neural circuit builds upon non-linear interactions -- that are not just output rectification -- to account for simple cell properties. We show that the classic (static) linear simple cell model of [18] needs to be extended to be compatible with several more recent physiological studies that have revealed important simple cell non-linearities (e.g., [6,15]. The static model is formally described in Section 3 which presents the model equations, as well as a description of its functionality. The computational properties of the model are investigated in Section 4 by employing natural and synthetic images. We show that it produces better localization of contrast changes when compared to a similar linear scheme of the type used in computer vision. In particular, we illustrate the noise tolerance of the model by showing accurate contour localization in the presence of non-trivial noise levels. We also show that the model is capable of accounting for biological data that reveals important simple cell non-linearities. The last part of the paper is concerned with motion processing (Section 5). After reviewing the relevant literature we show how the static model can be extended so that it is able to correctly detect motion energy (i.e., avoid ``erroneous'' responses). We suggest that another important cell type is involved, namely cortical complex cells . The extended model is used to model motion detection and account for the psychophysical phenomena of first- and second-order motion perception.

2 Simple Cell Physiology

The classical work of Hubel and Wiesel [18] revealed that cortical simple cells possess an elongated RF composed of a series of polarity-sensitive sub-fields. In the case of Figure 2B (left) there are two sub-fields, namely an ON sub-field and an OFF sub-field. Light shone on the ON sub-field excites the cell (the sub-field is sensitive to light increments), while light shone on the OFF sub-field inhibits the cell (the subfield is sensitive to light decrements). Therefore, a cell exhibiting a RF such as shown in Figure 2B (left) will be sensitive to light-dark contrast transitions (including edges). Cells with the reverse arrangement of sub-fields are also found and respond to dark-light transitions ( Figure 2B , right)).

Retinal cells display a concentric center-surround structure, an organization that is preserved at the lateral geniculate nucleus (LGN) of the thalamus, the major projection site of the retina. If both retinal and LGN cells display such organization, how do cortical simple cells exhibit elongated RFs? Some transformation must be capable of generating the new cellular properties from the inputs received. [18] hypothesized that a simple cell sub-field is generated directly by excitatory synaptic inputs from a row of LGN neurons whose RF centers overlap the sub-field. The ON sub-field would be generated from ON-center LGN cells, and the OFF subfield by OFF-center LGN cells. As Figure 3 illustrates, it is the proper alignment of the RFs of the projection LGN cells that generates the elongated cortical RF.

Figure 3 : Cortical simple cell receptive fields (RFs). Hubel and Wiesel hyphotesized that LGN cells with concentric, center-surround RFs project to the cortex with the proper arrangement so as to generate elongated RFs from unoriented ones. The ON regions of LGN cells project to ON regions of cortical cells, the same occurring with OFF regions. (A) Schematic representation of the combination of LGN signals. (B) Diagram highlighting the connections to cortical simple cells. Flashing a bar with an orientation that stimulates more of the excitatory centers (+) will stimulate the cell. Changing the orientation so that more of the inhibitory (-) surrounds are stimulated will diminish the cell's response.

Originating from the Hubel and Wiesel proposal, a long-standing view of simple cell response is that it depends on the linear sum of ON and OFF LGN signals (for a review see [16]). In other words, simple cells linearly sum (or pool) all of their inputs. Signals from ON LGN cells contribute to generate the ON simple cell sub-field response, while signals from OFF LGN cells contribute to generate the OFF sub-field response. Moreover, the final simple cell response is obtained by linearly combining the sub-field responses such that the activity of the OFF sub-field is subtracted from that of the ON sub-field.

The careful consideration of a large number of physiological studies indicates that the view of simple cells as linear devices is untenable, in general. For example, there are many studies that have used compound gratings to demonstrate (non-linear) response suppression when non-optimal grating frequency or orientation components are added to a stimulus [2,32]. Also, when the ON channel is pharmacologically blocked -- silencing retinal and LGN ON cells while leaving OFF cells largely unaffected -- simple cell responsiveness has been shown to be reduced by more than 50% [31]. Note that a linear combination scheme between the sub-fields predicts that blocking a single channel would lead to responses roughly around 50% as potent as regular responses.

The most vivid example of simple cell non-linearity is provided by the Hammond and MacKay study [15]. They initially investigated the length-summation characteristics of simple cells in which the integration properties of a RF is probed by stimulating it with bars of varying length ( Figure 4A ; in this case a dark bar). As observed in other studies, the mean simple cell response increased almost linearly (until response saturation) as the bar length increased within the RF ( Figure 5 , dashed line). Hammond and MacKay then probed cells with bars composed of opposite polarity segments (i.e., light and dark segments). Thus, for example, in Figure 4B , the OFF sub-field is stimulated by a dark bar that includes two light endings (light-dark-light, or LDL, stimuli). The reverse organization of the stimulus in which the two extremes were dark and the inner segment was light was also investigated (DLD stimuli). At the same time, the ON sub-field can be probed with light bars with dark segments at the two extremes, or the reverse stimulus arrangement (i.e., inner dark segment and light endings). Figure 5 shows the responses of probing the OFF subfield of a simple cell. We see that the inclusion of "wrong" polarity segments (as in Figure 4B ) largely suppresses cells responses, regardless of the specific placement of the opposite polarity segment (see inset).

Figure 4 : Stimuli used in the Hammond and MacKay study [15]. (A) For length-summation, bars of varying length stimulate the RF and assess the cell's integration characteristics. In this case a dark bar stimulates the OFF sub-field of the simple cell. (B) Introducing opposite polarity segments into the RF largely suppresses the cell's response.

Some decrease in cell response is, of course, expected insofar as the wrong polarity segments do not match the RF structure. Thus, dark segments do not provide appropriate stimulation for the ON sub-field and light segments are ineffective for the OFF sub-field. Linear summation would imply that cell responses drop in proportion to the length of the opposite polarity segments used. The larger the size of the ineffective segments, the weaker the cell response. Hammond and MacKay found instead that the degree of suppression was much greater than that predicted by linear summation, with often total suppression produced by small light segments (such as the addition of 0.1 deg. symmetrical light endings for the 0.8 deg. bar curve). These results are not only inconsistent with a linear summation scheme but are highly suggestive of non-linear, gating-like interactions .

Figure 5 : Hammond and MacKay [15] study of simple cell length-summation for stimuli of different configurations. The dashed line shows the cell's lenght-summation for a regular dark bar (i.e., darker than the background). Dash-dotted lines show responses ("o" and" x") for opposite polarity bars (shown in the inset) showing the influence of adding light segments onto the stimulus. The simple cell had a receptive field consisting of adjacent ON and OFF sub-regions. The abscissa is expressed as total bar length (bar + light segments). The dash-dotted lines show the response to stimuli of constant bar length but increasing length of light segments. Endpoints (see arrows) on the linear dashed line (length summation curve) show results predicted by length-aummation alone.

3 Simple Cell Neural Model

Figure 6A shows the simple cell circuit. The circuit is composed of two streams or channels, namely, ON and OFF, and includes both retinal and cortical stages. The model ¹ consists of a series of processing stages, each of which consists of a two-dimensional field (or grid) of processing units, or cells. The input is also encoded as a two-dimensional field of activity. All connections between model stages are topographically organized such that a spatial location ( i , j ) at a given stage connects to location ( i , j ) in the target field. The activation level at individual model stages represents the output value of the respective stages.

Retinal interactions . The input luminance distribution, I , is processed by cells with center-surround, antagonistic, RFs, such as retinal ganglion cells. The model includes both ON and OFF pathways that measure the degree of local luminance contrast in input images. Thus at every spatial location there are ON and OFF cells. These two fields of cells implement lateral inhibition that contrast-enhance, or sharpen, the input luminance distribution -- cells respond strongly to (unoriented) luminance discontinuities.

Retinal interactions can be approximated by filtering the input distribution with difference-of-Gaussian filters ² such as were originally proposed to model the receptive field structure of retinal ganglion cells [5,30]:

c ⁺ = I * G ⁺ and c ^- = I * G ^- (1)

where c ⁺ and c ^- are ON and OFF retinal contrast responses, I is the input luminance distribution, G ⁺ and G ^- are the respective Gaussian filters, and * implements the convolution operator.

In the classical proposal, the center and surround contributions combine linearly to determine the cell's response. The present approach adopts a multiplicative, or shunting, formalism [7,10,17], and thus departs from the specification given by Equation 1. Instead we follow a specification that takes into account visual adaptation processes that render cell responses sensitive to luminance ratios . Such formalization produces invariant retinal responses for a given image regardless of overall illumination (and constant illumination gradients such as produced by a light source), such as exhibited by retinal ganglion cells. Details of the retinal circuits can be found in [12] and [28]. Essentially, retinal contrast responses are obtained by dividing the signals produced by Equation 1 by the local average illumination. In all, initial retinal processing renders cell responses DC-level free (uniform regions produce zero output) and sensitive to input luminance ratios.

Figure 6 : (a) Proposed circuit of simple cell sensitive to light-dark contrast polarity. (b) Final competition of simple cells of opposite contrast polarity at each spatial location.

ON/OFF input . The input to the simple cell circuit comprises ON and OFF retinal signals. Simple cell RF elongation is obtained by blurring (by convolution) the activity distribution in the ON ( c ⁺ ) and OFF ( c ^- ) input channels with elongated Gaussian weighting functions. Formally, the ON and OFF inputs are p ⁺ = ( c ⁺ - c ^- ) * l and p ^- = ( c ^- - c ⁺ ) * l (see Figure 6 ), where the spatio-temporal kernel ³ l determines the orientation and motion direction preference of a given simple cell. The kernels l were spatially offset slightly between p ⁺ and p ^- to collect offset LGN responses, which are used to determine contrast polarity changes, such as the presence of edges or other abrupt luminance variations. In our simulations l was a Gaussian weighting function, which was shifted in time and in space to collect motion signals. Multiple spatial frequency selectivity is achieved by using weighting functions of different spatial extent in length and width (i.e., different spatial scales).

Sub-field opponent interactions . The second stage of the circuit contains direct excitatory inputs from each ON or OFF sub-field (second term in equation below), as well as an opponent (inhibitory) interaction between channels (third term). The activity of the ON subfield is given by

= - aq⁺ + p⁺ - bq⁺ p^- (2)

where a and b are constants.

Direct input and post-opponency signals combined . The third stage receives channel-specific inputs from both the first (excitatory; second term in equation below) and second (inhibitory; third term) stages. Formally,

= - gr⁺ + p⁺ - dr⁺ q⁺ (3)

where g and d are constants. OFF channel activations are obtained by exchanging "+" and" -" indices above.

The final response for a light-dark cell is obtained by summing ON and OFF activities as in z ^LD = r ⁺ + r ^- . The dark-light response, z ^DL , is obtained in a similar manner.

Mutual inhibition of cells . Simple cells of opposite polarity and same spatial position undergo mutual inhibition (e.g., [6]), as indicated in Figure 6B . The final simple cell responses are computed as Z ^LD = z ^LD - z ^DL and Z ^DL = z ^DL - z ^LD .

3.1 Simple Cell Functionality

The receptive field profile shown in Figure 7 indicates the spatial structure of the inputs to simple cells but does not reflect the non-linear combination of signals between ON and OFF sub-fields. The original simplified one-dimensional (static) version of the simple cell model [23] was inspired by brightness perception data and designed to meet two functional requirements [28]: (a) produce strong responses whenever ON and OFF retinal responses occur next to each other, such as at a luminance edge ⁴ ; (b) produce some non-zero response when only ON or OFF retinal responses are present, such as due to shallow luminance gradients (e.g., luminance ramps, or a luminance sinusoid). Such properties were retained in the present version of the model, whose cell responses are of the form:

h(ON + OFF) + m (ON x OFF) (4)

where h and m are model parameters (see below) and ON and OFF correspond to the retinal inputs to the simple cell circuit.

Figure 7 : Model simple cell for s _m =3 and s _m =6: surface plot (left) and projection to the xy -plane (right).

In all, the model behaves as a soft AND-gate that boosts adjacent ON and OFF signal configurations but that also responds to other configurations -- albeit in a reduced manner. In terms of the circuit shown in Figure 6A , it is the opponent inhibition of the second stage (signals q ⁺ and q ^- ) associated the within-channel inhibition (from the second to the third stage) that implements the soft AND-gate behavior. These interactions produce a mechanism for disinhibition that generates large outputs only when both ON and OFF channel inputs are large. To understand this consider the case where, say, only ON signals are input to the model; the OFF pathways are thus shut down. Simple cell responses will be small since the (within channel) inhibition from the second to the third stage will attenuate the input signal. Now consider the case when there are potent inputs to both ON and OFF channels. The cross inhibition between the first and second stages will largely reduce second stage activities (signals q ⁺ and q ^- ). These by their turn will not be able to inhibit stage three signals and the original inputs will be able to combine at the last stage ( z ^LD and z ^DL ) since they reach it via the "side pathways" from the first to third stage. We see that the presence of inputs in both channels leads to a disinhibition in the circuit and hence powerful responses. In all, the circuit detects when there are adjacent ON and OFF signals, hence the soft AND-gate behavior.

The key idea behind the simple cell model is that contrast changes are better localized by a circuit that is extremely sensitive to abrupt luminance transitions. These transitions will be invariably associated with adjacent ON and OFF retinal contrast signals (for some spatial scale). A linear combination scheme is, of course, sensitive to such transitions, but not sensitive enough . As we show below, our non-linear scheme not only accounts for physiological data on simple cells but provides a robust edge detector for image processing -- as well as having been inspired by psychophysical data on brightness perception.

It should be pointed out that our scheme shares some of the important features of the Marr and Hildreth edge detection proposal [21]. Marr and Hildreth's main idea was that it was possible to detect edges by linking the outputs of ON and OFF center-surround cells (such as LGN cells) through a logical AND-gate. While our model provides a circuit capable of realizing an AND-gate like behavior, it does not compute a logical AND-gate, but instead a soft gate. This property is important since image contours are present not only at edges but at other luminance distributions (see. e.g., Figure 8 of [26]).

It is possible to gain insight into the precise functionality of the model, especially with regard to its AND-gate behavior, by assuming that cells reach equilibrium fast (i.e., dx / dt = 0 ) and using the corresponding expressions. The activation of r -nodes in the ON and OFF branches can thus be computed analytically. An assumed symmetry relationship between both channels can be achieved by the identity d = bg . In this case, for a light-dark cell we get

and (5)

(6)

The final response for a light-dark simple cell is computed as z ^LD = r ⁺ + r ^- . We get

(7)

The dark-light response, z ^DL , is obtained in a similar manner. This demonstrates the non-linear interaction of activity between the two branches. Input is integrated linearly from both channels ( a ( p ⁺ + p ^- ) ) and spatially adjacent activity (in the ON and OFF pathways) is signaled by an additional correlational (gating-type) component (2 b p ⁺ p ^- ). The relative contribution of additive and gated activity is controlled by the (shunting) parameters a and b in Equation 2. Moreover, the activity self-normalizes with respect to the total input activity from the ON and OFF channel (1/( ag + d ( p ⁺ + p ^- ))). The model thus resembles the scheme proposed by [3] to normalize activity of cortical neurons through division of pooled activity from a large number of cells. Note that Equation 7 formalizes and makes precise the meaning of Equation 4.

4 Static Computer Simulations

Figure 8 : Processing a real camera image (left): Pooled activity of all orientation fields generated by the non-linear simple cell circuit (center) vs. the linear model (right).

In this section we show a series of computer simulations that demonstrate the functionality of the model. In order to better assess its behavior, the non-linear processing given by the specification of Section 3 is compared to results of a model that integrates activity from ON and OFF branches in a linear fashion ⁵ . In all simulations, simple cell responses are shown after mutual inhibition ( Z ^LD and Z ^DL ). The model parameters of the non-linear circuit were set to a = 1.0 , b = 2.0 , g = 0.5 , and d = 1.0 . Their specific choice is non-critical as long as the linear components a and g are not large compared to the cross-channel inhibition effect. The Gaussian weighting functions were elongated by a 2:1 ratio ( s _M : s _m ); the variance s _m is measured in pixels along the short axis. The separation t grows linearly with the variance. Eight discrete, equally spaced orientations were processed. Figure 7 shows a surface plot and the projection on the xy -plane of a model simple cell with s _m = 3 , t = 3 , and orientation j = 0 ^o .

4.1 Image processing

Natural Images . Natural images provide a good test of the image processing capabilities of our simple cell model. In particular we can assess the contrast localization properties of the model by comparing its output with that produced by an analogous linear scheme. Figure 8 illustrates the better contrast localization properties of our circuit when compared to the linear scheme. We see that much sharper "edge signals" are generated by the circuit effectively registering the contour outlines present in the image.

Noise tolerance . A stricter test of an image processing algorithm consists in probing it with noisy images. Figure 9A shows a synthetic image containing an elliptic region embedded in a lighter background. The entire image was corrupted by Gaussian noise (half width 50% amplitude). Figure 9 (middle) shows the output of the simple cell model revealing that it is capable of accurate contour localization even in the presence of non-trivial noise levels. Note that no final thresholding operations were performed -- these could be used to remove the low intensity spurious signals due to noise. It is also instructive to compare the performance of the simple cell with the linear scheme. As shown in Figure 9 (right), the linear scheme is much less robust to noise.

Figure 9 : Synthetic image with noise. Upper left: input luminance distribution corrupted by additive Gaussian noise with 50% amplitude of contrast height. Upper right: pooled activity of all orientation fields generated by the simple cell model. Bottom: Output of the linear scheme.

4.2 Hammond and MacKay data

Figure 10 : Simulation stimulus set for one bar length. Stimuli were generated consisting of a dark bar (intensity 64, width 2 pixels) of lengths l = 2, 4, 8, 16, 24, 32, 40, and 64 pixels shown against a gray background (intensity 128). For reverse contrast stimulation, light segments (intensity 192) were added either at the center (DLD (dark-light-dark) stimuli) or at both ends of the dark bar (LDL stimuli). The graph shows the stimulus set for a dark bar of length l = 16 pixels.

Figure 10 shows the set of stimuli used in the series of model simulations performed to compare the circuit's behavior with the Hammond and MacKay (1983) data [15]. The results for these simulations were generated in the following way. Instead of plotting the results of probing a single simple cell, we plot the results of summing the responses of five like-oriented cells arranged on a straight line (i.e., along their orientation axis). This corresponds to a simple cell model with the RF profile shown in Figure 11 . This was done since for a single cell with a Gaussian RF structure as shown in Figure 7 , bar segments far removed from the RF center are not effective in stimulating the cell. Thus, for large stimuli in which the" wrong" polarity segments are at the extremes of the bar, the segments will not be capable of affecting the cell response (since they have a very small effect on the cell's response).

Figure 11 : Simple cell corresponding to the summation of the responses of five like-oriented cells arranged along their orientation axis.

Figure 12 : Simple cell response to luminance gradient reversal.

Figure 12 shows the results produced by the model. When probing the circuit with a dark bar of varying length we obtain a roughly linear length-summation curve that saturates (continuous line), in good agreement with the experimental data. This shows that although the simple cell circuit includes some important non-linearities, it can display linear integration properties for simple bar stimuli. The responses of the model to a dark bar with a central light segment (dotted lines, DLD configuration) are also in good accordance with the biological data. The addition of a central light segment results in an almost gating-like suppression of the cell response, independent of bar length (as seen in the data). If we add light segments to the ends of the bar (LDL configuration), the degree of suppression depends on bar length (continuous lines). For a short bar we still observe a gating-like behavior. The degree of suppression, however, decreases significantly with increasing bar length.

In summary, for DLD stimuli we obtain excellent data fits (compare Figure 5 (lower inset; "x" points) with Figure 12 (dotted lines)). For LDL stimuli the model deviates from the actual data, especially at larger stimulus lengths (compare Figure 5 (upper inset; "o" points) with Figure 12 (continuous lines)). We hypothesize that the deviations from the actual data occur because, for large stimuli, the opposite polarity segments are ineffective in modulating the cell's response (i.e., they are in a region of the RF profile that contributes only weakly to stimulus response). If this is correct, the deviation can be minimized by pooling over a larger number of simple cells. We are currently investigating this issue.

5 Motion Processing

In this section we show how the static model can be extended to process moving stimuli. The extension builds upon physiological and psychophysical work revealing key non-linearities in the visual system's processing of motion.

5.1 Motion Detection

Finding edges in a scene corresponds to the detection of oriented contrast transitions. The two dimensions within which the orientation is to be detected are therefore spatial (say, x and y ). The detection of motion is analogous to orientation detection, except that one of the spatial dimensions, say y , is replaced by time . Thus the detection of motion can be viewed as the detection of orientation in space-time (as can be seen at the left of Figure 13 , a moving stimulus is oriented in a space-time diagram; see [1]). In praxis, this means that input signals at some spatio-temporal position have to be combined with signals from a different spatio-temporal position. Clearly, the spatial difference divided by the temporal difference then gives the velocity; in other words, such a motion detector has a preferred velocity. For example, combining ON sub-field signals at time t and location i with past signals from time t - d t at location i - d i , and doing the same for the OFF sub-field, will naturally endow a simple cell with direction selectivity .

Under laboratory conditions motion perception can be studied by employing, say, a bright bar that is moving. When contrast remains the same through time, it can be said to constitute the stimulus' defining (or first-order) feature, and such (standard) motion is called first-order motion [4]. This type of motion can be easily detected within the framework described in the previous paragraph if the inputs to a static oriented contrast (or edge) detector are replaced by spatio-temporal (directionally-oriented) signals. Therefore, a cortical simple cell can become a motion detector if it also receives from the LGN temporal information. That this is the case is attested by the fact that simple cells in striate cortex exhibit direction selectivity [18].

When the bar reverses its contrast each time it moves, (i.e., it alternates between being darker and brighter than the background) its first-order feature (i.e., contrast) continually changes. However, it is possible to define the contrast transition (independent of direction of contrast) as a second-order feature. Motion of this type is called second-order motion [4]; see Figure 13 bottom left. For second-order motion, energy is actually moving in a direction that is opposite to the moving bar. This can be understood considering how simple-cell-like RFs respond to a contrast-reversing bar as opposed to a standard bar ( Figure 13 ). Each time the bar reverses its contrast, the bright and the dark side at the contrast transition trade places. Therefore, the ON and OFF responses at the retina and LGN trade places. Such trading of places would send contradictory signals to a cell that simply integrates (i.e., sums) LGN signals. In fact, the units selective for the direction of motion in which the bar is moving are weakened since their RFs do not match the spatio-temporal LGN signals (compare to the situation in Figure 13 (top) in which a simple-cell-like RF produces a good match with LGN signals). Instead, at many places (such as shown at the bottom part of the figure), cells tuned to the opposite direction of motion would be excited -- indicating that the stimulus is moving in a direction opposite to the bar movement direction, a type of "erroneous" motion signal.

Figure 13 : First and second order motion. Space is along the horizontal axis, time along the vertical. LGN ON and OFF responses are schematized by + and - respectively. Top: first-order motion is characterized by motion of a stimulus feature, for example a contrast increment (at right edge of moving bar). With the moving bar, ON and OFF responses at the LGN also move. Simple cell can detect motion as a spatio-temporal orientation. Bottom: second-order motion is characterized by motion of a second-order stimulus feature, here a contrast transition that is direction-of-contrast independent (alternation between brighter and darker than the background at each edge). LGN ON and OFF responses trade places, thus sending contradictory motion signals to a cortical simple-cell-like RF that can thus generate an "erroneous" motion signal in the opposite direction. Notice that complex cells signal the correct motion direction in both cases since they are not sensitive to the direction-of-contrast of LGN signals.

In a theory of the human visual system we need to know what humans see when viewing a second-order stimulus. Psychophysical studies have shown that humans can detect first- and second-order motion. In other words, even when the contrast of a moving bar reverses, humans are able to detect the direction of motion -- and they are not confused by the actual motion energy going in the opposite direction (as illustrated in Figure 13 ). It has been found, however, that the response characteristics to the two types of motion differ, suggesting that two distinct mechanisms are involved in the detection of first- and second-order motion. The ability to correctly perceive the direction of a contrast reversing bar indicates that the motion detection process must include important non-linearities, since all of the (linear) motion energy is moving in the opposite direction.

The RFs of cortical simple cells are composed of spatially adjacent sub-fields, as seen in Figure 2B . Hence they are sensitive to the contrast polarity of a stimulus (the left RF in Figure 2B responds to dark-light transitions). We say that simple cells exhibit phase sensitivity. On the other hand, cortical complex cells are not sensitive to the contrast polarity of a stimulus and hence are phase insensitive ( Figure 2C ). Can these cell properties help account for the separate detection of first- and second-order motion? Before addressing this questions let us first review how cortical cells respond to first- and second-order motion.

Zhou & Baker [33] showed that in area 17 of the cat cortex no simple cells responded to second-order motion. Complex cells, on the other hand, respond to both first- and second-order motion. This is an important finding, since a direction selective simple cell such as shown in Figure 13 should respond (at least weakly) to second-order motion going opposite to its preferred direction -- remember that second-order motion will produce (linear) motion energy moving in the direction opposite to the moving bar; therefore, a second-order stimulus moving in the direction opposite to that preferred by the simple cell should excite it somewhat. Since this was not found, it follows that the "erroneous" second-order response of simple cells is suppressed in vivo . How could this have been accomplished?

As we formalize below, we propose that a second-order motion detector inhibits simple cells, thereby suppressing potential first-order motion responses by such cells. Thus, when the second-order detector responds to the contrast-reversing bar, it suppresses the "erroneous" response that could be produced by a simple cell. A potential problem with this proposal is that a second-order motion detector will most likely also respond to first-order motion -- since it responds to a contrast-reversing bar it should also respond to a regular bar. If this occurs, the inhibition from the second-order detector to the simple cell could potentially destroy simple cell responses to standard moving bars. As we show in simulations below, a careful balance of LGN excitation and second-order detector inhibition at simple cells, solves this problem. Hence, simple cells display first-order responses while avoiding second-order responses. Given the results by [33], we identify the above second-order detectors with complex cells.

In simple cells phase sensitivity can be achieved by careful separation of ON and OFF sub-fields. Phase insensitivity in complex cells can in principle be achieved in two ways. Hubel and Wiesel [18] suggested that complex cells pool activities from simple cells of opposite phase preferences. If a complex cell responds if either of its input simple cells responds, then the complex cell will be phase insensitive. However, this scheme has been impossible to verify thus far using anatomical and physiological techniques. We therefore suggest an alternative, whereby complex cells themselves receive direct input from LGN units. Unlike the case of simple cells, however, ON and OFF sub-fields are not separated. Hence, for a given location in space, complex cells receive both ON and OFF input from the LGN. This will naturally make complex cells phase insensitive and respond to both light-dark and dark-light luminance transitions. In this scheme complex cells obtain their orientation selectivity through spatial elongation of their receptive fields. Indeed, many complex cells show a strong elongation along their orientation preference axis [8].

Since complex cells pool ON and OFF signals from the LGN, they can detect first-order motion just like simple cells do. In the case of second-order motion, however, the picture is slightly more complex. As outlined above, in second order motion ON and OFF signals trade position vis a vis the edge to which they belong. But since complex cells pool ON and OFF signals from identical locations, their responses are independent of which LGN subpopulation (ON or OFF) is active at the LGN. Therefore, complex cells will respond to second-order motion stimuli.

5.2 Simple Cell Circuit

As outlined above, the simple cell circuit is identical to that used in the static simulations. However, since motion probes dynamic features of simple cell responses, all the underlying equations (e.g., Equations 2 and 3) are numerically integrated. Note also, that the convolution kernel l is oriented in space-time, endowing simple cells with direction selectivity. In addition, simple cells receive complex cell inhibition. This is introduced at the final stage where simple cells of opposite polarity inhibit each other:

Z ^LD = z ^LD - z ^DL - I _c

where I _c is the influence from complex cells. The same complex cells inhibits both Z ^LD and Z ^DL . Inhibition is defined as I _c = K _c * C , where K _c is an interaction kernel between complex and simple cells, and (capital) C is the complex cell activity, as defined below. K _c is defined such that a given complex cell will inhibit simple cells of the opposite direction preference.

5.3 Complex Cell Circuit

Complex cells are modeled very similarly to simple cells. Thus we only point out the key differences. Phase insensitivity is obtained because LGN ON and OFF responses are combined after convolution with l :

E = c ⁺ + c ^- ,

where c ⁺ and c ^- are retinal ON and OFF contrast responses (see Equation 1). Subsequent simple cell mechanisms (in particular q and r ) that enhance the sensitivity to contrast polarity are also omitted from model complex cells. Feedforward inhibition sharpens direction selectivity of complex cells:

,

where W is a scaling parameter, and i is a spatial position index ⁶ . Inhibition and excitation are combined in a final shunting equation:

= - x + (1 - x ) E - (1 + x ) I .

The final output from the complex cell stage is then C = max (0, x ) . Note that complex cells feed into simple cells of opposite direction selectivity as described in the previous section.

5.4 Motion Simulations

First-Order Motion . We probed our model with a first-order motion stimulus. In our simulations a bright bar is first moving rightwards and then leftwards on a gray background. Figure 14 shows the input to the model, and the responses of simple and complex cells. According to the model both simple and complex cells respond to the first-order motion stimulus. Note that simple cells respond only at one edge because they are sensitive to the direction-of-contrast of the bar (light-dark transitions), while complex cells respond at both edges. This concurs with physiological data [18].

Figure 14 : First-order motion simulation. Responses of a network of simulated neurons over time. The neurons are aligned from left to right in each figure. Time increases from bottom. Response strenght is indicated by the grey level. Black: no response, white: strong response. (A) mode input. (B) simple cell responses tuned to rightwards motion, and a dark-light transition. (C) complex cell responses tuned to rightwards motion.

Second-Order Motion . A critical test of our model is to verify simple and complex cell responses to a second-order stimulus. The input ( Figure 15 ) was a contrast reversing bar that was first moving rightwards and then leftwards, moving along the same trajectory as in the first-order simulations. The bar reversed between black and white, moving on a gray background. As can be seen, model simple cells were largely silenced, demonstrating that the inhibitory complex cell feedback was effective in suppressing" erroneous" responses. Complex cells, on the other hand, fired vigorously, in accordance with physiological data from cat area 17 [33].

Figure 15 : Second-order motion simulation. Same conventions as in Figure 14 . Note that no simple cells get activated by second order input.

6 Conclusions

This paper has advanced the notion that the integration of physiological and psychophysical data can be used to devise biological neural networks. These not only advance our understanding of brain function (and can be used to suggest novel physiological and psychophysical experiments and tests), but suggest powerful computational schemes that can be incorporated into artificial vision systems -- and be part of important technological innovations.

We have addressed the specific problem of detecting image luminance discontinuities, a fundamental process in vision that starts early on with retinal processing, and continues at the level of the LGN and striate cortex (and higher). Hence we have addressed how cortical simple cells combine ON and OFF signals to produce more than linear contrast detection. As shown through computer simulations, the non-linear pooling leads to higher positional selectivity of contrast changes (i.e., more accurate edge localization) and strong noise resistance, two extremely desirable properties for an artificial vision system. Moreover, the non-linear pooling is able to account for observed cellular non-linearities.

The static model was extended to do motion processing. Classical motion detectors, such as the Reichardt detector [29], or the spatio-temporal energy model [1], only respond to first-order motion, because they were essentially designed to respond only when luminance-defined stimuli (such as bars) are moving in the image. The present work shows how detectors that receive ON and OFF inputs can detect first- and second-order motion within a single framework. In all, the confluence of ON and OFF signals at the level of the cortex emerges as a powerful method to obtain orientation and direction selectivity in vision.

Acknowledgements

L. Pessoa, H. Neuman and E. Littmann were supported in part by the German-Brazilian Academic Research Collaboration Program DAAD-Capes, Probral 1997.

References

[1] E. H. Adelson and J. R. Bergen. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A , 2: 284-299, 1985

[2] A. B. Bonds. Role of inhibition in the specification of orientation selectivity of cells in the cat striate cortex. Vis. Neurosci. , 2:41-55, 1989

[3] M. Carandini, and D. J. Heeger. Summation and division by neurons in primate visual cortex. Science , 264:1333-1336,1994

[4] C. Chubb and G. Sperling. Drift-balanced random stimuli: a general basis for studying non-Fourier motion perception. J. Opt. Soc. Am. A , 5:1986-2007, 1988.

[5] C. Enroth-Cugell and J. G. Robson. The contrast sensitivity of retinal ganglion cells of the cat. J. Physiol. , 187:517-552, 1966

[6] D. Ferster. Spatially opponent excitation and inhibition in simple cells of the cat visual cortex. J. Neurosci. , 8:1172-1180, 1988

[7] G. G. Furman. Comparison of for subtractive and shunting lateral-inhibition in receptor-neuron fields. Kybernetyk , 2:257-274, 1965

[8] C. D. Gilbert. Laminar differences in receptive field properties of cells in cat primary visual cortex. J. Physiol. , 268:391-421, 1977

[9] S. Grossberg. 3D vision and figure-ground separation by visual cortex. Perception & Psychophysics , 55:48-120, 1994.

[10] S. Grossberg. Neural pattern discrimination. Journal of Theoretical Biology , 27:291-337, 1970.

[11] S. Grossberg and E. Mingolla. Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading. Psychological Review , 92:173-211, 1985

[12] S. Grossberg and D. Todorovic. Neural dynamics of 1-D and 2-D brightness perception: a unified model of classical and recent phenomena. Perception & Psychophysics , 43:723-742, 1988.

[13] S. Grossberg and A. Grunewald. Cortical synchronization and perceptual framing. Journal of Cognitive Neuroscience , 9:117-132, 1977.

[14] A. Grunewald and M. J. M. Lankheet. Orthogonal motion after-effect illusion predicted by a model of cortical motion processing. Nature , 384:358-360, 1966.

[15] P. Hammond and D. MacKay. Influence of luminance gradient reversal on simple cells in feline striate cortex. J. Physiol. , 337:69-87, 1983.

[16] R. von der Heydt. Approaches to visual cortical function. Rev. Physiol. Biochem. Pharmacol. , 108: 69-150, 1987.

[17] A. L. Hodgkin. The conduction of the nervous impulse. Liverpool UNiversity Press, Liverpool, 1964.

[18] D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. , 160:106-154, 1962.

[19] T. Kohonen. Self-organization and associative memory (3rd ed.). Springer-Verlag, 1989.

[20] E. Littmann, H. Neumann and L. Redoloux. Extraction of illusory contours by perceptual grouping. Mustererkennung 1996 (B. Jähne & F. Hering, eds.). Springer-Verlag, 1966.

[21] E. Littmann, A. Drees, H. Ritter et al . Visual gesture-based robot guidance with a modular neural system. Proceedings of the 9th Annual Conference on Neural Information Processing Systems (NIPS'95), Advances in Neural Information Processing Systems 8, D. Touretzky, M. Mozer, & M. Hasselmo (Eds.). MIT Press, 1996.

[22] D. Marr and E. Hildreth. Theory of edge detection. Proc. Royal Soc. of London (B) , 207:187-217, 1980.

[23] H. Neumann and L. Pessoa. A simple cell model with multiple spatial frequency selectivity and linear/non-linear response properties. Proceedings of the World Congress on Neural Networks (WCNN-94), San Diego, Vol. IV, IV-290-298, 1994

[24] H. Neumann and W. Sepp. A model of cortico-cortical integration of visual information: Receptive fields, grouping and illusory contours. Proc. of the Workshop on Vision & Action , Boston, Mass. Book of Abstracts, 1997.

[25] L. Pessoa. Mach band attenuation by adjacent stimuli: experiment and filling-in simulations. Perception , 25: 425-442, 1996.

[26] L. Pessoa. Mach bands: How many models are possible? Recent experimental findings and modeling attempts. Vision Research , 36:3205-3227, 1996.

[27] L. Pessoa and W. D. Ross. A contrast/filling-in model of 3-D lightness perception. Proceedings of the 9th Annual Conference on Neural Information Processing Systems (NIPS'95), Denver: Advances in Neural Information Processing Systems 8 , D. Touretzky, M. Mozer, & M. Hasselmo (Eds.). MIT Press, 1996.

[28] L. Pessoa, E. Mingolla, H. Neumann, H. et al . A contrast- and luminance-driven multiscale network model of brightness perception. Vision Research , 35:2201-2223, 1995.

[29] W. Reichardt. Autokorrelationsauswertung als Funktionsprinzip des Zentralnervensystems. Z. Naturforsch., 12b:447-457, 1957.

[30] R. W. Rodieck. Quantitative analysis of cat retinal ganglion cell responses to visual stimuli. Vision Research }, 5: 583-601, 1965.

[31] W. D. Ross and L. Pessoa. Lightness from contrast: A network model. Submitted to Perception & Psychophysics , 1996.

[31] P. Schiller. The ON and OFF channels of the visual system. Trends in Neurosciences , 15:86-91, 1992.

[32] R. De Valois and R. Tootell. Spatial-frequency-specific inhibition in cat striate cortex cells. J. Physiol. , 336:359-376, 1983.

[33] Y. X. Zhou and C. L. Baker. A processing stream in mammalian visual cortex neurons for non-Fourier responses. Science , 261:98-101,1993.

¹

The current version of the model lumps together retinal and LGN stages given the similar cell responses at these structures.

²

A difference-of-Gaussian filter assumes that both the center and surround components of retinal RFs can be modeled as Gaussian profiles of sensitivity. The larger spatial extent of the surround is simply obtained by employing a Gaussian with larger standard deviation.

³

We describe the simple cell model in terms of spatio-temporal kernels in order to extend the static model for motion processing, as discussed in Section 5. For a purely static circuit, the kernels can be understood as purely spatial filters, such as Gaussian functions.

⁴

At an edge, strong ON responses will occur at the light side of the edge and strong OFF responses will occur at the dark side of the edge.

⁵

The linear integration model with elongated Gaussian lobes for ON and OFF sub-fields approximates a first order derivative operation.

⁶

Note that a one-dimensional spatial implementation was adopted for the motion simulations. A two-dimensional extension (as in the case of the static model) is straightforward.

Publication Dates

• Publication in this collection
  07 Oct 1998
• Date of issue
  July 1997