Abstract

The Role Played by Intralayer and Interlayer Feedback Connections in Recurrent Neural Networks used for Planning

Aluizio F. R. Araújo and Hélio D’Arbo Jr.

Universidade de São Paulo - Depto. de Engenharia Elétrica

C.P. 359, São Carlos, SP Brazil

aluizioa@lasi01.sel.eesc.sc.usp.br and heliod@lasi01.sel.eesc.sc.usp.br

Abstract - This paper proposes five partially recurrent neural networks architectures to evaluate the different roles played by interlayer and intralayer feedback connections in planning a temporal sequence of states. The first model has only one-to-one feedback connections from the output towards the input layer. This topology is taken as the reference one. The other models have interlayer and/or intralayer all-to-all feedback connections added to them. All feedback connections, but the one-to-one feedback links, are trainable. The models yield a sequence which take four blocks from an initial to a goal state, when these states are presented to the network. The models showed good performance for planning in different levels of complexity. The results suggest that the models have poor generalization power.

Keywords: Planning, partially recurrent neural networks, intralayer feedback connection, interlayer feedback connection.

1. Introduction

Recurrent neural networks, in particular partially recurrent networks, are often used to deal with processes which require current input data and past states of the process to produce an output. A partially recurrent network [1] is defined as a network formed by feedforward and feedback connections in which the former are predominant but the latter are crucial to deal with three types of tasks involving temporal sequences. In sequence recognition, a particular input sequence should produce the corresponding output pattern. In sequence reproduction, part of a given sequence should yield the rest of it. Finally, in temporal association an input sequence should generate an output sequence.

The local-recurrent-global-feedforward (LRGF) architectures constitute a class of partially recurrent networks in which the feedback connections occur exclusively from one unit to itself [2]. These models present three different kinds of feedback connections. The local activation feedback allows combinations of delayed versions of the unit weighted summation as inputs. The local output feedback consists of feeding back combinations of delayed versions of the unit output. The local synapse feedback entails a feedback structure in each ‘synapse’.

The main interest of this paper is to discuss the role played by different types of recurrences in dynamic systems. Thus, five partially recurrent network are proposed to study such differences. The first model has only one-to-one feedback connections from the output to the input layer. The second and third models have the same type of interlayer feedback connections and an extra set of all-to-all intralayer feedback connections. Such links are taken as an extension of local output feedback. They are localized in the input and hidden layers for the second and third architecture respectively. The fourth model is a modification of the second one in which there are all-to-all output-input feedback connections. This kind of links are also included in the fifth model to change the architecture of the third model. In sum, a reference model is established and four alternatives of topologies are proposed. The options are reached by means of modifying the feedback connections in each case.

The work focuses on planning. The task is to establish a sequence of states which takes a set of four blocks from an initial spatial position to a final one. A block can be placed in one out of 3 columns with 4 possible positions each. The functional differences of the models are analyzed by comparing the performance of each model when applied to four different types of trajectories.

This paper is structured as follows. Section 2 surveys recurrent networks. Section 3 introduces the problem to test the proposed models which are described in Section 4. The results on the capacity of the models to correctly recall each sequence starting from the trained initial point or from an arbitrary sequence point are presented in Section 5 and discussed in Section 6.

2. The Recurrent Neural Networks

The class of recurrent networks considered in this work derives from the Multilayer Perceptrons (MLP) trained by backpropagation with a single hidden layer. In this case, there are two main classes of training algorithms: the fixed point and nonfixed point learning algorithms [3]. The fixed point learning algorithms involve networks that converge to stable fixed points. A set of initial states or a constant external input presented to the network produce an output state: the network fixed point. This approach is often applied to constraint satisfaction and associative memory tasks. The recurrent backpropagation [4], [5] and deterministic Boltzmann machines [6] are typical examples of this class of algorithms.

The nonfixed point learning algorithms entail error minimization as function of the temporal behavior of the network. Archetypes of this class of models are backpropagation through time (BTT) [7], the Williams-Zipser architecture [8], [9], and the Elman-Jordan model [10], [11]. The last two models are a discrete version of BTT in which the temporal history is cut off to one or two time steps. The scope of this paper is constrained to deal with nonfixed point learning algorithms, and nondynamic training techniques.

The Backpropagation Through Time [7] is a totally recurrent neural network, i.e., an algorithm in which an unit may be connected to any other one. The activation level of the processing unit is function of the previous output of the network and the last input , if any. Thus, the update rule is given by:

(1)

BTT was originally conceived as having discrete activation with synchronous upgrade. The training phase involves duplicating the network times, where is the number of time steps of the sequence. This process produces an unfolded feedforward network that can be trained.

Williams and Zipser [8], [12], [9] introduced an on-line learning rule to change the weights while a sequence is being presented instead of after the sequence is totally introduced. This approach employs a teacher forcing technique in which the undesired activation of units are corrected when a target is available during the training stage. The dynamics of this model is identical to that of BTT. The length of the sequence does not need to be predetermined.

Figure 1: (a) The Elman model. In this models and all the others from now onwards the bold lines denote all-to-all connections and the ordinary arrows represent one-to-one connections. (b) The model proposed by Jordan.

Elman [10] proposed a three layer network in which the input layer is formed by a set of input units and a set of context units (Figure 1.a). The information is propagated feedforward from the input layer toward the output layer. The activation state of each hidden unit is copied into the corresponding positions of the context units. In this model, the feedforward connections are adaptable whilst the feedback connections are set to one.

The activation state of the context unit is given as:

(2)

Jordan [11], proposed an architecture with the same feedforward topology as Elman network (Figure 1.b). However, the context units receive information from the output units and from themselves. The feedback weights do not vary. This model is a teacher forcing one and this strategy can be extended to any model with output feedback.

The activation state of the context unit is defined as:

(3)

In summary, the Elman-Jordan architecture is characterized by an extra set of processing units in which the network copies the delayed activation state of the hidden or output layer. These feedback connections are fixed with value equal to one. There are also connections from each unit to itself in the context units of Jordan model. The model is also typified by existence of context units, non-trainable self-connections, backpropagation training to the feedforward connections, and synchronous update. Furthermore, the models may fail to reach stability, present very slow convergence process, and decrease its error in a see-saw manner. In order to overcome some of these limitations a class of architectures that involves only local feedback connections was proposed: The local-recurrent-global-feedforward (LRGF) architectures [2].

Stornetta et al. [13] and Mozer [14] proposed local output feedback models. The first one is a feedforward model with three layers in which the context units, forming the input layer, feed their output back to themselves (Figure 2.a). The recurrence preprocesses the input signal in order to consider the past of the input signal in the present context.

Figure 2: (a) The model proposed by Stornetta et al. (b) The model proposed by Mozer.

Thus, the activation of the context unit is influenced by its own output and the stimulus provided by the environment. This is condensed in the following equation:

(4)

Mozer introduced a local output feedback architecture (Figure 2.b) which is formed by four feedforward layers. The context layer has one-to-one adjustable feedback connections which allows the network to consider the context history. Furthermore, the input and context layers are full-connected. The trainable context units acts as an integrator to consider the previous context states. Moreover, the trainable feedback connections allow the model to find an adequate decay rate to the context layer.

Mozer proposed a learning rule quite similar to that derived by Williams and Zipser. The activation of the context unit is given by:

(5)

Frasconi et al. [15] extended the former works to a more general local output feedback model. They presupposed a number of time delays in the feedback links to the hidden layer and a number of inputs from other layers. They proposed a new activation rule for the hidden unit:

(6)

where are the activation of the previous layer; is the threshold input; and is the number of delayed feedback.

Back and Tsoi [16] introduced an LRGF architecture with local synapse feedback. The self-connections are also localized in the hidden layer. In this model, the ‘synapses’ have a linear transfer functions with poles and zeros. The activation function of the processing unit is:

(7)

The transfer function in the ‘synapses’ is given as follows:

(8)

where ; are constants different from zero ; and are the number of poles and zeros respectively.

This model allows weighting on past activation states of the hidden units layer and signals from all input units.

The presented networks are typical examples of LRGF architectures. A general topology for these models involves possible feedback connections from the output, the weighted sum, or the ‘synapse’ of the processing unit. In Figure 3, is the synapse feedback structure, and are transfer functions, and is a non-linear function.

Figure 3: A general topology for the LRGF networks.

The presented models do perform tasks of different natures. Elman architecture can recognize sequences or produce short continuations of given sequences. Jordan model can produce a set of output sequences triggered by distinct input patterns or can recognize input sequences. Mozer network can reproduce sequences when a cluster of context units is added to storage output signals. All the remaining models, which involve only local recurrence, are more suitable to recognize sequences than to generate them.

The analysis of the models above suggests that local feedback enables a network to recognize temporal sequences. Also, architectures with feedback between layers seems to be good at reproducing short sequences. However, these are implemented with addition of units to copy activation of either the output or hidden layers. A model combining both kinds of recurrences might reproduce temporal sequences. Therefore, the objective of this paper is to introduce new architectures that can generate a temporal sequence of states when presented to the initial and final points of this sequence. These models will be tested to the class of problems defined in the next section.

The strategy to propose new architectures is discussed in section 4.

3. Planning Block Movements

The problem treated in this paper consists of moving four blocks (A, B, C, D) from an initial situation to a goal situation. Each block may occupy one of 12 positions established by three columns with four allowed positions each. The movements have the following constrains:

• Each block may occupy any vacant position in any of the three columns;

• Any allowed position may be the initial and final positions;

• It is permitted to move only one block per time;

• A block may be transferred only if its top is free;

• A block may be placed on the free top of other block or on the table;

• A block may be moved for more than one column in a single move.

The Figure 4 illustrates a typical sequence of movements.

Figure 4: Sequence of states from a given initial position to a goal position.

Recently, McAllester and Rosenblitt [17] introduced their SNLP to solve this problem. This algorithm produces a totally ordered sequence of actions which results from search in the partial plans space. The method produces a satisfactory full plan by extending the partial plans. The result of the search is a sequence of operators that may transform a set of assertions that defines the initial state in another one which outlines the final state. This elegant algorithm presents limitations such as: all possible outputs of this plan need to be known and there is the possibility of exponential explosion during the search.

One of the current alternatives to deal with intractability of planning is to use stored solutions of planning problems to produce new plans. This approach is called Case-based Planning or Memory-based Planning [18], [19]. For instance, Miyashita and Sycara [20] introduced a Case-based Planning method to optimize the context-dependent user preferences and tradeoffs. Such an optimization is used to improve the quality of a deliberative schedule or a reactive schedule.

Alternatively, D’Arbo and Araújo [21] proposed neural networks models to perform Case-based Planning. Thus, the aim of the models is to generate the whole set of situations, from the initial state to the final one. The resulting trajectory is based on the trajectories learned by the network. Therefore, the following assumptions are assumed in order to use a recurrent neural network to perform Case-based Planning:

• The planning constraints are absorbed during the learning stage of the network;

• Operators emerge from the network dynamics which produces the allowed operations as function of the present output, previous history, goal, and constraints;

• During the test phase, the network inputs are only the initial and final states of each sequence.

Ideally, the neural planner should generalize, that is, the sequence should be retrieved when started from a sequence state which is different from that one previously trained as initial state.

Figure 5: Trajectories which have in common: (a) At least the goal point; (b) only the initial point; (c) no points at all; (d) any point within the sequence, but the initial one.

The complexity of planning problems may be varied by changing the features of the given inputs and the desired output of such problems. In this paper, the complexity is varied by planning trajectories with distinct types of intersections between them. Thus, the neural planner should be able to deal with trajectories that start from distinct points and converge to a unique goal, trajectories that start from a single initial state and terminate into different goals, trajectories without common points, and trajectories in which any point, but the initial one, can be shared with another path. These trajectories are sketched in Figure 5.

4. The Proposed Models

This paper employs five partially recurrent models to reproduce temporal sequences. Thus, if the initial and final points of a trained sequence are presented to any of the networks, the models should produce the complete sequence. The proposed models employ nonfixed point learning algorithms.

The five models have three layers: the input, the hidden and the output layers. The feedforward structure is the same for all the models. The input layer is totally connected with the hidden layer which is fully connected with the output layer.

The number of units in the input, hidden, and output layers of any architecture is constant and equal to 16, 22, and 16 respectively. The number of units in the input and output layers is function of the chosen representation. The number of units in the hidden layer was established as the minimal number of units that could reproduce all the cases of the first type of trajectory employing the first architecture.

The feedback connections are of two types: output feedback connections and local feedback connections. The first group is present in the five models while the second group exists in last four models. The output feedback connections are trainable only in the fourth and fifth architectures whilst the intralayer connections are trainable connections in the four last models. The intralayer connections are an extension of local feedback links because they are constrained to a particular layer. All feedback connections are delayed in one time step. In sum, the architectures are proposed according to the following strategy. The first one is the architecture to be the reference, it employs interlayer feedback connections to reproduce temporal sequences. Architectures 2 and 3 are proposed to evaluate the effect of local output feedback connections upon the performance of the network. Finally, architectures 4 and 5 transform the one-to-one interlayer feedback connections into all-to-all links in order to evaluate the influence of distributing the interlayer feed back upon the network performance

The propagation, activation and learning rules regarding the feedforward connections are identical to those of the MLP trained by backpropagation. Moreover, the models use on-line learning in conjunction with the teacher forcing strategy for the output feedback. The activation state of units receiving feedback connections are case dependent, thus they will be presented. During the training stage, the feedback connections become a regular set of input connections in which the input values are the output values of other layers delayed of one time step.

The main characteristics of the four last architectures differ in several important ways from the models previously discussed. Firstly, there is not an extra set of units, the context units. Secondly, the feedback links are said local in an extended way: they are all connections within the same layer. Thirdly, all the feedback connections are trainable. Finally, the number of layers and units are constant in all cases.

Figure 6: (a) The first recurrent architecture: the activation of the output layer in fed back into the input layer. (b) The second recurrent architecture: the input layer receives signals from the output layer and from itself.

4.1. The Reference Model

The first proposed model (Figure 6.a) is a partially recurrent network in which the output signal is fed back into the input layer. This model is expected to reproduce simple and short sequences. All the results of the remaining topologies will be compared with the first one in order to evaluate the role played by the different types of recurrence in the Case-based Planning.

The one-to-one feedback links are fixed and equal to the unit. The activation function of the input unit is:

(9)

where and are the activation state of the i-th input and output unit at the time .

In the training stage, the initial point of the sequences having simply one-to-one feedback connections as incoming signals to the input layer is given as:

(10)

where and are the initial and the goal state of the i-th input unit.

4.2. Including Local Feedback

The next set of architectures includes the second and third models. Both have additional self-connections added to them. Such links are expected to solve conflicting situations between trajectories based on the history of the self-connected layer. The local recurrence preprocesses the input signal in order to consider its past states. The trainable connections acts as an integrator to achieve the objective above.

The second model (Figure 6.b) has an additional set of all-to-all feedback connections in the input layer. The activation rule of the input unit becomes:

(11)

where is the activation state of the input unit at the time step ; is a weight, and is the number of units in the input layer.

The function is defined as:

(12)

where is a real number.

The third architecture (Figure 7.a) has all-to-all connections from the hidden layer to itself. This option adds inertia to the mapping realized by the hidden layer. The activation rule of the hidden unit becomes:

(13)

where are weights and is the number of units in the hidden layer.

The activation rule for the input units follows equations (9) and (10).

Figure 7: (a) The third recurrent architecture: the activation of the output layer is fed back into the input layer and the output of the hidden layer is fed back into the input layer. (b) The fourth recurrent architecture: the input layer receives signals from the output layer and from itself through all-to-all feedback links.

Figure 8: The fifth recurrent architecture: the input and hidden layers receive feedback signals from the output and hidden layers respectively.

4.3. Distributed Interlayer Feedback

The next two models change the second and third topologies, transforming the one-to-one self-connections in all-to-all feedback links. This aims at decreasing the failures of the network to recall sequences through increasing association between states in a particular time step and the next one.

The fourth architecture (Figure 7.b) is a modification of the second model. The activation state of the input unit is given as:

(14)

whereis the activation state of the output unit at the time step and is the number of units in the output layer.

The fifth topology (Figure 8) inserts all-to-all feedback connections to those defined to the third model. The activation state of the input unit is given as:

(15)

The activation state of the hidden unit is given as:

(16)

The final step to define the models is to establish the representation of the training patterns. Note that a pattern is defined as a whole temporal sequence.

4.4. The Pattern Representation

A block can be situated in one of the four positions of a column which has an address, formed by combinations of 0’s and 1’s (Figure 9.a). Thus, the spatial position of a block is represented by a set of four units which assumes the address of the block position as activation state. Thus, the activation of each group of four units represents the address ( spatial position) of a block out of 12 possible positions. For instance, if the four blocks are placed as sketched in Figure 9.b, then the blocks A, B, C, and D are represented by: 0 0 0 1, 0 0 1 1, 0 0 1 0, 0 0 0 0. respectively.

Figure 9: (a) The possible addresses for each spatial position of each block; (b) A particular spatial configuration of the blocks.

In sum, each set of block spatial positions is represented by a 16-dimensional vector defined as stated above.

5. The Performance of Each Model

This section reports on the performance of each model. All programs were written in ANSI C and implemented in a PC-Pentium of 100 MHz. The models were trained to reproduce the a number of sequences following the constraints discussed in Section 3. A test consists in presenting the initial and goal states to the network in order to generate the sequence of states between the given points. The initial point may be a trained one or any point in the trajectory but the extreme ones. This strategy allows one to evaluate the networks capacity to recall the trained sequences and to generalize.

In all cases, the training stage stops when all the sequences are correctly recalled or when the error function variation from one epoch to the next one does not decreases any longer. The error function is defined as:

(17)

where is the number of sequences, is the number of patterns in a sequence, is the number of output units, and are the desired and the obtained activation of the output unit respectively.

The variable number of training sequences is consonant with the difficulty to find them. Hence, this research defined 91, 4, 6, and 30 temporal sequences following the restrictions sketched in Figure 5. These sequences were 6, 7, 7, and 7 states long in average. The numbers of epochs to finish the training stages are plotted in Figure 10.a.

Figure 10: (a) Number of epochs of the learning stage per trajectory. From this point onwards Arc stands for architecture in the figures. (b) Percentage of sequences correctly recalled.

The trajectories 2 and 3 are quickly trained, because there are quite a few sequences in these cases. Note that the trajectory 4 takes longer to be trained than the sequence, even though the number of sequences of the latter has tripled that one of the former. For more complex tasks the results suggest that local feedback connections slows down the training speed.

A sequence is correctly recalled if all states from the initial to the final considered point are retrieved. A failure is characterized when the goal state is reached through a sequence distinct of the trained ones or if the goal state is not reached at all.

Figure 10.b shows the percentage of sequences correctly recalled when the initial and the final trained points are presented to the network.

These results suggest that architecture 2 only works if there are few sequences to be memorized. Otherwise, the intralayer self-connections do not lead to sequence reproduction. The remaining four topologies have similar performance in the three first types of trajectories. The best performance for the most complex type of trajectory are achieved by architecture 4. This option illustrates the necessity of additional connections in order to deal with more complex problems, in the input layer.

Figure 11 shows the percentage of sequences correctly retrieved when all possible sequence states, but the trained initial and final points, are presented to the networks as initial state.

The overall generalization capacity is low. However, the results suggest that local links help the generalization.

The architecture 1 has good performance reproducing all, but the fourth, trajectories. This suggests that the interlayer connections do not identify correctly sequences in situation of conflict (the common points of trajectories type 4).

The architecture 2 showed itself inadequate to this type of task. One can infer that the influence of the local connections is significantly greater than that of the interlayer connections, thus most complex sequences can not be reproduced.

The architecture 3 presents performance very similar to architecture 1. However, the former is slower to be trained than the latter. This results suggest that the intralayer connections in the hidden layers are not necessary to solve this task.

Figure 11: Percentage of sequences correctly recalled for non-trained initial states.

The architecture 4 presents the best results in retrieving trained sequences. The combination of the intra- and interlayer connections seem to increase the network power to reproduce and identify sequences. This architecture shows, in average, the best generalization capacity.

The architecture 5 seems to confirm the benefits of the combination mentioned above. However, the position of the intralayer connections are very important for both reproduction of trained sequences and generalization.

The Appendix illustrates several sequence retrievals in each class of trajectory.

6. Conclusions

This work discusses the performance of five partially recurrent neural networks. The models are trained to provide the intermediate situations in order to transport blocks from any given initial state to a goal one, subject to constraints. This problem has its level of complexity varied through trajectories with distinct types of common states.

The analysis of the training stage suggests that the learning capacity of the models varies meanly with the complexity of the trajectory. Moreover, addition of connections does not mean addition of training epochs in all situations.

The results of the tests suggest that the intra- interlayer links increase the network power to reproduce and identify trained or non-trained sequences. Furthermore, the models can reproduce temporal sequences of variable length.

Appendix

This appendix presents a number of illustrations of the sequences retrieved in this research. Figure A1 shows a eight-step sequence out of the ninety one first type of trajectory.

Figure A1: Example of the first type of trajectory correctly recalled.

Figure A2 illustrate a trajectory of the second type.

Figure A2: Example of the second type of trajectory correctly recalled.

Figure A3 shows a trajectory of the third type.

Figure A3: Example of the third type of trajectory correctly recalled.

Figures A4 and ^A5 shows a trajectory of the forth type.

Figure A4: Example of the fourth type of trajectory recalled.

Figure A5: Example of the fourth type of trajectory correctly recalled.

Publication Dates