Units are connected to one another. A real
number is associated with each connection, which is called the weight of the connection.
It is denoted by Wij, the weight
of connection from unit ui to unit
uj. The weight matrix W represents the pattern whose elements are the weights Wij.
Connections include: excitatory and inhibitory. The positive weights are excitatory
connection while the negative weights are an inhibitory connection.
The activity of output layer is shown by
following two step procedure.
First, the total
weighted input xj is calculated, using the formula:
Where yi represents
activity level of the jth unit in the previous layer and Wij represents the weight of the link between
the ith and the jth unit.
Next, the activity yjis shown using some function which gives the
total weighted input. The sigmoid function is used:
As soon as the activities of all output
units have been found, the error E is
computed by the network, which is given by the following expression:
Where yj represents
the activity level of the jth unit in the top layer and dj gives the output of the jth unit desired.
The back-propagation algorithm has four steps:
1. How fast the error changes with the
activity of an output unit. This error derivative (EA) is the difference
between the actual and the desired activity.
2. How fast the error changes with the
total input that is received by an output unit is changed. This quantity (EI)
is the answer from step 1 multiplied by the rate of change in the output of a
unit with its total input.
3. How fast the error changes with the
weight on the connection into the change in an output unit. This quantity (EW)
is the answer from step 2 multiplied by the level of activity of the unit from
which the connection emanates.
4. How fast the error changes with the
activity of a unit in the previous layer so that back propagation can be
applied to multilayer networks. When the activity of a particular unit in the
previous layer is changed, the activity of all the output units to which it is
connected is changed. So in order to find the overall effect on the error, all
these separate effects on output units are added. But each effect is simple to
calculate. It is the answer in step 2 multiplied by the weight on the
connection to that output unit.
By the use of steps 2 and 4, the EAs of
one layer of units into EAs for the previous layer can be calculated. This
procedure can be repeated to get the EAs for as many previous layers as
desired. Once the EA of a unit is found, steps 2 and 3 can be used to compute the EWs on its incoming