C++ Neural Networks and Fuzzy Logic

C++ Neural Networks and Fuzzy Logic

by Valluru B. Rao

MTBooks, IDG Books Worldwide, Inc.

ISBN: 1558515526   Pub Date: 06/01/95

Previous Table of Contents Next

Chapter 8

BAM: Bidirectional Associative Memory

Introduction

The Bidirectional Associative Memory (BAM) model has a neural network of two layers and is fully connected

from each layer to the other. That is, there are feedback connections from the output layer to the input layer.

However, the weights on the connections between any two given neurons from different layers are the same.

You may even consider it to be a single bidirectional connection with a single weight. The matrix of weights

for the connections from the output layer to the input layer is simply the transpose of the matrix of weights for

the connections between the input and output layer. If we denote the matrix for forward connection weights by

W, then W

T

 is the matrix of weights for the output layer to input layer connections. As you recall, the

transpose of a matrix is obtained simply by interchanging the rows and the columns of the matrix.

There are two layers of neurons, an input layer and an output layer. There are no lateral connections, that is,

no two neurons within the same layer are connected. Recurrent connections, which are feedback connections

to a neuron from itself, may or may not be present. The architecture is quite simple. Figure 8.1 shows the

layout for this neural network model, using only three input neurons and two output neurons. There are

feedback connections from Field A to Field B and vice−versa. This figure also indicates the presence of inputs

and outputs at each of the two fields for the bidirectional associative memory network. Connection weights

are also shown as labels on only a few connections in this figure, to avoid cluttering. The general case is

analogous.

Figure 8.1

  Layout of a BAM network

Inputs and Outputs

The input to a BAM network is a vector of real numbers, usually in the set { –1, +1 }. The output is also a

vector of real numbers, usually in the set { –1, +1 }, with the same or different dimension from that of the

input. These vectors can be considered patterns, and the network makes heteroassociation of patterns. If the

output is required to be the same as input, then you are asking the network to make autoassociation, which it

does, and it becomes a special case of the general activity of this type of neural network.

For inputs and outputs that are not outside the set containing just –1 and +1, try following this next procedure.

You can first make a mapping into binary numbers, and then a mapping of each binary digit into a bipolar

digit. For example, if your inputs are first names of people, each character in the name can be replaced by its

ASCII code, which in turn can be changed to a binary number, and then each binary digit 0 can be replaced by

C++ Neural Networks and Fuzzy Logic:Preface

Chapter 8 BAM: Bidirectional Associative Memory

152

–1. For example, the ASCII code for the letter R is 82, which is 1010010, as a binary number. This is mapped

onto the bipolar string 1 –1 1 –1 –1 1 –1. If a name consists of three characters, their ASCII codes in binary

can be concatenated or juxtaposed and the corresponding bipolar string obtained. This bipolar string can also

be looked upon as a vector of bipolar characters.

Weights and Training

BAM does not modify weights during its operation, and as mentioned in Chapter 6, like the Hopfield network,

uses one−shot training. The adaptive variety of BAM, called the Adaptive Bidirectional Associative Memory,

(ABAM) undergoes supervised iterative training. BAM needs some exemplar pairs of vectors. The pairs used

as exemplars are those that require heteroassociation. The weight matrix, there are two, but one is just the

transpose of the other as already mentioned, is constructed in terms of the exemplar vector pairs.

The use of exemplar vectors is a one−shot learning—to determine what the weights should be. Once weights

are so determined, and an input vector is presented, a potentially associated vector is output. It is taken as

input in the opposite direction, and its potentially associated vector is obtained back at the input layer. If the

last vector found is the same as what is originally input, then there is resonance. Suppose the vector B is

obtained at one end, as a result of C being input at the other end. If B in turn is input during the next cycle of

operation at the end where it was obtained, and produces C at the opposite end, then you have a pair of

heteroassociated vectors. This is what is basically happening in a BAM neural network.

NOTE:  The BAM and Hopfield memories are closely related. You can think of the

Hopfield memory as a special case of the BAM.

What follow are the equations for the determination of the weight matrix, when the k pairs of exemplar

vectors are denoted by ( X

i

, Y

i

), i ranging from 1 to k. Note that T in the superscript of a matrix stands for the

transpose of the matrix. While you interchange the rows and columns to get the transpose of a matrix, you

write a column vector as a row vector, and vice versa to get the transpose of a vector. The following equations

refer to the vector pairs after their components are changed to bipolar values, only for obtaining the weight

matrix W. Once W is obtained, further use of these exemplar vectors is made in their original form.

W =  X

1T

 Y

1

 + ... + X

kT

 Y

k

and

W

T

 =  Y

1T

X

1

 + ... + Y

kT

 X

k

Example

Suppose you choose two pairs of vectors as possible exemplars. Let them be:

     X

1

 = (1, 0, 0, 1), Y

1

= (0, 1, 1)

and

     X

2

 = (0, 1, 1, 0), Y

2

 = (1, 0, 1)

These you change into bipolar components and get, respectively, (1, –1, –1, 1), (–1, 1, 1), (–1, 1, 1, –1), and

(1, –1, 1).

C++ Neural Networks and Fuzzy Logic:Preface

Weights and Training

153

      1 [−1 1 1]   −1 [1 −1 1]   −1  1  1     −1  1 −1    −2  2 0

 W = −1           + 1          =  1 −1 −1  +   1 −1  1  =  2 −2 0

     −1             1             1 −1 −1      1 −1  1     2 −2 0

      1            −1            −1  1  1     −1  1 −1    −2  2 0

and

       −2    2    2    −2

W

T

 =    2   −2   −2     2

        0    0    0     0

Previous Table of Contents Next

Copyright ©

 IDG Books Worldwide, Inc.

C++ Neural Networks and Fuzzy Logic:Preface

Weights and Training

154

C++ Neural Networks and Fuzzy Logic

by Valluru B. Rao

MTBooks, IDG Books Worldwide, Inc.

ISBN: 1558515526   Pub Date: 06/01/95

Previous Table of Contents Next

You may think that the last column of W being all zeros presents a problem in that when the input is X

1

 and

whatever the output, when this output is presented back in the backward direction, it does not produce X

1

,

which does not have a zero in the last component. There is a thresholding function needed here. The

thresholding function is transparent when the activations are all either +1 or –1. It then just looks like you do

an inverse mapping from bipolar values to binary, which is done by replacing each –1 by a 0. When the

activation is zero, you simply leave the output of that neuron as it was in the previous cycle or iteration. We

now present the thresholding function for BAM outputs.

     1  if  y

j

 > 0   1   if  x

i

 > 0

b

j

|

t+1

 =  b

j

|

t

    if  y

j

 = 0   and   a

i

|

t+1

 =   a

i

|

t

  if  x

i

 = 0

    0   if  y

j

 < 0                                     0   if  x

i

 < 0

where x

i

 and y

j

 are the activations of neurons i and j in the input layer and output layer, respectively, and b

j

|

t

refers to the output of the jth neuron in the output layer in the cycle t, while a

i

|

t

 refers to the output of the ith

neuron in the input layer in the cycle t. Note that at the start, the a

i

 and b

j

 values are the same as the

corresponding components in the exemplar pair being used.

If X

1

 = (1, 0, 0, 1) is presented to the input neurons, their activations are given by the vector (–4, 4, 0). The

output vector, after using the threshold function just described is (0, 1, 1). The last component here is

supposed to be the same as the output in the previous cycle, since the corresponding activation value is 0.

Since X

1

 and Y

1

 are one exemplar pair, the third component of Y

1

 is what we need as the third component of

the output in the current cycle of operation; therefore, we fed X

1

 and received Y

1

. If we feed Y

1

 at the other

end (B field) , the activations in the A field will be (2, –2, –2, 2), and the output vector will be (1, 0, 0, 1),

which is X

1

.

With A field input X

2

 you get B field activations (4, −4, 0), giving the output vector as (1, 0, 1), which is Y

2

.

Thus X

2

 and Y

2

 are heteroassociated with each other.

Let us modify our X

1

 to be (1, 0, 1, 1). Then the weight matrix W becomes

        1  [−1  1  1]       −1  [1  −1  1]       −2    2    0

 W =   −1              + 1               =    2   −2    0

        1                    1                    0    0    2

        1                   −1                   −2    2    0

and

                  −2     2     0    −2

W

T

 =    2    −2     0     2

                   0     0     2     0

Now this is a different set of two exemplar vector pairs. The pairs are X

1

 = (1, 0, 1, 1),

1

 = (0, 1, 1), and X

2

 =

(0, 1, 1, 0), Y

2

 = (1, 0, 1). Naturally, the weight matrix is different, as is its transpose, correspondingly. As

stated before, the weights do not change during the operation of the network with whatever inputs presented to

C++ Neural Networks and Fuzzy Logic:Preface

Weights and Training

155

it. The results are as shown in Table 8.1 below.

Table 8.1 Results for the Example

Input vectoractivationoutput vector

X

1

 = (1, 0, 1, 1)(−4, 4, 2)(0, 1, 1) = Y

1

X

2

 = (0, 1, 1, 0)(2, −2, 2)(1, 0, 1) = Y

2

Y

1

 = (0, 1, 1)(2, −2, 2, 2)(1, 0, 1, 1) = X

1

Y

2

 = (1, 0, 1)(−2, 2, 2, −2)(0, 1, 1,0) = X

2

You may think that you will encounter a problem when you input a new vector and one of the neurons has

activation 0. In the original example, you did find this situation when you got the third output neuron’s

activation as 0. The thresholding function asked you to use the same output for this neuron as existed in the

earlier time cycle. So you took it to be 1, the third component in (0, 1, 1). But if your input vector is a new X

vector for which you are trying to find an associated Y vector, then you do not have a Y component to fall

back on when the activation turns out to be 0. How then can you use the thresholding function as stated?

What guidance do you have in this situation? If you keep track of the inputs used and outputs received thus

far, you realize that the Field B (where you get your Y vector) neurons are in some state, meaning that they

had some outputs perhaps with some training vector. If you use that output component as the one existing in

the previous cycle, you have no problem in using the thresholding function.

As an example, consider the input vector X

3

 = ( 1, 0, 0, 0), with which the activations of neurons in Field B

would be (−2, 2, 0). The first component of the output vector is clearly 0, and the second clearly 1. The third

component is what is in doubt. Considering the last row of the table where Y

2

 gives the state of the neurons in

Field B, you can accept 1, the last component of Y

2

, as the value you get from the thresholding function

corresponding to the activation value 0. So the output would be the vector (0, 1, 1), which is Y

1

. But Y

1

 is

heteroassociated with X

1

. Well, it means that X

3

 = ( 1, 0, 0, 0) is not heteroassociated with any X vector.

Previous Table of Contents Next

Copyright ©

 IDG Books Worldwide, Inc.

C++ Neural Networks and Fuzzy Logic:Preface

Weights and Training

156

C++ Neural Networks and Fuzzy Logic

by Valluru B. Rao

MTBooks, IDG Books Worldwide, Inc.

ISBN: 1558515526   Pub Date: 06/01/95

Previous Table of Contents Next

Recall of Vectors

When X

1

 is presented at the input layer, the activation at the output layer will give ( –4, 4, 2) to which we

apply the thresholding function, which replaces a positive value by 1, and a negative value by 0.

This then gives us the vector (0, 1, 1) as the output, which is the same as our Y

1

. Now Y

1

 is passed back to the

input layer through the feedback connections, and the activation of the input layer becomes the vector (2, –2,

2, 2), which after thresholding gives the output vector (1, 0, 1, 1), same as X

1

. When X

2

 is presented at the

input layer, the activation at the output layer will give (2, –2, 2) to which the thresholding function, which

replaces a positive value by 1 and a negative value by 0, is applied. This then gives the vector (1, 0, 1) as the

output, which is the same as Y

2

. Now Y

2

 is passed back to the input layer through the feedback connections to

get the activation of the input layer as (–2, 2, 2, –2), which after thresholding gives the output vector (0, 1, 1,

0), which is X

2

.

The two vector pairs chosen here for encoding worked out fine, and the BAM network with

four neurons in Field A and three neurons in Field B is all set for finding a vector under

heteroassociation with a given input vector.

Continuation of Example

Let us now use the vector X

3

 = (1, 1, 1, 1). The vector Y

3

 = (0, 1, 1) is obtained at the output layer. But the

next step in which we present Y

3

 in the backward direction does not produce X

3

, instead it gives an X

1

 = (1, 0,

1, 1). We already have X

1

 associated with Y

1

. This means that X

3

 is not associated with any vector in the

output space. On the other hand, if instead of getting X

1

 we obtained a different X

4

 vector, and if this in the

feed forward operation produced a different Y vector, then we repeat the operation of the network until no

changes occur at either end. Then we will have possibly a new pair of vectors under the heteroassociation

established by this BAM network.

Special Case—Complements

If a pair of (distinct) patterns X and Y are found to be heteroassociated by BAM, and if you input the

complement of X, complement being obtained by interchanging the 0’s and 1’s in X, BAM will show that the

complement of Y is the pattern associated with the complement of X. An example will be seen in the

illustrative run of the program for C++ implementation of BAM, which follows.

C++ Implementation

In our C++ implementation of a discrete bidirectional associative memory network, we create classes for

neuron and network. Other classes created are called exemplar, assocpair, potlpair, for the exemplar pair

of vectors, associated pair of vectors, and potential pairs of vectors, respectively, for finding heteroassociation

between them. We could have made one class of pairvect for a pair of vectors and derived the exemplar and

C++ Neural Networks and Fuzzy Logic:Preface

Recall of Vectors

157

so on from it. The network class is declared as a friend class in these other classes. Now we present the

header and source files, called bamntwrk.h and bamntwrk.cpp. Since we reused our previous code from the

Hopfield network of Chapter 4, there are a few data members of classes that we did not put to explicit use in

the program. We call the neuron class bmneuron to remind us of BAM.

Program Details and Flow

A neuron in the first layer is referred to as anrn, and the number of neurons in this layer is referred to as

anmbr. We give the namebnrn to the array of neurons in the second layer, and bnmbr denotes the size of

that array. The sequence of operations in the program is as follows:

•  We ask the user to input the exemplar vectors, and we transform them into their bipolar versions.

The trnsfrm ( ) function in the exemplar class is for this purpose.

•  We give the network the X vector, in its bipolar version, in one exemplar pair. We find the

activations of the elements of bnrn array and get corresponding output vector as a binary pattern. If

this is the Y in the exemplar pair, the network has made a desired association in one direction, and we

go on to the next.step. Otherwise we have a potential associated pair, one of which is X and the other

is what we just got as the output vector in the opposite layer. We say potential associated pair because

we have the next step to confirm the association.

•  We run the bnrn array through the transpose of the weight matrix and calculate the outputs of the

anrn array elements. If , as a result, we get the vector X as the anrn array, we found an associated

pair, (X, Y). Otherwise, we repeat the two steps just described until we find an associated pair.

•  We now work with the next pair of exemplar vectors in the same manner as above, to find an

associated pair.

•  We assign serial numbers, denoted by the variable idn, to the associated pairs so we can print

them all together at the end of the program. The pair is called (X, Y) where X produces Y through the

weight matrix W, and Y produces X through the weight matrix which is the transpose of W.

•  A flag is used to have value 0 until confirmation of association is obtained, when the value of the

flag changes to 1.

•  Functions compr1 and compr2 in the network class verify if the potential pair is indeed an

associated pair and set the proper value of the flag mentioned above.

•  Functions comput1 and comput2 in the network class carry out the calculations to get the

activations and then find the output vector, in the proper directions of the bidirectional associative

memory network.

Previous Table of Contents Next

Copyright ©

 IDG Books Worldwide, Inc.

C++ Neural Networks and Fuzzy Logic:Preface

Program Details and Flow

158

C++ Neural Networks and Fuzzy Logic

by Valluru B. Rao

MTBooks, IDG Books Worldwide, Inc.

ISBN: 1558515526   Pub Date: 06/01/95

Previous Table of Contents Next

Program Example for BAM

For our illustration run, we provided for six neurons in the input layer and five in the output layer. We used

three pairs of exemplars for encoding. We used two additional input vectors, one of which is the complement

of the X of an exemplar pair, after the encoding is done, to see what association will be established in these

cases, or what recall will be made by the BAM network.

Header File

As expected, the complement of the Y of the exemplar is found to be associated with the complement of the X

of that pair. When the second input vector is presented, however, a new pair of associated vectors is found.

After the code is presented, we list the computer output also.