Lecture Notes in Computer Science

bet	9/88
Sana	16.12.2017
Hajmi	12.42 Mb.
	#22381

1 ... 5 6 7 8 9 10 11 12 ... 88

−1

−0.8

−0.6

−0.4

−0.2

0.2

0.4

0.6

0.8

(b)

Fig. 2. Clustering the single-ring data encoded using the range-based encoding method:

(a) The single-ring data, (b) data space quantization due to the range-based encoding,

and (c) data space representation by the output neurons

−5

−3

−1

−5

−3

−1

(a)

−15

−10

−5

−15

−10

−5

(b)

Fig. 3. Improper subclusters formed when the data is encoded with the range-based

encoding method for (a) the double-ring data and (b) the spiral data

layer. This binding is observed to form during the initial iterations of learning

when the ﬁring thresholds of neurons are low. The established binding cannot be

unlearnt in the subsequent iterations, leading to improper clustering at the out-

put layer. An encoding method that overcomes the above discussed limitations

is proposed in the next section.

Region-Based Encoding Using Multi-dimensional

Gaussian Receptive Fields

Using multi-dimensional GRFs for encoding helps in capturing the correlation

present in the data. One approach would be to uniformly place the multi-

dimensional GRFs covering the whole range of the input data space. However,

this results in an exponential increase in the number of neurons in the input

layer with the dimensionality of the data. To circumvent this, we propose a

region-based encoding method that places the multi-dimensional GRFs only in

the data-inhabited regions, i.e., the regions where data is present. The mean

vectors and the covariance matrices of these GRFs are computed from the data

in the regions, thus capturing the correlation present in the data.

To identify the data-inhabited regions in the input space, ﬁrst the k-means

clustering is performed on the data to be clustered, with the value of k being

larger than the number of actual clusters. On each of the regions, identiﬁed using

the k-means clustering method, a multi-dimensional GRF is placed by comput-

ing the mean vector and the covariance matrix from the data in that region.

Response of the i

GRF for a multi-variate input pattern a is computed as,

Region-Based Encoding Method Using Multi-dimensional Gaussians

(a) = exp

−

1

2

− μ

)

−1

− μ

) ,

(4)

where, μ

and Σ

are the mean vector and the covariance matrix of the i

GRF

respectively, and f

(a) is the activation value of that GRF. As discussed in Sec-

tion 3, these activation values are translated into ﬁring times in the range 0 to 9

milliseconds and the non-optimally stimulated input neurons are marked as NF.

By deriving the covariance for a GRF from the data, the region-based encoding

method captures the correlation present in the data. The regions identiﬁed by

k-means clustering and the data space quantization resulting from this encoding

method, for the single-ring data used in Section 3, are shown in Fig 4(a) and

4(b) respectively. The boundary given by the MDSNN with the region-based

encoding method is shown in Fig. 4(c). This boundary is more like the desired

circle-shaped boundary, as against the combination of linear segments observed

with the range-based encoding method (see Fig. 2(c)).

−2

−1.5

−1 −0.5

0.5

1.5

−2

−1.5

−1

−0.5

0.5

1.5

(a)

−2.5 −2 −1.5 −1 −0.5

0.5

1.5

2.5

−2.5

−2

−1.5

−1

−0.5

0.5

1.5

2.5

Fig. 4. (a) Regions identiﬁed by k-means clustering with k = 8, (b) data space quan-

tization due to the region-based encoding and (c) data space representation by the

neurons in the output layer

Next, we study the performance of the proposed region-based encoding

method in clustering complex 2-D and 3-D data sets. For the double-ring data,

the k-means clustering is performed with k = 20 and the resulting regions are

shown in Fig 5(a). Over each of the regions, a 2-D GRF is placed to encode

the data. A 20-8-2 MDSNN is trained using the multi-stage learning method,

discussed in Section 2. It is observed that out of 8 neurons in the hidden layer 3

neurons do not win for any of the training examples and the data is represented

by the remaining 5 neurons as shown in Fig 5(b). These 5 neurons provide the

input in the second stage of learning to form the ﬁnal clusters (Fig 5(c)). The

resulting cluster boundaries are seen to follow the data distribution as shown in

Fig 5(d).

Similarly, the spiral data is encoded using 40, 2-D GRFs. The regions of data

identiﬁed using the k-means clustering method are shown in in Fig 6(a). A

40-20-2 MDSNN is trained to cluster the spiral data. As shown in Fig 6(b), 14

subclusters are formed in the hidden layer that are combined in the next layer to

form the ﬁnal clusters as shown in Fig 6(c). The region-based encoding method

helps in proper subcluster formation at the hidden layer (Fig 5(b) and Fig 6(b)),

against the range-based encoding method (Fig 3).

The proposed method is also used to cluster 3-D data sets namely, the in-

terlocking donuts data and the 3-D ring data. The interlocking donuts data is

L.N. Panuku and C.C. Sekhar

−4

−3

−2

−1

−4

−3

−2

−1

(a)

−4

−3

−2

−1

−4

−3

−2

−1

(b)

−4

−3

−2

−1

−4

−3

−2

−1

(c)

Fig. 5. Clustering the double-ring data: (a) Regions identiﬁed by k-means clustering

with k = 20, (b) subclusters formed at the hidden layer, (c) clusters formed at the

output layer and (d) data space representation by the neurons in the output layer

−15

−10

−5

−15

−10

−5

(a)

−15

−10

−5

−15

−10

−5

(b)

−15

−10

−5

−15

−10

−5

(c)

Fig. 6. Clustering the spiral data: (a) Regions identiﬁed by k-means clustering with

k = 40, (b) subclusters formed at the hidden layer, (c) clusters formed at the output

layer and (d) data space representation by the neurons in the output layer

−2

−1

0

3 −2

−1

−2

−1

(a)

−2

−1

−2

−1

−2

−1.5

−1

−0.5

0.5

1.5

(b)

−2

−1

−2

−1

(c)

−2

−1

−2

−1.5

−1

−0.5

0.5

1.5

−1

(d)

Fig. 7. Clustering of the interlocking donuts data and the 3-D ring data: (a) Regions

identiﬁed by k-means clustering on the interlocking donuts data with k = 10 and (b)

clusters formed at the output layer. (c) Regions identiﬁed by k-means clustering on the

3-D ring data with k = 5 and (d) clusters formed at the output layer.

encoded with 10, 3-D GRFs and a 10-2 MDSNN is trained to cluster this data.

The k-means clustering results and the ﬁnal clusters formed by the MDSNN

are shown in Fig 7(a) and (b) respectively. The clustering results for the 3-D

ring data with the proposed encoding method are shown in Fig 7(c) and 7(d).

For comparison, the performance of the range-based encoding method and the

region-based encoding method, for diﬀerent data sets, is presented in Table 1. It

is observed that the region-based encoding method outperforms the range-based

encoding method for clustering complex data sets like the double-ring data and

the spiral data. For the cases, where both the methods give the same or almost

the same performance, the number of neurons used in the input layer is given

in the parentheses. It is observed that the region-based encoding method always

maintains a low neuron count, there by reducing the computational cost. The dif-

ference between the neuron counts for the two methods may look small for these

2-D and 3-D data sets. However, as the dimensionality of the data increases, this

Region-Based Encoding Method Using Multi-dimensional Gaussians

Table 1. Comparison of the performance (in %) of MDSNNs using the range-based

encoding method and the region-based encoding method for clustering. The numbers

in parentheses give the number of neurons in the input layer.

Data set

Encoding method

Range-based Region-based

encoding

Double-ring data

74.82

100.00

Spiral data

66.18

100.00

Single-ring data

100.00 (10)

100.00 (8)

Interlocking cluster data 99.30 (24)

100.00 (6)

3-D ring data

100.00 (15)

100.00 (5)

Interlocking donuts data 97.13 (21)

100.00 (10)

diﬀerence can be signiﬁcant. From these results, it is evident that the proposed

encoding method scales well to higher dimensional data clustering problems,

while keeping a low count of neurons. Additionally, and more importantly, the

nonlinear cluster boundaries given by the region-based encoding method follow

the distribution of the data or shapes of the clusters.

Conclusions

In this paper, we have proposed a new encoding method using multi-dimensional

GRFs for MDSNNs. We have demonstrated that the proposed encoding method

eﬀectively uses the correlation present in the data and positions the GRFs in the

data-inhabited regions. We have also shown that the proposed method results

in a low neuron count as opposed to the encoding method proposed in [14]

and the simple approach of placing multi-dimensional GRFs covering the data

space. This in turn results in low computational cost for clustering. With the

encoding method proposed in [14], the cluster boundaries obtained for clustering

nonlinearly separable data are observed to be combinations of linear segments

and the MDSNN is failed to cluster the double-ring data and the spiral data.

We have experimentally shown that with the proposed encoding method, the

MDSNNs could cluster complex data like the double-ring data and the spiral

data, while giving smooth nonlinear boundaries that follow the data distribution.

In the existing range-based encoding method, when the data consists of clusters

with diﬀerent scales, i.e., narrow and wider clusters, then the GRFs with diﬀerent

widths are used. This technique is called multi-scale encoding. However, in the

region-based encoding method the widths of the multi-dimensional GRFs are

automatically computed from the data-inhabited regions. The widths of these

GRFs can be diﬀerent. In the proposed method, for clustering the 2-D and 3-D

data, the value of k is decided empirically and the formation of subclusters at

the hidden layer is veriﬁed visually. However, for higher dimensional data, it is

necessary to ensure the formation of subclusters automatically.

L.N. Panuku and C.C. Sekhar

References

1. Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall PTR,

Englewood Cliﬀs (1998)

2. Kumar, S.: Neural Networks: A Classroom Approach. Tata McGraw-Hill, New

Delhi (2004)

3. Maass, W.: Networks of Spiking Neurons: The Third Generation of Neural Network

Models. Trans. Soc. Comput. Simul. Int. 14(4), 1659–1671 (1997)

4. Bi, Q., Poo, M.: Precise Spike Timing Determines the Direction and Extent

of Synaptic Modiﬁcations in Cultured Hippocampal Neurons. Neuroscience 18,

10464–10472 (1998)

5. Maass, W., Bishop, C.M.: Pulsed Neural Networks. MIT-Press, London (1999)

6. Gerstner, W., Kistler, W.M.: Spiking Neuron Models. Cambridge University Press,

Cambridge (2002)

7. Maass, W.: Fast Sigmoidal Networks via Spiking Neurons. Neural Computation 9,

279–304 (1997)

8. Verstraeten, D., Schrauwen, B., Stroobandt, D., Campenhout, J.V.: Isolated Word

Recognition with the Liquid State Machine: A Case Study. Information Processing

Letters 95(6), 521–528 (2005)

9. Bohte, S.M., Kok, J.N., Poutre, H.L.: Spike-Prop: Error-backpropagation in Tem-

porally Encoded Networks of Spiking Neurons. Neural Computation 48, 17–37

(2002)

10. Natschlager, T., Ruf, B.: Spatial and Temporal Pattern Analysis via Spiking Neu-

rons. Network: Comp. Neural Systems 9, 319–332 (1998)

11. Ruf, B., Schmitt, M.: Unsupervised Learning in Networks of Spiking Neurons using

Temporal Coding. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.)

ICANN 1997. LNCS, vol. 1327, pp. 361–366. Springer, Heidelberg (1997)

12. Hopﬁeld, J.J.: Pattern Recognition Computation using Action Potential Timing

for Stimulus Representations. Nature 376, 33–36 (1995)

13. Gerstner, W., Kempter, R., Van Hemmen, J.L., Wagner, H.: A Neuronal Learning

Rule for Sub-millisecond Temporal Coding. Nature 383, 76–78 (1996)

14. Bohte, S.M., Poutre, H.L., Kok, J.N.: Unsupervised Clustering with Spiking Neu-

rons by Sparse Temporal Coding and Multilayer RBF Networks. IEEE Transac-

tions on Neural Networks 13, 426–435 (2002)

15. Panuku, L.N., Sekhar, C.C.: Clustering of Nonlinearly Separable Data using Spik-

ing Neural Networks. In: de S´

a, J.M., Alexandre, L.A., Duch, W., Mandic, D. (eds.)

ICANN 2007. LNCS, vol. 4668, Springer, Heidelberg (2007)

Firing Pattern Estimation of Biological Neuron

Models by Adaptive Observer

Kouichi Mitsunaga

, Yusuke Totoki

, and Takami Matsuo

1

Control Engineering Department, Oita Institute of Technology, Oita, Japan

Department of Architecture and Mechatronics, Oita University, 700 Dannoharu,

Oita, 870-1192, Japan

Abstract. In this paper, we present three adaptive observers with the

membrane potential measurement under the assumption that some of

parameters in HR neuron are known. Using the Strictly Positive Realness

and Yu’s stability criterion, we can show the asymptotic stability of the

error systems. The estimators allow us to recover the internal states and

to distinguish the ﬁring patterns with early-time dynamic behaviors.

Introduction

In traditional artiﬁcial neural networks, the neuron behavior is described only

in terms of ﬁring rate, while most real neurons, commonly known as spiking

neurons, transmit information by pulses, also called action potentials or spikes.

Model studies of neuronal synchronization can be separated in those where mod-

els of the integrated-and-ﬁre type are used and those where conductance-based

spiking and bursting models are employed[1]. Bursting occurs when neuron ac-

tivity alternates, on slow time scale, between a quiescent state and fast repet-

itive spiking. In any study of neural network dynamics, there are two crucial

issues that are: 1) what model describes spiking dynamics of each neuron and

2) how the neurons are connected[3]. Izhikevich considered the ﬁrst issue and

compared various models of spiking neurons. He reviewed the 20 types of real

(cortical) neurons response, considering the injection of simple dc pulses such

as tonic spiking, phasic spiking, tonic bursting, phasic bursting. Through out

his simulations, he suggested that if the goal is to study how the neuronal be-

havior depends on measurable physiological parameters, such as the maximal

conductance, steady-state (in)activation functions and time constants, then the

Hodgkin-Huxley type model is the best. However, its computational cost is the

highest in all models. He also pointed out that the Hindmarsh-Rose(HR) model

is computationally simple and capable of producing rich ﬁring patterns exhibited

by real biological neurons. Nevertheless the HR model is a computational one of

the neuronal bursting using three coupled ﬁrst order diﬀerential equations[5,6],

it can generate a tonic spiking, phasic spiking, and so on, for diﬀerent parame-

ters in the model equations. Charroll simulated that the additive noise shifts the

neuron model into two-frequency region (ı.e. bursting) and the slow part of the

M. Ishikawa et al. (Eds.): ICONIP 2007, Part I, LNCS 4984, pp. 83–92, 2008.

c Springer-Verlag Berlin Heidelberg 2008

K. Mitsunaga, Y. Totoki, and T. Matsuo

responses allows being robust to added noises using the HR model[7]. The pa-

rameters in the model equations are important to decide the dynamic behaviors

in the neuron[12].

From the measurement theoretical point of view, it is important to estimate

the states and parameters using measurement data, because extracellular record-

ings are a common practice in neuro-physiology and often represent the only way

to measure the electrical activity of neurons[8]. Tokuda

et al. applied an adaptive

observer to estimate the parameters of HR neuron by using membrane potential

data recorded from a single lateral pyloric neuron synaptically isolated from other

neurons[13]. However, their observer cannot guarantee the asymptotic stability

of the error system. Steur[14] pointed out that HR equations could not trans-

formed into the adaptive observer canonical form and it is not possible to make

use of the adaptive observer proposed by Marino[10]. He simpliﬁed the three

dimensional HR equations and write as one-dimensional system with exogenous

signal using contracting and the wandering dynamics technique. His adaptive

observer with ﬁrst-order diﬀerential equation cannot estimate the internal states

of HR neurons.

We have recently presented adaptive observers with full states measurement

and with the membrane potential measurement[15]. However, the estimates of

the states by the observer with output measurement are not enough to recover

the immeasurable internal states. In this paper, we present three adaptive ob-

servers with the membrane potential measurement under the assumption that

some of parameters in HR neuron are known. Using the Kalman-Yakubovich

lemma, we can show the asymptotic stability of the error systems based on the

standard adaptive control theory[11]. The estimators allow us to recover the

internal states and to distinguish the ﬁring patterns with early-time dynamic

behaviors. The MATLAB simulations demonstrate the estimation performance

of the proposed adaptive observers.

Review of Real (Cortical) Neuron Responses

There are many types of cortical neurons responses. Izhikevich reviewed 20 of the

most prominent features of biological spiking neurons, considering the injection

of simple dc pulses[3]. Typical responses are classiﬁed as follows[4]:

– Tonic Spiking (TS): The neuron ﬁres a spike train as long as the input

current is on. This kind of behavior can be observed in the three types

of cortical neurons: regular spiking excitatory neurons (RS), low-threshold

spiking neurons (LTS), and ﬁrst spiking inhibitory neurons (FS).

– Phasic Spiking (PS): The neuron ﬁres only a single spike at the onset of the

input.

– Tonic Bursting: The neuron ﬁres periodic bursts of spikes when stimulated.

This behavior may be found in chattering neurons in cat neocortex.

– Phasic Bursting (PB): The neuron ﬁres only a single burst at the onset of

the input.

Firing Pattern Estimation of Biological Neuron Models

– Mixed Mode (Bursting Then Spiking) (MM): The neuron ﬁres a phasic burst

at the onset of stimulation and then switch to the tonic spiking mode. The in-

trinsically bursting excitatory neurons in mammalian neocortex may exhibit

this behavior.

– Spike Frequency Adaptation (SFA): The neuron ﬁres tonic spikes with de-

creasing frequency. RS neurons usually exhibit adaptation of the interspike

intervals, when these intervals increase until a steady state of periodic ﬁring

is reached, while FS neurons show no adaptation.

Single Model of HR Neuron

The Hindmarsh-Rose(HR) model is computationally simple and capable of pro-

ducing rich ﬁring patterns exhibited by real biological neurons.

3.1

Dynamical Equations

The single model of the HR neuron[1,5,6] is given by

x = ax

− x

Download 12.42 Mb.

Do'stlaringiz bilan baham:

1 ... 5 6 7 8 9 10 11 12 ... 88