Lecture Notes in Computer Science
Download 12.42 Mb. Pdf ko'rish
|
1 −1 −0.8 −0.6 −0.4
−0.2 0 0.2 0.4 0.6
0.8 1 (b) Fig. 2. Clustering the single-ring data encoded using the range-based encoding method: (a) The single-ring data, (b) data space quantization due to the range-based encoding, and (c) data space representation by the output neurons −5 −3 −1 1 3 −5 −3 −1 1 3 x y (a)
−15 −10
−5 0 5 10 15 −15 −10 −5 0 5 10 15 x y (b) Fig. 3. Improper subclusters formed when the data is encoded with the range-based encoding method for (a) the double-ring data and (b) the spiral data layer. This binding is observed to form during the initial iterations of learning when the firing thresholds of neurons are low. The established binding cannot be unlearnt in the subsequent iterations, leading to improper clustering at the out- put layer. An encoding method that overcomes the above discussed limitations is proposed in the next section. 4 Region-Based Encoding Using Multi-dimensional Gaussian Receptive Fields Using multi-dimensional GRFs for encoding helps in capturing the correlation present in the data. One approach would be to uniformly place the multi- dimensional GRFs covering the whole range of the input data space. However, this results in an exponential increase in the number of neurons in the input layer with the dimensionality of the data. To circumvent this, we propose a region-based encoding method that places the multi-dimensional GRFs only in the data-inhabited regions, i.e., the regions where data is present. The mean vectors and the covariance matrices of these GRFs are computed from the data in the regions, thus capturing the correlation present in the data. To identify the data-inhabited regions in the input space, first the k-means clustering is performed on the data to be clustered, with the value of k being larger than the number of actual clusters. On each of the regions, identified using the k-means clustering method, a multi-dimensional GRF is placed by comput- ing the mean vector and the covariance matrix from the data in that region. Response of the i th GRF for a multi-variate input pattern a is computed as, Region-Based Encoding Method Using Multi-dimensional Gaussians 79 f i (a) = exp − 1
(a − μ
i ) t Σ −1 i (a − μ
i ) ,
(4) where, μ
i and Σ
i are the mean vector and the covariance matrix of the i th GRF
respectively, and f i (a) is the activation value of that GRF. As discussed in Sec- tion 3, these activation values are translated into firing times in the range 0 to 9 milliseconds and the non-optimally stimulated input neurons are marked as NF. By deriving the covariance for a GRF from the data, the region-based encoding method captures the correlation present in the data. The regions identified by k-means clustering and the data space quantization resulting from this encoding method, for the single-ring data used in Section 3, are shown in Fig 4(a) and 4(b) respectively. The boundary given by the MDSNN with the region-based encoding method is shown in Fig. 4(c). This boundary is more like the desired circle-shaped boundary, as against the combination of linear segments observed with the range-based encoding method (see Fig. 2(c)). −2 −1.5
−1 −0.5 0 0.5 1 1.5
2 −2 −1.5 −1 −0.5
0 0.5
1 1.5
2 (a)
−2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5
2 2.5
−2.5 −2 −1.5 −1 −0.5
0 0.5
1 1.5
2 2.5
b) Fig. 4. (a) Regions identified by k-means clustering with k = 8, (b) data space quan- tization due to the region-based encoding and (c) data space representation by the neurons in the output layer Next, we study the performance of the proposed region-based encoding method in clustering complex 2-D and 3-D data sets. For the double-ring data, the k-means clustering is performed with k = 20 and the resulting regions are shown in Fig 5(a). Over each of the regions, a 2-D GRF is placed to encode the data. A 20-8-2 MDSNN is trained using the multi-stage learning method, discussed in Section 2. It is observed that out of 8 neurons in the hidden layer 3 neurons do not win for any of the training examples and the data is represented by the remaining 5 neurons as shown in Fig 5(b). These 5 neurons provide the input in the second stage of learning to form the final clusters (Fig 5(c)). The resulting cluster boundaries are seen to follow the data distribution as shown in Fig 5(d). Similarly, the spiral data is encoded using 40, 2-D GRFs. The regions of data identified using the k-means clustering method are shown in in Fig 6(a). A 40-20-2 MDSNN is trained to cluster the spiral data. As shown in Fig 6(b), 14 subclusters are formed in the hidden layer that are combined in the next layer to form the final clusters as shown in Fig 6(c). The region-based encoding method helps in proper subcluster formation at the hidden layer (Fig 5(b) and Fig 6(b)), against the range-based encoding method (Fig 3). The proposed method is also used to cluster 3-D data sets namely, the in- terlocking donuts data and the 3-D ring data. The interlocking donuts data is 80 L.N. Panuku and C.C. Sekhar −4 −3
−1 0 1 2 3 4 −4 −3 −2 −1 0 1 2 3 4 (a) −4 −3 −2 −1 0 1 2 3 4 −4 −3 −2 −1 0 1 2 3 4 (b)
−4 −3 −2 −1 0 1 2 3 4 −4 −3 −2 −1 0 1 2 3 4 (c) Fig. 5. Clustering the double-ring data: (a) Regions identified by k-means clustering with k = 20, (b) subclusters formed at the hidden layer, (c) clusters formed at the output layer and (d) data space representation by the neurons in the output layer −15 −10
−5 0 5 10 15 −15 −10 −5 0 5 10 15 (a) −15
−10 −5 0 5 10 15 −15 −10
−5 0 5 10 15 (b) −15 −10
−5 0 5 10 15 −15 −10 −5 0 5 10 15 (c) Fig. 6. Clustering the spiral data: (a) Regions identified by k-means clustering with k = 40, (b) subclusters formed at the hidden layer, (c) clusters formed at the output layer and (d) data space representation by the neurons in the output layer −2 −1
1 2 3 −2 −1 0 1 2 −2 −1 0 1 2 (a) −2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1.5
−1 −0.5
0 0.5
1 1.5
2 (b)
−2 −1 0 1 2 −2 −1 0 1 2 −1 0 1 (c)
−2 −1 0 1 2 −2 −1.5 −1 −0.5 0 0.5
1 1.5
2 −1 0 1 (d)
Fig. 7. Clustering of the interlocking donuts data and the 3-D ring data: (a) Regions identified by k-means clustering on the interlocking donuts data with k = 10 and (b) clusters formed at the output layer. (c) Regions identified by k-means clustering on the 3-D ring data with k = 5 and (d) clusters formed at the output layer. encoded with 10, 3-D GRFs and a 10-2 MDSNN is trained to cluster this data. The k-means clustering results and the final clusters formed by the MDSNN are shown in Fig 7(a) and (b) respectively. The clustering results for the 3-D ring data with the proposed encoding method are shown in Fig 7(c) and 7(d). For comparison, the performance of the range-based encoding method and the region-based encoding method, for different data sets, is presented in Table 1. It is observed that the region-based encoding method outperforms the range-based encoding method for clustering complex data sets like the double-ring data and the spiral data. For the cases, where both the methods give the same or almost the same performance, the number of neurons used in the input layer is given in the parentheses. It is observed that the region-based encoding method always maintains a low neuron count, there by reducing the computational cost. The dif- ference between the neuron counts for the two methods may look small for these 2-D and 3-D data sets. However, as the dimensionality of the data increases, this Region-Based Encoding Method Using Multi-dimensional Gaussians 81 Table 1. Comparison of the performance (in %) of MDSNNs using the range-based encoding method and the region-based encoding method for clustering. The numbers in parentheses give the number of neurons in the input layer. Data set Encoding method Range-based Region-based encoding
encoding Double-ring data 74.82 100.00
Spiral data 66.18
100.00 Single-ring data 100.00 (10) 100.00 (8) Interlocking cluster data 99.30 (24) 100.00 (6) 3-D ring data 100.00 (15) 100.00 (5) Interlocking donuts data 97.13 (21) 100.00 (10) difference can be significant. From these results, it is evident that the proposed encoding method scales well to higher dimensional data clustering problems, while keeping a low count of neurons. Additionally, and more importantly, the nonlinear cluster boundaries given by the region-based encoding method follow the distribution of the data or shapes of the clusters. 5 Conclusions In this paper, we have proposed a new encoding method using multi-dimensional GRFs for MDSNNs. We have demonstrated that the proposed encoding method effectively uses the correlation present in the data and positions the GRFs in the data-inhabited regions. We have also shown that the proposed method results in a low neuron count as opposed to the encoding method proposed in [14] and the simple approach of placing multi-dimensional GRFs covering the data space. This in turn results in low computational cost for clustering. With the encoding method proposed in [14], the cluster boundaries obtained for clustering nonlinearly separable data are observed to be combinations of linear segments and the MDSNN is failed to cluster the double-ring data and the spiral data. We have experimentally shown that with the proposed encoding method, the MDSNNs could cluster complex data like the double-ring data and the spiral data, while giving smooth nonlinear boundaries that follow the data distribution. In the existing range-based encoding method, when the data consists of clusters with different scales, i.e., narrow and wider clusters, then the GRFs with different widths are used. This technique is called multi-scale encoding. However, in the region-based encoding method the widths of the multi-dimensional GRFs are automatically computed from the data-inhabited regions. The widths of these GRFs can be different. In the proposed method, for clustering the 2-D and 3-D data, the value of k is decided empirically and the formation of subclusters at the hidden layer is verified visually. However, for higher dimensional data, it is necessary to ensure the formation of subclusters automatically. 82 L.N. Panuku and C.C. Sekhar References 1. Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall PTR, Englewood Cliffs (1998) 2. Kumar, S.: Neural Networks: A Classroom Approach. Tata McGraw-Hill, New Delhi (2004) 3. Maass, W.: Networks of Spiking Neurons: The Third Generation of Neural Network Models. Trans. Soc. Comput. Simul. Int. 14(4), 1659–1671 (1997) 4. Bi, Q., Poo, M.: Precise Spike Timing Determines the Direction and Extent of Synaptic Modifications in Cultured Hippocampal Neurons. Neuroscience 18, 10464–10472 (1998) 5. Maass, W., Bishop, C.M.: Pulsed Neural Networks. MIT-Press, London (1999) 6. Gerstner, W., Kistler, W.M.: Spiking Neuron Models. Cambridge University Press, Cambridge (2002) 7. Maass, W.: Fast Sigmoidal Networks via Spiking Neurons. Neural Computation 9, 279–304 (1997) 8. Verstraeten, D., Schrauwen, B., Stroobandt, D., Campenhout, J.V.: Isolated Word Recognition with the Liquid State Machine: A Case Study. Information Processing Letters 95(6), 521–528 (2005) 9. Bohte, S.M., Kok, J.N., Poutre, H.L.: Spike-Prop: Error-backpropagation in Tem- porally Encoded Networks of Spiking Neurons. Neural Computation 48, 17–37 (2002) 10. Natschlager, T., Ruf, B.: Spatial and Temporal Pattern Analysis via Spiking Neu- rons. Network: Comp. Neural Systems 9, 319–332 (1998) 11. Ruf, B., Schmitt, M.: Unsupervised Learning in Networks of Spiking Neurons using Temporal Coding. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 361–366. Springer, Heidelberg (1997) 12. Hopfield, J.J.: Pattern Recognition Computation using Action Potential Timing for Stimulus Representations. Nature 376, 33–36 (1995) 13. Gerstner, W., Kempter, R., Van Hemmen, J.L., Wagner, H.: A Neuronal Learning Rule for Sub-millisecond Temporal Coding. Nature 383, 76–78 (1996) 14. Bohte, S.M., Poutre, H.L., Kok, J.N.: Unsupervised Clustering with Spiking Neu- rons by Sparse Temporal Coding and Multilayer RBF Networks. IEEE Transac- tions on Neural Networks 13, 426–435 (2002) 15. Panuku, L.N., Sekhar, C.C.: Clustering of Nonlinearly Separable Data using Spik- ing Neural Networks. In: de S´ a, J.M., Alexandre, L.A., Duch, W., Mandic, D. (eds.) ICANN 2007. LNCS, vol. 4668, Springer, Heidelberg (2007)
Firing Pattern Estimation of Biological Neuron Models by Adaptive Observer Kouichi Mitsunaga 1 , Yusuke Totoki 2 , and Takami Matsuo 2 1
2 Department of Architecture and Mechatronics, Oita University, 700 Dannoharu, Oita, 870-1192, Japan Abstract. In this paper, we present three adaptive observers with the membrane potential measurement under the assumption that some of parameters in HR neuron are known. Using the Strictly Positive Realness and Yu’s stability criterion, we can show the asymptotic stability of the error systems. The estimators allow us to recover the internal states and to distinguish the firing patterns with early-time dynamic behaviors. 1 Introduction In traditional artificial neural networks, the neuron behavior is described only in terms of firing rate, while most real neurons, commonly known as spiking neurons, transmit information by pulses, also called action potentials or spikes. Model studies of neuronal synchronization can be separated in those where mod- els of the integrated-and-fire type are used and those where conductance-based spiking and bursting models are employed[1]. Bursting occurs when neuron ac- tivity alternates, on slow time scale, between a quiescent state and fast repet- itive spiking. In any study of neural network dynamics, there are two crucial issues that are: 1) what model describes spiking dynamics of each neuron and 2) how the neurons are connected[3]. Izhikevich considered the first issue and compared various models of spiking neurons. He reviewed the 20 types of real (cortical) neurons response, considering the injection of simple dc pulses such as tonic spiking, phasic spiking, tonic bursting, phasic bursting. Through out his simulations, he suggested that if the goal is to study how the neuronal be- havior depends on measurable physiological parameters, such as the maximal conductance, steady-state (in)activation functions and time constants, then the Hodgkin-Huxley type model is the best. However, its computational cost is the highest in all models. He also pointed out that the Hindmarsh-Rose(HR) model is computationally simple and capable of producing rich firing patterns exhibited by real biological neurons. Nevertheless the HR model is a computational one of the neuronal bursting using three coupled first order differential equations[5,6], it can generate a tonic spiking, phasic spiking, and so on, for different parame- ters in the model equations. Charroll simulated that the additive noise shifts the neuron model into two-frequency region (ı.e. bursting) and the slow part of the M. Ishikawa et al. (Eds.): ICONIP 2007, Part I, LNCS 4984, pp. 83–92, 2008. c Springer-Verlag Berlin Heidelberg 2008 84 K. Mitsunaga, Y. Totoki, and T. Matsuo responses allows being robust to added noises using the HR model[7]. The pa- rameters in the model equations are important to decide the dynamic behaviors in the neuron[12]. From the measurement theoretical point of view, it is important to estimate the states and parameters using measurement data, because extracellular record- ings are a common practice in neuro-physiology and often represent the only way to measure the electrical activity of neurons[8]. Tokuda et al. applied an adaptive observer to estimate the parameters of HR neuron by using membrane potential data recorded from a single lateral pyloric neuron synaptically isolated from other neurons[13]. However, their observer cannot guarantee the asymptotic stability of the error system. Steur[14] pointed out that HR equations could not trans- formed into the adaptive observer canonical form and it is not possible to make use of the adaptive observer proposed by Marino[10]. He simplified the three dimensional HR equations and write as one-dimensional system with exogenous signal using contracting and the wandering dynamics technique. His adaptive observer with first-order differential equation cannot estimate the internal states of HR neurons. We have recently presented adaptive observers with full states measurement and with the membrane potential measurement[15]. However, the estimates of the states by the observer with output measurement are not enough to recover the immeasurable internal states. In this paper, we present three adaptive ob- servers with the membrane potential measurement under the assumption that some of parameters in HR neuron are known. Using the Kalman-Yakubovich lemma, we can show the asymptotic stability of the error systems based on the standard adaptive control theory[11]. The estimators allow us to recover the internal states and to distinguish the firing patterns with early-time dynamic behaviors. The MATLAB simulations demonstrate the estimation performance of the proposed adaptive observers. 2 Review of Real (Cortical) Neuron Responses There are many types of cortical neurons responses. Izhikevich reviewed 20 of the most prominent features of biological spiking neurons, considering the injection of simple dc pulses[3]. Typical responses are classified as follows[4]: – Tonic Spiking (TS): The neuron fires a spike train as long as the input current is on. This kind of behavior can be observed in the three types of cortical neurons: regular spiking excitatory neurons (RS), low-threshold spiking neurons (LTS), and first spiking inhibitory neurons (FS). – Phasic Spiking (PS): The neuron fires only a single spike at the onset of the input. – Tonic Bursting: The neuron fires periodic bursts of spikes when stimulated. This behavior may be found in chattering neurons in cat neocortex. – Phasic Bursting (PB): The neuron fires only a single burst at the onset of the input.
Firing Pattern Estimation of Biological Neuron Models 85 – Mixed Mode (Bursting Then Spiking) (MM): The neuron fires a phasic burst at the onset of stimulation and then switch to the tonic spiking mode. The in- trinsically bursting excitatory neurons in mammalian neocortex may exhibit this behavior. – Spike Frequency Adaptation (SFA): The neuron fires tonic spikes with de- creasing frequency. RS neurons usually exhibit adaptation of the interspike intervals, when these intervals increase until a steady state of periodic firing is reached, while FS neurons show no adaptation. 3 Single Model of HR Neuron The Hindmarsh-Rose(HR) model is computationally simple and capable of pro- ducing rich firing patterns exhibited by real biological neurons. 3.1 Dynamical Equations The single model of the HR neuron[1,5,6] is given by ˙ x = ax 2 − x
Download 12.42 Mb. Do'stlaringiz bilan baham: |
ma'muriyatiga murojaat qiling