Chapter Evolving Connectionist and Fuzzy Connectionist Systems: Theory and Applications for Adaptive, On-line Intelligent Systems

7.1. The AVIS framework for integrated auditory and visual information
processing (Kasabov, Postma and Herik)
0
5
10
15
20
25
30
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1
2
3
4
5
6
7
8
9
10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1

---

132
A block diagram of the AVIS framework is shown in fig.7. It consists of three
main parts: auditory subsystem that processes auditory information; visual
subsystem that processes visual information; and higher level decision subsystem
that combines the outputs from the two subsystems. Each of the auditory and 
the
visual sub-systems consists of five modules as described in [43]. The auditory
subsystem and the visual subsystem can each operate as separate subsystems and
have their results combined at a higher-level. Single-modality input streams, or
multimodal input streams, can be used by each of the subsystems. Audio and
visual inputs can be used by either of the subsystems, by the two subsystems
simultaneously, and by the higher-level conceptual subsystem. There are several
different modes of operation allowing for a comparative study of how to use
visual and/or audio information in a different combination [43].
7.2. Using ECOS and EFuNNs for the implementation of AVIS
In an on-line, adaptive mode of operation each of the sub-systems and the modules
of them would develop as the system operates. ECOS and 
EFuNNs are suitable
techniques to be used for such implementation. Here a preliminary 
EFuNN
implementation of some of the modules from the auditory sub-system are
presented and discussed.