Firm foundation in the main hci principles, the book provides a working

Figure 9.9 Finger-based interaction using the Leap Motion. (From Leap Motion, Leap Motion 
Controller, 2014, http://www.leapmotion.com [7]).
Figure 9.10 Wristband type of EMG sensor for simple gesture recognition (http://www.thalmic.com).

---

14 9
F U T U R E O F H C I
to the usual keyword text-driven approach, e.g., when the name of the 
object is not known or when it happens to be more convenient to take 
the photo than typing in or voicing the input. Rather, the underlying 
technology of image recognition is more meaningful as an impor-
tant part of object motion tracking (e.g., face/eye recognition for gaze 
tracking, human body recognition for skeleton tracking, and object/
marker recognition for visual augmentation and spatial registration).
Lately, image understanding has become even more important, 
as the core technology for mixed and augmented reality (MAR) has 
attracted much interest. MAR is the technology for augmenting our 
environment with useful information (Figure 9.11). With the spread 
of smartphones equipped with high-resolution cameras, GPUs, and 
light and fashionable see-through projection glasses (not to mention 
near 2-GHz processing power), MAR has started to find its way into 
mainstream usage and may soon revolutionize the way we interact 
with everyday objects. Moreover, with the cloud infrastructure, the 
MAR service can become even more robust and high quality. Finally, 
image recognition can also assume a very important supplemen-
tary role in multimodal interaction. It can be used to extract affect 
properties (e.g., facial expression), disambiguation of spoken words 
(e.g., deictic gestures), and lip movements). See Figures 9.12 and 9.13.
9.1.4 Multimodal Interaction
Throughout this chapter, I have alluded to the need for multimodal 
interaction on many occasions. Even though machine recognition 
rates in most modalities are approaching 100% (with the help of the 
(a)
(b)
(c)
Figure 9.11 Image recognition for (a) face, (b) object/marker (Sony Smart AR, http://www.sony.
net/SonyInfo/News/Press/201105/11-058E), and (c) hand and their applications for motion tracking 
and augmented reality.

---

15 0
H U M A N – C O M P U T E R I N T E R A C T I O N 
cloud–client platform), the usability is not as high as we might expect 
because of a variety of operational restrictions (e.g., ambient noise 
level, camera/sensor field of view, interaction distance, line of sight, 
etc.). To reiterate, this is where multimodal interaction can be of great 
help. In this vein, multimodal interaction has been an active field 
of research in academia beginning with the first pioneering system, 
called “put that there,” developed by Bolt et al. at MIT in the early 
1980s [8]. Since then, various ways of combining multiple modalities 
for effective interaction have been devised (Figure 9.14). Although 
we have already outlined them in Chapter 3, we list them again here.
“Put That There”
Figure 9.12 Bolt’s pioneering “put that there” system. The target object of interest is identified 
from voice and deictic gestures. (From Bolt, R.A., Proceedings of ACM SIGGRAPH, Association for 
Computing Machinery, New York, 1980, pp. 262–270 [8].)
Figure 9.13 Applying image understating to information search and augmentation on a wear-
able display device (Google
®
Glass, https://plus.google.com).

---