Lecture Notes in Computer Science
Download 12.42 Mb. Pdf ko'rish
|
- Bu sahifa navigatsiya:
- Acknowledgments
|
4 Discussion We have investigated the ability of the neuromuscular system to control the direction of the impedance of the arm during movement. Subjects adapted to three differently oriented position dependent unstable force fields. We found that the endpoint stiffness rotated towards the direction of the instability in each force field. The change in the stiffness matrix occurred in the appropriate components to compensate for the environment. The three fields differed only in the K yx component: the component which determines the force applied in the y-axis in response to an error in the x-axis. In response, while the K xx and K
yy terms were increased in the two rotated DF fields, the main change occurred in the K yx term of the endpoint stiffness of the arm. This change was opposite to the imposed force field K yx term in order to compensate for it. Such a change was brought about by changing the antisymmetric component of the endpoint stiffness. Any changes in the antisymmetric stiffness are thought to be produced by changes in the reflex response, particularly in terms of heteronymous reflex responses [8]. Investigation of the reflex responses produced by the perturbations demonstrated that the reflex gains had been modulated by the CNS. The long latency responses had been either inhibited or excited from the baseline level depending on the force field in which they were moving. Evidence of feedforward changes in the feedback gain have been seen previously in cyclic activities such as cycling [13], ball catching [14] and arm reaching movements [15]. However, here we show that these reflex responses are tuned according to the instability in the environment as part of the impedance controller. The CNS carefully modified the long latency reflex responses of the limb, tuning it to appropriately counteract the disturbing force field. This work demonstrates that the neuromuscular system attempts to selectively increase the impedance in the direction of instability rather than globally. This provides compelling evidence for the existence and utility of an impedance controller in the CNS. It also shows that this impedance controller does not only control the feedforward co-activation of the muscles but also changes the reflex gains in order to appropriately tune the overall response of the neuromuscular system to the environment. Acknowledgments We thank T. Yoshioka for his assistance in running the experiments as well as for programming the PFM. DWF is supported by a fellowship from NSERC, Canada.
922 G. Liaw et al. References 1.
Rancourt, D., Hogan, N.: Stability in force-production tasks. J. Mot. Behav. 33, 193–204 (2001)
2.
Harris, C.M., Wolpert, D.M.: Signal-dependent noise determines motor planning. Nature 394, 780–784 (1998) 3.
Burdet, E., Osu, R., Franklin, D.W., Milner, T.E., Kawato, M.: The central nervous system stabilizes unstable dynamics by learning optimal impedance. Nature 414, 446–449 (2001) 4.
model formation when reaching in a randomly varying dynamical environment. J. Neurophysiol. 86, 1047–1051 (2001) 5.
315–331 (1985) 6.
Franklin, D.W., So, U., Kawato, M., Milner, T.E.: Impedance control balances stability with metabolically costly muscle activation. J. Neurophysiol. 92, 3097–3105 (2004) 7.
stiffness of the arm is directionally tuned to instability in the environment. J. Neurosci. (2007)
8.
Mussa-Ivaldi, F.A., Hogan, N., Bizzi, E.: Neural, mechanical, and geometric factors subserving arm posture in humans. J. Neurosci. 5, 2732–2743 (1985) 9.
Gomi, H., Kawato, M.: Human arm stiffness and equilibrium-point trajectory during multi-joint movement. Biol. Cybern. 76, 163–171 (1997) 10.
measuring endpoint stiffness during multi-joint arm movements. J. Biomech. 33, 1705– 1709 (2000) 11.
characteristics for interaction with environments. J. Neurosci. 18, 8965–8978 (1998) 12.
Ito, T., Murano, E.Z., Gomi, H.: Fast force-generation dynamics of human articulatory muscles. J. Appl. Physiol. 96, 2318–2324 (2004) 13.
Grey, M.J., Pierce, C.W.P., Milner, T.E., Sinkjær, T.: Soleus stretch reflex during cycling. Motor Control 1, 36–49 (2001) 14.
Lacquaniti, F., Maioli, C.: Anticipatory and reflex coactivation of antagonist muscles in catching. Brain Res. 406, 373–378 (1987) 15.
Gomi, H., Saijyo, N., Haggard, P.: Coordination of multi-joint arm reflexes is modulated during interaction with environments. In: 12th Annual Meeting of Neural Control of Movement, E-09, Naples, Florida (2002) Analysis of Variability of Human Reaching Movements Based on the Similarity Preservation of Arm Trajectories Takashi Oyama 1 , Yoji Uno 2 , and Shigeyuki Hosoe 1 1
Shimoshidami, Moriyama-ku, Nagoya 403-0003, Japan 2 Graduate School of Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8601, Japan Abstract. Human movements exhibit some variability. Based on the assumption that the main factor of movement variability is noise adding to motion commands in motion execution, it can be considered that a planned trajectory is the same during the same task. However, the human might not be able to plan the same trajectory during the same task because of some factors (e.g., the uncertainty of target location perception). We analyzed the similarity preservation of trajectories in reaching movements. The similarity preservation of trajectories cannot be reproduced by random noise in motion execution. We argue that the movement variability breaks out in motion planning. 1 Introduction Human movements exhibit some variability, namely the variance or standard de- viation of the points of trajectories. Some hypotheses have been suggested with regard to the factor concerning this variability. Harris & Wolpert [1] suggested “signal dependent noise” (SDN) as the factor of movement variability. SDN is the noise added to a motor command during its execution, and the magnitude of SDN increases with the magnitude of the motor command. van Beers et al. [2] measured the reaching movements along many directions and distances, in- dicating that SDN could reproduce the variability of movements. Based on the assumption that the main factor of movement variability is SDN, it can be con- sidered that a planned trajectory is the same during every trial. For example, a minimum torque change model [3] and a minimum commanded torque change model [4] determine exactly one desired trajectory when the start and target positions are provided. The perception of a target location must be accurate to calculate a trajectory that reaches only the target. However, large deviation and variability exist between the visual and proprioceptive perception of a target location [5]. Sakamoto et al. [6] suggested the uncertainty of target localization in motion planning as the factor of movement variability. We conjecture that the influence of noise on variability during motion execution is not significant and the variability is mainly attributed to the uncertainty of target localization in motion planning. If trajectories vary with noise, the similarity between the M. Ishikawa et al. (Eds.): ICONIP 2007, Part I, LNCS 4984, pp. 923–932, 2008. c Springer-Verlag Berlin Heidelberg 2008 924 T. Oyama, Y. Uno, and S. Hosoe (a) Trajectory variability caused by the noise in motion execution; the second parts (B) of the trajectories are not similar even if the first parts (A) are similar. (b) Trajectory variablility caused by uncertainty in motion planning; the second parts (B) of the trajectories are similar if the first parts (A) are similar. A B
B Fig. 1. Difference in the trajectory similarities depending on the factor of variability trajectories in the first parts may not be kept in the latter half because of the noise (Fig. 1a). On the other hand, if movement variability breaks out with the uncertainty of target localization in motion planning, the similarity between the trajectories will be preserved (Fig. 1b). In order to investigate whether the factor of movement variability is the noise in motion execution or the uncertainty of target localization in motion planning, the similarities between the torque pro- files of actual measured trajectories, trajectories computed with noise in motion execution, and minimum commanded torque change trajectories with variable end points that were made on the basis of the uncertainty of target localization in motion planning were analyzed. 2 Experiment 2.1 Measurement Five subjects participated in this experiment. Human right arm 2-joint (a shoul- der and an elbow) reaching movements between two points in the horizontal plane were examined. When a subject’s right shoulder position was regarded as the origin of Cartesian coordinates, a start position and an end position of the movements were located at (-0.25 m, 0.3 m) and (0.05 m, 0.45 m), respectively. The shape of the target is a circle of which diameter is approximately 1 cm. The subjects could perceive their limbs and the target on the basis of visual informa- tion prior to the motion execution. The subjects were asked to close their eyes as soon as they noticed a signal indicating the onset of movement and to aim at the target in the absence of any visual information. The subjects were asked to aim using a single motion; they were allowed to open their eyes and see the trial result after each movement. A total of 100 trials were measured for each subject. 2.2 Simulated Trajectories with Noise The procedure to compute the trajectories under the influence of noise is shown in Fig. 2 [2]. The mean measured trajectory was computed by averaging the points of the measured trajectories in each subject; the movement duration of each trajectory was normalized. The magnitude of the added noise depends on the motor command and is expressed as σ(t) = N (0, 1) k 2 SDN u 2 (t) + k 2 CON ST Analysis of Variability of Human Reaching Movements 925
Inverse kinematics Inverse dynamics Mean measured trajectory Joint angles Forward kinematics Forward dynamics Noise
Joint torques Motor commands Joint angles Motor commands Joint angles Deformed trajectories Fig. 2. Procedure to compute trajectories deformed by noise mean trajectory (b) intersection points with the normal plane of the mean trajectory: intermediate points variability (a) 95% concentration ellipse of the end points of the trajectories: end points variability Fig. 3. Index of the variability of trajectories; (a) the 95% concentration ellipse of the end points is used as the index of the end points variability, and (b) the standard deviations of the intersection points with the normal plane of the mean trajectory of measured trajectories is used as the index of the intermediate points variability as shown in van Beers et al. [2]. Here, N (0, 1) denotes a Gaussian noise with zero mean value and unit variance, k SDN
determines the extent of noise depending on the motor command, and k CON ST determines the extent of noise indepen- dent of the motor command. u(t) is the motor command and is expressed as u(t) = t
e t a ¨ τ (t) + (t e + t
a ) ˙τ (t) + τ (t) as shown in van der Helm & Rozendaal [7]. Here, t e and t a represent time constants with values of 30 and 40 ms, re- spectively. Further, the movement duration of the trajectories was also changed stochastically according to the normal distribution N (μ md , k
T IM E ). Here, μ md is the mean of the movement duration of measured trajectories. The movement duration of a measured trajectory was defined as the time interval during which the tangential velocity of movement was greater than 0.01 m/s. As shown by Hollerbach & Flash [8], when the mean of the movement duration is μ md and that of the computed trajectory is cμ md , the motor command of the computed trajectory is set to 1/c 2 times that of the original value. The value of noise parameters k SDN
, k CON ST
and k T IM E
were modulated so that the variability of (a) end points or (b) intermediate points of measured trajectories were reproduced. The 95% concentration ellipse of the end points was used as the index of the end points variability, and the standard deviations of the intersection points with the normal plane of the mean trajectory of measured trajectories was used as the index of the intermediate points variability (Fig. 3). (a) Simulated trajectories with noise that reproduces end points vari- ability. The value of noise parameters k SDN , k
CON ST and k
T IM E were
modulated so that the variability of end points of measured trajectories were 926 T. Oyama, Y. Uno, and S. Hosoe Table 1. The values of noise parameters k SDN
, k CON ST
and k T IM E
that reproduce the variability of the end points of measured trajectories Subject A
C D E k SDN
0.018 0.162 0.061 0.069 0.042 k CON ST 0.589 0.568 0.943 0.605 0.353 k T IM E 0.909 1.195 1.365 1.260 1.045 Subject A Subject C Subject D Subject E Y X
4 cm Noise
Measured Fig. 4. The 95% concentration ellipses of the end points of measured trajectories (solid lines) and N oise end
(dashed lines) 0 1 2 3 4 5 400
350 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured Noise
0 1 2 3 4 5 6 7 8 400 350
300 250
200 150
100 Intermediate variability [cm] Time [ms] Measured
Noise 0 2 4 6 8 10 12
14 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured Noise
Subject A Subject B Subject C 0 1 2 3 4 5 6 400 350 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured Noise
0 1 2 3 4 5 300 250
200 150
100 Intermediate variability [cm] Time [ms] Measured
Noise Subject D Subject E Fig. 5. The standard deviations of the intersection points of measured trajectories (solid line) with the normal plane of the mean trajectory and N oise end
(dashed lines) reproduced (Table 1). Hereinafter, these simulated trajectories with noise are referred as N oise end
. The 95% concentration ellipses of the end points of mea- sured trajectories (solid lines) and simulated trajectories with noise (dashed lines) are shown in Fig. 4, and the standard deviations of the intersection points of measured trajectories (solid lines) with the normal plane of the mean trajec- tory and simulated trajectories with noise (dashed lines) are shown in Fig. 5. The horizontal axis illustrates the passage of time from the start point. The variability of the intermediate points of the simulated trajectories was less than that of the measured trajectories; the variability of the actual trajectories could not be reproduced. (b) Simulated trajectories with noise that reproduces intermediate points variability. The value of noise parameters k SDN
, k CON ST
and k T IM E
were modulated so that the variability of intermediate points of measured Analysis of Variability of Human Reaching Movements 927
Table 2. The values of noise parameters k SDN
, k CON ST
and k T IM E
that reproduce the variability of the intermediate points of measured trajectories Subject A
C D E k SDN
0.292 0.586 0.608 0.547 0.307 k CON ST 0.100 0.567 0.566 0.271 0.100 k T IM E 0.909 1.195 1.365 1.260 1.045 Y X 4 cm 4 cm
Subject A Subject B Subject C Subject D Subject E Noise
Measured Fig. 6. The 95% concentration ellipses of the end points of measured trajectories (solid lines) and N oise inter
(dashed lines) 0 1 2 3 4 5 400
350 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured Noise
0 1 2 3 4 5 6 7 8 400 350
300 250
200 150
100 Intermediate variability [cm] Time [ms] Measured
Noise 0 2 4 6 8 10 12
14 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured Noise
Subject A Subject B Subject C 0 1 2 3 4 5 6 400 350 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured Noise
0 1 2 3 4 5 300 250
200 150
100 Intermediate variability [cm] Time [ms] Measured
Noise Subject D Subject E Fig. 7. The standard deviations of the intersection points of measured trajectories (solid line) with the normal plane of the mean trajectory and N oise inter
(dashed lines) trajectories were reproduced (Table 2). In this case, these simulated trajecto- ries with noise are referred as N oise inter
. Observe that parameters k T IM E
are the same with the ones in Table 1. This indicates that the influence of the vari- ability of movement duration on the variability of intermediate points is small. The 95% concentration ellipses of the end points of measured trajectories (solid lines) and simulated trajectories with noise (dashed lines) are shown in Fig. 6, and the standard deviations of the intersection points of measured trajectories (solid line) with the normal plane of the mean trajectory and simulated tra- jectories with noise (dashed lines) are shown in Fig. 7. The variability of the end points of the simulated trajectories was greater than that of the measured trajectories; namely the variability of the actual trajectories could not be well reproduced in this case.
928 T. Oyama, Y. Uno, and S. Hosoe 0.5 0.4
0.3 0.1
0 -0.1
-0.2 -0.3
Y [m] X [m]
0.5 0.4
0.3 0.1
0 -0.1
-0.2 -0.3
Y [m] X [m]
Measured trajectories M CT C
un Fig. 8. Measured trajectories (left) and M CT C un (right). The 95% concentration el- lipse of end points is the same for both the cases. 0 1 2 3 4 5 400
350 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured MCTC
0 1 2 3 4 5 6 7 8 400 350
300 250
200 150
100 Intermediate variability [cm] Time [ms] Measured
MCTC 0 2 4 6 8 10 12
14 300
250 200
150 100
Intermediate variability [cm] Time [ms] Measured MCTC
Subject A Subject B Subject C 0 1 2 3 4 5 6 7 400 350
300 250
200 150
100 Intermediate variability [cm] Time [ms] Measured
MCTC 0 1 2 3 4 5 6 300 250 200
150 100
Intermediate variability [cm] Time [ms] Measured MCTC
Subject D Subject E Fig. 9. The standard deviations of the intersection points of measured trajectories (solid line) with the normal plane of the mean trajectory and M CT C un (dashed lines) Although both the variability of end points and intermediate points could be reproduced by well modulating the noise parameters, the values of parameters are not near in each condition (Table 1 and 2). It appeared difficult to reproduce both variability of the end points and intermediate points simultaneously. 2.3 Simulated Trajectories with the Uncertainty of Target Perception As described in the introduction, a planned trajectory may also exhibit variabil- ity if we assume the uncertainty of the target localization in planning stage [5,6]. In this section, the minimum commanded torque change trajectories between two points (start and end points) were computed to reproduce the uncertainty of target localization in motion planning. Hereinafter, these simulated trajec- tories with the uncertainty of target perception are referred as M CT C un . The end points were imparted variability so that the 95% concentration ellipse of the end points of the measured trajectories was reproduced. The measured trajec- tories and the corresponding minimum commanded torque change trajectories are shown in Fig. 8. The variability of the intermediate points of the measured Analysis of Variability of Human Reaching Movements 929
trajectories (solid lines) and the minimum commanded torque change trajecto- ries (dashed lines) are shown in Fig. 9. The variability of the intermediate points of the minimum commanded torque change trajectories were approximately in agreement with that of the measured trajectories, although the minimum com- manded torque change trajectories were calculated so that only the 95% concen- tration ellipse of the end points of the measured trajectories was reproduced. 2.4 Analysis of the Similarity Preservation of Trajectories The time series of the joint torques were used to evaluate the similarity between the trajectories. It is possible to distinguish different trajectories according to the motor commands even if the paths appear similar. Denote by S m , S ne , S ni and S
mctc the set of all the measured trajectories, simulated trajectories with noise (N oise end
and N oise inter
) and minimum commanded torque change trajectories (M CT C un ), respectively. To each set S m , S
ne , S
ni and S
mctc , the
similarity between the trajectories is evaluated as follows: (1) Select any one of the trajectories in S (S presents either S m , S
ne , S
ni or S mctc ). (2) Compute the squared sum of the differences of torques during a certain time interval A (see Fig. 1) between the selected and other trajectories. (3) Choose N trajectories from S in the ascending order of the squared sum. This set of these N trajectories is referred to as S A . (4) Do procedures (2) and (3) by shifting time interval A to B, where the time interval B comes after A (see Fig. 1). Denote by S B the set of the corre- sponding N trajectories. (5) Denote by M the number of trajectories that belong to both S A and S
B . (6) Repeat (1) to (5) for all selection of the trajectories in S. (7) Average M over all the selection of trajectories in (1). This presents the index of the similarity preservation of S. These procedures were applied to the experimental data. When the parame- ters are selected as N = 20, A =100–200 ms, and B =250–350 ms, the order of the squared sum of the differences of torques for a particular trajectory during intervals A and B is shown in Fig. 10. The black circles in this figure represent trajectories within the 20th order of the squared sum of the differences of torques during time interval A. Those correspond to black circles for time interval B. The number of black circles within the 20th order during time interval B repre- sents the index of the similarity preservation of trajectories (=M ). For example, M is 8 for a trajectory verified in Fig. 10. The procedure was applied for 100 measured trajectories, 100 trajectories with noise (N oise end
and N oise inter
), and 100 minimum commanded torque change trajectories (M CT C un ) for all the subjects. Time intervals A and B were selected depending on the mean of movement duration (μ md ) in each subject (A = 0.2μ md –0.4μ md , B = 0.5μ md –0.7μ
md ,), and N = 20 was provided. We also tested other parameters set that is described later.
930 T. Oyama, Y. Uno, and S. Hosoe 0 20
40 60
80 100
10 20 30 40 50 60 70 80 90 100 Squared difference [(Nm)²] Order 0
40 60
80 100
120 140
160 10 20 30 40 50 60 70 80 90 100 Order Squared difference [(Nm)²] A: interval 100–200 ms B: interval 250–350 ms Fig. 10. Example of trajectory similarity evaluation. Black circles represent trajectories in which the squared sum of the differences of torques is small during the interval of 100–200 ms (left). The number of black circles comprising the top 20 ranks during the interval of 250–350 ms (right) indicates the measure of trajectory similarity. 0 5
15 20
E D C B A Number M Subject Measured
Noise (end) Noise (intermediate) MCTC Fig. 11. The mean of a number of trajectories preserving similarity M (not filled: measured trajectory; light mesh: N oise end
; thick mesh: N oise inter
; filled: M CT C un ; errorbar means standard deviation) 2.5
Results The mean of a number M of trajectories preserving a similarity between the mea- sured trajectories (not filled), N oise end
(light mesh), N oise inter
(thick mesh), and M CT C un (filled) are shown in Fig. 11. In all the subjects, the number M of the trajectories preserving the similarity between the trajectories with noise was smaller than that of the measured trajectories with a significant difference (p < 0.001). No significant difference was observed between the number of tra- jectories preserving similarity M of the measured trajectories and the minimum commanded torque change trajectories for subjects C and E. Although a signifi- cant difference existed between M of the measured trajectories and the minimum commanded torque change trajectories for subjects A, B and D the tendency of the magnitude relation of M was inconsistent. The parameters N (the number of trajectories considered in the ascending order of the squared sum of the differences of torques) and the set of A and B (over which the squared sum of the differences of torques is evaluated) employed during the procedure to calculate the number of trajectories preserving similarity M were modified to a different value and the procedure was executed. When N
Analysis of Variability of Human Reaching Movements 931
was 15 and 30, and the set of A and B was moved before, after, or extended tem- porally (e.g., A = 0.3μ md –0.5μ
md and B = 0.6μ md –0.8μ
md ; A = 0.2μ md –0.5μ
md and B = 0.5μ md –0.8μ
md ), the value of M for the trajectories with noise was always less than that of the measured trajectories with a significant difference (p < 0.001). 3 Discussion In this study, the similarity between the trajectories was investigated to test whether the primary factor of movement variability was “noise in motion execu- tion” or “the uncertainty of location perception in motion planning.” If the vari- ability of the actual trajectories results from random noise in motion execution, the preservation of the similarity between the trajectories would depend on the probability distribution of noise. In contrast, if variability is included in motion planning, the similarity between the realized trajectories is well preserved. The number of trajectories preserving similarity M for the trajectories with noise was significantly smaller than that of the measured trajectories. The minimum com- manded torque change trajectories in which the end points exhibited almost the same variability as the measured trajectories (based on the assumption of the un- certainty of location perception in motion planning) could reproduce M of the measured trajectories better than that of trajectories with noise. Further, the min- imum commanded torque change trajectories could roughly reproduce the vari- ability of the intermediate points of the measured trajectories even though they were not specified. It is appropriate that the primary factor of movement variabil- ity is the uncertainty of location perception in motion planning as compared to randomly added noise to a motor command during motion execution. In this ex- periment, the subjects were asked not to use visual feedback during motion execu- tion. The influence of available somatosensory feedback on movement variability should be considered. If a subject correctly localizes a target position in motion planning and uses lots of somatosensory feedback in motion execution, trajectories would tend to converge on the perceived position as time passes and the movement variability will decrease. As shown in Fig. 5, however, the variability of interme- diate points of measured trajectories increases as time passes. We conjecture that the subjects use almost feedforward control to perform reaching movements and the influence of somatosensory feedback is small on the results. Schmidt et al. [9] showed that the variability of end points increases when the movement distance and movement duration are longer. How are these results explained under the assumption that movement variability breaks out in motion planning? The first explanation is that when a subject rapidly moves his hand toward a target to perform a task, the application of feedback control in the vicinity of the target is difficult. Since the subject is instructed to stop his hand at the end of the movement certainty during a typical reaching task, he may be more concerned about stopping his hand rather than the accuracy of reach- ing; hence, the variability of the end points increases. The second explanation is that the difficulty of computing a desired trajectory increases the amount of
932 T. Oyama, Y. Uno, and S. Hosoe computation since the movement duration increases. When a subject performs a slow and time-consuming movement, the amount of planned trajectory data may increase, and it increases the movement variability. We have measured very slow reaching movements (movement duration of 6 sec) and fast reaching move- ments (movement duration of 0.5 sec) and revealed that the variability of the intermediate points of the slow movements was greater than that of the fast movements [10]. Movement variability involves the computation difficulty of a planned trajectory. We propose the uncertainty of location perception as one of the factors of variability in motion planning. A planned trajectory is not necessarily invari- able, such as a mean trajectory for the same task, and it exhibits a large vari- ability. The manner in which a planned trajectory is expressed and computed, and the required information of perception should be investigated. The method to estimate a planned trajectory from a measured trajectory [11] is useful to investigate the variability in motion planning. Acknowledgment. This work was supported by Grant-in-Aid for scientific Re- search (B) 18360202 from JSPS. References 1. Harris, C.M., Wolpert, D.M.: Signal-dependent noise determines motor planning. NATURE 394, 780–784 (1998) 2. van Beers, R.J., Haggard, P., Wolpert, D.M.: The role of execution noise in move- ment variability. J. Neurophysiol. 91, 1050–1063 (2004) 3. Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement. Biol. Cybern 61, 89–101 (1989) 4. Nakano, E., Imamizu, H., Osu, R., Uno, Y., Gomi, H., Yoshioka, T., Kawato, M.: Quantitative examinations of internal representations for arm trajectory planning: minimum commanded torque change model. J. Neurophysiol. 81, 2140–2155 (1999) 5. Kitagawa, T., Fukuda, H., Fukumura, N., Uno, Y.: Investigation of error in human perception of hand position during arm movement (in Japanese). IEICE J89-D, 1429–1439 (2006) 6. Sakamoto, T., Fukumura, N., Uno, Y.: Variability in human reaching move- ments depends on perception of targets (in Japanese). Technical Report of IEICE NC2002-175, 19–24 (2003) 7. van der Helm, F.C.T., Rozendaal, L.A.: Musculoskeletal systems with intrinsic and proprioceptive feedback. In: Winters, J.M., Crag, P.E. (eds.) Biomechanics and Neural Control of Posture and Movement, pp. 164–174 (2000) 8. Hollerbach, J.M., Flash, T.: Dynamic interaction between limb segments during planar arm movement. Biol. Cybern 44, 67–77 (1982) 9. Schmidt, R.A., Zelaznik, H., Hawkins, B., Frank, J.S., Quinn Jr., J.T.: Motor- output variability: a theory for the accuracy of rapid motor acts. Psychological Review 86, 415–451 (1979) 10. Oyama, T., Uno, Y.: Variability of human arm reaching movements occurs in mo- tion planning (in Japanese). In: BPES20th, pp. 149–152 (2005) 11. Oyama, T., Uno, Y.: Estimation of a human planned trajectory from a measured trajectory (in Japanese). IEICE J88-DII, 800–809 (2005)
Directional Properties of Human Hand Force Perception in the Maintenance of Arm Posture Yoshiyuki Tanaka and Toshio Tsuji Department of Artificial Complex Systems Engineering, Hiroshima University, 1-4-1 Kagamiyama, Higashi-hiroshima, 739-8527, Japan {ytanaka,tsuji}@bsys.hiroshima-u.ac.jp http://www.bsys.hiroshima-u.ac.jp Abstract. This paper discusses the directional properties of human hand force perception during maintained arm posture in operation of a robotic device by means of robotic and psychological techniques. A series of perception experiments is carried out using an impedance-controlled robot depending on force magnitudes and directions in the different arm postures. Experimental results demonstrate that human hand force per- ception is much affected by the stimulus direction and can be expressed with an ellipse. Finally, the relationship between hand force perception properties and hand force manipulability is analyzed by means of human force manipulability ellipse. Keywords: Force perception, multi-joint arm, human force manipula- bility. 1
Humans can control dynamic properties of his/her own body according to tasks by utilizing the perceived information of environmental characteristics. For ex- ample, in the door open-close task usually appearing as a constrained task in our daily activities, we can carry out a smooth operation without feeling an ex- cessive load by controlling hand movements and arm configurations at the same time depending on kinematical and dynamical characteristics of the maneuver- ing door. If the relationship between human sensation and dynamic properties of movements for environmental characteristics can be described quantitatively, it would be useful to evaluate and develop a novel human-machine system in which the operator can manipulate the machine comfortably. In the field of robotics, there have been several methods to evaluate the ma- nipulability of a robot with a serial link mechanism for each posture from the kinematical and dynamical viewpoints in the operational task space [2,3]. These methods can quantitatively indicates the directional dependence of robot per- formance by representing the shape and size of an ellipse for the specified pos- ture. Some researches applied robot manipulability into the analysis of human movements [4] and the evaluation of welfare equipments [5] to develop more comfortable and safety mechanical interfaces for a human operator. The robot M. Ishikawa et al. (Eds.): ICONIP 2007, Part I, LNCS 4984, pp. 933–942, 2008. c Springer-Verlag Berlin Heidelberg 2008 934 Y. Tanaka and T. Tsuji manipulability, however, cannot consider kinetic characteristics of the human musculoskeletal system that should be taken into account for evaluating hu- man movements. For such a problem, Tanaka and Tsuji developed human force manipulability based on human joint-torque characteristics that can estimate human force capability in good agreement with experimental data, and applied to the layout problem of driving interfaces of a human-robotic system [6][7]. How- ever, these previous researches did not concern how a human operator would feel dynamic properties of designed devices although the improvement of operational feeling was stated as a major study purpose. On the other hand, a large number of studies on the properties of human perception have been carried out in the psychometric field so far, and there are some well-known law based on experimental findings, e.g., Weber-Fechner law; a just-noticeable difference in a stimulus is proportional to the magnitude of the original stimulus [8]. Many experimental studies have been also reported on human force perception [10]–[15]. Especially, Jones [13] examined the per- ceived magnitude of forces exerted to muscle groups in the finger, upper-arm, and forearm, and reported that human force perception ability is much affected by muscle size. This experimental evidence suggests that directional properties might be involved in human hand force perception since the muscles used change depending on the direction of the force exerted in multi-joint arm movement. No research, however, has been reported on the directional dependency of human hand force perception as far as the authors know. The objective of the present paper is to clarify whether the force direction affects human hand force perception properties during maintained arm posture in operation of a robotic device. Experimental findings will be useful as basic data for designing kinematic and dynamic properties of a robotic system as well as the control strategy to assist operator’s motion more safety and comfortably. This paper is organized as follows: Section 2 explains the experimental system and method using an impedance-controlled robot for investigating human hand force perception, and presents typical experimental results. Section 3 discusses the directional dependency of human hand force perception and the relationship between hand force perception properties and hand force manipulability by using human force manipulability ellipse [7]. 2 Hand Force Perception Experiment 2.1 Experimental System Fig. 1 shows an overview of the constructed experimental apparatus. The robot is composed of two linear motor tables with one degree of freedom (NSK Ltd., maximum driving force: x axis 100 [N], y axis 400 [N], encoder resolution: x axis 4 [μm], y axis 4 [μm]), where the two tables are placed orthogonally in order to carry out the two-dimensional hand motion exercise [16]. Hand force generated by a human operator is also measured using a six-axis force/torque sensor on the handle (Nitta Corp., resolution: force x and y axes 200 [N], z axis 400 [N],
Directional Properties of Human Hand Force Perception 935
Robot Force
Position Bio-feedback Impedance control Human
Force sensor Display
x y DSP system (a) φ
d X φ
d X X 0
0 : Hand position : Target position of equilibrium : Initial position : Target direction (b)
Fig. 1. Experimental apparatus for the human hand force perception torque 18 [Nm]). Hand position X ∈ 2
the linear motor table. The robotic device is impedance-controlled [17], and can provide force loads F ∈
to the operator’s hand by adjusting the impedance parameters. Thus, dynamics of the robot is as follows: − F (t) = M ¨ X(t) + B ˙ X(t) + K(X(t) − X
v (t))
(1) where M = diag.(m r , m
r ), B = diag.(b r , b
r ), K = diag.(k r , k
r ) ∈ 2 ×2 is the robot inertia, viscosity, and stiffness; and X v ∈ 2 is the equilibrium of K. The robot control is performed using a DSP system (A & D Company, AD5410) that can provide stable control based on the real-time simulation output of the Matlab/Simulink (Mathworks Inc.) as well as high-quality data measurement in high sampling. The biofeedback display, programmed in Open GL, shows the target direction of the hand force φ with an arrow and the current hand position X with a circle. 2.2
Experimental Method Experiments were carried out based on a magnitude estimation method [8]. A human subject was seated in the front of the robot as shown in Fig. 2. The shoulders of the subject were restrained by a shoulder harness belt to the chair back, and the elbow of the right arm was hung from the ceiling by a rope to maintain his right arm posture in the horizontal plane without excessive co-contractions of the muscles. The right wrist and hand were tightly fixed by a molded plastic cast to the robot handle to eliminate the influence of tactile sensation in the experiment as much as possible [9]. The subject was instructed to keep his hand position at the initial position X 0
position of robot stiffness X v was smoothly moved to the target position X d 936 Y. Tanaka and T. Tsuji θ 1
2 φ
y Fig. 2. Experimental condition on the arm posture located on the circle with radius 4 [cm] in the specified direction φ within two seconds: The force stimulus was gradually increased to the target value for two seconds. After that, the subject perceived the reaction force for three seconds, and reported the perceived value in percentage terms with respect to the stan- dard force stimulus. The presented force series was composed of the six different magnitude F = 5, 10, 15, 20, 25, 30 [N], and the eight different directions φ = 0, 45, 90, 135, 180, 225, 270, 315 [deg.] in consideration of the performance of the employed robot and a human subject. The presented values of force F and direction φ were randomly determined by the experimental system. The standard force magnitude was set at F s = 15 [N] and the standard force direction was at φ s = 0 [deg.] in this paper, i.e., the standard force stimulus was at (F s , φ s ) = (15 [N], 0 [deg.]). Each magnitude of the different six force stimuli was provided five times in each direction, and the total number of trials was 240 (= 30 × 8) for each arm posture. The standard force stimulus was presented after every three trials so that a human subject can remember the standard force stimulus during the perception test. Under the above conditions, the arm posture θ = (θ 1 , θ
2 ) ∈ 2 was changed as θ 1 = 30, 60, 90 [deg.] under θ 2 = 60 [deg.]. The presented force magnitude was generated by changing the robot stiff- ness k
r . The robot inertia was set at m r = 5 [kg] and the robot viscosity b r was automatically adjusted for critical damping to avoid the oscillation of hand movements during the perception test. The sampling frequency for robot control was 1 [kHz] in this study. Fig. 3 shows typical measured signals during the perception test with the standard force stimulus. The hand displacement from the initial position for each axis, and the hand force for each axis are presented in the order from the top. It can be seen that the smooth change of force amplitude is realized using the developed experimental system, and that hand motion is almost constant during the perception term for three seconds. Directional Properties of Human Hand Force Perception 937
-0.02 0 0.02 0.04 0.06
-0.02 0 0.02 0.04 0.06
-0.02 0 0.02 0.04 0.06
-20 -10
0 0 2 4 6 8 -20 -10
0 Equilibrium position,
[m]
Hand displacement, dx [m]
Hand displacement, dy [m]
Hand force, fy [N]
Hand force, fx [N]
5 5
Time [s] Fig. 3. Time profiles of the measured signals during a force perception test with the standard force stimulus 2.3
Experimental Results Six right-handed volunteers (male university students, aged 23 - 25) were partic- ipated in the perception test. The subjects carried out some practice until they understood the point of the perception test and had enough confidence in their answers. Fig. 4 shows the results of the force perception test for Subject A in the case of (θ 1 , θ 2 ) = (30 [deg.], 60 [deg.]) where the vertical axis in each graph is the perceived force amplitude F p , and the horizontal axis is the true force F t normalized with the standard force magnitude F s (= 15 [N]). The solid line for each direction denotes a regression curve with a logarithmic function, y = A ln(x) + B, obtained by fitting the all data (30 samples) using the least squares method, and the dotted line is 95 % prediction interval assuming that the normal distribution is satisfied in the data. The black circle • is the perception result for F s , and r 2 represents the coefficient of determination between true and perceived values. 938 Y. Tanaka and T. Tsuji y = 58.6 ln (x) - 148.4 y = 60.6 ln (x) - 161.9 y = 61.6 ln (x) - 173.6 y = 55.1 ln (x) - 138.0 y = 48.6 ln (x) - 95.8 y = 59.5 ln (x) - 150.3 y = 62.5 ln x - 178.0 y = 58.0 ln (x) - 151.7 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] r = 0.94 r = 0.96 r = 0.96 r = 0.94 r = 0.98 r = 0.88 r = 0.98 r = 0.94 x y 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 0 50 100 150 200 0 50 100 150
200 Perception value, F p [%]
True value, Ft [%] 2 2 2 2 2 2 2 2 Fig. 4. Relationship between true and perceived hand forces under θ 1 = 30 [deg.] (Subject A) The subject almost correctly perceived the standard force stimulus (F s , φ
s ) in each specified arm posture, and there exist ranges where the subject perceives hand force as larger or smaller than the true one. The perceived force tends to be larger than the true one when the presented force magnitude was smaller than the standard one, while smaller when the presented force magnitude was larger. The coefficient of determination is over 0.88, and the relationship with the logarithm of the present force is almost proportional. That is, the Weber- Fechner law is almost satisfied in hand force perception for all combination of the specified directions and arm postures. The significant difference of perceived force to the standard force magnitude (a mark • in Fig. 4) was observed in the perception direction φ for each of the specified arm posture by the analysis of variance (ANOVA) with significant level p < 0.001: F(7, 32) = 20.94 for θ 1 = 30 [deg.]; F(7, 32) = 8.14 for θ 1 = 60 [deg.]; F(7, 32) = 6.24 for θ 1 = 90 [deg.]. These characteristics on hand force perception mentioned here were observed for the other subjects. 3 Directional Properties of Human Hand Force Perception 3.1
Human Force Perception Ellipse Further analysis on the directional properties of human hand force perception was executed using “Point of Subjective Equality (PSE)” [8] with respect to the standard force stimulus. The PSE in this paper was defined as the value of Directional Properties of Human Hand Force Perception 939
a regression curve at F p = 100, that is the force magnitude to make a human subject perceive the standard force stimulus. The PSE was calculated backward using the regression curve for each of directions and arm postures. Fig. 5 shows the results of the PSE analysis for all subjects depending on the arm posture θ 1 , where the vertical axis is the force magnitude corresponding to PSE and the horizontal axis is the perception direction φ. It can be seen that the PSE changes in the perception direction as well as the arm posture although some individual differences exist. Next, the changes of PSE in the direction φ was expressed with an ellipse defined by the following quadratic form as: F p cos φ − d
F p sin φ − e T a b b c F p cos φ − d
F p sin φ − e = 1
(2) where a, b, c, d and e are determined by fitting the PSE shown in Fig. 5 with a least squares method. Fig. 6 shows the results of a human hand force perception ellipse (HFPE) depending on the arm posture for Subs. A, B, and C, where the radius between the center of a HFPE and the white circle denotes the value of PSE. It can be seen that the directional changes of PSE is well expressed with an ellipse. The major axis of an ellipse represents the direction where a larger force than the standard force F s (= 15 [N]) will be required to make a human subject have the illusion that he perceives F s , while the minor axis the direction where a smaller force will be required. Accordingly, the HFPE indicates that a human subject perceives F s as smaller toward the direction of the major axis while he perceives as larger toward the direction of the minor axis. It can be also found that the HFPE changes depending on the specified arm posture, and that the major axis tends to be oriented toward the shoulder point. Similar tendencies were observed for all subjects, although some individual exceptions were found. 3.2 Relationship between Human Force Manipulability Finally, the directional properties of human hand force perception were asso- ciated with the human hand force manipulability ellipse (HFME) [7] defined by f
(J (θ)T (θ) −1 )(J (θ)T (θ) −1 ) T f ≤ 1,
(3) where f denotes a hand force generated by a human, J denotes a Jacobian matrix on the hand position with respect to the arm posture, and T denotes a matrix representing the characteristics of human arm joint-torque with maxi- mum effort. The size and shape of HFME can be utilized as a performance index in generating the maximum hand force according to the operational direction under the specified arm posture θ. Large operational force can be easily exerted in the major axis direction, while it is difficult toward the minor axis direction. The results of HFME for each of the arm postures is drawn with a gray ellipse under the HFPE as shown in Fig. 6, where the human arm is modeled as a rigid 940 Y. Tanaka and T. Tsuji (c) Subject C (f) Subject F Direction, [deg.] φ PSE [N] 0 5 10 15 20 25 30 0 45 90 135
180 225
270 315
Direction, [deg.] φ PSE [N] 0 5 10 15 20 25 30 0 45 90 135
180 225
270 315
(b) Subject B (e) Subject E Direction, [deg.] φ PSE [N] 0 5 10 15 20 25 30 0 45 90 135
180 225
270 315
Direction, [deg.] φ PSE [N] 0 5 10 15 20 25 30 0 45 90 135
180 225
270 315
(a) Subject A (d) Subject D Direction, [deg.] φ PSE [N] 0 5 10 15 20 25 30 0 45 90 135
180 225
270 315
Direction, [deg.] φ PSE [N] 0 5 10 15 20 25 30 0 45 90 135
180 225
270 315
= 30 [deg.] = 60 [deg.] = 90 [deg.] θ 1 θ 1 θ 1 = 30 [deg.] = 60 [deg.] = 90 [deg.] θ 1
1 θ 1 = 30 [deg.] = 60 [deg.] = 90 [deg.] θ 1 θ 1 θ 1 = 30 [deg.] = 60 [deg.] = 90 [deg.] θ 1
1 θ 1 = 30 [deg.] = 60 [deg.] = 90 [deg.] θ 1 θ 1 θ 1 = 30 [deg.] = 60 [deg.] = 90 [deg.] θ 1
1 θ 1 Fig. 5. PSE depending on the arm postures and the direction of motion for the subjects (a) Subject A (b) Subject B (c) Subject C Shoulder Elbow
Elbow Shoulder
Shoulder Elbow
150 [N] 15 [N]
HFME HFPE
Fig. 6. HFPE and HFME for the three subjects (Subjects A, B and C) link structure with two rotational joints (See Fig. 2). The length of forearm and upper-arm was measured for each subject and the joint-torque matrix was determined based on the previous work [7]. It can be found that the orientation of HFME is similar to one of HFPE in each arm posture for the subjects. The orientation difference between HFPE and HFME Δψ is then summarized for all six subjects in Fig. 7. Although there exists some differences, the orien- tation of HFPE almost agrees with one of HFME. The results indicate that a human perceives a reaction force exerted on the hand as smaller than the actual force in the direction where he can easily generate a larger hand force, and vise versa. The possible reasons why the anisotropy of HFPE is not clear compared than HFME are 1) the HFME expresses the directional properties of generating the maximum hand force under maximum effort, and 2) a human perceives an re- action force as smaller than the actual one when the force magnitude is enough large since Weber-Fechner law is almost satisfied in hand force perception. Directional Properties of Human Hand Force Perception 941
0 20 40 Sub.A Sub.C
Sub.B Sub.D
Sub.E Sub.F
Orientation dif ference
between HFPE and HFME, [deg.] Δψ θ
= 90 [deg.] θ 1 = 60 [deg.] θ 1 = 30 [deg.] Fig. 7. Orientation differences between HFPE and HFME for all subjects A series of experimental and simulated results demonstrates the directional properties of human hand force perception and its close relationship with the motor performance of hand force generation. 4 Conclusion This paper experimentally analyzed the influence of direction on the perception of human hand force during maintained arm posture by means of robotic and psychological techniques. The main results are summarized as follows: 1) Humans have directional hand force perception properties that change the magnitude of subjective sensation according to the direction of the force exerted, 2) The directional properties change according to arm posture, and can expressed with an ellipse, and 3) There is the close relationship between HFPE and HFME. These findings on human sensorimotor characteristics will be useful in designing an advanced user-friendly machine, a virtual reality system and a rehabilitation system using robotic devises. The future research will be directed to examine for other arm configurations with considerations of the muscle co-contractions and the sensation of active movements during the force perception test in order to clarify the mechanism of human hand force perception physiologically. It is also planned to evaluate and design human-machine systems, such as a neuro-rehabilitation system, based on human sensorimotor characteristics. Acknowledgments. This research work was supported in part by a Grant-in- Aid for Scientific Research from the Japanese Ministry of Education, Science and Culture (18760193, 15360226). The authors would like to appreciate the kind supports of the A&D Co. Ltd. on the DSP instrument. References 1. Ito, K.: Physical wisdom systems, Kyoritsu Shuppan (in Japanese) (2005) 2. Asada, H.: Geometrical representation of manipulator dynamics and its application to arm design. Transaction of the ASME, Journal of Dynamic Systems, Measure- ment, and Control 105, 105–135 (1983)
942 Y. Tanaka and T. Tsuji 3. Yoshikawa, T.: Analysis and control of robot manipulators with redundancy. In: Brady, M., Paul, R. (eds.) Robotic Research, the 1st International Symposium, pp. 735–747. MIT Press, Cambridge (1984) 4. Hada, M., Yamada, D., Tsuji, T.: Equivalent inertia of human-machine systems under constraint environments. In: Proceedings of the third Asian Conference on Multibody Dynamics 2006, August 2006, vol. 00643 (2006) 5. Otsuka, A., Tsuji, T., Fukuda, O., Shimizu, M.E., Sakawa, M.: Development of an internally powered functional prosthetic hand with a voluntary closing system and thumb flexion and radial abduction. In: Proceedings of the 2000 IEEE International Workshop on Robot and Human Interactive Communication, pp. 405–410 (2000) 6. Tanaka, Y., Yamada, N., Masamori, I., Tsuji, T.: Manipulability Analysis of lower Extremities Based on Human Joint-Torque Characteristics (in Japanese). Trans- action of the Society of Instrument and Control Engineers 40(6), 612–618 (2004) 7. Tanaka, Y., Yamada, N., Nishikawa, K., Masamori, I., Tsuji, T.: Manipulability Analysis of Human Arm Movements during the Operation of a Variable-Impedance Controlled Robot. In: Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robotics and Systems, pp. 3543–3548 (2005) 8. Oyama, T., Imai, S., Wake, T.: Sensory and perceptual handbook, Seishin Shobo (in Japanese) (1994) 9. Tanaka, Y., Abe, T., Tsuji, T., Miyaguchi, H.: Motion dependence of impedance perception ability in human movements. In: Proceedings of the First International Conference on Complex Medical Engineering, pp. 472–477 (2005) 10. Gandevia, S.C., McClosky, D.I.: Sensations of heaviness. Brain 100, 345–354 (1977) 11. Jones, L.A.: Role of central and peripheral signals in force sensation during fatigue. Experimental Neurology 81, 497–503 (1983) 12. Miall, R.C., Ingram, H.A., Cole, J.D., Gauthier, G.M.: Weight estimation in a “deafferented” man and in control subjects: are judgments influenced by peripheral or central signal? Experimental Brain Research 133, 491–500 (2000) 13. Jones, L.A.: Perceptual constancy and the perceived magnitude of muscle forces. Experimental Brain Research 151, 197–203 (2003) 14. Yamakawa, S., Fujimoto, H., Manabe, S., Kobayashi, Y.: The necessary conditions of the scaling ratio in master-slave systems based on human difference limen of force sense. IEEE Transactions on System, Man, and Cybernetics: Part A 35(2), 275–282 (2005) 15. Sasaki, H., Fujita, K.: Experimental analysis of role of visual information in hard- ness cognition displayed by a force display system and effect of altered visual information (in Japanese). Journal of the Virtual Reality Society of Japan 5(1), 795–802 (2000) 16. Tanaka, Y., Matsushita, K., Tsuji, T.: Sensorimotor characteristics in human arm movements during a virtual curling task (in Japanese). Transactions of the Society of Instrument and Control Engineers 42(12), 1288–1294 (2006) 17. Hogan, N.: Impedance Control: An approach to manipulation, Parts I, II, III. ASME Journal of Dynamic Systems, Measurement, and Control 107(1), 1–24 (1985)
Computational Understanding and Modeling of Filling-In Process at the Blind Spot Shunji Satoh and Shiro Usui Laboratory for Neuroinformatics, RIKEN Brain Science Institute, Japan shun@brain.riken.jp Abstract. A visual model for filling-in at the blind spot is proposed. The general scheme of standard regularization theory is used to derive a visual model deductively. First, we indicate problems of the diffusion equation, which is frequently used for various kinds of perceptual com- pletion. Then, we investigate the computational meaning of a neural property discovered by Matsumoto and Komatsu (J. Neurophysiology, vol. 93, pp. 2374–2387, 2005) and introduce second derivative quantities related to image geometry into a priori knowledge of missing images on the blind spot. Moreover, two different information pathways for filling-in (slow conductive paths of horizontal connections in V1, and fast feedfor- ward/feedback paths via V2) are regarded as the neural embodiment of adiabatic approximation between V1 and V2 interaction. Numerical simulations show that the outputs of the proposed model for filling-in are consistent with a neurophysiological experimental result, and that the model is a powerful tool for digital image inpainting. 1 Introduction The blind spot is the area in the visual field that corresponds to the lack of photoreceptors on the retina. Although no receptors exists to detect light stim- ulus at the blind spot (BS), we do not see a black disk or a strange pattern but we perceive the same color or pattern as the surroundings. This phenomenon is referred to as perceptual filling-in at the blind spot. Perceptual filling-in is not a special phenomenon limited to the blind spot. It is observed in various situations, e.g., an artificial scotoma produced by transcranial magnetic stim- ulation [1], patients with retinal scotoma [2], normal visual field (no defect of visual field) for stabilized retinal image [3], and others [4]. Those findings imply that perceptual filling-in is a general and common process in our visual system. Consequently, research about BS computation contributes to understanding of the general process of the visual system. Komatsu and colleagues reported a quantitative analysis of V1 neural re- sponses of awake macaque monkeys to bar stimuli presented on the blind spot This work was partially supported by the Grant-in-Aid for Young Scientists (#17700244), the Ministry of Education, Culture, Sports, Science and Technology, Japan. M. Ishikawa et al. (Eds.): ICONIP 2007, Part I, LNCS 4984, pp. 943–952, 2008. c Springer-Verlag Berlin Heidelberg 2008 944 S. Satoh and S. Usui Fig. 1. (a). Schematic examples of bar stimuli. BS represents the blind spot. RF is the receptive field of a recorded neuron. (b) Retinal inputs corresponding to the above stimuli. (c) Responses of the recorded neuron. (d) Abstract model proposed by Matsumoto & Komatsu. (Adapted from Fig. 3 in J. Neurophysiology, vol. 93, pp. 2374–2387, 2005) This is also the neural representation of the proposed algorithm for filling-in in this article. (e). Simulation result of the proposed model in this article. [5,6]. The receptive fields (RFs) of the recorded neurons overlapped with the BS area (Fig. 1(a)). Bar stimuli of various lengths were presented at the BS. One end of the bar stimulus was fixed and the other end was varied (Figs. 1(a1)– 1(a4)). Some V1 neurons showed a significant increase in their activities when the bar end exceeded the BS (Fig. 1(c)), although those activities remained constant as long as the end was in the BS area. These results indicate that V1 neurons perform the filling-in process in addition to orientation detection. In addition, Matsumoto and Komatsu posited the existence of two different pathways with different velocities of visual signal: (i) fast feedforward [ff] and feedback [fb] connections via V2, and (ii) slow horizontal connections [hc] connecting V1 neurons (see Fig. 1D). However, no discussion exists about the necessities of those properties. Why are the different velocities necessary? What is the significance of the two different pathway velocities? As shown in Fig.1(c), the abrupt increase of neural response was observed even though the retinal stimulation increased only slightly; one end of a bar appears or not (see Fig. 1(b3) and 1(b4)). This phenomenon implies the complexity and nonlinearity of the filling-in process at the BS. It also shows the difficulties in obtaining appropriate patterns. The different velocities of pathways and the existence of V2 neurons might be the keys to solving the difficult problem of the filling-in process. We therefore studied the computational necessities of the different conduc- tance velocities and the role of V2 neurons from a computational point of view. For this purpose, we interpreted the physiological findings given by Matsumoto and Komatsu [6] from a theoretical point of view. We constructed a visual model so that it reproduced the data shown in Fig. 1(c). We will deductively obtain Computational Understanding and Modeling of Filling-In Process 945
(a) (b)
(c) (d)
Fig. 2. (a) Filling-in process is applied in BS area B (a gray rectangle). (b) Example of a desired filling-in pattern. (c) Filling-in by the diffusion equation. (d) Filling-in by the proposed visual model. our visual model based on standard regularization theory because the theory enables us to understand our filling-in model as an optimizer for a pre-defined evaluation function. Moreover, the dynamics derived from standard regulariza- tion is generally a reaction-diffusion equation with few parameters. It can be implemented easily as a neural network. Application of standard regularization theory to explain the filling-in process seems a trivial approach because the theory is applicable to various types of completion problems to recover missing information, e.g., estimation of optical flow, restoration of surface, aperture problems [7], and so on. However, no suc- cessful work exists on BS filling-in based on the standard regularization theory reflecting the physiological characteristics described before. One reason why no mathematical model for BS filling-in exists would be the high nonlinearity and the complexity of the filling-in problem. The neural properties related to the BS should be keys to solving the complex problem to reiterate. We expect that the filling-in algorithm in our visual system should be an effective image-processing algorithm. For example, digital image inpainting (DII) is a technique that can repair a region of a damaged or removed image using automatic mechanisms [8]. Other applications of DII include restoration of video images, image transmission through narrow band systems, and so on [9]. Our visual model is merely an algorithm for DII when we regard the BS area as the area to be restored using DII. In addition to the initial purpose of this article, we also evaluate the effectiveness of our visual model for a DII algorithm using color input images. 2 Evaluation Function for Filling-In 2.1 Problems of Diffusion Equation for Filling-In Process To formulate the filling-in problem, we define an evaluation function E[I] (or an energy function, a functional for I) for filling-in patterns I(x) so that E takes a small value when an image I(x) is a desired one, where I signifies the brightness and x = (x, y) is a spatial position within the BS area. The BS area and the boundary are referred to as B and ∂B, respectively. A desired filling-in for 946 S. Satoh and S. Usui Fig. 2(a) is a completed bar, as shown in Fig. 2(b), in which the spatial change of I(x) in B should be small (no intensity fluctuation along x-axis). A simple functional that evaluates intensity change is formulated as E 1
1 2 B d 2 x ||∇I(x, t)|| 2 = 1 2 B d 2 x I 2 ξ (x), (1) where I
ξ is the directional derivative of I in the direction ξ, which is parallel to the gradient of I at a point. We obtain I ξ = ∂ ∂ξ I = (cos ξ) ∂ ∂x I + (sin ξ) ∂ ∂y I (2)
Hereafter, we indicate the directional derivative using subscripts. For example, I ξη means the second order directional derivative I in the directions ξ and η. As shown in Fig. 3, the direction ∇I (the gradient vector of I) is referred to as ξ, and the orthogonal vector as μ. The algorithm we use is an iterative update method of I such that E[I] decreases as time progresses. We obtain the dynamics (update rule) of I by applying the steepest descent method to the functional E: ∂ ∂t I(x, t) = ∇ 2 I(x, t), if x
∈ B ∇ 2 I(x, t) − ∂ ∂x + ∂ ∂y I(x, t),
if x ∈ ∂B
(3) Equation (3) is the diffusion equation of brightness I. The steady state of I is referred to as ¯ I; it is a result of the filling-in process. Figure 2(c) is the result of a numerical simulation of (3). We find that the result is far from our expectation. We confirmed this was not attributable to local minima trapped by the steepest descent method (the value of E 1 for Fig. 2(c) was smaller than that for Fig. 2(b)). To obtain Fig. 2(b) as the steady state ¯ I, we applied other possible meth- ods, e.g., multi-resolution, multi-grid, and anisotropic diffusion. However, every method failed. Apparently, we will never obtain the desired result as long as we use (1) as the evaluation function. 2.2 New Functional with Curvature Terms Term I ξ in (1) is the first-order derivative of the image, and it represents the output of a V1 neuron selective to ξ-orientation. This computational aspect reveals a flaw of (1) that the functional E 1 uses only V1-coded visual information. On the other hand, Matsumoto and Komatsu found that V2 neurons con- tribute to filling-in. That is, from a computational viewpoint, we should in- troduce visual information coded by V2 neurons into the evaluation function. Hence, we introduce angular information of contours because some V2 neurons are selective to angles embedded within V-shaped patterns [10]. Important quantities associated with contour angles are (i) κ: curvature of level-set and (ii) μ: curvature of flow line; the former signifies the curvature of
Computational Understanding and Modeling of Filling-In Process 947
Fig. 3. Quantities used in this work (see the text for detail) contour lines of I, the latter is the curvature of lines crossing at right angles with contour lines (see Fig. 3(b)). Both κ and μ represent smoothness of edges or image contours. We then propose the following new functional E based on E 1
E = 1 2 d 2 x ¯ κ 2 (x, t) + ¯ μ 2 (x, t) I 2 ξ (x, t). (4) In that equation, ¯ κ = κI
ξ = I
ηη = I 2 y I xx − 2I
x I y I xy + I 2 x I yy I 2 x + I
2 y (5) ¯ μ = μI
ξ = I
ξη = (I 2 x − I 2 y )I xy − I
x I y (I y y − I x x) I 2 x + I 2 y . (6)
3 Dynamics for Filling-In at the Blind Spot We expect that the dynamics for filling-in and the corresponding neural network will emerge deductively by applying steepest descent method to (4). However, the resultant dynamics is very complex, as shown below. ∂ ∂t I = ( I 6 y I yyyy
I 2 x + 3I 4 y I yyyy
I 4 x + 3I 2 y I yyyy
I 6 x + · · · (74 terms) + · · · + 3I 6
I 2 x I xxxx
+ 3I 4 y I 4 x I xxxx
+ I 2 y I 6 x I xxxx
) / I 2 x + I 2 y 3/2 (7)
The problematic complex equation (7) comprises 80 terms. It seems impossible to analyze all 80 terms to investigate their physiological meanings. Similarly, we can not expect a physiologically plausible visual model from (7). Again, we attempt to use a physiological phenomenon as a key to solving the problem. The key is the different pathway with different conductance velocities: fast pathways via V2, and slow pathways within V1 using horizontal connections. The faster conductance velocity of the visual pathway via V2 implies the faster optimization of ¯ κ(x, t) and ¯ μ(x, t) in time than that of I ξ (x, t). In the 948 S. Satoh and S. Usui Fig. 4. (a) Spatial distributions of receptive fields formulated as Gaussian derivatives (σ = 1). The RF model is selective to horizontal bars when θ = 90 ◦ . (b) ˜
μ ξθθ
is the sum of four neural outputs selective to about 27 ◦ angular difference of lines when θ = 90 ◦ . (c) ˜ κ ηθθ
is the sum of four neural outputs selective to those four patterns when θ = 90 ◦ . extreme situation of velocity difference, we can assume a constant value of I ξ (x, t) with respect to time t. This assumption corresponds to variable separation and adiabatic approximation. Applying the steepest descent method to (4) using this assumption, we obtain the following dynamics: ∂ ∂t I(x, t) = ∂ ∂η ˜ κ +
∂ ∂ξ ˜ μ − ∂ 2 ∂ξ 2 + ∂ 2 ∂η 2 ˜ κ + λI ηη , (8) = ˜
κ η + ˜ μ ξ − Δ˜κ + λI ηη , (9) where ˜ κ = I
2 ξ κ, ˜ μ = I 2 ξ μ and λ is a positive constant. The fourth term of (9) is a consistency term which alleviates the drawbacks of variable separation and adiabatic approximation. Compared with (7) in the appendix, we find a simple, analyzable, neural im- plementable equation as a neural network in (9). However, we have no guarantee that desired filling-in patterns are obtained by (9). We will demonstrate the validity of (9) in section 5 using numerical simulations. 4 Neural Dynamics for Filling-In at the Blind Spot The intention of this study is the construction of a V1 model for filling-in. We next consider the dynamics of orientation selective neurons and the neural im- plementation. To simulate the neurophysiological experiments illustrated in Fig. 1(d), we must consider the dynamics of V1 neurons selective to bar stimuli. In this section, we derive orientation selective neurons dynamics from eq. (9). We apply a Gaussian derivative model (GD) as receptive fields (RFs) of V1 neurons [11]. The RF model selective to θ-orientation is written as ∂ 2 g
(x)/∂θ 2 , where g σ is the Gaussian function with σ 2 variance. Figure 4(a) shows examples of θ-preferring RFs. We use a linear neuron model. The output of a θ-preferring
|
