This page is a supplementary resource for a paper:
S. Wibirama, H.A. Nugroho, K. Hamamoto, “Depth Gaze and ECG Based Frequency Dynamics during Motion Sickness in Stereoscopic 3D Movie”, Entertainment Computing, Vol. 28, 2018, pp.117-127.
doi: https://doi.org/10.1016/j.entcom.2018.02.003

 

 

Fig.1. An illustration of a spectator watching a 3D content

 

ABSTRACT

The simulator sickness questionnaire (SSQ) has been a prevalent method to observe motion sickness in stereoscopic 3D motion picture. However, previous works do not provide adequate comprehension of the relationship between SSQ, depth gaze behavior, and heart rate variability in the stereoscopic 3D motion picture. To fill this research gap, we present a novel investigation of motion sickness in stereoscopic 3D movies using SSQ, electrocardiography (ECG), and 3D gaze tracking. Forty participants (N=40) watched only one of two 3D contents—3D content with a strong or a moderate sensation of vection. We observed that viewers of the 3D content with an intense feeling of vection more frequently reported symptoms of nausea (p<0.005) and disorientation (p<0.05) than their counterpart. SSQ, ECG, and 3D gaze tracking data show that sickness level could be reduced by persistently gazing at a particular point during exposure of 3D contents (p  <0.001). Additionally, we found that individuals who were prone to motion sickness experienced depth gaze oscillation during several provoking scenes in dynamic 3D contents. Our experimental results may be used as a guideline in the development of a motion sickness predictor for various stereoscopic 3D motion pictures.

KEYWORDS

ECG; Heart rate variability; 3D gaze analysis; Eye tracking; Motion sickness; Stereoscopic 3D; User experience

 

:: What is visually induced motion sickness ?

Have you ever felt dizzy and sweaty when seeing a stereoscopic 3D content ?
Or, do you often experience sickness when seeing 3D TV ?

3D contents can be found in many daily aspects, such as advertising billboards, televisions, computers, cinemas, and personal gaming systems. To perceive a 3D image in a stereoscopic display, an individual has to wear active shutter glasses that open and close the L/R shutter alternately (Fig.1). Generally, 60 Hz refresh rate of each shutter is enough to reveal various virtual 3D objects seamlessly. Despite of its wide usage, stereoscopic 3D motion pictures with dynamic motions produce a biomedical effect known as visually induced motion sickness (VIMS). VIMS is a condition during which spectators of dynamic 3D contents feel symptoms of nausea, dizziness, or visual fatigue during or after exposure while they are being physically still.

Decoupling of accommodation and vergence in human visual system during exposure of a dynamic 3D content was found to be the main contributing factor of VIMS. Furthermore, the mismatch of visual-vestibular stimulation was also argued as a prevalent factor that increased the possibility of VIMS. Different from common motion sickness, VIMS is caused by visual factors without any vestibular stimulation. VIMS becomes a major issue in the research of 3D technology since VIMS constrains the usage of stereoscopic 3D technology for longer duration as well as for wider purposes.

The Japan Times–a prominent English news paper in Japan–reported on July 10th, 2003 that thirty-six out of 294 junior high school students were taken to a hospital due to dizziness when watching an educational video (Fig.2). The students experienced dizziness and nausea provoked by swaying images in the video. This case was later on known as “The Matsue Case”, an important trigger to research on VIMS in Japan. Japan–one of major players in 3D contents industry–has put VIMS as one of important research theme nationwide.

matsue_case

Fig.2. News on The Japan Times reported VIMS in 2013

 

:: Our research in VIMS

One of active research in VIMS has been performed by Professor Dr. Kazuhiko Hamamoto of Tokai University Japan and Dr. Sunu Wibirama of Universitas Gadjah Mada, Indonesia. Professor Hamamoto and Dr. Wibirama have been interested on investigating VIMS in the large-scale immersive virtual environment (IVE) as well as in the desktop-based system. Previous research works of investigating VIMS in IVE have been performed mainly by Dr. Chompoonuch Jinjakam , a former PhD student of Professor Kazuhiko Hamamoto and now a faculty member in the King Mongkut’s Institute of Technology Ladkrabang, Thailand. Her research works mainly observed how a dynamic 3D content in a four-screens HoloStage induced motion sickness. She investigated VIMS mainly by using subjective judgment of the spectators–Simulator Sickness Questionnaire (SSQ). SSQ is a pen-and-paper evaluation method filled out by the spectators right after watching the dynamic 3D content [1-3, 12]. However, no objective measurement was undertaken in the previously published references [1-3]. Furthermore, users filled out the SSQ form after exposure of 3D contents. Thus, physiological effect of VIMS during exposure of 3D contents, duration of VIMS, and part of scene that might cause VIMS were still unknown.

There are two physiological indicators to objectively investigate VIMS: heart rate variability (HRV) and depth gaze behavior. HRV is extracted from electro-oculogram (ECG) data. ECG measures heartbeats and their variability through several attached electrodes on the human body. In a normal condition, the harmonious mechanism of sympathetic and parasympathetic nerves of autonomic response system to govern cardiopulmonary function is almost similar as two opposite sine waves [4]. In contrast, when a human is experiencing symptoms of VIMS or being mentally stressed, the ideal balance of the sympathetic and parasympathetic nerves will be broken. The sympathetic nerves are generally found to be more dominant than the parasympathetic nerves.

On the other hand, the 3D gaze tracking estimates point of gaze using various computer vision techniques in not only X (horizontal) and Y (vertical) direction, but also Z (depth) direction. Gaze tracking is a useful method to investigate VIMS [5-7]. Nevertheless, most of gaze measurements in the previous research works were performed only in 2D direction, i.e. only X and Y gaze position were obtained during exposure of stimuli. Furthermore, instead of using stereoscopic 3D movie, optical flow contents or alternate black-and-white vertical stripes were generally presented to the user. Therefore, there is no information of how stereoscopic 3D movie affects 3D gaze behavior and heart rate variability at the same time. To the best knowledge of the author, there is no research work that combines SSQ, ECG, and 3D gaze tracking to investigate VIMS.

In this paper, we present a novel investigation of VIMS using SSQ, ECG, and 3D gaze tracking. We used SSQ as a tool to investigate the occurrence of VIMS. We then observed the duration and detail of VIMS occurrence using an ECG system. We used our 3D gaze tracking to measure 3D point of gaze in virtual space, anisoptropy maps, as well as to record depth gaze behavior during exposure of dynamic 3D contents [8-11]. Two-way statistical analysis of variance (ANOVA) on SSQ data shows that nausea and disorientation symptoms increase as amount of dynamic motions increase. To reduce VIMS, SSQ and ECG data suggest that the user should perform voluntary gaze fixation at a particular point when experiencing a vertical motion (up or down) and a horizontal motion (turn left and right) in the dynamic 3D content. Observation of 3D gaze tracking data reveals that individuals who experienced VIMS tended to have unstable depth gaze than individuals who did not experience VIMS.

More about our research in VIMS and 3D gaze tracking can be found here:

  • S. Wibirama, H.A. Nugroho, K. Hamamoto, “Evaluating 3D Gaze Tracking in Virtual Space: A Computer Graphics Approach,” Entertainment Computing, Vol. 21, June 2017, pp.11-17. (Q2 ScimagoJR). doi: http://doi.org/10.1016/j.entcom.2017.04.003 | Download PDF
  • S.Wibirama and K. Hamamoto, “3D Gaze Tracking on Stereoscopic Display using Optimized Geometric Method”, IEEJ Transactions on Electronics, Information, and Systems, Vol. 134, No. 3, 2014, pp. 345-352. (Q3 ScimagoJR). PDF | DOI .
  • S. Wibirama, R. R. Mahesa, H.A. Nugroho, K. Hamamoto, “Estimating 3D gaze in physical environment: a geometric approach on consumer-level remote eye tracker,” Proc. SPIE 10225, Eighth International Conference on Graphic and Image Processing (ICGIP 2016), Tokyo, Japan, 29-31 October 2016. pp. 102251H1-102251H6. doi: dx.doi.org/10.1117/12.2266109
  • T. Wijayanto, S. Wibirama, Z. Z. Maryoto, M. N. Winadi, M. Bahit, “Effects of Morning-night Difference and Sleep Deprivation on Situation Awareness and Driving Performance”, submitted to The 2016 IEEE International Conference on Industrial Engineering and Engineering Management, Bali, Indonesia, 4-7 December 2016, pp.267 – 271. doi : 10.1109/IEEM.2016.7797878
  • M. Bahit, S. Wibirama, H. A. Nugroho, T. Wijayanto, M. N.Winadi, “Investigation of Visual Attention in Day-Night Driving Simulator during Cybersickness Occurrence”, Proceedings of The 8th 2016 International Conference on Information Technology and Electrical Engineering (ICITEE 2016), Yogyakarta, Indonesia, 5-6 Oktober 2016, pp. 213-216. doi: doi.org/10.1109/ICITEED.2016.7863260
  • S. Mujahidin, S. Wibirama, H.A. Nugroho, K. Hamamoto, “3D gaze tracking in real world environment using orthographic projection,” Proceedings of The 2016 Conference on Fundamental and Applied Science for Advanced Technology, AIP Publishing Vol. 1746, Yogyakarta, Indonesia, 25-26 January 2016, pp. 020072 (1-6). doi : dx.doi.org/10.1063/1.4953997
  • S. Wibirama, T. Wijayanto, H. A. Nugroho, M. Bahit, M. N. Winadi, “Quantifying visual attention and visually induced motion sickness during day-night driving and sleep deprivation,” in Proceedings of The 2015 International Conference on Data and Software Engineering, Yogyakarta, Indonesia, 25-26 November 2015, pp. 191-194. doi: dx.doi.org/10.1109/ICODSE.2015.7436996

Below is a short video that summarize our ongoing research in VIMS:

:: Bibliography

  1. C. Jinjakam and K. Hamamoto, “Parallax, position, and height difference effects on simulator sickness in immersive virtual environment,” in Proceedings of Biomedical Engineering International Conference (BMEiCON), 2013 6th, pp. 1-4, 2013.
  2. C. Jinjakam and K. Hamamoto, “Analysis of Simulator Sickness in IVE by PCA,” Transaction of the Japan Society for Simulation Technology, 2012, 4, pp. 194-199, 2012.
  3. C. Jinjakam and K. Hamamoto, “Study on Parallax Affect on Simulator Sickness in One-screen and Three-screen Immersive Virtual Environment,” in Proceedings of The School of Information and Telecommunication Engineering Tokai University, vol. 4, pp. 34-39, 2011.
  4. M. Nakagawa, T. Iwao, S. Ishida, H. Yonemochi, T. Fujino, T. Saikawa, and M. Ito, “Circadian rhythm of the signal averaged electro- cardiogram and its relation to heart rate variability in healthy subjects”, Heart, vol. 79, no. 5, pp. 493–496, 1998.
  5. C. Diels, K. Ukai, and P. A. Howarth, “Visually Induced Motion Sickness with Radial Displays: Effects of Gaze Angle and Fixation,” Aviation, Space, and Environmental Medicine, vol. 78, no. 7, pp. 659–665, 2007.
  6. C. Diels and P. A. Howarth, “Visually induced motion sickness: Single-versus dual-axis motion,” Displays, vol. 32, no. 4, pp. 175–180, 2011.
  7. J. Yang, C. Guo, R. So, and R. Cheung, “Effects of eye fixation on visually induced motion sickness: are they caused by changes in retinal slip velocity?,” in Proceedings of the human factors and ergonomics society 55th annual meething, pp. 1220– 1224, 2011.
  8. S.Wibirama and K. Hamamoto, “3D Gaze Tracking on Stereoscopic Display using Optimized Geometric Method,” IEEJ Transactions on Electronics, Information, and Systems, Vol. 134, No. 3, pp. 345-352, 2014.
  9. S. Wibirama,  S. Tungjitkusolmun, and C. Pintavirooj, “Dual-Camera Acquisition for Accurate Measurement of Three-Dimensional Eye Movements,” IEEJ Transactions on Electrical and Electronic Engineering, Vol.8, No.3 pp. 238-246, 2013.
  10. S. Wibirama & K. Hamamoto, “Design and Implementation of Gaze Tracking Headgear for Nvidia 3D Vision®,”  in Proceeding of The 5th International Conference on Information Technology and Electrical Engineering (ICITEE) 2013, pp. 84-87, 2013.
  11. S. Wibirama & K. Hamamoto, “3D Gaze Tracking System for Nvidia 3D Vision®,” in Proceeding of The 35th Annual International Conference of the IEEE EMBS,  pp. 3194-3197, 2013.
  12. R. Kennedy and N. Lane, “Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness,” The International Journal of Aviation Psychology, vol. 3, no. 3, pp. 203–220, 1993.

:: Acknowledgments

Authors would like to thank Mr. Suzuki Satoshi for his assistance in operating ECG system during experiment, Dr. Chompoonuch Jinjakam for her useful comments, and Tokai University – Japanese International Cooperation Agency (JICA) for providing research equipments and funding.

 

:: Appendices

1. Sample of informed consent to perform experiment with human subjects

 

2. Matlab source code to simulate  3D gaze tracking algorithm

Download : depthmed.zip (4kb)
You need to download “Robotic and Machine Vision Toolbox” to use my source code properly.
Please cite the following work if you find the code is useful for your research:
S.Wibirama and K. Hamamoto, “3D Gaze Tracking on Stereoscopic Display using Optimized Geometric Method”, IEEJ Transactions on Electronics, Information, and Systems, Vol. 134, No. 3, 2014, pp. 345-352.

 

3. Useful links to performs statistical analysis

4. Useful links about stereoscopic 3D and computer graphics