ZTE Communications ›› 2019, Vol. 17 ›› Issue (1): 48-55.DOI: 10.12142/ZTECOM.201901008
-
收稿日期:2019-01-11出版日期:2019-02-20发布日期:2019-11-14
Quality-of-Experience in Human-in-the-Loop Haptic Communications
LIU Qian1, ZHAO Tiesong2
- 1. Dept. of Computer Science and Technology, Dalian University of Technology, Dalian, Liaoning 116024, China
2. College of Physics and Information Engineering, Fuzhou University, Fuzhou, Fujian 350108, China
-
Received:2019-01-11Online:2019-02-20Published:2019-11-14 -
About author:LIU Qian (qianliu@dlut.edu.cn) currently works as an associate professor at the Dept. of Computer Science and Technology, Dalian University of Technology, China. She received her B.S. and M.S. degrees from Dalian University of Technology in 2006 and 2009, respectively, and the Ph.D. degree from The State University of New York at Buffalo, USA, in 2013. She served as a post-doctoral fellow with the Ubiquitous Multimedia Laboratory, State University of New York at Buffalo from 2013 to 2015. She received Alexander von Humboldt Fellowship and worked with the Chair of Media Technology and the Chair of Communication Networks, Technical University of Munich from 2016 to 2017. Her research interests include haptic-audio-visual multimodal signal processing and communications, wireless multimedia communications, and human-machine interactions.|ZHAO Tiesong received the B.S. degree in electrical engineering from the University of Science and Technology of China in 2006, and the Ph.D. degree in computer science from the City University of Hong Kong, China in 2011. He served as a research associate with the Department of Computer Science, City University of Hong Kong, from 2011 to 2012, a post-doctoral fellow with the Department of Electrical and Computer Engineering, University of Waterloo, Canada from 2012 to 2013, and a research scientist with the Ubiquitous Multimedia Laboratory, State University of New York at Buffalo until 2015. He is currently a professor with the College of Physics and Information Engineering, Fuzhou University, China. His research interests include image/video signal processing, visual quality assessment, and video coding and transmission.
引用本文
. [J]. ZTE Communications, 2019, 17(1): 48-55.
LIU Qian, ZHAO Tiesong. Quality-of-Experience in Human-in-the-Loop Haptic Communications[J]. ZTE Communications, 2019, 17(1): 48-55.
Figure 1. A comparison of conventional audio-visual communications and audio-visual-haptic communications.
Figure 1. A comparison of conventional audio-visual communications and audio-visual-haptic communications.
Figure 2. Roadmap of this survey.
Figure 2. Roadmap of this survey.
Table 1 Psychophysical factors, corresponding human perception mechanism and exemplar signal generation approaches
| Kinesthetic sense | Tactile sense | |
|---|---|---|
| Signal type | Position, velocity, angular velocity, force, and torque | Surface texture and friction |
| Human perception mechanism | Sensed by the muscles, joints, and tendons of the body | Sensed by different types of mechanoreceptors in the skin |
| Exemplar signal generation solutions | Using high torque motors to generate kinesthetic force feedback, such as Geomagic Touch (used to be called as Phantom Omni [ | Multi-pin display attached to the human skin [ |
Table 1 Psychophysical factors, corresponding human perception mechanism and exemplar signal generation approaches
| Kinesthetic sense | Tactile sense | |
|---|---|---|
| Signal type | Position, velocity, angular velocity, force, and torque | Surface texture and friction |
| Human perception mechanism | Sensed by the muscles, joints, and tendons of the body | Sensed by different types of mechanoreceptors in the skin |
| Exemplar signal generation solutions | Using high torque motors to generate kinesthetic force feedback, such as Geomagic Touch (used to be called as Phantom Omni [ | Multi-pin display attached to the human skin [ |
Figure 3. Application-level QoE management in an exemplar haptic teleoperation application with two teleoperation sessions.
Figure 3. Application-level QoE management in an exemplar haptic teleoperation application with two teleoperation sessions.
Figure 4. Perceptual deadband principle. The perception thresholds (boundaries of gray zones) are a function of the stimulus intensity I. Samples that fall within the deadbands can be dropped (adapted from [25]).
Figure 4. Perceptual deadband principle. The perception thresholds (boundaries of gray zones) are a function of the stimulus intensity I. Samples that fall within the deadbands can be dropped (adapted from [25]).
Table 2 Overview of the human perception models and corresponding data reduction solutions (reproduced from [3])
| Haptic type | Human perception model | Data reduction solutions | ||||
|---|---|---|---|---|---|---|
| Kinesthetic | Weber’s law (linear) [ | Without considering network conditions | considering network conditions | |||
| Perceptual deadband schemes:Single DoF [ | Solutions | Knownconst. delay | Unknown const. delay | Time-varying delay | ||
| WV+ PD | [ | [ | - | |||
| TDPA+ PD | [ | [ | [ | |||
| MMT+ PD | [ | [ | - | |||
| Tactile | Data-driven ML models | [ | - | |||
| Similar to speech signal | [ | - | ||||
Table 2 Overview of the human perception models and corresponding data reduction solutions (reproduced from [3])
| Haptic type | Human perception model | Data reduction solutions | ||||
|---|---|---|---|---|---|---|
| Kinesthetic | Weber’s law (linear) [ | Without considering network conditions | considering network conditions | |||
| Perceptual deadband schemes:Single DoF [ | Solutions | Knownconst. delay | Unknown const. delay | Time-varying delay | ||
| WV+ PD | [ | [ | - | |||
| TDPA+ PD | [ | [ | [ | |||
| MMT+ PD | [ | [ | - | |||
| Tactile | Data-driven ML models | [ | - | |||
| Similar to speech signal | [ | - | ||||
Figure 5. An application-level multiplexing scheme for synchronized multi-sensory data delivery.
Figure 5. An application-level multiplexing scheme for synchronized multi-sensory data delivery.
Figure 6. a) Hypothesis between quality of experience and delay for different control schemes; b) subjective tests results of a virtual environment-based spring-damper teleoperation system (adopted from [60]).
Figure 6. a) Hypothesis between quality of experience and delay for different control schemes; b) subjective tests results of a virtual environment-based spring-damper teleoperation system (adopted from [60]).
| [1] | ZHAO T S, LIU Q, CHEN C W . QoE in Video Transmission: A User Experience-Driven Strategy[J]. IEEE Communications Surveys & Tutorials, 2017,19(1):285-302. DOI: 10.1109/comst.2016.2619982 |
| [2] | PARK P, COLERI ERGEN S, FISCHIONE C , et al. Wireless Network Design for Control Systems: A Survey[J]. IEEE Communications Surveys & Tutorials, 2018,20(2):978-1013. DOI: 10.1109/comst.2017.2780114 |
| [3] | ANTONAKOGLOU K, XU X, STEINBACH E , et al. Toward Haptic Communications over the 5G Tactile Internet[J]. IEEE Communications Surveys & Tutorials, 2018,20(4):3034-3059. DOI: 10.1109/comst.2018.2851452 |
| [4] |
OKAMOTO S, NAGANO H, YAMADA Y . Psychophysical Dimensions of Tactile Perception of Textures[J]. IEEE Transactions on Haptics, 2013,6(1):81-93. DOI: 10.1109/toh.2012.32
DOI URL |
| [5] | CHA J, HO Y-S, KIM Y , et al. A Framework for Haptic Broadcasting[J]. IEEE Multimedia, 2019,16(3): pp. 16-27. |
| [6] | STEINBACH E, HIRCHE S, ERNST M , et al. Haptic Communications[J]. Proceedings of the IEEE, 2012,100:937-956. DOI: 10.1109/JPROC.2011. 2182100 |
| [7] | SILVA A, RAMIREZ O , et al. PHANToM OMNI Haptic Device: Kinematic and Manipulability [C]//Electronics, Robotics and Automotive Mechanics Conference (CERMA), Cuernavaca, Morelos, 2009: 193-198 |
| [8] | FORCE DIMENSION . Force Dimension User Manual of Omega.x Haptic Device, Version 1.7 [Z]. Nyon, Switzerland, 2013. |
| [9] | BREWSTER S, CHOHAN F, BROWN L . Tactile Feedback for Mobile Interactions [C]//SIGCHI Conference on Human Factors in Computing Systems, Gaithersburg, USA, 2007: 159-162 |
| [10] | DREWING K, FRITSCHI M, ZOPF R , et al. First Evaluation of a Novel Tactile Display Exerting Shear Force Via Lateral Displacement[J]. ACM Transactions on Applied Perception, 2005,2(2):118-131. DOI: 10.1145/1060581.1060586 |
| [11] | BENALI-KHOUDJA M, HAFEZ M, ALEXANDRE J M , et al. Tactile Interfaces: A State-of-the-Art Survey [C]//International Symposium of Robotics, Paris, France, 2004: 23-26 |
| [12] | CHOUVARDAS V G, MILIOU A N, HATALIS M K . Tactile Display Applications: A State-of-the-Art Survey [C]//2nd Balkan Conference in Informatics, Ohrid, Macedonia, 2005: 290-303 |
| [13] | WINFIELD L, GLASSMIRE J, COLGATE J E , et al. T-PaD: Tactile Pattern Display Through Variable Friction Reduction [C]//Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07), Tsukaba, Japan, 2007: 421-426. DOI: 10.1109/WHC.2007.105 |
| [14] | BAU O, POUPYREV I, ISRAR A et al. TeslaTouch: Electrovibration for Touch Surfaces[C]// User Interface Software and Technology (UIST), New York, USA:283-292, 2010. |
| [15] | WEBER E. The Sense of Touch [M]. Cambridge, USA: Academic Press, 1978 |
| [16] | GESCHEIDER G A , Psychophysics: The Fundamentals[M]. 3rd ed. Mahwah, USA: Lawrence Erlbaum, 1997 |
| [17] | DEHAENE S . The Neural Basis of the Weber-Fechner Law: A Logarithmic Mental Number Line[J]. Trends in Cognitive Sciences, 2003,7(4):145-147. DOI: 10.1016/S1364-6613(03)00055-X |
| [18] | DEHAENE S . Subtracting Pigeons: Logarithmic or Linear?[J] Phychological Science, 2001,12(3):244-246. DOI: 10.1111/1467-9280.00343 |
| [19] | OKAMURA A M, DENNERLEIN J T, HOWE R D . Vibration Feedback Models for Virtual Environments [C]//IEEE International Conference on Robotics and Automation (Cat. no.98CH36146), Leuven, Belgium, 1998: 674-679. DOI: 10.1109/ROBOT.1998.677050 |
| [20] | KUCHENBECKER K J, ROMANO J, MCMAHAN W . Haptography: Capturing and Recreating the Rich Feel of Real Surfaces[M]//KUCHENBECKER K J, ROMANO J, MCMAHAN W. eds. Springer Tracts in Advanced Robotics. Berlin, Heidelberg, Germany: Springer Berlin Heidelberg, 2011: 245-260. DOI: 10.1007/978-3-642-19457-3_15 |
| [21] | GURUSWAMY V L, LANG J, LEE W S . IIR Filter Models of Haptic Vibration Textures[J]. IEEE Transactions on Instrumentation and Measurement, 2011,60(1):93-103. DOI: 10.1109/tim.2010.2065751 |
| [22] | SHIN S, OSGOUEI R H, KIM K D , et al. Data-Driven Modeling of Isotropic Haptic Textures Using Frequency-Decomposed Neural Networks [C]//IEEE World Haptics Conference (WHC), Evanston, USA, 2015: 131-138. DOI: 10.1109/WHC.2015.7177703 |
| [23] | YIM S, JEON S, CHOI S . Data-Driven Haptic Modeling and Rendering of Viscoelastic and Frictional Responses of Deformable Objects[J]. IEEE Transactions on Haptics, 2016,9(4):548-559. DOI: 10.1109/toh.2016.2571690 |
| [24] | CHOI H, CHO S, SHIN S , et al. Data-Driven Thermal Rendering: An Initial Study [C]//IEEE Haptics Symposium (HAPTICS), San Francisco, USA, 2018: 344-350. DOI: 10.1109/HAPTICS.2018.8357199 |
| [25] | STEINBACH E, HIRCHE S, KAMMERL J , et al. Haptic Data Compression and Communication[J]. IEEE Signal Processing Magazine, 2011,28(1):87-96. DOI: 10.1109/MSP.2010.938753 |
| [26] | HINTERSEER P, STEINBACH E, HIRCHE S , et al. A Novel, Psychophysically Motivated Transmission Approach for Haptic Data Streams in Telepresence and Teleaction Systems [C]// ICASSP’05. IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, USA, 2005: ii/1097-ii/1100. DOI: 10.1109/ICASSP.2005.1415600 |
| [27] | HINTERSEER P, STEINBACH E, CHAUDHURI S . Perception-Based Compression of Haptic Data Streams Using Kalman Filters [C]//IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, Toulouse, France, 2006. DOI: 10.1109/ICASSP.2006.1661315 |
| [28] | HINTERSEER P, STEIBACH E, CHAUDHURI S . Model Based Data Compression for 3D Virtual Haptic Teleinteraction [C]//International Conference on Consumer Electronics, Las Vegas, USA, 2006: 23-24. DOI: 10.1109/ICCE. 2006.1598291 |
| [29] | HINTERSEER P, HIRCHE S, CHAUDHURI S , et al. Perception-Based Data Reduction and Transmission of Haptic Data in Telepresence and Teleaction Systems[J]. IEEE Transactions on Signal Processing, 2008,56(2):588-597. DOI: 10.1109/tsp.2007.906746 |
| [30] | KAMMERL J, CHAUDHARI R, STEINBACH E . Combining Contact Models with Perceptual Data Reduction for Efficient Haptic Data Communication in Networked VEs[J]. IEEE Transactions on Instrumentation and Measurement, 2011,60(1):57-68. DOI: 10.1109/tim.2010.2065670 |
| [31] | KAMMERL J, VITTORIAS I, NITSCH V , et al. Perception-Based Data Reduction for Haptic Force-Feedback Signals Using Velocity-Adaptive Deadbands[J]. Presence: Teleoperators and Virtual Environments, 2010,19(5):450-462. DOI: 10.1162/pres_a_00008 |
| [32] | KAMMERL J, CHAUDHARI R, STEINBACH E . Exploiting Directional Dependencies of Force Perception for Lossy Haptic Data Reduction [C]//IEEE International Symposium on Haptic Audio Visual Environments and Games, Phoenix, USA, 2010: 1-6. DOI: 10.1109/HAVE.2010.5623975 |
| [33] | BRANDI F, KAMMERL J, STEINBACH E . Error-Resilient Perceptual Coding for Networked Haptic Interaction [C]//ACM International Conference on Multimedia, At Firenze, Italy. 2010: pp. 351-360. DOI: 10.1145/1873951.1874000 |
| [34] | HIRCHE S, BUSS M . Transparent Data Reduction in Networked Telepresence and Teleaction Systems. Part II: Time-Delayed Communication[J]. Presence: Teleoperators and Virtual Environments, 2007,16(5):532-542. DOI: 10.1162/pres.16.5.532 |
| [35] | VITTORIAS I, KAMMERL J, HIRCHE S , et al. Perceptual Coding of Haptic Data in Time-Delayed Teleoperation [C]//World Haptics 2009—Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Salt Lake City, USA, 2009: 208-213. DOI: 10.1109/WHC.2009.4810811 |
| [36] | XU X, SCHUWERK C, CIZMECI B , et al. Energy Prediction for Teleoperation Systems that Combine the Time Domain Passivity Approach with Perceptual Deadband-Based Haptic Data Reduction[J]. IEEE Transactions on Haptics, 2016,9(4):560-573. DOI: 10.1109/toh.2016.2558157 |
| [37] |
XU X, CIZMECI B, AL-NUAIMI A , et al. Point Cloud-Based Model-Mediated Teleoperation with Dynamic and Perception-Based Model Updating[J]. IEEE Transactions on Instrumentation and Measurement, 2014,63(11):2558-2569. DOI: 10.1109/tim.2014.2323139
DOI URL |
| [38] | XU X, KAMMERL J, CHAUDHARI R , et al. Hybrid Signal-Based and Geometry-Based Prediction for Haptic Data Reduction [C]//IEEE International Work shop on Haptic Audio Visual Environments and Games, Qinghuangdao, China, 2011: 68-73. DOI: 10.1109/HAVE.2011.6088394 |
| [39] | XU X, STEINBACH E . Towards Real-Time Modeling and Haptic Rendering of Deformable Objects for Point Cloud-Based Model-Mediated Teleoperation [C]//IEEE International Conference on Multimedia and Expo Workshops (ICMEW), Chengdu, China, 2014: 1-6. DOI: 10.1109/ICMEW.2014.6890578 |
| [40] | XU X, CHEN S L, STEINBACH E . Model-Mediated Teleoperation for Movable Objects: Dynamics Modeling and Packet Rate Reduction [C]//IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), Ottawa, Canada, 2015: 1-6. DOI: 10.1109/HAVE.2015.7359474 |
| [41] | OKAMOTO S, YAMADA Y . Perceptual Properties of Vibrotactile Material Texture: Effects of Amplitude Changes and Stimuli beneath Detection Thresholds [C]//IEEE/SICE International Symposium on System Integration, Sendai, Japan, 2010: 384-389. DOI: 10.1109/SII.2010.5708356 |
| [42] | CRAIG J C . Difference Threshold for Intensity of Tactile Stimuli[J]. Perception & Psychophysics, 1972,11(2):150-152. DOI: 10.3758/bf03210362 |
| [43] | CHAUDHARI R, ÇIZMECI B, KUCHENBECKER K J , et al. Low Bitrate Source-Filter Model Based Compression of Vibrotactile Texture Signals in Haptic Teleoperation [C]//20th ACM International Conference on Multimedia. New York, USA, 2012: 409-418 |
| [44] | ITU-T. Coding Of Speech At 8 Kbit/S Using Conjugate-Structure Algebraic-Code-Excited Linear-Prediction (CS-ACELP): Reduced Complexity 8 kbit/s CSACELP Speech Codec[S]. 1996 |
| [45] | CHAUDHARI R, SCHUWERK C, DANAEI M , et al. Perceptual and Bitrate-Scalable Coding of Haptic Surface Texture Signals[J]. IEEE Journal of Selected Topics in Signal Processing, 2015,9(3):462-473. DOI: 10.1109/jstsp.2014.2374574 |
| [46] | LAWRENCE D A . Stability and Transparency in Bilateral Teleoperation[J]. IEEE Transactions on Robotics and Automation, 1993,9(5):624-637. DOI: 10.1109/70.258054 |
| [47] | EID M, ALAMRI A, SADDIK A E . MPEG-7 Description of Haptic Applications Using HAML [C]//IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE 2006), Ottawa, Canada, 2006: 134-139. DOI: 10.1109/HAVE.2006.283780 |
| [48] | GAO Y, OSMAN H A, EL SADDIK A . MPEG-V Based Web Haptic Authoring Tool [C]//IEEE International Symposium on Haptic Audio Visual Environments and Games (HAVE), Istanbul, Turkey, 2013: 87-91. DOI: 10.1109/HAVE. 2013.6679616 |
| [49] | KING H H, HANNAFORD B, KAMMERLY J , et al. Establishing Multimodal Telepresence Sessions Using the Session Initiation Protocol (SIP) and Advanced Haptic Codecs [C]//IEEE Haptics Symposium, Waltham, USA, 2010: 321-325. DOI: 10.1109/HAPTIC.2010.5444637 |
| [50] | SCHUWERK C, XU X, CHAUDHARI R , et al. Compensating the Effect of Communication Delay in Client-Server: Based Shared Haptic Virtual Environments[J]. ACM Transactions on Applied Perception, 2015,13(1):1-22. DOI: 10.1145/2835176 |
| [51] | SCHUWERK C, PAGGETTI G, CHAUDHARI R , et al. Perception-Based Traffic Control for Shared Haptic Virtual Environments[J]. Presence: Teleoperators and Virtual Environments, 2014,23(3):320-338. DOI: 10.1162/pres_a_00196 |
| [52] | SCHUWERK C, XU X, STEINBACH E . On the Transparency of Client/Server-Based Haptic Interaction with Deformable Objects[J]. IEEE Transactions on Haptics, 2017,10(2):240-253. DOI: 10.1109/toh.2016.2612635 |
| [53] | MILLER R B , Response Time in Man-Computer Conversational Transactions [C]//AFIPS’68 (Fall, part I), San Francisco, USA, 1968: 267-277 |
| [54] | YUAN Z H, BI T, MUNTEAN G M , et al. Perceived Synchronization of Mulsemedia Services[J]. IEEE Transactions on Multimedia, 2015,17(7):957-966. DOI: 10.1109/tmm.2015.2431915 |
| [55] | EID M, CHA J, EL SADDIK A . Admux: An Adaptive Multiplexer for Haptic-Audio-Visual Data Communication[J]. IEEE Transactions on Instrumentation and Measurement, 2011,60(1):21-31. DOI: 10.1109/tim.2010.2065530 |
| [56] | CIZMECI B, XU X, CHAUDHARI R , et al. A Multiplexing Scheme for Multimodal Teleoperation[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2017,13(2):1-28. DOI: 10.1145/3063594 |
| [57] | CIZMECI B, CHAUDHARI R, XU X , et al. A Visual-Haptic Multiplexing Scheme for Teleoperation Over Constant-Bitrate Communication Links[M] //CIZMECI B, CHAUDHARI R, XU X, et al. eds. Haptics: Neuroscience, Devices, Modeling, and Applications. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014: 131-138. DOI: 10.1007/978-3-662-44196-1_17 |
| [58] | EID M, CHA J, EL SADDIK A . An Adaptive Multiplexer for Multi-Modal Data Communication [C]//IEEE International Workshop on Haptic Audio Visual Environments and Games, Lecco, Italy, 2009: 111-116. DOI: 10.1109/HAVE.2009.5356121 |
| [59] | LIU S W, LI M L, XU X , et al. QoE-Driven Uplink Scheduling for Haptic Communications over 5G Enabled Tactile Internet [C]//IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), Dalian, China, 2018: 1-5. DOI: 10.1109/HAVE.2018.8547503 |
| [60] | XU X, LIU Q, STEINBACH E . Toward QoE-Driven Dynamic Control Scheme Switching for Time-Delayed Teleoperation Systems: A Dedicated Case Study [C]//IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), Abu Dhabi, United Arab Emirates, 2017: 1-6. DOI: 10.1109/HAVE.2017.8240352 |
| [61] | AIJAZ A . Toward Human-In-The-Loop Mobile Networks: A Radio Resource Allocation Perspective on Haptic Communications[J]. IEEE Transactions on Wireless Communications, 2018,17(7):4493-4508. DOI: 10.1109/twc.2018. 2825985 |
| [62] | CONDOLUCI M, MAHMOODI T, STEINBACH E , et al. Soft Resource Reservation for Low-Delayed Teleoperation over Mobile Networks[J]. IEEE Access, 2017,5:10445-10455. DOI: 10.1109/access.2017.2707319 |
| [63] | SONG R, WANG Y L, HAN Y , et al. Statistically Uniform Intra-Block Refresh Algorithm for very Low Delay Video Communication[J]. Journal of Zhejiang University SCIENCE C, 2013,14(5):374-382. DOI: 10.1631/jzus.c1200333 |
| [64] | LIU T, CHOUDARY C . Real-Time Content Analysis and Adaptive Transmission of Lecture Videos for Mobile Applications [C]//12th Annual ACM International Conference on Multimedia, New York, USA, 2005: 400-403 |
| No related articles found! |
| 阅读次数 | ||||||
|
全文 |
|
|||||
|
摘要 |
|
|||||