Quality-of-Experience in Human-in-the-Loop Haptic Communications

doi:10.12142/ZTECOM.201901008

Abstract

Abstract:

With the worldwide rapid development of 5G networks, haptic communications, a key use case of the 5G, has attracted increasing attentions nowadays. Its human-in-the-loop nature makes quality of experience (QoE) the leading performance indicator of the system design. A vast number of high quality works were published on user-level, application-level and network-level QoE-oriented designs in haptic communications. In this paper, we present an overview of the recent research activities in this progressive research area. We start from the QoE modeling of human haptic perceptions, followed by the application-level QoE management mechanisms based on these QoE models. High fidelity haptic communications require an orchestra of QoE designs in the application level and the quality of service (QoS) support in the network level. Hence, we also review the state-of-the-art QoS-related QoE management strategies in haptic communications, especially the QoS-related QoE modeling which guides the resource allocation design of the communication network. In addition to a thorough survey of the literature, we also present the open challenges in this research area. We believe that our review and findings in this paper not only provide a timely summary of prevailing research in this area, but also help to inspire new QoE-related research opportunities in haptic communications.

Key words: QoE, human-in-the-loop, haptic communications, kinesthetic signals, tactile signals, haptic

LIU Qian, ZHAO Tiesong. Quality-of-Experience in Human-in-the-Loop Haptic Communications[J]. ZTE Communications, 2019, 17(1): 48-55.

Figures/Tables 8

References 64

[1]	ZHAO T S, LIU Q, CHEN C W . QoE in Video Transmission: A User Experience-Driven Strategy[J]. IEEE Communications Surveys & Tutorials, 2017,19(1):285-302. DOI: 10.1109/comst.2016.2619982
[2]	PARK P, COLERI ERGEN S, FISCHIONE C , et al. Wireless Network Design for Control Systems: A Survey[J]. IEEE Communications Surveys & Tutorials, 2018,20(2):978-1013. DOI: 10.1109/comst.2017.2780114
[3]	ANTONAKOGLOU K, XU X, STEINBACH E , et al. Toward Haptic Communications over the 5G Tactile Internet[J]. IEEE Communications Surveys & Tutorials, 2018,20(4):3034-3059. DOI: 10.1109/comst.2018.2851452
[4]	OKAMOTO S, NAGANO H, YAMADA Y . Psychophysical Dimensions of Tactile Perception of Textures[J]. IEEE Transactions on Haptics, 2013,6(1):81-93. DOI: 10.1109/toh.2012.32 DOI URL
[5]	CHA J, HO Y-S, KIM Y , et al. A Framework for Haptic Broadcasting[J]. IEEE Multimedia, 2019,16(3): pp. 16-27.
[6]	STEINBACH E, HIRCHE S, ERNST M , et al. Haptic Communications[J]. Proceedings of the IEEE, 2012,100:937-956. DOI: 10.1109/JPROC.2011. 2182100
[7]	SILVA A, RAMIREZ O , et al. PHANToM OMNI Haptic Device: Kinematic and Manipulability [C]//Electronics, Robotics and Automotive Mechanics Conference (CERMA), Cuernavaca, Morelos, 2009: 193-198
[8]	FORCE DIMENSION . Force Dimension User Manual of Omega.x Haptic Device, Version 1.7 [Z]. Nyon, Switzerland, 2013.
[9]	BREWSTER S, CHOHAN F, BROWN L . Tactile Feedback for Mobile Interactions [C]//SIGCHI Conference on Human Factors in Computing Systems, Gaithersburg, USA, 2007: 159-162
[10]	DREWING K, FRITSCHI M, ZOPF R , et al. First Evaluation of a Novel Tactile Display Exerting Shear Force Via Lateral Displacement[J]. ACM Transactions on Applied Perception, 2005,2(2):118-131. DOI: 10.1145/1060581.1060586
[11]	BENALI-KHOUDJA M, HAFEZ M, ALEXANDRE J M , et al. Tactile Interfaces: A State-of-the-Art Survey [C]//International Symposium of Robotics, Paris, France, 2004: 23-26
[12]	CHOUVARDAS V G, MILIOU A N, HATALIS M K . Tactile Display Applications: A State-of-the-Art Survey [C]//2nd Balkan Conference in Informatics, Ohrid, Macedonia, 2005: 290-303
[13]	WINFIELD L, GLASSMIRE J, COLGATE J E , et al. T-PaD: Tactile Pattern Display Through Variable Friction Reduction [C]//Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07), Tsukaba, Japan, 2007: 421-426. DOI: 10.1109/WHC.2007.105
[14]	BAU O, POUPYREV I, ISRAR A et al. TeslaTouch: Electrovibration for Touch Surfaces[C]// User Interface Software and Technology (UIST), New York, USA:283-292, 2010.
[15]	WEBER E. The Sense of Touch [M]. Cambridge, USA: Academic Press, 1978
[16]	GESCHEIDER G A , Psychophysics: The Fundamentals[M]. 3rd ed. Mahwah, USA: Lawrence Erlbaum, 1997
[17]	DEHAENE S . The Neural Basis of the Weber-Fechner Law: A Logarithmic Mental Number Line[J]. Trends in Cognitive Sciences, 2003,7(4):145-147. DOI: 10.1016/S1364-6613(03)00055-X
[18]	DEHAENE S . Subtracting Pigeons: Logarithmic or Linear?[J] Phychological Science, 2001,12(3):244-246. DOI: 10.1111/1467-9280.00343
[19]	OKAMURA A M, DENNERLEIN J T, HOWE R D . Vibration Feedback Models for Virtual Environments [C]//IEEE International Conference on Robotics and Automation (Cat. no.98CH36146), Leuven, Belgium, 1998: 674-679. DOI: 10.1109/ROBOT.1998.677050
[20]	KUCHENBECKER K J, ROMANO J, MCMAHAN W . Haptography: Capturing and Recreating the Rich Feel of Real Surfaces[M]//KUCHENBECKER K J, ROMANO J, MCMAHAN W. eds. Springer Tracts in Advanced Robotics. Berlin, Heidelberg, Germany: Springer Berlin Heidelberg, 2011: 245-260. DOI: 10.1007/978-3-642-19457-3_15
[21]	GURUSWAMY V L, LANG J, LEE W S . IIR Filter Models of Haptic Vibration Textures[J]. IEEE Transactions on Instrumentation and Measurement, 2011,60(1):93-103. DOI: 10.1109/tim.2010.2065751
[22]	SHIN S, OSGOUEI R H, KIM K D , et al. Data-Driven Modeling of Isotropic Haptic Textures Using Frequency-Decomposed Neural Networks [C]//IEEE World Haptics Conference (WHC), Evanston, USA, 2015: 131-138. DOI: 10.1109/WHC.2015.7177703
[23]	YIM S, JEON S, CHOI S . Data-Driven Haptic Modeling and Rendering of Viscoelastic and Frictional Responses of Deformable Objects[J]. IEEE Transactions on Haptics, 2016,9(4):548-559. DOI: 10.1109/toh.2016.2571690
[24]	CHOI H, CHO S, SHIN S , et al. Data-Driven Thermal Rendering: An Initial Study [C]//IEEE Haptics Symposium (HAPTICS), San Francisco, USA, 2018: 344-350. DOI: 10.1109/HAPTICS.2018.8357199
[25]	STEINBACH E, HIRCHE S, KAMMERL J , et al. Haptic Data Compression and Communication[J]. IEEE Signal Processing Magazine, 2011,28(1):87-96. DOI: 10.1109/MSP.2010.938753
[26]	HINTERSEER P, STEINBACH E, HIRCHE S , et al. A Novel, Psychophysically Motivated Transmission Approach for Haptic Data Streams in Telepresence and Teleaction Systems [C]// ICASSP’05. IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, USA, 2005: ii/1097-ii/1100. DOI: 10.1109/ICASSP.2005.1415600
[27]	HINTERSEER P, STEINBACH E, CHAUDHURI S . Perception-Based Compression of Haptic Data Streams Using Kalman Filters [C]//IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, Toulouse, France, 2006. DOI: 10.1109/ICASSP.2006.1661315
[28]	HINTERSEER P, STEIBACH E, CHAUDHURI S . Model Based Data Compression for 3D Virtual Haptic Teleinteraction [C]//International Conference on Consumer Electronics, Las Vegas, USA, 2006: 23-24. DOI: 10.1109/ICCE. 2006.1598291
[29]	HINTERSEER P, HIRCHE S, CHAUDHURI S , et al. Perception-Based Data Reduction and Transmission of Haptic Data in Telepresence and Teleaction Systems[J]. IEEE Transactions on Signal Processing, 2008,56(2):588-597. DOI: 10.1109/tsp.2007.906746
[30]	KAMMERL J, CHAUDHARI R, STEINBACH E . Combining Contact Models with Perceptual Data Reduction for Efficient Haptic Data Communication in Networked VEs[J]. IEEE Transactions on Instrumentation and Measurement, 2011,60(1):57-68. DOI: 10.1109/tim.2010.2065670
[31]	KAMMERL J, VITTORIAS I, NITSCH V , et al. Perception-Based Data Reduction for Haptic Force-Feedback Signals Using Velocity-Adaptive Deadbands[J]. Presence: Teleoperators and Virtual Environments, 2010,19(5):450-462. DOI: 10.1162/pres_a_00008
[32]	KAMMERL J, CHAUDHARI R, STEINBACH E . Exploiting Directional Dependencies of Force Perception for Lossy Haptic Data Reduction [C]//IEEE International Symposium on Haptic Audio Visual Environments and Games, Phoenix, USA, 2010: 1-6. DOI: 10.1109/HAVE.2010.5623975
[33]	BRANDI F, KAMMERL J, STEINBACH E . Error-Resilient Perceptual Coding for Networked Haptic Interaction [C]//ACM International Conference on Multimedia, At Firenze, Italy. 2010: pp. 351-360. DOI: 10.1145/1873951.1874000
[34]	HIRCHE S, BUSS M . Transparent Data Reduction in Networked Telepresence and Teleaction Systems. Part II: Time-Delayed Communication[J]. Presence: Teleoperators and Virtual Environments, 2007,16(5):532-542. DOI: 10.1162/pres.16.5.532
[35]	VITTORIAS I, KAMMERL J, HIRCHE S , et al. Perceptual Coding of Haptic Data in Time-Delayed Teleoperation [C]//World Haptics 2009—Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Salt Lake City, USA, 2009: 208-213. DOI: 10.1109/WHC.2009.4810811
[36]	XU X, SCHUWERK C, CIZMECI B , et al. Energy Prediction for Teleoperation Systems that Combine the Time Domain Passivity Approach with Perceptual Deadband-Based Haptic Data Reduction[J]. IEEE Transactions on Haptics, 2016,9(4):560-573. DOI: 10.1109/toh.2016.2558157
[37]	XU X, CIZMECI B, AL-NUAIMI A , et al. Point Cloud-Based Model-Mediated Teleoperation with Dynamic and Perception-Based Model Updating[J]. IEEE Transactions on Instrumentation and Measurement, 2014,63(11):2558-2569. DOI: 10.1109/tim.2014.2323139 DOI URL
[38]	XU X, KAMMERL J, CHAUDHARI R , et al. Hybrid Signal-Based and Geometry-Based Prediction for Haptic Data Reduction [C]//IEEE International Work shop on Haptic Audio Visual Environments and Games, Qinghuangdao, China, 2011: 68-73. DOI: 10.1109/HAVE.2011.6088394
[39]	XU X, STEINBACH E . Towards Real-Time Modeling and Haptic Rendering of Deformable Objects for Point Cloud-Based Model-Mediated Teleoperation [C]//IEEE International Conference on Multimedia and Expo Workshops (ICMEW), Chengdu, China, 2014: 1-6. DOI: 10.1109/ICMEW.2014.6890578
[40]	XU X, CHEN S L, STEINBACH E . Model-Mediated Teleoperation for Movable Objects: Dynamics Modeling and Packet Rate Reduction [C]//IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), Ottawa, Canada, 2015: 1-6. DOI: 10.1109/HAVE.2015.7359474
[41]	OKAMOTO S, YAMADA Y . Perceptual Properties of Vibrotactile Material Texture: Effects of Amplitude Changes and Stimuli beneath Detection Thresholds [C]//IEEE/SICE International Symposium on System Integration, Sendai, Japan, 2010: 384-389. DOI: 10.1109/SII.2010.5708356
[42]	CRAIG J C . Difference Threshold for Intensity of Tactile Stimuli[J]. Perception & Psychophysics, 1972,11(2):150-152. DOI: 10.3758/bf03210362
[43]	CHAUDHARI R, ÇIZMECI B, KUCHENBECKER K J , et al. Low Bitrate Source-Filter Model Based Compression of Vibrotactile Texture Signals in Haptic Teleoperation [C]//20th ACM International Conference on Multimedia. New York, USA, 2012: 409-418
[44]	ITU-T. Coding Of Speech At 8 Kbit/S Using Conjugate-Structure Algebraic-Code-Excited Linear-Prediction (CS-ACELP): Reduced Complexity 8 kbit/s CSACELP Speech Codec[S]. 1996
[45]	CHAUDHARI R, SCHUWERK C, DANAEI M , et al. Perceptual and Bitrate-Scalable Coding of Haptic Surface Texture Signals[J]. IEEE Journal of Selected Topics in Signal Processing, 2015,9(3):462-473. DOI: 10.1109/jstsp.2014.2374574
[46]	LAWRENCE D A . Stability and Transparency in Bilateral Teleoperation[J]. IEEE Transactions on Robotics and Automation, 1993,9(5):624-637. DOI: 10.1109/70.258054
[47]	EID M, ALAMRI A, SADDIK A E . MPEG-7 Description of Haptic Applications Using HAML [C]//IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE 2006), Ottawa, Canada, 2006: 134-139. DOI: 10.1109/HAVE.2006.283780
[48]	GAO Y, OSMAN H A, EL SADDIK A . MPEG-V Based Web Haptic Authoring Tool [C]//IEEE International Symposium on Haptic Audio Visual Environments and Games (HAVE), Istanbul, Turkey, 2013: 87-91. DOI: 10.1109/HAVE. 2013.6679616
[49]	KING H H, HANNAFORD B, KAMMERLY J , et al. Establishing Multimodal Telepresence Sessions Using the Session Initiation Protocol (SIP) and Advanced Haptic Codecs [C]//IEEE Haptics Symposium, Waltham, USA, 2010: 321-325. DOI: 10.1109/HAPTIC.2010.5444637
[50]	SCHUWERK C, XU X, CHAUDHARI R , et al. Compensating the Effect of Communication Delay in Client-Server: Based Shared Haptic Virtual Environments[J]. ACM Transactions on Applied Perception, 2015,13(1):1-22. DOI: 10.1145/2835176
[51]	SCHUWERK C, PAGGETTI G, CHAUDHARI R , et al. Perception-Based Traffic Control for Shared Haptic Virtual Environments[J]. Presence: Teleoperators and Virtual Environments, 2014,23(3):320-338. DOI: 10.1162/pres_a_00196
[52]	SCHUWERK C, XU X, STEINBACH E . On the Transparency of Client/Server-Based Haptic Interaction with Deformable Objects[J]. IEEE Transactions on Haptics, 2017,10(2):240-253. DOI: 10.1109/toh.2016.2612635
[53]	MILLER R B , Response Time in Man-Computer Conversational Transactions [C]//AFIPS’68 (Fall, part I), San Francisco, USA, 1968: 267-277
[54]	YUAN Z H, BI T, MUNTEAN G M , et al. Perceived Synchronization of Mulsemedia Services[J]. IEEE Transactions on Multimedia, 2015,17(7):957-966. DOI: 10.1109/tmm.2015.2431915
[55]	EID M, CHA J, EL SADDIK A . Admux: An Adaptive Multiplexer for Haptic-Audio-Visual Data Communication[J]. IEEE Transactions on Instrumentation and Measurement, 2011,60(1):21-31. DOI: 10.1109/tim.2010.2065530
[56]	CIZMECI B, XU X, CHAUDHARI R , et al. A Multiplexing Scheme for Multimodal Teleoperation[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2017,13(2):1-28. DOI: 10.1145/3063594
[57]	CIZMECI B, CHAUDHARI R, XU X , et al. A Visual-Haptic Multiplexing Scheme for Teleoperation Over Constant-Bitrate Communication Links[M] //CIZMECI B, CHAUDHARI R, XU X, et al. eds. Haptics: Neuroscience, Devices, Modeling, and Applications. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014: 131-138. DOI: 10.1007/978-3-662-44196-1_17
[58]	EID M, CHA J, EL SADDIK A . An Adaptive Multiplexer for Multi-Modal Data Communication [C]//IEEE International Workshop on Haptic Audio Visual Environments and Games, Lecco, Italy, 2009: 111-116. DOI: 10.1109/HAVE.2009.5356121
[59]	LIU S W, LI M L, XU X , et al. QoE-Driven Uplink Scheduling for Haptic Communications over 5G Enabled Tactile Internet [C]//IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), Dalian, China, 2018: 1-5. DOI: 10.1109/HAVE.2018.8547503
[60]	XU X, LIU Q, STEINBACH E . Toward QoE-Driven Dynamic Control Scheme Switching for Time-Delayed Teleoperation Systems: A Dedicated Case Study [C]//IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), Abu Dhabi, United Arab Emirates, 2017: 1-6. DOI: 10.1109/HAVE.2017.8240352
[61]	AIJAZ A . Toward Human-In-The-Loop Mobile Networks: A Radio Resource Allocation Perspective on Haptic Communications[J]. IEEE Transactions on Wireless Communications, 2018,17(7):4493-4508. DOI: 10.1109/twc.2018. 2825985
[62]	CONDOLUCI M, MAHMOODI T, STEINBACH E , et al. Soft Resource Reservation for Low-Delayed Teleoperation over Mobile Networks[J]. IEEE Access, 2017,5:10445-10455. DOI: 10.1109/access.2017.2707319
[63]	SONG R, WANG Y L, HAN Y , et al. Statistically Uniform Intra-Block Refresh Algorithm for very Low Delay Video Communication[J]. Journal of Zhejiang University SCIENCE C, 2013,14(5):374-382. DOI: 10.1631/jzus.c1200333
[64]	LIU T, CHOUDARY C . Real-Time Content Analysis and Adaptive Transmission of Lecture Videos for Mobile Applications [C]//12th Annual ACM International Conference on Multimedia, New York, USA, 2005: 400-403

	Kinesthetic sense	Tactile sense
Signal type	Position, velocity, angular velocity, force, and torque	Surface texture and friction
Human perception mechanism	Sensed by the muscles, joints, and tendons of the body	Sensed by different types of mechanoreceptors in the skin
Exemplar signal generation solutions	Using high torque motors to generate kinesthetic force feedback, such as Geomagic Touch (used to be called as Phantom Omni [7]) and Omega 3 [8]	Multi-pin display attached to the human skin [9]-[11], or using vibrators to display vibrotactile stimuli [12], such as TPad [13] and TeslaTouch [14]

	Kinesthetic sense	Tactile sense
Signal type	Position, velocity, angular velocity, force, and torque	Surface texture and friction
Human perception mechanism	Sensed by the muscles, joints, and tendons of the body	Sensed by different types of mechanoreceptors in the skin
Exemplar signal generation solutions	Using high torque motors to generate kinesthetic force feedback, such as Geomagic Touch (used to be called as Phantom Omni [7]) and Omega 3 [8]	Multi-pin display attached to the human skin [9]-[11], or using vibrators to display vibrotactile stimuli [12], such as TPad [13] and TeslaTouch [14]

Haptic type	Human perception model	Data reduction solutions
Kinesthetic	Weber’s law (linear) [16]; Fechner’s law (logarithmic) [17]	Without considering network conditions	considering network conditions
		Perceptual deadband schemes:Single DoF [2];3-DoF [29], [30]	Solutions	Knownconst. delay	Unknown const. delay	Time-varying delay
			WV+ PD	[34]	[35]	-
			TDPA+ PD	[36]	[36]	[36]
			MMT+ PD	[37]	[37]	-
Tactile	Data-driven ML models	[41], [42]	-
Tactile	Similar to speech signal	[43], [45]	-

Haptic type	Human perception model	Data reduction solutions
Kinesthetic	Weber’s law (linear) [16]; Fechner’s law (logarithmic) [17]	Without considering network conditions	considering network conditions
		Perceptual deadband schemes:Single DoF [2];3-DoF [29], [30]	Solutions	Knownconst. delay	Unknown const. delay	Time-varying delay
			WV+ PD	[34]	[35]	-
			TDPA+ PD	[36]	[36]	[36]
			MMT+ PD	[37]	[37]	-
Tactile	Data-driven ML models	[41], [42]	-
Tactile	Similar to speech signal	[43], [45]	-