Double Deep Q-Network Decoder Based on EEG Brain-Computer Interface

doi:10.12142/ZTECOM.202303002

Abstract

Abstract:

Brain-computer interfaces (BCI) use neural activity as a control signal to enable direct communication between the human brain and external devices. The electrical signals generated by the brain are captured through electroencephalogram (EEG) and translated into neural intentions reflecting the user's behavior. Correct decoding of the neural intentions then facilitates the control of external devices. Reinforcement learning-based BCIs enhance decoders to complete tasks based only on feedback signals (rewards) from the environment, building a general framework for dynamic mapping from neural intentions to actions that adapt to changing environments. However, using traditional reinforcement learning methods can have challenges such as the curse of dimensionality and poor generalization. Therefore, in this paper, we use deep reinforcement learning to construct decoders for the correct decoding of EEG signals, demonstrate its feasibility through experiments, and demonstrate its stronger generalization on motion imaging (MI) EEG data signals with high dynamic characteristics.

Key words: brain-computer interface (BCI), electroencephalogram (EEG), deep reinforcement learning (Deep RL), motion imaging (MI) generalizability

REN Min, XU Renyu, ZHU Ting. Double Deep Q-Network Decoder Based on EEG Brain-Computer Interface[J]. ZTE Communications, 2023, 21(3): 3-10.

Figures/Tables 6

Figure 1 Reinforcement learning (RL)-based BCI decoding structure

Figure 2 (a) Classical dataset and (b) BCI Competition IV-2a (BCI-2a) dataset are set to a center-out task. The center is located at the origin (0,0), represented by a green square, and each class target is a purple circle

Figure 3 Network structure of electroencephalogram (EEG) signal decoder based on double deep Q-network (DDQN)

Figure 4 Learning curves of double deep Q-network (DDQN) on (a) Classical (CLA) dataset and (b) BCI competition IV-2a EEG data (BCI-2a) dataset, where each color represents the average success rate of 10 Monte Carlo trials

Figure 5 The generalizability of DDQN is compared with other classical algorithms for decoding based on Feature 1 (left) and Feature 2 (right) on the (a)(b)BCI-2a dataset and (c)(d) Classical (CLA) dataset, respectively

Figure 6 Results of experiments on (a) BCI-2a and (b) CLA under 10 different random seeds

References 34

1	WILLETT F R, AVANSINO D T, HOCHBERG L R, et al. High-performance brain-to-text communication via handwriting [J]. Nature, 2021, 593(7858): 249–254. DOI: 10.1038/s41586-021-03506-2 DOI URL
2	CRUZ A, PIRES G, LOPES A, et al. A self-paced BCI with a collaborative controller for highly reliable wheelchair driving: experimental tests with physically disabled individuals [J]. IEEE transactions on human-machine systems, 2021, 51(2): 109–119. DOI: 10.1109/THMS.2020.3047597 DOI URL
3	SCHWARZ A, HÖLLER M K, PEREIRA J, et al. Decoding hand movements from human EEG to control a robotic arm in a simulation environment [J]. Journal of neural engineering, 2020, 17(3): 036010. DOI: 10.1088/1741-2552/ab882e DOI URL
4	SONG Y H, WU W F, LIN C Q, et al. Assistive mobile robot with shared control of brain-machine interface and computer vision [C]//4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). IEEE, 2020: 405–409. DOI: 10.1109/ITNEC48623.2020.9085096 DOI URL
5	ANG K K, CHIN Z Y, WANG C C, et al. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b [J]. Frontiers in neuroscience, 2012, 6: 39. DOI: 10.3389/fnins.2012.00039 DOI URL
6	TONIN L, CARLSON T, LEEB R, et al. Brain-controlled telepresence robot by motor-disabled people [C]//Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2011: 4227–4230. DOI: 10.1109/IEMBS.2011.6091049 DOI URL
7	GROSSE-WENTRUP M, BUSS M. Multiclass common spatial patterns and information theoretic feature extraction [J]. IEEE transactions on biomedical engineering, 2008, 55(8): 1991–2000. DOI: 10.1109/TBME.2008.921154 DOI URL
8	JIN J, XIAO R, DALY I, et al. Internal feature selection method of CSP based on L1-norm and Dempster⁃Shafer theory [J]. IEEE transactions on neural networks and learning systems, 2020, 32(11): 4814–4825. DOI: 10.1109/TNNLS.2020.3015505 DOI URL
9	JIN J, MIAO Y, DALY I, et al. Correlation-based channel selection and regularized feature optimization for MI-based BCI [J]. Neural networks, 2019, 118: 262–270. DOI: 10.1016/j.neunet.2019.07.008 DOI URL
10	FU R R, TIAN Y S, BAO T T, et al. Improvement motor imagery EEG classification based on regularized linear discriminant analysis [J]. Journal of medical systems, 2019, 43(6): 169. DOI: 10.1007/s10916-019-1270-0 DOI URL
11	LIU Y X, ZHOU W D, YUAN Q, et al. Automatic seizure detection using wavelet transform and SVM in long-term intracranial EEG [J]. IEEE transactions on neural systems and rehabilitation engineering, 2012, 20(6): 749–755. DOI: 10.1109/TNSRE.2012.2206054 DOI URL
12	SAMUEL O W, GENG Y J, LI X X, et al. Towards efficient decoding of multiple classes of motor imagery limb movements based on EEG spectral and time domain descriptors [J]. Journal of medical systems, 2017, 41(12): 194. DOI: 10.1007/s10916-017-0843-z DOI URL
13	LIN C T, CHUANG C H, HUNG Y C, et al. A driving performance forecasting system based on brain dynamic state analysis using 4-D convolutional neural networks [J]. IEEE transactions on cybernetics, 2021, 51(10): 4959–4967. DOI: 10.1109/TCYB.2020.3010805 DOI URL
14	AMIN S U, ALSULAIMAN M, MUHAMMAD G, et al. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion [J]. Future generation computer systems, 2019, 101: 542–554. DOI: 10.1016/j.future.2019.06.027 DOI URL
15	TAYLOR D M, TILLERY S I H, SCHWARTZ A B. Direct cortical control of 3D neuroprosthetic devices [J]. Science, 2002, 296(5574): 1829–1832. DOI: 10.1126/science.1070291 DOI URL
16	GAGE G J, LUDWIG K A, OTTO K J, et al. Naïve coadaptive cortical control [J]. Journal of neural engineering, 2005, 2(2): 52–63. DOI: 10.1088/1741-2560/2/2/006 DOI URL
17	VELLISTE M, PEREL S, SPALDING M C, et al. Cortical control of a prosthetic arm for self-feeding [J]. Nature, 2008, 453(7198): 1098–1101. DOI: 10.1038/nature06996 DOI URL
18	SUTTON R S, BARTO A G. Reinforcement learning: an introduction [M]. Cambridge, USA: MIT press, 2018
19	DIGIOVANNA J, MAHMOUDI B, FORTES J, et al. Coadaptive brain: machine interface via reinforcement learning [J]. IEEE transactions on biomedical engineering, 2009, 56(1): 54–64. DOI: 10.1109/TBME.2008.926699 DOI URL
20	ITURRATE I, MONTESANO L, MINGUEZ J. Robot reinforcement learning using EEG-based reward signals [C]//IEEE International Conference on Robotics and Automation. IEEE, 2010: 4822–4829. DOI: 10.1109/ROBOT.2010.5509734 DOI URL
21	MATSUZAKI S, SHIINA Y, WADA Y. Adaptive classification for brain-machine interface with reinforcement learning [C]//18th International Conference on Neural Information Processing. ICONIP, 2011: 360–369. DOI: 10.1007/978-3-642-24955-6_44 DOI URL
22	MAHMOUDI B, SANCHEZ J C. A symbiotic brain-machine interface through value-based decision making [J]. PLoS one, 2011, 6(3): e14760. DOI: 10.1371/journal.pone.0014760 DOI URL
23	SANCHEZ J C, TARIGOPPULA A, CHOI J S, et al. Control of a center-out reaching task using a reinforcement learning brain-machine interface [C]//5th International IEEE/EMBS Conference on Neural Engineering. IEEE, 2011: 525–528. DOI: 10.1109/NER.2011.5910601 DOI URL
24	POHLMEYER E A, MAHMOUDI B, GENG S J, et al. Using reinforcement learning to provide stable brain-machine interface control despite neural input reorganization [J]. PLoS one, 2014, 9(1): e87253. DOI: 10.1371/journal.pone.0087253 DOI URL
25	MARSH B T, TARIGOPPULA V S A, CHEN C, et al. Toward an autonomous brain machine interface: integrating sensorimotor reward modulation and reinforcement learning [J]. Journal of neuroscience, 2015, 35(19): 7374–7387. DOI: 10.1523/jneurosci.1802-14.2015 DOI URL
26	BAE J, CHHATBAR P, FRANCIS J T, et al. Reinforcement learning via kernel temporal difference [C]//Annual International Conference of IEEE Engineering in Medicine and Biology Society. IEEE, 2011: 5662–5665. DOI: 10.1109/IEMBS.2011.6091370 DOI URL
27	THAPA B R, TANGARIFE D R, BAE J. Kernel temporal differences for EEG-based reinforcement learning brain machine interfaces [C]//44th Annual International Conference of IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2022: 3327–3333. DOI: 10.1109/EMBC48229.2022.9871862 DOI URL
28	VAN HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning [C]//AAAI Conference on Artificial Intelligence. ACM, 2016: 2094–2100. DOI: 10.5555/3016100.3016191 DOI URL
29	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning [C]//NIPS Deep Learning Workshop. NIPS, 2013. DOI: 10.48550/arXiv.1312.5602 DOI URL
30	KAYA M, BINLI M K, OZBAY E, et al. A large electroencephalographic motor imagery dataset for electroencephalographic brain computer interfaces [J]. Scientific data, 2018, 5(1): 1–16. DOI: 10.1038/sdata.2018.211 DOI URL
31	TANGERMANN M, MÜLLER K R, AERTSEN A, et al. Review of the BCI competition IV [J]. Frontiers in neuroscience, 2012, 6: 55. DOI: 10.3389/fnins.2012.00055 DOI URL
32	QI F, WANG W, XIE X, et al. Single-trial eeg classification via orthogonal wavelet decomposition-based feature extraction [J]. Frontiers in Neuroscience, 2021, 15: 715855. DOI: 10.3389/fnins.2021.715855 DOI URL
33	MUSALLAM Y K, ALFASSAM N I, MUHAMMAD G, et al. Electroencephalography-based motor imagery classification using temporal convolutional network fusion [J]. Biomedical signal processing and control, 2021, 69: 102826. DOI: 10.1016/j.bspc.2021.102826 DOI URL
34	KEERTHI KRISHNAN K, SOMAN K P. CNN based classification of motor imaginary using variational mode decomposed EEG-spectrum image [J]. Biomedical engineering letters, 2021, 11(3): 235–247. DOI: 10.1007/s13534-021-00190-z DOI URL