1. Introduction
The human brain controls the body by passing signals through a peripheral nervous system. This process is started with the human’s intent and continues through peripheral nerves until the destination body part is reached. Recent advances in electrophysiological recording technology offer alternative ways to bypass the peripheral nervous system and control a device directly by the brain. Such a system that is responsible from translating brain activity to device control command is called a Brain Computer Interface (BCI) .Reference Graimann, Allison and Pfurtscheller 1
A BCI measures the brain activity patterns produced by the user’s intent and uses it for applications such as communication or control. This can be very useful for patients with motor disabilities. However the application of BCI is not limited to people with disabilities. BCI can be used in a variety of applications, from communication tools for Locked-In State (CLIS) patients to video gaming for healthy people.
An overview of a BCI system is given in Figure 1. The device to be controlled may be a wheelchair, a neuroprothesis, a computer, a game console or any other device. In a BCI system, the user represents his/her intention by a mental activity. The resulting brain signals are transmitted to a computer and processed to generate a control signal for the device to be manipulated. The control signal is used to change the state of the device controlled and a feedback about the new state of the device is provided to the user. The loop continues as the user changes his or her mental activity according to the new state of the device.
Figure 1 Overview of a BCI system.
Any BCI system interacts with the user by using different types of feedback signals. Using these feedbacks provides the adaptation of the user to the system and also the system to the user. Subjects learn to regulate their brain activities by using the online feedback signals sent by the BCI system. The information collected may also be used to train the BCI system through machine learning algorithms.
There are different control paradigms that define how the user interacts with the BCI system. In asynchronous control, users can interact with a BCI any time without worrying about timing. However, in a synchronous control system there are specific time intervals that the user should respond to only in these periods. This is the easiest and probably the most common paradigm in BCI applications.
People with motor disabilities can use BCI to control their environment. Controlling the TV, lights, or room temperature can improve the quality of life for these people.Reference Sellers, Vaughan and Wolpaw 2 Locomotion is another BCI application that helps people with physical impairments to control their wheelchairs autonomously.Reference Rebsamen, Guan, Zhang, Wang, Teo, Ang and Burdet 3 Improvements in BCI technology have opened a new way to extend BCI use by non-disabled people. BCI provides a new interaction modality to play video games or use computers. In some recent studies, simple video games, such as Pacman, are being controlled by motor imagery.Reference Krepki, Blankertz, Curio and Muller 4
Speech communication, which is also called silent speech, is one of the main applications of BCI for people who have communication disabilities. There have been a lot of studies in the field. In these studies, a variety of brain activities have been used to select the target letter from an on-screen display. One of the most popular paradigms for BCI control in communication applications is to use P300 event-related brain potentials. These signals are used in many speech communication studies in BCI.Reference Farwell and Donchin 5 – Reference Li, Nam, Shadden and Johnson 9 Steady State Visual Evoked Potentials (SSVEP) are other types of control signals that have been used frequently for speech communication.Reference Cheng, Gao, Gao and Xu 10 – Reference Segers, Combaz, Manyakov, Chumerin, Vanderperren, Van Huffel and Van Hulle 13 Motor imagery signals are also popular in speech communication.Reference Obermaier, Muller and Pfurtscheller 14 – Reference Blankertz, Müller, Krusienski, Schalk, Wolpaw, Schlögl, Pfurtscheller, Millan, Schröder and Birbaumer 16
The aim of this paper is to review BCI systems with an emphasis on speech communication applications, This application is chosen since there are several studies in the literature and it is most appropriate for showing how different approaches can be used for the same purpose in BCI. In the next section, how to measure brain activity in general is explained. In Section 3, different modalities for EEG signal acquisition are explained since EEG is the most convenient and widely used approach in BCI systems. In Section 4, how to process EEG signals is explained along with information about the toolboxes, software libraries and datasets available for BCI applications. Then, in Section 5, the existing BCI systems for silent speech are summarized and how to measure the performances of such systems is also explained. Finally, conclusions are provided in Section 6.
2. Measuring Brain Activity
Brain activity produces electrophysiological and haemodynamic activities. There are different sensors that can detect different types of activities in the brain. Signal acquisition methods can be categorized in to two main groups: invasive and non-invasive techniques. Table 1, which is an extended version of the table provided in Ref. Reference Nicolas-Alonso and Gomez-Gil17, summarizes different signal acquisition methods.
Table 1 Properties of different signal acquisition methods.
Invasive methods record the brain signals using sensors implanted inside the body. Micro-electrode arrays (MeA) are highly invasive since they are implanted inside the brain.Reference Suner, Fellows, Vargas-Irwin, Nakata and Donoghue 18 Electrocorticographic (ECoG) activity recording is another invasive approach in which the sensors are placed not inside but on the surface of the brain.Reference Freeman, Holmes, Burke and Vanhatalo 19 Despite the accurate signal recording ability of the invasive methods, surgery risks and implant-related problems make these methods less preferable for BCI applications. However, there are some studies that used EcoGReference Levine, Huggins, BeMent, Kushwaha, Schuh, Passaro, Rohde and Ross 20 and MeAReference Kennedy, Kirby, Moore, King and Mallory 21 for BCI applications.
Non-invasive techniques involve all the methods that record brain activity from outside of the body boundaries. These methods can measure two groups of signals: signals from haemodynamic (blood oxygenation levels) activities Reference Gelenbe, Feng and Krishnan 105 and signals from electrophysiological (neuronal) activities.Reference Wolpaw, Loeb, Allison, Donchin, do Nascimento, Heetderks, Nijboer, Shain and Turner 22 The first group of signals can be detected with functional magnetic resonance imaging (fMRI) or near-infrared spectroscopy (NIRS) methods. In fMRI, blood oxygenation level-dependent (BOLD) signals associated with cortical activation are being measured. Different oxygen levels of the blood can also be measured by NIRS, which is a portable device with a higher temporal resolution but lower spatial resolution compared with fMRI.Reference Bauernfeind, Leeb, Wriessnegger and Pfurtscheller 23 There are few studies that use fMRI for BCI applications.Reference Ward and Mazaheri 24 This is because of the difficulty of real time measurement of the brain activity. On the other hand, fNIRS has been used in several BCI studies in recent years, although it has lower spatial resolution.Reference Coyle, Ward and Markham 25 , Reference Power, Kushki and Chau 26
Magnetoencephalography (MEG) and electroencephalography (EEG) methods are two basic modalities for measuring brain electrophysiological activities. MEG measures the brain activity with high resolution by measuring the magnetic fields induced by the neuron’s electric current. However, MEG equipment is large and expensive, which makes it a poor choice for BCI applications.Reference Nicolas-Alonso and Gomez-Gil 17 Some studies have used MEG for BCI applications, though.Reference Lal, Schröder, Hill, Preissl, Hinterberger, Mellinger, Bogdan, Rosenstiel, Hofmann, Birbaumer and Schölkopf 27 – Reference Jinyin, Sudre, Xin, Wei, Weber and Bagic 29
EEG also records brain activity by measuring the electrical fields produced by firing neurons. EEG signals have comparatively low spatial resolution but, high temporal resolution with cheap and easy to use equipment.Reference Nicolas-Alonso and Gomez-Gil 17 These features make the EEG a proper choice for BCI applications. EEG is used as the signal acquisition method in plenty of BCI studies and therefore is explained in detail in the following section.
3. Brain Activities Used in EEG-based BCI
In EEG, sensor electrodes are placed over the head to measure the brain activity. The number of electrodes can vary from 1 to more than 100. To accurately place the electrodes over the head and measure the activities in different parts of the brain, the International 10–20 System is being used. In this system, the distances between the electrodes are 10% or 20% of the front–back or right–left distance of the skull. Each region has a letter corresponding to the brain lobe (F frontal, T temporal, C central, P parietal, and O occipital) and a number specifying the hemisphere location. The electrode placement according to the International 10–20 System is shown in Figure 2.
Figure 2 Electrode locations in the international 10–20 System.
The brain, as a result of conscious or unconscious mechanisms, may generate different brain activity signals. The function of most of these signals is not understood. However, the physiological phenomena of some of these signals are understood and are being used in BCI applications. These signals are P300 evoked potentials, Steady State Visual Evoked Potentials (SSVEP), Slow Cortical Potentials (SCPs) and Sensory-Motor Rhythms and Motor Imagery.
3.1. P300
P300-evoked potentials are positive peaks in the EEG because of infrequent task-related stimuli. These potentials appear in the EEG signal, approximately 300 ms after the stimulus. To evoke P300, the user is given a series of random stimuli. Whenever the target (infrequent) stimulus is observed, P300 appears in the EEG.Reference Farwell and Donchin 5 P300 signals are used widely in BCI applications from controlling cursersReference Citi, Poli, Cinel and Sepulveda 30 and robotsReference Bell, Shenoy, Chalodhorn and Rao 31 to speech communication.Reference Farwell and Donchin 5 – Reference Li, Nam, Shadden and Johnson 9
3.2. SSVEP
SSVEP signals are oscillations observable at the occipital lobe, because of visual stimulation. The frequencies of the oscillations are the same as the frequencies of the stimulation.Reference Herrmann 32 When the subject focuses on a stimulus, the amplitude in the corresponding frequency bands is increased. SSVEP signals are used mostly in speech communication studies.Reference Cheng, Gao, Gao and Xu 10 – Reference Segers, Combaz, Manyakov, Chumerin, Vanderperren, Van Huffel and Van Hulle 13
3.3. SCP
SCP appears as a slow voltage shift in the EEG in the frequency range 1–2 Hz. A decrease in cortical excitability causes negative SCPs and an increase in cortical excitability causes positive SCPs. It is shown in Ref. Reference Hinterberger, Schmidt, Neumann, Mellinger, Blankertz, Curio and Birbaumer33 that users can be trained to control their SCPs by using visual or auditory feedback signals. SCP signals are used to provide communication for ALS patients.Reference Iversen, Ghanayim, Kübler, Neumann, Birbaumer and Kaiser 34
3.4. Sensory-Motor Rhythms and Motor Imagery
According to brain state, different oscillations happen in brain activity. These oscillations are categorized into four different groups based on their frequency band in EEG: delta (1–4 Hz), theta (4–8 Hz), mu (8–13 Hz), beta (13–25 Hz), and gamma (25–40 Hz). Sensory-Motor Rhythms (SMR) refers to oscillatory activities observed in somatosensory and motor areas. The activations in different parts of the body are mapped to different regions in the sensorimotor cortex of the brain. An activity in a particular part of the body causes a decrease in SMR activity in the related brain area. This decrease is called event-related desynchronization (ERD).Reference Pfurtscheller and da Silva 35 Correspondingly, event-related synchronization (ERS) is the increase in SMR activity during the relaxation period after the body movements. These ERD and ERS activities also happen when the subject is imagining the body movement and not actually moving the body. ERS/ERD oscillations can be observed in EEG in beta and mu frequency bands.
The term ‘motor imagery’ refers to moving a body part in imagination without actually moving it. As discussed above, this imagination causes ERD activities in the brain that can be observed in EEG. However, ERD/ERS patterns of all body parts cannot be discriminated in EEG. The produced patterns should be large enough to be distinguished from the background EEG. Currently, there are four types of motor imagery actions that can be detected via EEG. These actions are the movements of the left hand, right hand, feet and tongue. These four motor imagery signals can be used to control BCI after attending sufficient training sections.Reference Schlögl, Lee, Bischof and Pfurtscheller 36
MI related signals are usually recorded by using C3, C4 and Cz electrodes in EEG. Activity invoked by imagining the movement of right hand can be observed mostly in electrode location C3. Left hand movement imagery can be observed mostly in location C4. Movement imageries of left and right feet are not distinguishable since the corresponding motor rhythm origination areas take part in a sulcus (groove in the cerebral cortex). Therefore, the measured potentials on the scalp are spatially close. They both invoke activity mostly over the Cz area.Reference Pfurtscheller, Brunner, Schlogl and da Silva 37 Motor imagery is used in wide range of BCI applications to send the desired command. In Ref. Reference Fabiani, McFarland, Wolpaw and Pfurtscheller38 motor imagery is used for curser movement. It is also used for controlling a wheelchairReference Long, Li, Wang, Yu, Pan and Li 39 and a robot arm.Reference Horki, Solis-Escalante, Neuper and Müller-Putz 40 Speech communication is another popular application of motor imagery.Reference Obermaier, Muller and Pfurtscheller 14 - Reference Blankertz, Müller, Krusienski, Schalk, Wolpaw, Schlögl, Pfurtscheller, Millan, Schröder and Birbaumer 16
4. EEG Signal Processing
Once the brain activity patterns are measured, the next step is to process these signals in order to translate them to the appropriate control commands. This stage has three steps: preprocessing, feature extraction and classification.
4.1. Preprocessing
The goal of the preprocessing step is to improve the quality of the desired patterns in EEG and enhance the signal-to-noise ratio (SNR). There are three main steps in EEG signal preprocessing: referencing, temporal filtering and signal enhancement. Preprocessing also involves the removal of undesired EEG artefacts.
Referencing
The choice of referencing in EEG-based BCI applications can change the results dramatically. There are three main referencing strategies in EEG.
Common reference: In this approach an electrode far from the other electrodes is selected as a reference. This method is widely used in BCI applications.
Average reference: In this method, the average of the activity of all electrodes is subtracted from the measurements.
Current source density (CSD): It is ‘the rate of change of current flowing into and through the scalp’.Reference Al-ani and Trad 41 This quantity can be derived from EEG data, and it may be interpreted as the potential difference between an electrode and a weighted average of their surrounding electrodes.
Temporal Filtering
Informative brain signals for BCIs are found in the frequencies below 30 Hz. Therefore, all other content with higher frequencies can be removed using a low pass filter. Specific frequency bands may also be selected using band-pass filters.
Signal Enhancement
Because of the volume conduction, potentials from a large area affect the measured potential in one electrode. To estimate the contribution of each electrode, a linear transformation may be applied to the EEG signal. Methods such as Common Average Reference (CAR) and Laplacian filter preserve the original values of electrodes. Some other methods, such as Principal Component Analysis (PCA)Reference Jolliffe 42 and Independent Component Analysis (ICA),Reference Comon 43 try to find independent sources without a direct reference to original channels. Some of these methods are explained in further sections.
EEG Artefacts
The EEG signal includes undesired potentials that corrupt the brain signals. These signals are called artefacts and should be cleaned before the processing step. Artefacts may originate from outside the human body (non-physiological) or inside human body (physiological). The first type of artefacts may originate due to recording equipment. There are some activities inside the human body that may also cause artefacts. Ocular artefacts, caused by eye blinking and pupil movement, and muscular artefacts, caused by movement of body parts, are two main groups of physiological artefacts.
Artefacts can be handled by using three different strategies: avoiding, rejecting and removing. Artefacts may be avoided by asking the subjects to avoid moving and eye blinking. Artefacts can also be identified and rejected by an expert in offline applications. An artefact removal approach attempts to detect and remove the artefacts automatically during the signal processing step. Because of the online application of BCI, this approach is the preferred method for BCI studies. In the literature there are several methods for artefact removal, such as linear filtering, linear combination and regression, and Principle component analysis (PCA). Some of these methods are explained in the further sections.
4.2. Feature Extraction
The goal of the signal processing stage of a BCI system is to separate brain patterns related to a subject’s intention from the other patterns. Therefore, we deal with a pattern recognition problem where different patterns should be classified according to their features. Selecting suitable features is a challenging issue. The values recorded from one electrode may contain overlapped signals from different sources. In this section, we briefly discuss most common feature extraction methods for BCI applications.
Time and Frequency Domain Features
Time domain features can be used when event related potentials are present in the signal. The relevant information can be separated based on the EEG signal amplitude by using methods such as band-pass filtering, windowing and down-sampling. Frequency domain features are derived from oscillations in the EEG signal. These features are mostly used in BCI systems based on SSVEP and motor imagery tasks. Different types of timeReference An, Kuang, Guo, Zhao and He 44 and frequencyReference Ince, Arica and Tewfik 45 , Reference Kaiser, Bauernfeind, Kreilinger, Kaufmann, Kübler, Neuper and Müller-Putz 46 domain features have been used in BCI studies. In Ref. 47, a fourth-order Butterworth band-pass filter is used to select the frequency bands 6–30 Hz, including mu and beta bands that correspond to limb movements. Then, different frequency bins and time segments are selected as features. Event related desynchronization (ERD) and event-related synchronization (ERS) can also be used as features. ERD and ERS are defined as the percentage of power decrease (ERD) or power increase (ERS) in a defined frequency band in relation to the reference interval with second duration before the verification of an event.Reference Kaiser, Bauernfeind, Kreilinger, Kaufmann, Kübler, Neuper and Müller-Putz 46 The band powers can be used as features in the classification algorithms.
Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is an orthogonal linear transformation method that transforms the data to a new basis according to variance of the data. The axes of the new coordinate system, which are called the principal components, are ordered with decreasing variance and the components having high variance are used to represent the data. PCA is commonly used for reducing dimensionality of the data set since correlated variables are also eliminated while projecting data to the lower dimensional space.Reference Jolliffe 42 PCA is proven to reduce noise and improve the classification accuracy. This method has been used in several EEG BCI applications. PCA is used to reduce the dimension of the feature space before classificationReference Ince, Arica and Tewfik 45 , Reference Lin and Hsieh 49 – Reference Talukdar, Sakib, Pathan and Fattah 51 and also to remove the EEG artefacts and reduce noise.Reference Boye, Kristiansen, Billinger, do Nascimento and Farina 48
Independent Component Analysis (ICA)
ICA is a statistical method that assumes the recorded value of the EEG signal is a combination of independent sources coming from different cognitive activities inside the brain. No further previous information is used about the signals. The recorded EEG signal is expressed by a linear or nonlinear function of the independent sources.Reference Te-Won, Lewicki, Girolami and Sejnowski 52 The number of independent components is usually assumed to be fewer than or equal to the number of EEG channels. Like PCA, ICA uses information from channels to identify patterns in brain activity related to different mental tasks. ICA is usually used to remove artefacts from the EEG signal before the classification.Reference Gao, Yang, Lin, Wang and Zheng 53 However, it can also be used as a classification method.Reference Erfanian and Erfani 54
Common Spatial Pattern (CSP)
CSP tries to map EEG channels into a subspace where the differences between channels are maximized and the similarities are reduced. The variances of the signals filtered by CSP can be directly used as features for classification.Reference Ramoser, Muller-Gerking and Pfurtscheller 55 CSP is designed to solve two-class problems but can be extended to deal with multi-class problems too. This method has been used in many BCI applications, especially for motor imagery tasks.Reference Grosse-Wentrup and Buss 56 , Reference Ang, Chin, Zhang and Guan 57
Genetic Algorithm (GA)
GA is originally an optimization method, which may be used for selecting efficient features.Reference Holland 58 In BCI studies, GA has been used to extract the optimal set of features automatically. In this method, first a random population of chromosomes is constructed. Each chromosome has binary value for each feature. Then, in each iteration/generation a portion of chromosomes with best fitting values are selected for the next generation. These chromosomes are then modified by cross-over and mutation operations. In cross-over, two chromosomes are mixed to make new chromosomes. In mutation, random changes happen at chromosomes. Fitness is defined as classification accuracy for each chromosome. When the termination condition is reached, the best chromosome is selected as the feature set for classification. In BCI area, GA is used to select features from the power spectral density (PSD) of each EEG channel during the motor imagery taskReference Corralejo, Hornero and Alvarez 59 and to select features for P300 classification.Reference Seno, Matteucci and Mainardi 60
AdaBoost
AdaBoost is a machine-learning algorithm first introduced for adaptive boosting.Reference Freund and Schapire 61 The main idea is to combine weak classifiers to construct a new strong classifier. The features are selected by using the discriminative properties of the target and non-target classes. AdaBoost performs dimension reduction by selecting a subset of features according to the information provided in training data and eliminating the unselected features. In BCI studies, AdaBoost is used for feature selection and also for classification purposes.Reference Yıldırım and Halici 50 , Reference Boostani and Moradi 62
4.3. Classification
The classification step aims to determine the subject’s intention by using the features provided in the previous stage. These features are used to construct boundaries between classes in the training stage of the classifier and then they are used to discover the intention in the recognition stage. Some of the most popular classification methods used in BCI studies are discussed in the following.
K-Nearest Neighbour Classifier (k-NNC)
In this classifier, the test sample is classified into a class based on the distance between the features of the test sample and samples of different classes. K nearest neighbours (with less distance) are selected from trained samples and the test sample is assigned to the class with more neighbors.Reference Fix and Hodges 63 K-NNC is proven to be efficient when the dimension of the feature vector is low and is not very popular in BCI research.Reference Nicolas-Alonso and Gomez-Gil 17
Linear Discriminant Analysis (LDA)
LDA is a simple classifier with acceptable accuracy and low computational requirements.Reference Fukunaga 64 LDA is designed for classification of two classes but can be extended for multi-classes. For a two-class problem, LDA tries to define a hyperplane in the feature space that distinguishes the classes. This hyperplane is defined by a linear discrimination function. LDA has some drawbacks, such as failing in the presence of strong noise and not being stable. LDA can also be used for dimension reduction for feature extraction before classification. There are some improved algorithms based on LDA, like Fisher LDA (FLDA) and Bayesian LDA (BLDA).Reference Hoffmann, Vesin, Ebrahimi and Diserens 65 Because of the ability of online computation, this method has been applied in many BCI studies.Reference Scherer, Müller, Neuper, Graimann and Pfurtscheller 15 , Reference Nicolas-Alonso and Gomez-Gil 17 , Reference Ince, Arica and Tewfik 45 , Reference Kaiser, Bauernfeind, Kreilinger, Kaufmann, Kübler, Neuper and Müller-Putz 46 , Reference Garrett, Peterson, Anderson and Thaut 66
Support Vector Machine (SVM)
The main idea in SVM is to select the hyperplanes separating the classes in a way that the distance from the nearest training points of different classes is maximized.Reference Cortes and Vapnik 67 , Reference Burges 68 SVM was proposed originally for classification of two classes but it can be extended to multi-classes. It provides simple, robust and fast classification without needing a large training set. This method has been used in many BCI applications, especially to classify P300 evoked potentials.Reference Nicolas-Alonso and Gomez-Gil 17 , Reference Schlögl, Lee, Bischof and Pfurtscheller 36 , Reference Yıldırım and Halici 50 , Reference Garrett, Peterson, Anderson and Thaut 66 , Reference Blankertz, Curio and Muller 69 , Reference Rakotomamonjy and Guigue 70
Bayesian Statistical Classifier
Bayesian classifier assigns an observed vector x to a class y by maximizing the so-called a posteriori probability P(y|x). For a feature vector x, a posteriori probability is defined by Bayesian rule as P(y|x)=P(y)P(x|y)/P(x), where P(y) is the prior probability of class y and P(x|y) is the likelihood of x given class y.Reference Jensen 71 The likelihood function is usually assumed to have Gaussian form. The parameters of the Gaussian model are being estimated to achieve maximum likelihood or maximum a posteriori (MAP). The Expectation Maximization (EM) algorithm is usually used to predict these parameters.Reference Moon 72
Although Bayesian classifiers are not very popular in BCI applications, they have been used in some motor imagery and P300 studies.Reference Nicolas-Alonso and Gomez-Gil 17 , Reference Corralejo, Hornero and Alvarez 59
Hidden Markov Models (HMM)
An HMM is a stochastic process that has unobserved (hidden) states that can only be observed through another set of stochastic processes that produce the sequence of observed symbols.Reference Rabiner and Juang 73 Hidden Markov Models are well known for their application in temporal pattern recognition such as speech recognition, and they have been used in some BCI.Reference Obermaier, Guger, Neuper and Pfurtscheller 74 , Reference Zhong and Gosh 75
Artificial Neural Network (ANN)
ANNs are non-linear classifiers that have been used in a wide variety of pattern recognition applications. The multilayer perceptron (MLP) is a popular ANN structureReference Rumelhart, Hinton and Williams 76 but several other models are also used.Reference Gelenbe and Fourneau 106 The backpropagation algorithm is the most widely used algorithm for training MLP. In the backpropagation algorithm, a labelled training set is fed to the network and the difference between the output produced by the network and the desired output is computed. Then optimization methods such as gradient descent are used to minimize this difference by changing network weights. The trained network can then be used for classification of the new samples. There are a variety of other NN structures.Reference Gelenbe, Mao and Li 107 , Reference Gelenbe and Timotheou 108
Neural Networks are used in many BCI applications to classify two or more tasks.Reference Masic and Pfurtscheller 77 - Reference Hamedi, Salleh, Noor and Mohammad-Rezazadeh 83 They have also been used in the preprocessing step of EEG studies to improve the classification accuracy.Reference Nicolas-Alonso and Gomez-Gil 17
Deep Neural Networks
Deep neural networks is a recent approach in neural networks, allowing the network to extract much more complex features of the input by using several hidden layers. Each layer has a nonlinear activation function. In this way, deep networks can represent more functions in a compact form. Due to the complexity of the deep networks, training is a difficult task. The algorithms used to train the deep neural networks are called Deep Learning. One approach is to pre-train a deep network work by training each layer in turn. This approach is utilized in stacked autoencoder networks. In a stacked autoencoder, multiple layers of autoencoders are connected to each other consecutively.Reference Gelenbe and Yin 112 The parameters of each layer are learned separately, and the activation units of the layer are computed. Then, the computed neuron outputs are used as raw input for the next layer. Mapping from the last hidden layer to the output can be performed by classification methods such as logistic regression. To improve the results, a fine-tuning by backpropagation can be applied to tune the change of all layers at the same time. Convolutional Neural Network,Reference LeCun, Bottou, Bengio and Haffner 84 Stacked auto encoders,Reference Bengio, Lamblin, Popovici and Larochelle 85 and Deep Boltzmann MachineReference Hinton 86 are the most widely used deep networks for various applications.
Deep neural networks have been used in some recent BCI studies. Convolutional neural networks are used for classification of P300 in Ref. 87. Stacked Auto Encoder and Deep Boltzmann Machine have been used for classification of EEG motor imagery signals.Reference An, Kuang, Guo, Zhao and He 44 , Reference Junhua and Cichocki 88 Convolutional neural networks and Stacked Auto Encoder are used together in Ref. 89 to classify motor imagery signals.
4.4. Tools, Libraries and Datasets
EEGLABReference Delorme and Makeig 90 is a Matlab toolbox that may be used to analyse the EEG data and different brain patterns. BCILABReference Kothe and Makeig 91 is another Matlab toolbox for designing and testing brain computer interface experiments. BioSigReference Vidaurre, Sander and Schlögl 92 is an open source software library that provides signal processing algorithms for biomedical applications. To design an experiment with visual or auditory feedbacks and also in connection with the EEG device, PsychtoolboxReference Brainard 93 may be used in Matlab.
There are plenty of online datasets including BCI signals. The most popular datasets are BCI competition datasets. BCI competition 2003Reference Blankertz 94 includes several datasets with SCP, P300 and motor imagery signals. BCI Competition IIIReference Blankertz 95 includes different P300 and motor imagery datasets with different paradigms. BCI competition IVReference Blankertz 96 also has different motor imagery datasets. There are also other online available datasets, such as OpenVIBE dataset, that provide BCI signals.Reference Renard, Lotte, Gibert, Congedo, Maby, Delannoy and Lécuyer 97
5. BCI Applications for Silent Speech
Silent Speech applications, which are BCI systems developed for speech communication, do not use voice, but only brain signals. These systems can be categorized in three main groups according to the brain response they use: event-related potentials (ERP), steady state evoked potential (SSVEP), and motor imagery (MI).
5.1. BCIs Based on P300
The best-known representative of this group is the P300 speller. The first speller based on P300 was proposed in Ref. Reference Farwell and Donchin5 and different modifications of it have been studied afterwards.Reference Townsend, LaPallo, Boulay, Krusienski, Frye, Hauser, Schwartz, Vaughan, Wolpaw and Sellers 6 – Reference Li, Nam, Shadden and Johnson 9 , Reference Blankertz 94 , Reference Blankertz, Dornhege, Krauledat, Schroder, Williamson, Murray-Smith and Müller 98
In such applications, a matrix of characters is displayed to the subject. The rows and columns of the matrix are intensified sequentially and the subject attends to the target character. A sample character matrix used in P300 spelling paradigm is shown in Figure 3.
Figure 3 P300 Spelling Paradigm character matrix that is displayed to user.Reference Blankertz 94
The attention of the subject to an intensified character evokes an enhanced P300 component. A classifier can be trained to detect the target character by using the combination of intensified rows and columns. For signal processing, a time window is usually applied to select the EEG samples related to P300 evoked potentials. Then, different samples are selected from each channel and used as feature vectors for training and testing. In the literature, different classification methods such as SVM, neural networks and Bayesian linear discriminant analysis are used for classification. Different spelling paradigms based on P300 are used in speech communication studies.Reference Townsend, LaPallo, Boulay, Krusienski, Frye, Hauser, Schwartz, Vaughan, Wolpaw and Sellers 6 – Reference Li, Nam, Shadden and Johnson 9
Even though there has been a lot of research in the P300 speller area, the most recent systems are still not applicable for clinical use. The proposed systems lack robustness across the users and the users cannot control the system easily.Reference Cecotti 99
5.2. BCIs Based on SSVEP
In this paradigm, flickering lights at different frequencies are used as the stimuli. For each flickering frequency band, Steady-State Visual Evoked Potential (SSVEP) oscillations happen in the visual cortex of the brain with the same frequency band and higher harmonics. By using this fact, it is possible to detect if the subject is looking at the display part with frequency f or 2f, 3f, etc. Several graphical interfaces have been proposed for this purpose. Figure 4 shows a simple form of SSVEP speller. In this example, symbol w is selected in three stages. Each stage is composed of four boxes in the display with different flickering frequencies.Reference Mora-Cortes, Manyakov, Chumerin and Van Hulle 100 Different forms of SSVEP-based spellers are introduced for speech communication.Reference Cheng, Gao, Gao and Xu 10 – Reference Segers, Combaz, Manyakov, Chumerin, Vanderperren, Van Huffel and Van Hulle 13
Figure 4 Character sets based on SSVEP.Reference Mora-Cortes, Manyakov, Chumerin and Van Hulle 100
Since the SSVEP is embedded in other ongoing brain activity and also noise, the recording interval should be long. Another limitation is that only flickering frequencies within a particular frequency range evoke a reasonable SSVEP response.Reference Jia, Gao, Hong and Gao 101 Further studies are needed to provide a SSVEP-based speller for commercial uses.
5.3. BCIs Based on MI
As described before, moving a body part or imagining it produces neural activity in the motor cortex of the brain that can be detected by EEG. Only a limited number of movements can be detected by using this method. So, a strategy should be used to combine these acts and produce characters.
Different spelling interfaces have been proposed in the literature for MI-based communication.
A speller is presented in Ref. Reference Blankertz, Dornhege, Krauledat, Schroder, Williamson, Murray-Smith and Müller98 by using only two commands: left hand and both feet. In this study, 30 different characters are divided into six hexagons around a circle (Figure 5). By left hand command, the arrow rotates in a clockwise manner showing the selected box, and by feet command the box is selected. A character can be selected in two stages.
Figure 5 MI based speller with six hexagons chosen by two MI tasks.Reference Blankertz, Dornhege, Krauledat, Schroder, Williamson, Murray-Smith and Müller 98
Another speller system based on MI systemReference D’albis, Blatt, Tedesco, Sbattella and Matteucci 102 is shown in Figure 6. This system is composed of four boxes. Twenty-six English characters and a space symbol are grouped in three boxes. The fourth box is used for undo command. The subject selects one of the boxes by imagining the movement of the corresponding body part. They have used left hand, right hand, both hands and both feet movement for command. The desired symbol can be selected in three stages.
Figure 6 MI-based speller with four boxes each chosen by one of four MI tasks.Reference D’albis, Blatt, Tedesco, Sbattella and Matteucci 102
5.4. Other Studies Use Similar Interfaces for Selecting Characters
Motor Imagery commands can be used in another manner to produce desired characters. Each character can be coded into a combination of motor imagery acts. In this way there is no need for a graphical interface. To our knowledge, there are only two studies considering this approach in the literature. Both of these studiesReference Palaniappan, Paramesran, Nishida and Saiwaki 103 , Reference Nicolaou and Georgiou 104 use motor imagery EEG signals from an EEG dataset recordedReference Keirn and Aunon 109 with different MI signals recorded separately. In other work, these signals are combined to synthesize new words. An actual experiment for spelling and performance analysis is not performed in these studies.
5.5. Measuring Speller Performance
It is difficult to measure the performance of different BCI speller systems and compare them in a meaningful way. BCI spellers use different spelling paradigms that make them very different from each other. One traditional way to measure performance is to compute typing accuracy. However, this doesn’t provide any information about spelling speed, which is also an important issue, and an information transfer rate (ITR) metric has been proposed to measure the performance of BCI speller applications.Reference Wolpaw, Birbaumer, McFarland, Pfurtscheller and Vaughan 110 ITR is the amount of information communicated per unit time. It takes into account the accuracy, the number of possible selectable commands that the interface supports, and the time required for communicating one command. However, this metric has some drawbacks, such as considering backspace command as a correct transformation of information. In addition, in spellers that use word compilation strategies, this metric can’t provide a fair performance measurement. Another strategy is to use character per minute measure beside bit per minute in ITR.
It has also been proposed to use output character per minute (OCM) measure for spelling performance measurement.Reference Ryan, Frye, Townsend, Berry, Mesa, Gates and Sellers 111 OCM is defined as the ratio of the total number of characters in the final text to the total time spent spelling it. This metric can be used to compare different BCI spellers with different paradigms and even different language models.
6. Conclusions
This article has discussed different parts of a Brain Computer Interface (BCI) system from signal acquisition to signal processing. How to measure brain activity in general, and especially how different modalities for EEG signal acquisition can be used, have been explained. Various EEG signal processing techniques used in BCI application for preprocessing, feature extraction and classification have been presented and information has been provided about the toolboxes, software libraries and datasets. Various speech communication systems based on neural activity are explained in detail. Current speech communication studies are discussed and different spelling paradigms and methods are explained.
Speech communication systems can provide a huge benefit for people with severe disabilities. Current spellers mostly use P300, SSVEP and motor imagery paradigms to provide communication. Signal processing and machine learning algorithms for BCI signals have been improved extensively in recent years. The classification performances of these methods are near acceptable. However, designing a spelling paradigm and graphical interface suitable for the daily life use of people with disabilities is still a challenge. Current studies provide slow communication rates that make them less preferable for common utilization. Improvements in signal processing algorithms as well as designing easy to use and fast spellers are needed to make a BCI-based speller in the future. New portable signal acquisition methods can also help a lot to make a usable spelling device.
Acknowledgements
This study is partially supported under projects BAP-03-01-2015-001, BAP-07-02-2015-005, BAP-03-2016-003 and BAP-03-01-2016-002, Middle East Technical University, Ankara.
Yousef Rezaei Tabar received a BS degree in Electrical Engineering from Urmia University, Urmia, Iran, in 2005. He is currently working toward a PhD degree in Biomedical Engineering at Middle East Technical University, Ankara, Turkey. His current research interests include biomedical signal processing, deep learning and brain–computer interfaces.
Ugur Halici is a faculty member of the Department of Electrical and Electronics Engineering, Middle East Technical University, Ankara, Turkey and chairperson of the METU-Hacettepe University joint Neuroscience and Neurotechnology PhD programme that she helped establish in 2014. She has published two internationally co-edited books Intelligent Biometric Techniques in Fingerprint and Face Recognition with OCR Press 1999 and Innovations in ART Neural Networks with Springer Verlag in 2000, two books in Turkish, and over 100 journal/conference papers. Her research interests cover Computer Vision, Machine Learning, 3D modelling, Pattern Recognition, Intelligent Systems and Computational Neuroscience.
