CHAPTER 2 - TECHNICAL BACKGROUND: STATE OF THE ART OF BRAIN-COMPUTER INTERFACES
Artificial intelligence is an innovative technology that begins to abandon the laboratory and science fiction books to become a reality that can transform our society. People are surrounded by intelligent machines running on incredibly powerful learning software platforms. And this is just the beginning. For a long time, the fantasy of being able to communicate with machines just by using our thinking has captivated our imagination.
Humans are so accustomed to our bodies that we are not aware of the complex biological, chemical and electrical mechanisms that occur between our brain and our body to be able to communicate easily and interact with the world. New technological advances are making it
As an introduction to this new topic, an explanation of the relationship of neuroscience and neurotechnology with BCIs is presented below, followed by a brief presentation of historical events that have marked the beginning of this technology and to provide a quick overview of the development of BCIs over the past 50 years. This will lead to further analysis of the state of the art of the BCI as an emerging technology of man-computer interaction. Firstly, a definition of the BCI is introduced, followed by a classification scheme of it, along with different types of brain signal recording methods for the functioning of this new technology. Subsequently, several types of applications of the BCI are presented with emphasis on future researches. Finally, the most used and applied neurotechnology technique is explained, the electroencephalography, which is the object of discussion due to the different applications it has.
2.1.1 Related Neuroscience and Neurotechnology
The mission of neuroscience is the study of how the human brain works. The field of neuroscience has grown in recent years as new challenges have arisen in the detection and treatment of brain diseases, the control, and modification of behavior, and the improvement of individual performance. These discoveries and new methods that neuroscientists, aided by new technologies and therapies, are using in their research, help for a better understanding of physiological and pathological pathways, as well as determining which conditions may lead to disease. Its purpose is to know how the brain works and serve as a basis for a greater and more complete understanding of human nature and is characterized by the emergence of interdisciplinary fields. These fields include neurotechnology, neuroeconomics, neurophilosophy, neuroethics, neurolaw, neuromarketing, neuro-education among others.
Neurotechnology is at the intersection of human cognition and computer science involving one of the most fragile and sensitive parts of the human body. The deep integration of one of the most important technical components called BCI is giving way to a new dimension of direct links between the brain and a machine that makes it possible to carry out a task while avoiding nerves and muscles. BCIs allow people to use their thoughts to control computers, machines, prosthetics or other automated systems without the need for using the musculoskeletal system.
2.3 Brain-Computer Interfaces Origins
In the 1920s, a German scientist named Hans Berger was the first to demonstrate that the human brain was producing electric currents. Such currents reflected brain activity and could be measured on the scalp using electrodes. It was in 1924 that Berger built the first electroencephalogram (EEG), an instrument that makes it possible to graphically record brain electrical activity, measured on the surface of the skull. From this moment on, the EEG proved to be a key tool in neuroscience, especially to study cognitive functions and their neuronal correlations to understand or diagnose neuropathologies. Likewise, with the development of the EEG, the idea that brain activity could be used as a communication channel or information carrier also quickly emerged.
Now then, in 1963, neurophysiologist William Grey Walter conducted what is now considered the first experimental demonstration of a BCI in humans. He used electrodes to record the motor cortex of patients undergoing brain surgery. Patients were asked to press a button with their finger to lower a slide projector. Walter recorded what is called a preparation potential (RP) of the patient's brain signals that occurred shortly before pressing the button. In the process, patients were able to lower the projector just by thinking, and the preparation potential was used as a trigger connected directly to the projector's mechanics.
Subsequently Joe Kamiya, in 1968, remarkably demonstrated that the characteristics of EEG activity can be purposefully controlled by a human after some training. This was the beginning of neurofeedback, a field focused on training users to self-regulate their brain activity through real-time feedback on this activity. Later, the origins of BCI begin, when Jose Delgado (1969) and Eberhard Fetz (1969) began to work on it. Delgado developed an implantable chip (which he called "stimo-ceiver") that could be used both to stimulate the brain by radio and to send electrical signals of brain activity by telemetry, allowing the person to move freely. In a demonstration, Delgado used the "stimo-ceiver" to stop a charging bull on its way by pressing a remote-control button that provided electrical stimulation to the caudate nucleus in the region of the basal ganglia of the bull's brain. Shortly thereafter, Fetz demonstrated that monkeys can control the activity of a single brain cell by controlling a meter needle and earning food rewards.
A while later, in 1973, Belgian researcher Jacques J. Vidal explored the use of brain signals recorded on the scalp in humans to implement a simple non-invasive BCI based on "visually evoked potentials". It was there that he introduced the term "Brain-Computer Interface". Specifically, Vidal defines the BCI as the use of brain signals in a man-computer dialogue and as a meaning of control of external processes such as computers or prosthetic devices. It is worth mentioning that only concepts were proposed at that time, as implementations were still ongoing. However, several of the ideas proposed at that time are still being explored and are still being applied today.
While the field remained inactive in the 1970s and early 1980s, in the late 1980s and early 1990s a group of researchers from the USA and Europe pioneered the field of the BCI, proposing the first implementations of it in real-time and in operation, which defined several of the main paradigms used today. At the end of the last century and the beginning of the new century, BCI research became a field of research on its own, with many new research groups joining forces to make the field evolve rapidly. From then until now, the scope of this new technology has expanded, and many newer applications of it are now being explored, including among others, clinical, communication, gaming, and virtual reality use.
Thus, the most recent increase in interest in the BCI can be attributed to several factors such as the fact that computers are now faster and cheaper, there are advances in the knowledge of how the brain processes sensory information and produces motor production, in turn, there is greater availability of devices to record brain signals and finally there are more powerful algorithms for signal processing and machine learning.
2.4 State of the Art
As this brief historical chapter has shown, the BCI is now a slightly more mature and considered multidisciplinary research field due to the multiple disciplines, tools, and concepts that are required to master for the proper functioning and application of this new technology.
Over the last few decades, new technology has emerged whereby the human brain can communicate directly with the environment through a certain technology called, brain-computer interface (BCI), brain-machine interface (BMI), direct neuronal interface or mind-machine interface (MMI). The term brain-machine interface (BMI) was used as early as 1985 to describe implanted devices that stimulate the brain, however, it was not specifically applied to devices that provide new outputs until recently. In practice, the term BMI has been applied primarily to systems that use the cortical neuronal activity recorded by implanted microelectrodes. Today, BCI and BMI are synonymous terms, and the choice between them is largely a matter of personal preference. One reason for preferring BCI to BMI is that the word "machine" in BMI suggests an inflexible conversion of brain signals into output commands and therefore does not reflect the reality that a computer and the brain are partners in the interactive adaptive control necessary for effective BCI (or BMI) operation. BCI is defined as a type of extension of our brain that allows the brain to communicate with an electronic or manual device. The brain computing interface is a system that analyzes brain activity and translates certain characteristics or impulses, which associate a person's intentions in control instructions to a device. Devices measure the activity of neurons to obtain a signal and its subsequent process. However, the current definition of BCIs has expanded; therefore, a BCI is currently defined as “a system that measures central nervous system (CNS) activity and converts it into artificial output that replaces, restores, enhances or supplements natural CNS output and thereby changes the ongoing interactions between the CNS and its external or internal environment”.
BCI systems consist of several sequential steps or stages, which can be divided into categories, each of which are discussed in more detail below: 1) recording and processing the brain activity, 2) deciphering brain activity with the purpose of giving it an interpretation, 3) translating the interpretation into a command to be sent to the output device, and 4) providing feedback to the BCI user to calibrate the brain-computer relationship for better control. It is important to mention that the following processing stages are typically involved to translate brain activity into control commands for the devices and/or stimulating the brain to provide sensory feedback or restore neurological function.
There are two classes of brain imaging technologies or devices in which invasive or non-invasive recording techniques can be used. On the one hand, invasive technologies perform the procedure of obtaining medical information from a sensor that is implanted through a surgical operation. The sensor measures the electrical activity of neurons found in small regions of the brain. Invasive BCI involves the implantation of recording electrodes, either placed on the surface of the cortex or penetrating deeper into the cortical cell layers. Invasive BCIs allow a more accurate understanding of brain activity than noninvasive BCIs since electrodes pick up activity directly from inside the brain, where the activity is generated. Therefore, this type of BCI is usually limited to the medical field.
On the other hand, there is the non-invasive recording technique, in which brain activity is measured using external sensors, i.e. no surgical operation is performed. These are the most used technologies because they do not produce medical risks since they measure the activity on the scalp. Generally, non-invasive devices are considered the safest and least expensive. However, these devices can only capture "weaker" human brain signals due to obstruction of the skull. In spite of everything, non-invasive BCIs are more practical as surgery is not necessary and has led to promising applications for both clinical and commercial use.
- Signal processing
Raw signals are preprocessed after acquisition and techniques are used for reducing artifacts and extracting characteristics for classification. For many of the techniques, EEG (Electroencephalography) is used as the non-invasive recording mode to illustrate the concepts involved, although the techniques can be applied to signals from other sources, such as Magnetoencephalography (MEG) and Electrocorticography (EcoG).
The signals recorded in the brain, whether invasive or non-invasive, often contain various types of noise or mixtures of signals from multiple neurons, so these techniques attempt to extract useful signals from the brain's raw signals. For non-invasive approaches, there is a wide range of feature extraction techniques based on the frequency domain, time domain, or wavelet analysis, which can be used in conjunction with spatial filtering techniques to reduce dimensionality (PCA), separate sources of mixtures (ICA), or improve discrimination between output classes (CSP). Some of these techniques can also be used to reduce artifacts originating outside the brain (e.g. line noise or muscle artifacts). While the goal of BCI research is to exploit various types of electrophysiological brain activity for the effective manipulation of devices, each of these types of activity reveals different sensitive information about the state of the brain that is important to consider in brain data privacy issues.
- Pattern recognition and machine learning
The field of automatic learning plays an essential role in the development of brain-computer interfaces by facilitating techniques that can learn to record neuronal activity with control commands. This stage consists of generating a control signal based on input patterns, typically using machine learning techniques. Once brain activity is recorded and processed, the next step is to automatically interpret the activity. To accomplish this, a variety of machine learning algorithms can be applied to brain signal data to properly classify them into pre-defined groups of activities. Almost all BCI systems rely on machine learning algorithms that include logistic regression, artificial neural networks, supportive vector machines, among others. Machine learning works by taking large and complex data, such as brain signal data from many recording electrodes, and applying a specific function/algorithm to the data. In this way, algorithms can independently construct a model for the data that can be used to organize them into patterns and make accurate predictions about new data, but not visible. It is important to emphasize that because algorithms learn these parameters on their own, data analysis is not coded by humans. Besides, many of the higher-performing algorithms create non-linear models of the data, making them very difficult for a human to understand the reasoning behind the algorithm's decisions. This raises some questions about data privacy laws that will be addressed in the following chapters.
- Sensory feedback
Users need feedback from the BCI system to improve their performance. Feedback is the system's response after a user action. It can be visual, auditory, or tactile and helps the user to control his or her brain activities and adapt them accordingly to improve the overall performance of the BCI. Also, it can speed up the training process and improve overall performance by motivating and sustaining the person's attention.
- Output devices and applications
Once the signals are identified, BCIs can associate a specific command to this identified mental state and send it to a specific application. Available BCI applications can be divided into two main categories. The first and most important category is clinical use. The main objective of BCIs is to serve people with disabilities by providing a new channel of communication and control. However, there is a second category that is part of the non-medical domain. Thus, even if BCIs are designed primarily for people with disabilities, they may also be of interest to healthy people. Some of the possible applications with current BCIs are described below:
- Personal communication applications: Its purpose is to allow the possibility of communicating with a person, who has a speech or muscular disability, with the outside through their brain waves. Communication devices based on BCI allow the patient to send confidential messages that provide great independence and self-determination.
- Vehicle operation applications: These are the applications that allow a vehicle to be driven through thought.
- Requests for medical assistance: They are applications that support the treatment of ailments such as autism, attention deficit, Parkinson's, epilepsy and migraines.
- Home automation or environmental control applications: They are applications to help people with disabilities to perform daily tasks: lighting control, air conditioning, changing channels, among others. Its use increases patient independence and reduces the workload of caregivers.
- Robotic applications: They develop applications to support people with motor disabilities for the control of a robotic arm or the manipulation of a wheelchair.
- Virtual reality and video game applications. In addition to medical and rehabilitation applications, there are a growing number of BCI multimedia applications, such as simple 2D video games and the most advanced 3D video games. There are BCI systems used to navigate virtual worlds and BMI systems used to select and/or manipulate virtual objects.
Hence, the rise of mobile technologies, communications, and cloud computing has led to a very rapid evolution of BCI technologies, allowing the exploration of new applications that were unthinkable a few years ago. For instance, Nissan has recently introduced "Brain-to-Vehicle" technology, which connects the driver's brain to the vehicle to create a more comfortable and safe driving experience. The project is based on a brain-computer interface that has led to breakthroughs in brain detection devices, EEG signal processing, and shared vehicle control. Similarly, Facebook has plans to interact with computers and someday will allow users to write quickly with the brain. Now, neuroscientists at the University of California, San Francisco (backed by Facebook's Reality Labs) have demonstrated a system that can translate speech to text in real-time using only brain activity with their mental keyboard.
2.5 Most Commonly Used Method in BCI: Electroencephalography (EEG)
The ideal brain measurement method should have a high temporal and spatial resolution and should be inexpensive, portable and easy to use. However, this method does not yet exist although there are several neuroimaging techniques for acquiring brain signals. Among the techniques listed are magnetoencephalography (MEG) which measures the magnetic fields produced by the electrical current occurring in the brain, functional magnetic resonance imaging (fMRI) which recognizes the changes in blood flow established by neural activity in the brain, and near-infrared functional spectroscopy (fNIRS) which, through the use of light in the near-infrared range, maps the dynamics of blood in the brain in order to detect neuronal activity. Finally, there is Electroencephalography (EEG) that records brain activity as electrical signals, using electrodes placed around the scalp. Among the techniques listed, the EEG is by far the most used in the BCI, due to its high temporal resolution and its usability. It is more suitable for clinical and commercial use than other neuroimaging techniques because it is a relatively low-cost method when compared to other methods that require expensive equipment and trained professionals to operate.
Due to these specific characteristics of the EEG method, this thesis only analyzes this type of BCI. Besides, because EEG is more suitable for commercial use than other neuroimaging techniques and the potential for releasing an extraordinary volume and variety of brain information is much more reasonable and attainable, the value that directly extrapolated data from the human brain is acquiring, also outside the clinical sphere, is now evident.
2.5.1 How does the EEG method work?
As mentioned at the beginning of this chapter, there are two recording methods for BCIs. The safest and most prevalent of these methods are non-invasive. This method requires equipment that touches the scalp, such as an electroencephalogram (EEG). A BCI based on the EEG technique has a simple structure composed of certain steps that were explained in the section above. The first step involves the measurement of brain activity to acquire the person's raw signals, that is, the acquisition of EEG is the first step in the functioning of the BCI.
The EEG measures electrical activity in the brain using surface electrodes or sensors placed on the scalp. The neurophysiological origin of the EEG signals is the pyramidal neurons of the cortex. The EEG sensors are placed on the person's head, then the electrodes non-invasively detect the person's brain waves. EEG sensors can record up to several thousand snapshots of electrical activity generated in the brain in a single second. The recorded brain waves are sent to amplifiers, then to a computer or the cloud to process the data. The amplified signals, which resemble wavy lines, can be recorded on a computer, mobile device, or in a cloud database. Cloud computing software is considered a critical innovation in EEG data processing, as it allows real-time analysis of scale records. In the early days of EEG measurement, waves were simply recorded on graph paper. Today, EEG systems in academic and commercial research often display the data as a time series or as a continuous flow of voltages. A crucial element in these technologies is electrical impulses. The essence of human brain activity consists of electrical impulses created by billions of neurons that transmit information through electrical and chemical signals. An electrical impulse is sent down the axon and into the synapse each time neurons are fired during excitation. Because electrical signals are not capable of crossing neuronal boundaries, a chemical reaction is created between neurons. This chemical reaction is triggered by electrical impulses and causes an action potential. An action potential is the process of depolarization of the neuron, followed by repolarization. Chemical information may begin to flow through the synaptic cleft when a neuron is at its level of polarization at rest. The flow causes depolarization, and repolarization is necessary before more chemical information can flow back through the synapse.
Another element that must be considered when interpreting an electroencephalogram is the neuronal oscillations observed in the EEG signals which are called "brain waves". These brain waves are identified by frequency (in hertz or cycles per second) and amplitude in the microvolt range (μV or 1/1,000,000 of a volt). Each brain wave has its own set of characteristics representing a specific level of brain activity and mental states. There are six different frequency bandswith different biological meanings:
- Delta (frequency < 4 Hz):
Delta activity is predominantly found in infants. Delta waves are associated with deep sleep stages in older people and have been documented in patients with absence seizures, involving short, sudden lapses in attention.
- Theta (frequency 4-7 Hz): The theta rhythm detected in EEG measurement is usually found in young adults, particularly in the temporal regions and during hyperventilation.
- Alpha (frequency 8-15 Hz): Alpha waves are often linked with a relaxed, calm and lucid state of mind. Alpha waves can be found in the occipital and posterior regions of the brain. They can be produced by closing the eyes and relaxing, and are rarely present during intense cognitive processes such as thinking, mental calculus, and problem-solving.
- Beta (frequency 16-31 Hz): Beta waves are more closely associated with being conscious or in an awake, attentive, and alert state. Low amplitude beta waves are associated with active concentration or a busy or anxious state of mind. Beta waves are also associated with motor decisions.
- Gamma (frequency > 32 Hz): Gamma waves are related to regional learning, processing and ideation of memory and language. They are shown away from brain signals for anesthesia caused by deep sleep.
Frequency patterns change depending on the specific state of the brain. Thus, knowing what region of the brain is involved and the intensity of the related activity allows to detect the state of the brain in real-time by differentiating, for example, delta brain waves reflect slow, strong and functional mental states that prevail during late sleep.
All applications of BCI technology need to constantly extract EEG data to function properly. It is important to mention that the BCI extraction process does not discriminate between necessary data for command operation and other less critical or important data, because it is the raw data that are sent and stored in the processing operation. Also, since the BCI technology is fundamentally based on a neuroimaging technique, it must consider that the amount of information shared with the device is massive by default and unknown to users, as only small parts of brain activity are under voluntary control. For the moment, however, it is essential to understand what can be inferred from the EEG signal, because the BCI picks up all brain
Besides, the technical explanation of EEG neurotechnologies shows that the information that is extracted consists mainly of two elements. First, there are the raw values related to the electrical activity of the brain, that is, the graphs with the electrical waves or the brain images obtained by the EEG. On the other hand, there is the interpretation that experts and scientists give of the raw values. These interpretations are capable of translating mere physiological measurements into meaningful information related to the health status of a person.
As science advances in its discoveries, the role of the brain in its relations with the mental faculties and with that time-space called consciousness becomes more noticeable, which remains an enigma to be deciphered. Differences can be highlighted between these three concepts, brain, mind, and consciousness, although they are usually used as synonyms. The brain is the physical support through which the functions of the mind are objectified, and different degrees and depths of consciousness are expressed, depending on the case. The brain is an organ like any other of the human body that we can see, touch and examine all its parts and is controlled by the intangible mind.
Thus, in different disciplines such as psychology, philosophy, neuroscience, cognitive anthropology, among others over the years have divided the mind into two parts: conscious and subconscious.
Having understood this, it is important to know that measuring electrical impulses does not mean reading the human mind. These raw values captured from the human brain need to be combined and interpreted through innovative technologies to decipher human thoughts or feelings. Thus, the more the raw values of neural activity are interpreted and processed through special techniques, the more the line of brain reading is crossed to enter mind reading.
It is extremely important to be aware that raw brain data are not just physiological values but can be translated by interpretation or new computational methods to reveal more personal and intimate information about the person, and so eventually, the EEG signal may reveal more sensitive information. For instance, neuroscientists at the University of Toronto Scarborough developed a technique by which images of what people perceive can be reconstructed based on their brain activity collected by the EEG. For the study, test people connected to the EEG equipment were shown images of faces. Their brain activity was recorded and then it was used to digitally recreate the image in the person's mind using a technique based on machine learning algorithms. This study validates that the EEG has the potential for this type of image reconstruction, something that many researchers doubted was possible because of its apparent limitations. The use of EEG data for image reconstruction has great theoretical and practical potential from a neurotechnological point of view, especially because, as mentioned above, it is relatively inexpensive and portable. As for the next steps, work is currently underway to test how image reconstruction based on EEG data could be performed using memory and applied to a wider range of objects beyond the faces. Thus, the fact that you can reconstruct what someone visually experiences based on their brain activity opens up many possibilities for mint reading. It reveals more sensitive information such as the subjective content of the mind and provides a way to access, explore and share the content of people's perception, memory, and imagination. Likewise, in a laboratory experiment, Martinovic et al at the USENIX 2012 Security Symposium, illustrated how the brain's response to a particular stimulus (Potential P300) can be used to infer this sensitive information. Paradoxically, by employing this technique, it is possible to efficiently collect the most sensitive information, since the more a stimulus relates to a person, the easier it is to capture the sensitive information because the response captured by the BCI is sharper.
The combination of information extrapolated by modern neurotechnologies and other methods of interpretation and data analysis involving the application of machine learning and algorithms are expanding the horizons of neuroscience, offering scientists and researchers new and unimaginable opportunities. The brain-computer interface device has come a long way since its first development in 1920. Its value
This chapter aimed to describe the different types of information extrapolated from the brain by the most popular BCI technology at the clinical and non-clinical level such as the EEG. Moreover, it has been seen that the values captured from the human brain and translated into brain wave graphs or brain images of the active areas of the brain, have no meaning by themselves, and both require high expertise to be read and interpreted. Also, BCI does not limit the collection of brain data to the one necessary for the required functional performance, but it collects and stores the whole raw EEG data. Furthermore, considering the basic distinction between brain and mind, modern techniques have been designed to attempt to read the human mind. Applications, software and, algorithms, combined with raw brain data extrapolated by EEG-based technologies are now closer than ever to be able to read personal intentions or thoughts in various fields of application, both inside and outside the medical field. Therefore, in dealing with the raw values recorded by the above-mentioned neurotechnologies, it is necessary to carefully consider the implications related to their potential interpretation and their combination with new mind-reading techniques and therefore must be legally protected from indiscriminate dissemination.
Undoubtedly, the biggest challenge is moving from the current raw form of mind-reading of the BCI to systems that can accurately decode the content of specific thoughts. Experience says that what a decade ago was thought to be just science fiction often becomes today's scientific reality. Therefore, rapid advances in neurotechnology would make the regulation of brain data outdated, with the consequence of increasing risks to people