Employees of the GC «Neurobotics» in conjunction with the MIPT Laboratory of Neurobototechnics have learned to recreate images that a person sees at the moment from the electrical activity of the brain.

This allows you to create a new type of device for post-stroke rehabilitation, controlled by brain signals. Preprint work is available at bioRxiv.

The experiment consisted of two parts. In the first part, researchers arbitrarily selected five different categories of YouTube videos: «abstractions», «waterfalls», «people's faces», «speed» (first-person video shooting of snowmobile races, jet skis, rallies) and «moving mechanisms», which showed subjects, while recording the EEG. The videos lasted 10 seconds, in total, the entire recording session for each subject was 20 minutes.

In this part of the experiment, scientists were able to prove that the frequency characteristics of wave activity (spectra) of EEG for different categories of videos are significantly different. This allowed us to analyze the reaction of the brain to videos in real time.

For the second part of the experiment, three categories from the video mentioned above were arbitrarily selected. Specialists developed two neural networks. One of them generated arbitrary images of the same categories from «noise». The second created a similar «noise» from the EEG. Then the authors of the work trained these neural networks to work together. Using the recorded EEG signal, frames were created similar to those seen by people at the time of recording.

For testing, subjects were shown completely new videos of the same categories. At the same time, the EEG was being shot, which was sent to the neural networks in real time. Neural networks did a good job of this. They created realistic frames by which in 90% of cases it was possible to determine the category of video.