A modification of a popular text-to-image generative artificial intelligence allows it to convert brain signals directly into images. However, the system requires extensive training with bulky and expensive imaging equipment, so reading minds in everyday life is far from reality.
Several research groups have previously generated images from brain signals using AI models that consume a lot of energy and require fine-tuning of millions to billions of parameters. Now, Shinji Nishimoto and Yu Takagi from Osaka University in Japan have developed a much simpler method using Stable Diffusion, a text-to-image generator launched by Stability AI. Their new method involves thousands, instead of millions, of parameters.
When used normally, Stable Diffusion converts text into an image by starting with random visual noise and adjusting it to produce images that resemble those in its training data that have similar text prompts.
Nishimoto and Takagi built two additional models for the AI to work with brain signals. The pair used data from four individuals who participated in a previous study where functional magnetic resonance imaging (fMRI) was used to scan their brains while they viewed 10,000 different images of landscapes, objects, and people.
Using around 90% of the brain image data, the pair trained a model to establish links between fMRI data from a brain region that processes visual signals, called the early visual cortex, and the images people were seeing.
They used the same dataset to train a second model that formed links between textual descriptions of the images – made by five annotators in the previous study – and fMRI data from a brain region that processes the meaning of images, called the ventral visual cortex. A study that analyzed the same data using a much more tedious approach.
“I couldn’t believe what I was seeing, I went to the bathroom and took a look in the mirror, then came back to my desk to take another look,” said Takagi. However, the study only tested the method with four people, and mind-reading AIs work better with some people than with others, says Nishimoto. “This is not at all practical for daily use,” says Sikun Lin from the University of California, Santa Barbara.
In the future, more practical versions of the method could allow people to make art or alter images with their imagination, or add new elements to the game, says Lin.
With information from Carissa Wong. Story originally written in Spanish by Carla Rodríguez in Ecoosfera.