Google’s new technology can read all your body language without a camera

This article was originally published on the WIRED blog, translated and shared by InfoQ Chinese.

What if your computer decides not to ring the notification bell because it notices that you’re not sitting at your desk? What if your TV saw you leave the couch to open the door, and automatically paused Netflix, then resumed it while you sat down? What if our computers could pick up more social cues from our movements and learn to be more considerate partners?

It sounds futuristic and, of course, maybe a little intrusive – a computer monitoring your every move? But if you know that these technologies don’t have to rely on cameras to observe your location and behavior, it’s less scary. Instead, they used radar. Google’s Advanced Technologies and Products division — better known as ATAP, which is behind quirky projects like touch-sensitive denim jackets — has spent the past year exploring how computers can use radar to learn about us needs or intentions and respond appropriately.

This isn’t the first time Google has used radar to provide spatial awareness to its small devices. In 2015, Google released Soli , a sensor that uses radar electromagnetic waves to precisely capture gestures and movements. It first appeared in the Google Pixel 4 and can detect simple gestures, allowing users to snooze alarms or snooze music without actually touching their smartphone. More recently, the second-generation Nest Hub smart display has also embedded radar sensors that can detect the movements and breathing patterns of people sleeping next to it. This way, the device is able to track that person’s sleep without requiring them to wear a smartwatch.

The same Soli sensor was used in the new round of research, but instead of using sensor input to directly control the computer, ATAP uses sensor data to enable the computer to recognize our daily movements and make new choices.

“We believe that it is only fair to ask technology to take more cues from us as it enters more and more of our lives,” said Leonardo Giusti, director of design at ATAP. Just as your mom might remind you to grab your umbrella before you leave the house, maybe a thermostat can convey the same message as you walk by, or a TV lowers the volume when it detects you’re asleep on the couch.

Humans enter the personal space of the computer (image provided by Google, click to view the animation)

Much of the research is based on spatial relations , the study of how people use the space around them to facilitate social interaction, Giusti said. As you get closer to a person, the more you expect increased engagement and intimacy. The ATAP team used this and other social cues to confirm that people and devices have their own notions of personal space.

Radar can detect you approaching the computer and entering its personal space. This could mean that the computer can choose to perform certain actions, such as a splash screen, without the need for you to press a button. Currently, Google Nest smart displays already have this interactive capability, but instead of radar, Google uses ultrasonic waves to measure the distance between people and devices. When the Nest Hub detects you’re approaching, it highlights current reminders, calendar events, or other important notifications.

Just getting close is not enough. What if you end up looking in a different direction and walk past the machine? To solve this problem, Soli captures more details in movements and gestures, such as the orientation of your body, which way you might be walking, and where your head is facing—and, with the help of machine-learning algorithms, refine that data further. This wealth of information captured by Radar helps it better predict whether you’re actually going to start interacting with a device, and what kind of interaction is likely.

This perception improvement came as the team performed a series of carefully crafted tasks in their own living room (they stayed home during the pandemic), using head-mounted cameras (to track their own movements) and real-time radar sensing.

Click to watch the video : https://youtu.be/r-eh2K4HCzI

Lauren Bedal, Senior Interaction Designer at ATAP, said, “We move in different ways, transform different actions, and then—given that we were using a real-time system at the time—we improvise, to some extent, based on real-time discovery Completed”.

Bedal has a dance background. The process, she says, is very similar to how a choreographer takes a basic movement idea (i.e. a movement theme) and explores its variations, such as how dancers move their center of gravity, or change the position and orientation of their bodies. Based on these studies, the team formalized a set of movements, all inspired by non-verbal communication and natural interactions with the device: approaching or leaving, passing, facing or back, and glancing.

Bedal gave several examples of computers responding to these actions. If the device senses you’re approaching, it can bring up touch controls; approach the device and it will highlight incoming emails; leave the room and the TV records your progress as you left and picks up where you left off when you return play. If the device determines you’re just passing by, it won’t bother you with low-priority notifications. If you’re cooking on video in the kitchen, the device can pause when you walk away to get ingredients, resume playback when you return, and express an intent to continue. If you glance at the smart display while you’re on a call, the device offers the option to go to a video call so you can put your phone down.

“From all these movements, we can get a glimpse into a way of interacting with computers in the future, taking advantage of our natural movements, which feel very hidden, and the idea is that the computer kind of takes a back seat and only helps us when it’s appropriate,” Bedal said. . “We’re really pushing the boundaries of what people think is possible for human-machine interaction.”

Using radar to influence how a computer responds to us has some challenges. For example, while radar can detect multiple people in a room, if the subjects get too close, the sensor sees a group of people as a loosely shaped blob, leading to confusion in decision-making. There’s still a lot of work to be done, which is why Bedal has emphasized (multiple times) that this work is largely in the research phase — so don’t expect it to appear in next-generation smart displays just yet.

ATAP’s radar technology can sense where you’re looking without using a camera (image courtesy of Google, click to view animation)

There’s good reason to think that radar can also help learn your day-to-day life patterns. ATAP’s Giusti says it’s an area on their research roadmap that could potentially offer advice on healthy habits related to personal goals. I imagine my smart display conjuring up a giant stop sign when it realizes I’m going for a snack in the middle of the night.

The device also needs to strike a balance when it comes to performing the set of actions it thinks you want. For example, what if I want the TV to be on while I’m cooking in the kitchen? The radar will detect that no one is watching the TV and put the TV on pause instead of leaving it on. “As we start working on these interaction patterns that feel very stealthy, fluid, and stutter-free, there needs to be the right balance of user control and automation,” Bedal said. “It doesn’t seem like a lot of trouble, but we should consider the amount of control or configuration that the user might want.”

The ATAP team chose to use radar because it allows for better privacy when collecting rich spatial data. (It has very low latency, can work in the dark, and has no influence on external factors like sound or temperature). Unlike cameras, radar does not capture and store discernible images of the body, face or other identifiers. “It’s more of an advanced motion sensor,” Giusti said. Soli has a detection range of about 9 feet — smaller than most cameras — but if you have multiple small devices with Soli sensors in your home, they can effectively cover your space and create an effective mesh network to track your whereabouts at home. (It’s worth noting that, currently, Soli sensor data in Google’s Nest Hub is processed locally, the raw data is never sent to the cloud).

A device with new ATAP technology inside can sense your approach, then guess what you might want to do and change its state. (The picture is provided by Google, click to view the animation)

Consumers will have to make trade-offs over personal privacy — after all, Google is “the world leader in monetizing data,” said Chris Harrison, a human-computer interaction researcher at Carnegie Mellon University who heads the Future Interfaces group — but He still thinks Google’s camera-free approach is largely user-first and privacy-first. “There’s no privacy violation,” Harrison said. “Everything is on the spectrum.”

Devices are bound to be equipped with sensors like Soli to collect more data in order to understand us better. Ultimately, Harrison expects to see improvements in human-machine interaction based on ATAP’s various technological assumptions.

“Humans can really understand human behavior, and computers do lead to some additional frustrating [situations] when they do,” Harrison said. “Bringing social scientists and behavioral scientists into computing can make that experience even more frustrating.” Happy people and more humane care.”

View the original English text: https://www.wired.com/story/google-soli-atap-research-2022

The text and pictures in this article are from InfoQ