A new 3D gaze forecasting system developed by researchers at Georgia Tech is helping augmented reality(AR) devices predict where users will look next.
The technology tracks attention in three-dimensional environments and predicts future gaze patterns several seconds in advance. This advance could make AR experiences faster, smoother, and more responsive.
Augmented reality(AR) devices are designed to blend digital content with the real world. Most current systems react after a user looks at something. This delay can affect how smooth and responsive the experience feels.
Researchers at Georgia Tech are working on a different approach. Their new system predicts where a person will look before it happens. This gives AR devices extra time to prepare digital content in advance.
The research is led by Fiona Ryan, a Ph.D. student in Georgia Tech’s School of Interactive Computing. She presented the work at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Denver. The study is titled Forecasting 3D Scanpaths in Egocentric Video.
Ryan’s research focuses on tracking gaze from a first-person perspective. The system analyzes how people move and interact with objects around them. It then predicts the likely path of their visual attention.
3D Gaze Prediction Works
Most previous gaze prediction studies focused on two-dimensional images. Those systems examined where people might look in a flat picture. However, real-world environments are far more complex.
READ ALSO: AI-Powered Laser Mosquito Killer Tracks and Eliminates Pests With Precision Targeting
People move through spaces, change directions, and interact with objects constantly. Their attention shifts based on tasks, surroundings, and movement. This makes predicting gaze in the real world much more challenging.
Ryan’s research addresses this challenge by modeling attention in three dimensions. Instead of predicting a single point on a screen, the system forecasts a path of visual attention through physical space. This creates a more realistic representation of how people observe their surroundings.
Much of the work was completed during Ryan’s internship at Meta. The project used data from Meta’s Aria Digital Twin dataset. The dataset contains first-person video recordings of people interacting with objects inside a detailed apartment environment.
Researchers selected the dataset because it includes highly accurate 3D reconstructions of indoor spaces. This allows eye movements to be mapped directly onto real-world objects. As a result, the team could establish reliable ground-truth gaze data for training and testing the model.
A demonstration of the system showed a user walking toward a table containing a cup. After the person picked up the cup, the software successfully predicted the direction of the user’s next movement. This demonstrated how attention often follows a logical sequence during everyday tasks.
Human vision naturally focuses on specific areas rather than processing every detail equally. People tend to look at objects related to their immediate goals. For example, someone picking up a cup often looks at the place where they plan to set it down next.
WATCH ALSO: China sends 3 astronauts, 4 mice to space station for mammal study in orbit
AR Speeds Future Robotics
According to the research team, the system can predict gaze up to three seconds into the future on average. In some situations, predictions can extend up to 10 seconds. Even a few seconds of advance notice can significantly improve AR performance.
This extra time allows AR devices to preload and render digital content before the user actually looks at it. The result is a smoother experience with reduced delays. Users could see information appear more naturally as they move through an environment.
The technology also helps reduce unnecessary processing. Instead of rendering every part of a scene at maximum detail, the system can focus computing resources on areas that users are expected to view next. This could improve both performance and battery efficiency in future wearable devices.
Researchers believe the work is still at an early stage. Future versions may incorporate information about a user’s goals and activities. Understanding what a person is trying to accomplish could improve prediction accuracy and reduce uncertainty.
READ ALSO: Driving Innovation: Engineering the Next Generation of Green Race Cars
The research may also benefit robotics. By studying where humans look while performing tasks, engineers can teach robots to prioritize important information in similar ways. This knowledge could support the development of machines that learn more naturally from human behavior.
As augmented reality moves closer to everyday use, technologies that anticipate user actions are becoming increasingly important. Predictive gaze systems could help create more responsive digital experiences while also providing valuable insights for robotics, human-computer interaction, and future wearable computing platforms.
