Designing Agentic AI Interactions in XR
From assistants to co-pilots: exploring a new paradigm of intelligence through context-aware XR interaction.
From Assistants to Co-Pilots
As XR environments become more spatial and context-rich, command-based assistants may not be enough. The team explored how an AI agent could understand context, anticipate intent and interact through more natural forms of communication.
Designing User–Agent Interactions
The team explored user–agent interactions for a futuristic, ubiquitous agent living inside an XR device — reimagining Bixby as a proactive, context-aware co-pilot rather than a purely rule-based assistant.
Beyond the Command
Three channels carry user intent. The agent reads them together with context to decide an appropriate response.
Gaze
Where attention rests becomes the first signal.
Voice
Spoken instruction, stated directly to the agent.
Gesture
Hand action that selects, moves or confirms.
Designing for Context, Not Just Commands
The interaction logic the team worked with, from a user's intent through to a confirmation or a fallback.
- ▪Gaze detection
- ▪Verbal command
- ▪Gesture command
- ▪Context interpretation
- ▪Confirmation
- ▪Manual input when confidence is insufficient
An Agent That Understands the Situation
The furniture-shopping and product-discovery scenario used to test the model end to end.
Giving the Agent a Presence
The agent was treated not only as a function, but as an interaction presence inside the XR space — identity, visual language and emotional state.
My Contribution Within the Team
Collaborative Master's project with five designers — Omkar, Richa, Lokanshu, Janice and Soumya.
[Replace with exact individual contribution.]
[Briefly explain how this contribution supported the overall interaction concept.]
From Research to Interaction Concept
Interaction models, user–agent flows, AI visual language and XR scenarios brought together as one concept.
What this project taught me
“Designing agentic interactions made me think beyond conventional interface design — toward systems that can interpret context, understand intent and decide how they should respond. The project expanded my understanding of how interaction design may evolve when interfaces become spatial, conversational and increasingly proactive.”
Explore the Full Project
See the complete research, interaction models and project development on Behance.
View Full Project on Behance →