FleetScape: A Mixed Reality Sandtable That Reimagines How Humans Supervise Drone Fleets

Main Story

Scaling a drone fleet beyond a handful of units forces a fundamental change in the human role: the operator can no longer pilot each aircraft individually and must instead function as a mission supervisor. Yet most ground control software still treats multi-drone management as a linear extrapolation of single-drone operation — more windows, more feeds, more alerts crammed into the same flat screen. A research team has taken a different approach.

Authored by Peisen Xu, Jérémie Garcia, Peter Cleveland, Ooi Wei Tsang, and Christophe Jouffrais, and accepted for UIST 2026, FleetScape rethinks the operator interface from first principles. The core argument is that fleet supervision is inherently a spatial problem, and that externalising mission state into three-dimensional, mixed reality space is a more natural fit than conventional 2D displays.

The system centres on a 3D Sandtable — an exocentric spatial miniature of the operational environment rendered inside an MR headset. Rather than tracking drones on a flat map, the operator perceives the full mission volume as a scaled physical model they can inspect from any angle. Layered on top of that miniature are real-time mission data, safety alerts, drone positions, and environmental information, all anchored to their precise geographic coordinates within the sandtable.

FleetScape also integrates a 2D Minimap, a vertical Fleet Status Board, and a Drone Control Panel as supporting interface elements. Together these components address a tension that has long troubled multi-robot HCI: how to move fluidly between strategic oversight and hands-on control of a single unit without losing the thread of the wider mission.

To that end, the system offers three distinct control modes: fully autonomous inspection, direct manual control via an individual drone's camera feed, and semi-autonomous route planning that allows the operator to interactively edit waypoints. Switching between modes is designed to be seamless, enabling the operator to, for example, pull one drone out of autonomous operation for a manual inspection pass and then return it to the swarm — all without relinquishing situational awareness of the rest of the fleet.

To test the system under realistic conditions, the team built a high-fidelity building inspection simulation that streams synchronised multi-drone positional data, point clouds, and safety events into the MR environment in real time. The scenario was designed to reflect a structured urban inspection mission with defined safety procedures — a context where spatial precision and safety-critical decision-making are both at a premium.

A user study recruited six experienced drone pilots to manage fleets of up to 15 drones using the prototype. The results confirmed that the layered spatial representations in FleetScape support situational awareness and reduce ambiguity when switching between control modes. However, the study also surfaced a significant finding: situational awareness degrades as fleet size increases, and pilots adapted by adopting qualitatively different supervisory strategies — some prioritising a high-level overview, others dipping in and out of individual drone views more frequently.

This fleet-size ceiling effect is the paper's most practically consequential finding. It suggests that MR spatial interfaces are a meaningful step forward, but not a complete solution to the scalability problem. The authors derive design implications for future systems targeting larger and more heterogeneous fleets.

Technical Breakdown

Industry Impact

For ground control station developers and UAS software vendors, FleetScape represents a direct design challenge to the dominant paradigm of 2D map-centric GCS interfaces. The sandtable metaphor has established precedent in military planning environments, but applying it to real-time drone fleet supervision at the software layer — rather than as a hardware planning tool — is a substantive architectural shift. Vendors should pay close attention to the control-mode transition findings: smooth hand-off between autonomy levels is identified as a key differentiator for operator experience.

For operators and fleet service providers, the study's fleet-size ceiling finding has immediate operational relevance. Current results suggest a single-operator MR interface can meaningfully support fleets up to around 15 units, but that situational awareness degrades beyond that threshold without additional interface or workflow design investment. Organisations planning commercial inspection or infrastructure monitoring deployments at larger scale will need to factor in how supervisory workload is allocated — whether through multi-operator models, AI-assisted alerting, or further interface innovation.

For the research and HCI community, the design implications section of the paper opens several productive directions: adaptive data-layer decluttering as fleet size grows, attention-aware interfaces that surface the most safety-critical events, and clearer signalling of autonomy mode state. The paper's cross-listing across cs.HC and cs.RO reflects how deeply drone fleet supervision has become an HCI problem, not just a robotics control problem.

For regulators and standards bodies, the work provides early empirical grounding for human factors requirements in scaled drone operations. As UTM frameworks globally begin to address multi-drone commercial operations, evidence-based minimum standards for operator interface design — particularly around mode awareness and fleet-size limits — will become increasingly necessary.