PIVOT: The Drone-Captured Dataset Exposing Hidden Cracks in 3D Reconstruction Benchmarks

Main Story

For years, the computer vision community has measured the progress of neural scene reconstruction by benchmarks that quietly favour laboratory conditions. Training and evaluation images are sampled from the same camera trajectory. Camera poses are handed to the model pre-optimised. Lens intrinsics are tuned per dataset. These conveniences are sensible for controlled research, but they are a poor proxy for what a drone, a robot, or an autonomous inspection system actually encounters in the field.

A new dataset and evaluation framework called PIVOT — Pose, Intrinsics and Viewpoint Oriented Testbed — takes direct aim at this gap. Published as a preprint on arXiv (cs.CV), PIVOT is designed to independently stress-test the three variables that realistic deployments disturb most: camera trajectory, pose source, and camera calibration.

The dataset was captured across five real-world outdoor scenes using a DJI Mini 4 Pro — a sub-249 g consumer quadcopter equipped with a 1/1.3-inch CMOS sensor capable of up to 48 MP stills and 4K video. The choice of platform is deliberate: the Mini 4 Pro sits at the accessible end of the professional drone market, weighing under the regulatory threshold that triggers mandatory registration in many jurisdictions, yet carrying imaging hardware representative of the kind of footage increasingly fed into reconstruction pipelines.

For each scene, PIVOT captures multiple structurally different camera trajectories rather than the single orbit or grid sweep typical of reconstruction datasets. Critically, the dataset retains both sensor-derived measured poses — the raw GPS/IMU-fused position estimates logged by the drone — and COLMAP-optimised poses, where the widely used Structure-from-Motion pipeline refines camera locations by matching visual features across frames. It also pairs each pose source with both factory-calibrated intrinsics and COLMAP-optimised intrinsics, giving researchers a 2×2 factorial design for pose and lens parameters alone.

COLMAP, an open-source photogrammetry tool that estimates camera poses and reconstructs sparse 3D point clouds from image sequences, has become the de facto pose provider for most NeRF and 3DGS pipelines. PIVOT's inclusion of raw sensor poses alongside COLMAP-optimised ones allows, for the first time in a drone-specific benchmark, a clean comparison of how sensitive reconstruction quality is to that upstream choice.

The framework defines three benchmark families:

1. Seen vs. unseen trajectory novel-view generalisation — does a model trained on one flight path render convincingly along a structurally different path?
2. Measured vs. optimised pose sensitivity — how much does reconstruction quality degrade when sensor-logged poses substitute for COLMAP-refined ones?
3. Calibrated vs. optimised intrinsics sensitivity — what is the cost of reusing a pre-calibrated lens profile rather than optimising intrinsics from scratch?

To quantify the geometric relationship between training and evaluation trajectories — an aspect previous benchmarks have not formalised — PIVOT introduces a directed pose-space Chamfer distance. This metric measures how well the spatial distribution of training viewpoints covers the evaluation trajectory, providing a principled way to predict when a model is being asked to extrapolate beyond its training coverage.

The evaluation toolchain is built on Nerfstudio, an open-source, modular Python framework for NeRF development originating from UC Berkeley, which supports multiple reconstruction methods and provides real-time visualisation of trained radiance fields. By grounding PIVOT's evaluation toolchain in Nerfstudio, the authors ensure reproducibility across the growing family of methods the platform supports — including both NeRF variants and 3D Gaussian Splatting.

Benchmark results from PIVOT v1 confirm what practitioners have long suspected but lacked a clean benchmark to demonstrate: there is a consistent, measurable quality gap between views held out from represented trajectories and views drawn from unseen trajectories. The results also show substantial sensitivity to both pose source and camera intrinsics — meaning that a model whose PSNR looks strong on a standard held-out split may fail meaningfully when given real sensor poses or deployed along a novel flight path.

PIVOT v1 ships with an open processing pipeline and is positioned as a living dataset, with the versioning suggesting future scene additions.

Technical Breakdown

Industry Impact

For researchers and benchmark designers, PIVOT introduces a structural argument that the field's standard evaluation practice — holding out frames from the same trajectory used for training — overstates generalisation performance. Any method claiming strong novel-view synthesis results will increasingly need to demonstrate performance on structurally distinct trajectories to be credible for real-world deployment.

For drone operators and survey professionals, the pose-sensitivity findings have immediate practical implications. Many field operators rely on onboard GPS/IMU logs rather than running COLMAP reconstruction to supply poses for downstream 3D modelling. PIVOT's benchmarks quantify the quality cost of that shortcut for the first time in a drone-native dataset, giving operators a data-driven basis for deciding when COLMAP optimisation is worth the additional compute.

For manufacturers and integrators building inspection, mapping, or digital-twin pipelines on top of consumer drones, the choice of camera intrinsics handling — whether to reuse a stored calibration or re-optimise per flight — now has benchmark evidence behind it. PIVOT's intrinsics sensitivity family gives system designers a concrete test for whether their calibration workflow is introducing a meaningful quality floor.

For the NeRF and 3DGS method development community, the directed pose-space Chamfer distance is a reusable geometric tool. It offers a way to characterise training–evaluation coverage in any dataset, not just PIVOT's, and could be adopted by future benchmarks as a standard diagnostic.

For Nerfstudio's ecosystem, PIVOT's adoption of the framework as its evaluation backbone adds a drone-specific, multi-trajectory benchmark to what has been primarily a lab and indoor-scene ecosystem. This expands the practical surface area against which Nerfstudio-compatible methods can be tested.

The regulatory angle is subtler but real: the DJI Mini 4 Pro's sub-249 g weight places it below the registration thresholds in many jurisdictions, meaning research teams worldwide can replicate PIVOT-style data collection without the permitting overhead associated with heavier platforms — lowering the barrier to extending the dataset with new scenes and trajectories.