Full-sphere pedestrian tracking

PanoPed: Beyond Bounding Boxes
for Sim-to-Real Panoramic
Pedestrian Tracking

PanoPed brings people tracking onto the full sphere, from stationary cameras to quadruped and drone viewpoints. Sextant reads visual evidence already inside a frozen tracker to place each person more precisely.

Paper coming soon Code Explore the data

60synthetic sequences

5real sequences

34,692Sextant parameters

Fixed · Quadruped · Drone

01 / Motivation

Why a box falls short
on the sphere.

Planar boxes are useful detector outputs, yet they cannot fully describe the visible angular support of a person on a sphere. PanoPed pairs a synthetic benchmark with consented real scenes and evaluates tracking in native spherical regions.

Video introductionUE third-person illustrations + released ERP footage · original instrumental audio

02 / The benchmark

Synthetic training.
Real-world evaluation.

Full-sphere tracking from simulated fixed and moving cameras to consented real panoramas, with each target protocol identified explicitly.

PanoPed-SUnreal Engine · v1.0.0

Controlled worlds.
Uncontrolled viewpoints.

60 one-minute sequences and 108,000 RGB frames, with fixed, quadruped-mounted and drone-mounted cameras. Released annotations include visible support, spherical regions, identities, depth and scene state.

30fixed

18quadruped

12drone

License: PanoPed-S Research-Only Data License v1.0.0. Noncommercial research/education; access requires acceptance of the release terms. Reconstructing or redistributing underlying meshes and textures is prohibited.

View release packages

PanoPed-RConsented real video · v0.9.0

Real people.
Real panoramic ambiguity.

Five fixed-camera sequences and 28,002 frames across corridors and open plazas. Three sequences have dense identity and full-sphere Box-fit annotations; the other two are released as unlabeled video.

5sequences

16,247labeled frames

3LOSO folds

License: PanoPed-R Research-Only Data License v0.9.0. Noncommercial research/education; access requires acceptance of the release terms. Identification and named-person surveillance uses are prohibited; secure storage is required.

View release packages

Three synthetic panoramas followed by three consented real panoramas with frozen automatic visible-support overlays — Fixed, quadruped, and drone views above; real corridors and plazas below. Real colored regions come from box-prompted automatic Support-Auto masks, not hand-drawn pixel labels. View full-size figure ↗

03 / The method

Sextant: localization
from frozen queries.

Sextant reuses the detector's frozen visual query. It predicts a local angular correction while the tracker's identities, scores, detections and history remain untouched.

Sextant diagram showing a frozen detector and identity stream, a 34,692-parameter angular localization head, and a native spherical region — Frozen tracker → query-conditioned readout → native spherical localization. The identity branch bypasses Sextant.

Read

256D detector query, initial geometry and confidence.

Correct

Lightweight 262→128→4 head, trained on native support targets.

Preserve

Final localization changes; scores, identities and tracker state do not.

04 / Paired results

Tracking gains under
defined protocols.

PanoPed-S TEST · MOTIP

Native full-sphere tracking

Geometric readout47.30

MOTIP + Sextant · seed 4249.49

HOTA ↑ · seed 43: 49.47

PanoPed-R-Support-Auto · HAT · source-only

Transfer without real-label fitting

Geometric readout31.71

HAT + Sextant · seed 4232.85

HOTA ↑ · different tracker and evaluation target from S TEST

05 / Scene library

Explore the benchmark footage.

Short excerpts from released RGB, encoded for this page. Synthetic labels come from release annotations; no generated or reconstructed people are added.

01 · PanoPed-S

Fixed camera

One full sphere, one stable viewpoint.

02 · PanoPed-S

Quadruped camera

Street-level movement through a crowd.

03 · PanoPed-S

Drone camera

Changing height and viewpoint.

04 · PanoPed-S

Released labels

L1 spherical regions over RGB.

05 · PanoPed-R

R-01 · Corridor

Seam crossing and lower-polar targets.

06 · PanoPed-R

R-02 · Open plaza

Multiple levels, stairs and scale variation.

07 · PanoPed-R

R-03 · Courtyard

Occlusion across a broad open scene.

08 · PanoPed-R · Unlabeled

R-U01 · Indoor passage

Released RGB footage, without dense identity labels.

09 · PanoPed-R · Unlabeled

R-U02 · Outdoor square