GuideTWSI: A Diverse Tactile Walking Surface Indicator Dataset from Synthetic and Real-World Images for Blind and Low-Vision Navigation

Tactile Walking Surface Indicators (TWSIs) are safety-critical landmarks that blind and low-vision (BLV) pedestrians use to locate crossings and hazard zones. From our observation sessions with BLV guide dog handlers, trainers, and an O&M specialist, we confirmed the critical importance of reliable and accurate TWSI segmentation for navigation assistance of BLV individuals.

Achieving such reliability requires large-scale annotated data. However, TWSIs are severely underrepresented in existing urban perception datasets, and even existing dedicated paving datasets are limited: they lack robot-relevant viewpoints (e.g., egocentric or top-down) and are geographically biased toward East Asian directional bars.

We introduce GuideTWSI, the largest and most diverse TWSI dataset, which combines a photorealistic synthetic dataset, carefully curated open-source tactile data, and quadruped real-world data collected and annotated by the authors. Notably, we developed an Unreal Engine-based synthetic data generation pipeline to obtain segmented, labeled data across diverse materials, lighting conditions, weather, and robot-relevant viewpoints.

Extensive evaluations show that synthetic augmentation improves truncated dome segmentation across diverse state-of-the-art models, with gains of up to +29 mIoU points, and enhances cross-domain robustness. Moreover, real-robot experiments demonstrate accurate stoppings at truncated domes, with high repeatability and stop success rates (96.15%).

• Photorealistic Synthetic Data Pipeline: Unreal Engine 4-based pipeline specifically designed for blind navigation, simulating multiple truncated dome textures under diverse viewpoints and lighting conditions, generating over 15K samples.
• Large-Scale GuideTWSI Dataset: Comprising (i) meticulously curated open-source annotations (RBar-22K), (ii) our synthetic data (SDome-15K), and (iii) real-world robot-collected data (RDome-2K), all released with code and pretrained model weights.
• Extensive Evaluations: State-of-the-art segmentation models showing up to +29 mIoU improvement with synthetic augmentation, demonstrating robustness across domains.
• Real-World Robot Deployment: Guide dog robot stopping at truncated domes with a success rate over 96% across diverse, previously unseen real-world environments.

Our synthetic data generation pipeline uses various sidewalk environments in Unreal Engine 4 with varying viewpoints, lighting, and weather conditions to simulate real-world variability. We use AirSim for automatic annotation of depth, instance masks, and bounding boxes.

Dataset Components:

We integrated our fine-tuned segmentation model into a fully untethered guide dog robot platform. The system is built upon the Unitree Go2 quadruped robot equipped with a RealSense D435 camera positioned with a downward tilt for truncated dome segmentation.

The model was converted to a TensorRT-optimized engine, achieving up to 43 FPS inference on an NVIDIA Jetson AGX Orin. Our system employs segmentation-based closest-point detection strategy, identifying the lowest truncated dome-pixel in the image frame as the point closest to the robot body.

Impact of Synthetic Data Augmentation on Truncated Dome Segmentation

Robot Stopping Performance

Qualitative comparison of segmentation models trained on real data only vs. real + synthetic data. Models trained with synthetic data produce sharper boundaries and fewer missed detections, especially under challenging textures and lighting.