FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations

Fedor Rodionov, Abdelrahman Eldesokey, Michael Birsak, John Femiani, Bernard Ghanem, Peter Wonka

Abstract

A diagnostic benchmark for spatial reasoning in LLMs, built on structured JSON/XML representations of indoor scenes and covering tasks like distance, visibility, pathfinding, and object placement. Evaluating 15 frontier models shows they can answer shallow spatial queries but often fail to respect physical constraints and scene coherence, even while staying fairly robust to small spatial perturbations.