Raising the safety awareness of robots
As robots make their way into our everyday lives, ensuring that they can act safely requires more than simply avoiding collisions or following predefined analytical constraints. They must also understand the semantics of the environment, reason about the context, distil pertinent constraints into computable forms, and then reliably translate these constraints into appropriate actions. “A seemingly simple instruction to heat food in a microwave requires the robot not only to recognise the microwave and the food but also to understand that metallic objects should be removed beforehand to avoid a potential fire hazard,” explains SiQi Zhou(opens in new window), an assistant professor of Computing Science at Simon Fraser University(opens in new window). While strong theoretical foundations exist for certifying robot actions with respect to a given set of constraints, existing frameworks often assume the constraints to be known a priori, as well as be explicitly defined in terms of robot states and actions. “Recent advances in machine learning allow robots to extract semantic information from images, understand natural-language instructions, and reason about their surroundings,” adds Zhou. “However, translating this semantic understanding into safe and contextually appropriate actions remains challenging.” Addressing this challenge is SSDM(opens in new window), an EU-funded project led by Zhou and hosted by Angela Schoellig at the Technical University of Munich(opens in new window) (TUM).
Semantic filter helps robots comply with common-sense safety constraints
The project, which received support from the Marie Skłodowska-Curie Actions(opens in new window) programme, is developing approaches that couple semantic perception with language-conditioned reasoning and safety-critical control. One of those approaches is a semantic safety filter(opens in new window) designed to help robots comply with common-sense safety constraints. The filter builds on an open-vocabulary 3D representation of the environment, uses large language models to identify context-dependent hazards, and translates these hazards into computable constraints that can be enforced by a robot controller. The solution was demonstrated on a Franka Emika FR3 manipulator in a real-world kitchen, where learned diffusion policies were used to complete object transportation tasks involving fragile objects and to recognise fire or electrical hazards. “The filter prevented unsafe actions such as placing a metal cup inside a microwave or a pressurised spray can on a stove,” notes Zhou, the project’s principal investigator. The framework achieved a 100 % constraint-satisfaction rate for both geometric and semantic safety requirements.
Solution allows robots to detect changes in the operating environment
Advances in computer vision have progressively moved robotic perception from geometric reconstruction to semantic understanding and, more recently, open-vocabulary mapping. However, many existing approaches still assume that the environment remains static over time. “This assumption breaks down in everyday settings, where furniture, appliances and smaller objects can be moved, removed or reintroduced,” remarks Zhou. “Failure to account for such changes will lead to an inaccurate map, hindering the reliability of downstream decision-making algorithms.” To address this limitation, the project developed an open-vocabulary map-maintenance approach that explicitly tracks object-level changes and maintains a reliable, up-to-date 3D representation of the environment for downstream safety reasoning and decision-making. In experiments conducted in real-world kitchens and offices, the system detected 95 % of the environment changes and maintained mapping precision within 3 % of fully reconstructed maps. “Taken together, our results provide theoretical foundations, algorithmic tools and experimental demonstrations for connecting semantic understanding with safe robot control,” concludes Zhou.