Glass-box Automated Driving: Insights and Future Trends
page under construction
page under construction
Abstract
Automated driving has advanced significantly through the use of black-box AI models, particularly in perception tasks. However, as these models have grown, concerns over the loss of explainability and interpretability have emerged, prompting a demand for creating 'glass-box' models. Glass-box models in automated driving aim to design AI systems that are transparent, interpretable, and explainable. While such models are essential for understanding how machines operate, achieving perfect transparency in complex systems like autonomous driving may not be entirely practicable nor feasible. This paper explores arguments on both sides, suggesting a shift of the focus towards balancing interpretability and performance rather than considering them as conflicting concepts.
Fully automated driving on roads has been a long-sought goal, with little significant progress for many years. The field has only recently advanced notably, driven by improvements in hardware computational capabilities and data-driven models with the promise of end-to-end automated driving in the near future. However, this progress comes at the cost of losing a clear connection to the fundamentals of process control. Automated driving on roads exemplifies the control of a complex system where data-driven models offer advantages, as creating detailed analytical models for every component is nearly infeasible.
Classical optimal control theory aims to design control processes that achieve optimal tracking of a desired reference signal, offering elegant analytical solutions to many problems. The theory begins with linear, scalar systems and extends to multivariate, non-linear processes, significantly increasing computational complexity and the system's level of abstraction, particularly due to the challenges of solving higher-dimensional and non-linear models.
From this perspective, a model that analytically captures reality at its fundamental level would be the ideal foundation for building deterministic, error-proof control systems. By definition, such a system would qualify as a glass-box model, offering complete interpretability and explainability. On the other hand, many fundamental physical processes remain poorly understood, and while the human instinct is to seek clear explanations, technological advancement often relies on approximate descriptions to manage complex control systems in an uncertain real world. A deeper understanding leads to improved functionality, highlighting the importance of interpretability. While black-box models may enable control systems to function effectively, only full analytical interpretability can unlock their true potential.
Following the example of automated driving, a full analytical and computational description of dynamic behavior—such as aerodynamics, tire friction, and engine response to driver commands—is complex and not entirely practical yet achievable. Steering robots capable of driving vehicles in structured environments, such as test tracks, have existed since the 1980s. The real challenge, however, lies in real-world interactions, where modeling the unpredictable behavior of other road users becomes increasingly difficult. In such scenarios, users expect a robotic driver to act deterministically and make safety-critical decisions within fractions of a second.
Following this line of research, this work discusses the following research questions while focusing on the problem of automated driving:
What are the practical motivations for building glass-box analytical models?
What are the challenges in achieving full explainability and interpretability in the field of automated driving?
What viable strategies exist for achieving transparency in automated driving systems?
This paper is structured as follows: Section 2 provides the motivation for applying glass-box models and presents our perspective on RQ1. Section 3 explores the challenges of developing such models in the field of automated driving, addressing RQ2. Finally, our proposed strategy for RQ3, which focuses on modularizing models and balancing interpretability with performance, is discussed in Section 4.
Motivation
Let's assume we want to design a system capable of taking the best action in every situation, following a data-driven approach. For simplicity, we can envision it as a perfect driver: one that always selects the optimal route from A to B while ensuring energy efficiency, minimal travel time, safety, and passenger comfort.
The process of building such a system would involve recording pairs of information points and optimal actions. Each information point can be considered as an abstract representation of everything the robotic driver perceives, including the internal vehicle status and any external sources of information (e.g., driving scenario, traffic conditions, other road users, etc.).
For simplicity Fig.~\ref{fig:actionInformationModel} provides an abstract representation of these information points on an action-information Cartesian map, where $i$ represents our information content and $d$ represents our best recorded decision in such a scenario. By considering all recorded information, represented as gray dots, one can construct a function—using any analytical fitting method—thus obtaining an action-information mapping function.
The goal of building such a function is to use the action-information model as a decision-making tool to estimate the best action as new information becomes available. If we assume that the blue line in Fig.~\ref{fig:actionInformationModel} represents our best model, when new information $i_{new}$ arrives, the model’s guess should be no other than $\hat{d}_{new}$ as the intersection point between the information and the model.
However, the optimal decision in that situation may be something slightly different, such as $d^{*}_{new}$, resulting in a gap between the model’s guess and the true optimal decision.
Such an gap in the optimality might result from:
Measurement errors in both the information used to build the model and the new data point.
Approximation errors due to complex interactions between variables not fully captured by the model.
Truly unknown circumstances, highlighting a difference between the current information point and previous knowledge of the system.
A change in the operational domain, where the model was not trained or calibrated for new conditions, leading to discrepancies in the decision-making process.
Sticking to the example of automated driving, the gap in the optimal decision can lead to a wide range of consequences, from minor delays in travel time or passenger discomfort to more severe outcomes like crashes.
"How can one identify the cause of a miscalculation in the decision-making process if the system is a fully enclosed black box?"
From this perspective, all points mentioned earlier—measurement errors, approximation errors, truly unknown circumstances, and changes in the operational domain—are equally valid sources of error with very little possible action to address and correct such errors to ensure the system functions properly across different scenarios. The current black-box approach to solving this problem tends to add information to the system indefinitely, leading to memorization rather than meaningful knowledge abstraction with minimal validation and verification possibilities.
Assuming that our action-information model is an analytical multivariate function underpinning our process, a gap in decision-making can be investigated, studied, and potentially debugged if, and only if, the system can be fully explained and interpreted.
Often, interpretability and explainability are seen as a written form of text providing an explanatory statement about driving decisions and specific actions \cite{9616449}.
However, this approach should not be conflated with true interpretability or accountability. The generated text, often produced by black-box LLMs (large language models), merely describes the situation without offering useful debugging information.
As LLMs have shown, it is always possible to generate ‘human-understandable’ explanations that can be terribly misleading and fail to reflect the system’s actual behavior at its fundamental level. Thus, this type of verbal explanation often falls more into the realm of psychological perspective rather than a genuine interpretation of system behavior.
Challenges
Without any doubt, full black-box models have demonstrated compelling capabilities in controlling complex systems such as autonomous vehicles. This achievement cannot be neglected, as it represents a significant opportunity to address long-term practical problems. However, glass-box models offer the opportunity to debug and refine these systems by providing full control and transparency. This allows for a deeper understanding and improvement of each system’s behavior.
The first significant challenge in building a glass-box control system for autonomous driving lies in managing the complexity of the environment. As previously mentioned, the challenge does not stem from driving along a predefined path at a precise velocity but rather from interacting effectively with the surrounding environment. Autonomous driving requires processing vast amounts of real-time data from sensors such as cameras, LiDAR, radar, and GPS. The decision-making process entails intricate interactions between perception, prediction, and control systems, which are often modeled using complex deep learning or neural networks—methods that are inherently difficult to interpret. Figure~\ref{fig:control} illustrates the control flow from low-level to high-level driving models.The concept is that any state-of-the-art controller, from PID to Lyapunov-based controllers, can effectively and precisely manage vehicle speed and steering angle, provided a simplified physical model of the vehicle is known. The assumptions required for these controllers to function are often unrealistic and ineffective at predicting dynamic environmental changes. The higher the complexity of the vehicle model, the better and more effective the controller. This capability is sufficient for driving in structured environments. The system’s internal components, highlighted with a dashed line in Fig.~\ref{fig:control}, can be fully explainable and interpretable PIDs. Literature offers a robust foundation of analytical analyses and formal solutions for the problem of following reference signals, enabling deterministic control in these scenarios.
As the research community approached this problem, it quickly became evident that addressing it required multiple levels of abstraction. At the very low level, classical automatic controllers, such as PID or model predictive controllers, perform their tasks effectively. However, tackling vehicle perception and routing with the same level of detail as low-level control, represented in the outer cycle of Fig.~\ref{fig:control}, is computationally and practically infeasible.
The routing problem itself can be subdivided into two distinct components: the generation of kinematically or dynamically feasible trajectories for short-range vehicle control, and waypoint generation for high-level vehicle routing. The former requires solving differential equations, which are computationally demanding and often limited to localized regions. The latter typically employs simpler, non-physically compliant algorithms, such as Dijkstra’s algorithm or rapidly-exploring random trees (RRT). Attempting to solve differential equations for every point along a long path is not only computationally prohibitive but also ineffective, as it assumes a static environment, thereby reducing the system's ability to adapt to dynamic changes. Conversely, ignoring differential equations altogether can result in unfeasible paths that violate the vehicle's physical constraints.
The routing problem exemplifies the necessity for different levels of abstraction to address complex challenges in autonomous driving. Each level retains its own interpretability: long-range planning may sacrifice physical precision but provides computational efficiency, while short-range trajectory generation maintains a detailed physical interpretation. This balance allows the system to adapt to a dynamic environment while ensuring feasibility at a local level.
On a different level, the perception problem, which includes tasks such as object detection and segmentation, is predominantly addressed using black-box models. On one hand, these models excel in providing detection capabilities that can even surpass human performance in certain scenarios, such as low-light conditions or when integrating data from multiple sensor sources. An illustrative example is shown in the two images in Fig.~\ref{fig:control} which are extracted from the IseAuto dataset and include the addition of a black-box segmentation result.
On the other hand, they lack an understanding of the semantic meaning of objects within a scene, treating a tree and a person merely as objects without recognizing their distinct roles or contextual importance. Contextual importance is inherently challenging to capture, as even human observers may assign varying interpretations and significance to similar objects in a scene. An interpretable and explainable model should not aim to produce verbose descriptions of a scene but rather to classify objects accurately while accounting for their high-level roles and importance. For instance, in such a framework, errors or flaws in detecting vehicles or pedestrians, compared to non-critical objects like rocks or foliage, could be identified, debugged, and addressed in a targeted manner.
Safety-Critical Nature
While it may seem that autonomous vehicles must prioritize split-second decision-making over immediate explanations, it is precisely in these high-stakes scenarios that the effort to build simple, classical control, and rule-based systems becomes essential. Transparency is critical for diagnostics and regulatory compliance, and it is often mischaracterized as compromising the vehicle's ability to respond effectively in emergencies. On the contrary, such transparency ensures that misunderstandings or misinterpretations are minimized, particularly in situations where they cannot be tolerated.
In \cite{10599385}, the authors clearly emphasize the safety concerns that deep learning poses to automated driving, covering aspects such as operational domain definition and limitations, as well as the methods used for data preparation and algorithm development. Moreover, regulations are placing increasing emphasis on the importance of AI explainability in safety-critical industries like transportation. Glass-box models are required to comply with stringent industry standards, such as ISO 26262 \cite{iso26262} (functional safety) and ISO/PAS 21448 (Safety of the Intended Functionality - SOTIF), to ensure reliability, accountability, and safety \cite{kirovskii2019driver}. Additionally, frameworks such as the European Union’s Ethics Guidelines for Trustworthy AI and the U.S. NIST’s initiatives on AI explainability advocate for more transparent and accountable AI systems.
The recently adopted AI Act (\textit{Regulation (EU) 2024/1689 laying down harmonized rules on artificial intelligence}) sets clear requirements and obligations for AI developers, emphasizing explainability and accountability in AI-based systems, including automated driving functionalities.