The MEDIATOR project and the Mediator system
The MEDIATOR project is working towards a system that mediates, in real time, between the automated functions of a vehicle and the driver/operator ensuring that the one that is most fit for the task at hand is in control. The Mediator system aims to reduce the risks related to the transition towards full automation, a phase that still relies on the human driver for taking over when the automation does not yet function at a sufficiently reliable level or in a limited number of situations.
The aim of the project is to develop a Mediator prototype for SAE levels 2 - 4 and have them tested in a number of relevant traffic scenarios. Each of these levels of automation provide different requirements to the system. Whereas SAE level 2 automation requires drivers to be ‘in-the-loop’ all the time, the higher SAE levels allows them to be ‘out-the-loop’ for shorter or longer periods of time.
In order to decide whether it is safer to have the driver or the automated system in control, the Mediator system must be able to assess the fitness of the driver and the fitness of the automation, as well as the requirements of the driving context. This not only includes an assessment of the situation right now, but also a prediction of the situation in the next few seconds up to minutes. Based on the assessment and prediction, the system has to apply the correct logic to decide if a transfer of control from vehicle to driver or vice versa is needed, whether such a transfer would require specific actions to improve the fitness of the driver or of the automation, while also taking account of driving comfort and driver preferences. In case of a transfer of control it has to be ensured that the human machine interface (HMI) conveys a message (the Mediator action) that is trustworthy and transparent for the driver in order to guarantee acceptance of the system and prevent unintended negative effects such as mode confusion and overreliance.
Aims and scope of this deliverable
The current report is the first Deliverable of the project. It aims to define what we need to know to assess the fitness of the human driver and the automation, what we already know based on the available literature, and what are the research gaps that need to be bridged in order to develop the Mediator system. Information and approaches of existing Mediator-like systems in road and in other transport domains were also considered. In a more or less iterative process the review of needed and available knowledge helped to further elaborate a set of feasible functional requirements for three use cases. The results form the basis for the concrete design and work plan for the technical development of the various components of the Mediator system and their interaction. The results also form the basis for the definition of a series of targeted experiments to fill the most prominent research gaps in the area of human fitness, automation fitness, and HMI, in order to make the best possible decision for a transfer of control, and define the best means of communication to the driver.
In separate chapters, this deliverable discusses the assessment of human fitness, the assessment of the fitness of the automation, the HMI requirements, the decision making logic, and the functional requirements in relation to the identified use cases.
Existing Mediator-like systems
A brief, non-exhaustive literature overview of early systems for assisting humans and of more recent driver monitoring systems was performed. This review showed that in the aviation and military domain, research and applications of systems that mediate between the human and automation date back several decades. However, in the automotive industry it is a relatively new concept. Even though there are obvious differences between these domains, the research in the other transport domains is in many aspects also applicable to the car domain. For instance, both domains benefit from understanding human information processing and decision making. Also monitoring and prediction of human states, such as workload and situation awareness, are integral to human-automation cooperative designs in all domains. In the automotive domain, research and development of driver monitoring systems has recently increased with the advancements in vehicle automation, and several ongoing or recently finished projects have been identified that can provide very useful input for the development of our Mediator system.
Assessment of human fitness
In the area of assessing the human fitness to drive, several potentially relevant factors were considered: personal factors related to, in particular, age, experience and gender, as well as feelings of comfort, emotions and trust in automation, and the more information-processing aspects related to mental workload, distraction, fatigue, and hazard perception. An overview is provided of factors affecting drivers’ performance in the context of automated driving. Based on these findings we identified key human-related variables that should be monitored by our Mediator system in order to determine the driver state (e.g., fatigued, distracted, bad mood). The exact factors, and the way they impact fitness to drive depend on the level of automation.
Assessment of automation fitness
Analogous to assessing the human fitness, the fitness of the automation has to be assessed. In other words, what has to be measured (and how) to decide whether the automated system is sufficiently fit to take over or continue the driving task? To better understand vehicle automation systems, a generalized functional architecture of driving automation systems is provided, accompanied with engineering concepts for analysing the driver task. For assessing the automation fitness and defining the corresponding appropriate actions, the Mediator system requires detailed information on the automation functioning now and in the next few seconds to minutes. The assessment would need to include reliability measures and reasons for degraded performance. Furthermore, context related information gathered by the vehicle automation should also be sent to the Mediator system, as it could be a source for improving human fitness in terms of, for example, situation awareness. Based on what the Mediator system requires, an initial overview of possible information sources within existing vehicle automation systems is provided.
HMI requirements
The Human Machine Interface (HMI) of a vehicle can be defined as set of all interfaces that allow the user of a vehicle to interact with the vehicle and/or devices connected to it. It is a fundamental aspect to ensure that the driver and the automated vehicle have a safe and acceptable exchange of roles. The HMI should take into consideration several demands that need to be evaluated and balanced: driver needs, available technology, applicable regulations, and the costs. Related challenges include trust, mode awareness, fatigue and distraction, information load, user acceptance, industry acceptance, as well as learning and unlearning. Quite a few studies have been identified dealing with each these challenges, both in the road transport section as in maritime and aviation. Nevertheless, some challenges were identified that were not yet or only partly solved. Moreover, whereas studies generally focus on individual challenges, knowledge on dealing with multiple challenges simultaneously is largely missing. This is specifically relevant because a solution for one challenge may have negative side-effects with regard to dealing with other challenges, requiring evidence-based trade-offs.
Decision making logic
Central to the Mediator system is what we called the Mediate Control component, i.e., the decision making component. The basic goal of the decision making component is making the decisions whether the human driver or the automation is most fit to control the vehicle, based on information about the driving context, the human driver state and capabilities, and the automation state and capabilities.
The core of the decision logic process will most likely be based on Markov Decision Processes (MDP). In the terminology of the MDP, this requires a description of the state space and the action space. The state space consists of the driving context, the current human driver state and capabilities, and the current automation state and capabilities. The action space refers to the set of actions the system can perform. At this stage, four main classes of actions were identified:
- Transfer driving task
- Improve/maintain fitness
- Maintain trust, comfort, transparency
- Requests
The main action of the Mediator system is to mediate the transfer of a driving task between automation and human. The other three actions can be seen as sub actions. These include:
- actions that ensure the human or the automation remains fit/becomes fitter, e.g., instructing the driver to put hands on the steering wheel again;
- actions that optimise trust, comfort and transparency, e.g. by providing information about the automation state in order to reduce overreliance or mode confusion; and
- actions requesting the driver for additional input in case of incomplete or uncertain information, e.g., indicating how fatigued he/she is, or requesting the automation to initiate a save-stop procedure.
Once an action has been decided, this action must be ‘negotiated’ or ‘managed’ with and by the interface to the human driver (HMI) and the automated driving system, leading to a safe and comfortable transfer.
Whereas the basic principles of the MDP seem to suitable for our Mediator system, it most likely requires several adaptations and related additional research. A list of key knowledge and development gaps were identified.
Use cases and functional requirements
In order to limit the scope of the development of the Mediator system during the project three use cases were identified which reflect the intended functioning of the Mediator system. Within the project an operational prototype of the Mediator system will be developed for these use cases.
- The ‘continuous mediation’ use case focuses at a lower level of automation which requires the driver to be involved in the driving task continuously. While the automation performs certain parts of the driving task, the driver performs other parts. Continuous mediation is needed between the automation and the driver. Maintaining adequate situational awareness, avoiding mode confusion, and underload are the main challenges here.
- The ‘driver stand-by’ use case focuses on a higher level of automation in which the driver can hand over full control to the automation and be “out of the loop” for some period of time. This is only possible for situations where the automation is confident it can function for the next moments. Hence, the driver should be prepared to resume control on short notice at any time. The main challenges here are determining how long it takes to regain driver fitness, how long automation is fit to drive, and how these times should be balanced.
- The time-to-sleep use case focuses at a level of automation that allows the driver to be completely out of the loop for prolonged periods, and do completely things unrelated to driving and monitoring, including sleeping. The main challenges in this use case are predicting the moment the take-over should take place with sufficient confidence, and bringing the driver back into the loop after a period of full absence.
For each of the above-mentioned use cases, an initial list of higher-level functional requirements was made, related to the assessment of the human fitness, the automation fitness and the Mediator actions, as well as more general description of the required properties of the system. In the next stage of the project, a limited number of specific sub-use cases or test scenarios will be defined, in order to test the functionalities of the Mediator system.
Overall conclusions and next steps
The main goal of the Mediator system is to determine who is fittest to drive, human or automation, and consequently define preferred actions to be sent to the HMI or automation in order to ensure safety and comfort of the human driver. To this end, four components of the system were defined: human state, automation state, HMI and decision logic. This deliverable described the state of the art knowledge and corresponding knowledge and development gaps related to these four components as well as a description of the high level (non) functional requirements and relevant use cases for the MEDIATOR project. The identified knowledge and development gaps will be further investigated in the next steps in the project. More detailed requirements will be defined as well as a clear structure for the integration of all components into one Mediator system. Detailed use cases will be defined to test the Mediator system throughout the project and help with the prioritization of investigations into the identified knowledge gaps.
MEDIATOR has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 814735.
