Page 86 - Australian Defence Magazine October 2019

PACIFIC
AI
to develop autonomy that could be trust- ed. One of such models considers trust as the combination of competency and integ- rity. Integrity carries a much higher weight than competency. That is, it is often easier to maintain or recover trust from a breach of competency. A breach of integrity, how- ever, is usually much harder to reconcile.
This model is very interesting in the context of autonomous systems. Integrity could be embedded in the system as part of the system design, while competency can be acquired through learning or re-design- ing and then be tested.
In discussions with Australian Defence personnel, time and again, the funda- mental factor for trust that came first is integrity and honesty in particular: it is important that an autonomous system can do what it communicates it can in a particular operational condition. A second aspect that figures highly is that personnel and autonomy must train together before deployment. This aligns with humans as- sessing the degree of competency about autonomous system technology.
How can the competency of an autono- mous system be assessed?
One attractive way of assessing the com- petency of autonomy is through the assess- ment of behaviours and the characterisa- tion of uncertainty about these behaviours. In practical terms, we define behaviours by considering performance indices that we can quantify and associating to them ranges of acceptable values. For example, we could require an autonomous surface vessel to manoeuvre and remain clear of other traffic and that it adheres to Colli- sion Regulations at Sea (ColRegS).
Considering behaviours in this way has several advantages. If we describe the oper- ation requirements of a system in terms of behaviours then we can use different tech- nologies to implement these behaviours. Also by testing the behaviours we may not require to consider industry proprietary information for test and evaluation.
What is essential to the competency as- sessment is the behaviour and not how the behaviour is implemented. This provides a parallel on how we currently qualify the competency of humans. Indeed, when we assess a person for a licence to operate a machine, we do not analyse the signals in the brain. Instead, we pose scenarios and we assess the person’s behaviour in these scenarios. Perhaps autonomy should be as- sessed in the same way.
Autonomy can be tested in a much more
From Metal Mike to Digital Mike, technology is always evolving.
thorough way than humans can the sys- tem can be tested more comprehensively using a large number of scenarios repeat- edly. Also, autonomy does not relax when being tested in a virtual environment, and therefore, virtual assessment can be more conclusive than it can be for humans.
How do we test trust?
A question that often arises is whether we can test every possible scenario that could arise from the interaction of the system and its environment. The answer is no. But we can test a lot, and we can test it more comprehensively that we test it on humans through the use of accelerated simulation techniques.
We anticipate operations requiring human-autonomy interaction. This poses one of the largest risks in deploying auton- omous systems in the short-term future without segregating their operational en- vironments. This segregation is common in automated production and fabrication where there are boundaries on the zones where robotic manipulators operate and where humans operate. These zones do not necessarily overlap.
Such segregation provides a means to ensure the safety of humans, but this lim- its the range of operations and the benefits envisaged to the technology, especially in the short term. Therefore, frameworks for assessing behaviours that can test au- tonomy, humans, and human-autonomy interaction become attractive.
We mentioned the example of an au- tonomous surface vessel that manoeuvres and remains clear of other traffic while adhering to ColRegS. Regulations and
certain aspects of the Law, such as neg- ligence, are often formulated in terms of behaviours. Many nations and bodies such as the UN are currently considering ethical frameworks for the development and deployment of autonomous systems and AI-related technologies. Ethical principles are formulated in terms of be- haviours that are acceptable within par- ticular societal contexts. Then from the perspective of qualifying competency of behaviours, we can test behaviours re- lated to safety and performance and some related to law and ethics.
A framework
If we adopt a behavioural approach as a means for qualifying competency of au- tonomy, then we can consider the follow- ing steps for the assessment:
1. Define the type of operations and mis- sions for the autonomous system;
2. Define relevant behaviours for each op- eration and mission;
3. Define the operational and interven- tional environment for assessment;
4. Asses the system over selected environ- mental conditions to quantify uncer- tainty;
5. Present information to stakeholders to inform their decision processes;
6. Support stakeholders with the decision process;
Steps 1 to 3 relate to system design speci- fications, and they may be augmented with specific needs for the purpose of the as- sessment. These steps can provide input to virtual testing environments based on com- puter simulations and to the planning of assessment through trials. In essence, these steps are akin to the process that Navies around the world, including the Royal Aus- tralian Navy, use to assess the behaviour of ships in waves -seakeeping analysis – to make decisions during tender processes.
But there are key differences. In steps 2 and 3, behaviours for an autonomous system must be considered in a wider sense including safety, performance, hu- man-autonomy requirements, and even autonomous decision making. The opera- tional environment must also incorporate anomalous conditions. Recall that one of Digital Mike’s functions is to manage contingencies, so the qualification must include how the autonomous system man- ages anomalous situations.
The latter can be internal to the system such as faults in sensors, force actuators, a drop of communication channels, or
86 | October 2019 | www.australiandefence.com.au
DEFENCE