By Preet Gandhi, NYUIntroduction In the era of data science, artificial intelligence is making impossible feats possible.
Driverless cars, IBM Watson’s question-answering system, cancer detection, electronic trading, etc.
are all made possible through the advanced decision making ability of artificial intelligence.
The deep layers of neural networks have a magical ability to recreate the human mind and its functionalities.
When humans make decisions, they have the ability to explain their thought process behind it.
They can explain the rationale; whether its driven by observation, intuition, experience or logical thinking ability.
Basic ML algorithms like decision trees can be explained by following the tree path which led to the decision.
But when it comes to complex AI algorithms, the deep layers are often incomprehensible by human intuition and are quite opaque.
Data scientists may have trouble explaining why their algorithm gave a decision and the laymen end-user may not simply trust the machine’s predictions without contextual proof and reasoning.
Explainability is a multifaceted topic.
It encompasses both individual models and the larger systems that incorporate them.
It refers not only to whether the decisions a model outputs are interpretable, but also whether or not the whole process and intention surrounding the model can be properly accounted for.
They try have an efficient trade-off between accuracy and explainability along with a great human-computer interface which can help translate the model to understandable representation for the end users.
There need to be three steps which should be fulfilled by the system :1) Explained the intent behind how the system affects the concerned parties2) Explain the data sources you use and how you audit outcomes3) Explain how inputs in a model lead to outputs.
The necessityExplainability is motivated due to lacking transparency of the black-box approaches, which do not foster trust and acceptance of AI generally and ML specifically.
Rising legal and privacy aspects, e.
g.
with the new European General Data Protection Regulations will make black-box approaches difficult to use in Business, because they often are not able to explain why a machine decision has been made.
IBM Watson shook the world when it won jeopardy beating the best human players.
But when it was marketed to hospitals to help the oncology department detect cancer, it failed miserably.
Cancer detection is a very difficult and serious topic.
The doctors as well as the patients were unable to trust the machine at each stage of consulting and treatment as Watson wouldn’t provide the reasons for its results.
Moreover when its results agreed with the doctor’s, it couldnt provide a diagnosis.
Using AI for military practices is a hotly debated topic.
Proponents claim that lethal autonomous weapon systems (LAWS) might cause less collateral damage but despite there being large training data to distinguish between combatants and civilians or targets and non-targets, it is very risky to leave the final decision making power to machines.
CIA has 137 AI projects, one of which is the automated AI-enabled drones where the lack of explainability of the AI software’s selection of the targets is controversial.
The extent of an explanation currently may be, “There is a 95 percent chance this is what you should do,” but that’s it.
When the algorithm cant describe it’s features that contributed towards identifying a legitimate target, it leaves room open for debate on racism and stereotype issues which bias the model.
Moreover in case of a false target, explanations would help diagnose the cause of failure and help improve the model.
New methods in academiaThere are two main set of techniques used to develop explainable systems; post-hoc and ante-hoc.
Ante-hoc techniques entail baking explainability into a model from the beginning.
Post-hoc techniques allow models to be trained normally, with explainability only being incorporated at testing time.
Ante-Hoc Methods:Post-Hoc Methods:x_j — the activation value for neuron j in layer lw_j,k — the weighing of the connection between neuron j in layer l and neuron k in layer l + 1R_j — Relevance scores for each neuron in layer lR_k — Relevance scores for each neuron in layer l+1Explainability vs AccuracyWill we need to ‘dumb down’ AI algorithms to make them explainable?.Explainable models are easily understandable but don’t work very well as they are simple.
Accurate models work well but aren’t explainable as they are complicated.
The main issue with explainable AI is whether it can accurately fulfill the task it was designed for.
The tradeoff decision to be made should depend on the application field of the algorithm and the end-user to whom its accountable.
When dealing with technical users who are acquainted with the field and have high trust level, especially tech companies or researchers, having highly accurate models would be preferred over high explainability as performance is very important.
But dealing with laymen users is a different scenario.
It would be difficult to garner their trust.
The machine learning approach is very different specially in banks, insurance companies, healthcare providers and other regulated industries.
The reason is mainly that they are prone to legal or even ethical requirements which tend to limit more and more the use of black box models.
Such institutions are answerable for their decisions and process and in this case a simple explainable albeit less efficient algorithm would do.
Hence the tradeoff should be decided according to the application domain and the users concerned.
If you want your system to be explainable, you’re going to have to make do with a simpler system that isn’t as powerful or accurate.
Simple systems can give a prediction to the user but the ultimate action should be taken by the user.
When performance matters most, even with a complicated model, opt for the trust that comes from making sure that you’re able to verify that your system does, in fact, work.
Use Case : DARPA XAIThe Department of Defense (DoD) is increasingly investing in AI research in collaboration with technological giants like Google to have artificial intelligence programs for military activities.
Due to ethical reasons, many government departments agree to have a “man in the loop” to control any lethal system of these “killer robots”.
Due to the increasing debates and strikes by the AI community to not contribute towards military AI, the DARPA division is pushing towards their $2 Billion Explainable Artificial Intelligence ( XAI ) program.
XAI is their hope to usher in the third-wave AI systems which can understand the context in which they function and can characterize the real world phenomena.
Its main aim is to:The final goal of XAI is to have a toolkit library of ML and HCI modules for more understandable AI implementations.
The Phase 1 of the project was completed in May 2018 and further advancements are expected in the coming years.
In case of a gray area in decision output, XAI aims to give the analyst reasons for red flags upon which the analyst can act to make a decision using human input.
ConclusionThe success of AI models is due to the machines’ own internal representations which are even more complicated then manually generated features leading to its inexplicable nature.
There are a lot of on-going research on the ante-hoc and post-hoc methods to make the AI more answerable and awareness to inculcate these methods in existing programs.
Good initiatives by leading institutions like DARPA, Google, DeepMind, etc.
are leading the necessary change.
Despite this, there will always be a tradeoff between explainability and accuracy whose balance depends on various factors like end-user, legal liability, technicality of the concerned parties and the field of application.
Artificial intelligence should not become a powerful deity which we follow blindly without understanding its reasoning but we shouldn’t forget about its beneficial insight it can have.
Ideally, we should build flexible and interpretable models that can work in collaboration with experts and their domain knowledge.
ReferencesBio: Preet Gandhi is a student at NYU and was a Data Science Intern at Apple Inc.
Resources:Related: var disqus_shortname = kdnuggets; (function() { var dsq = document.
createElement(script); dsq.
type = text/javascript; dsq.
async = true; dsq.
src = https://kdnuggets.
disqus.
com/embed.
js; (document.
getElementsByTagName(head)[0] || document.
getElementsByTagName(body)[0]).
appendChild(dsq); })();.