Insights into the black box of artificial intelligence

Researchers at FAU make data processing visible with coloured scatter diagrams.

At many banks, insurance companies and online retailers, self-learning computer algorithms are used to make decisions that have major consequences for customers. However, just how algorithms in artificial intelligence (AI) represent and process their input data internally is largely unknown. Researchers at the Pattern Recognition Lab and the Cognitive Computational Neuroscience Group at Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) and the Neuroscience Laboratory at the Department of Otorhinolaryngology – Head and Neck Surgery at Universitätsklinikum Erlangen, the Aix-Marseille Université in Marseille, France, and York University in Toronto, Canada, examined the problem of ‘black boxes’ in artificial intelligence and developed a method that makes these processes visible. They have published their results in the journal Neural Networks.

Non-reproducible decisions

‘What we call artificial intelligence today is based on deep artificial neural networks that roughly mimic human brain functions,’ explains Dr. Patrick Krauss from the Cognitive Computational Neuroscience Group at FAU. As is the case in children learning their native language without being aware of the rules of grammar, AI algorithms can learn to make the right choice by independently comparing a large amount of input data. What the systems achieve in terms of pattern recognition is astounding, says Dr. Krauss. However, we don’t know which steps the neural networks follow. ‘Decisions made by AI must be reproducible for ethical reasons.’ One obvious example of why this must be the case is when decisions concerning possible medical treatments are taken based on AI-supported medical diagnoses. The issue of whether the AI of an autonomous vehicle should aim to save the life of the driver or the pedestrian during an inevitable collision also needs to be clarified.

Artificial neural networks (ANN) are mathematical copies of how the brain processes stimuli. They are comprised of artificial neurons that are connected to each other. The algorithms process numerical values instead of the electrical or chemical signals in biological systems. The architecture of an ANN is usually made up of several layers where the output of the first layer becomes the input of the second layer.

Image recognition AI that should differentiate between cars and bicycles initially only roughly sorts the images according to the outline and then separates them layer by layer according to further criteria.  The research team introduced the GDV (generalized discrimination value) numerical value, which indicates how well the input data has been separated into classes. ‘The optimum layer depth is achieved when no further significant changes to the value are present,’ explains Dr. Krauss. Each additional layer does not improve the hit rate, but only increases the computation time.

Learning with databases

Artificial neural networks learn using image databases such as the Modified National Institute of Standards and Technology Database (MNIST). It contains 60,000 handwritten numbers from 0 to 9 that can be sorted into ten classes. The researchers were able to determine an optimum layer depth of four using this dataset. Eight layers are required, on the other hand, to differentiate between the grayscale images of ten different types of clothing in the Fashion-MNIST database. The optimum layer depth of the CIFAR-10 database, which contains 6,000 colour images each of ships, aircraft, trucks, cars, horses, deer, dogs, cats, birds and frogs, is 15. ‘The more complex the training data, the more layers are required for effective classification,’ says Patrick Krauss.

The link between the progression of the GDV and the increase in classification performance per layer was made visible by using dots on one layer when the layers were activated. Each dot represents a specific input into the neural network and the ten different classes were labelled with ten different colours. The deeper the layer depth, the more the dots grouped together into clusters of the same colour. ‘This new method enables us to compare different AI systems with each other, in order to find the most efficient architecture for a given problem,’ says Dr. Krauss. It could also be used to analyse which methods the AI system uses to order and represent the input data. The method could also become an important tool for neuroscience. ‘Our method enables us to quantify how well a specific model describes brain function.’

Further information:

DOI: 10.1016/j.neunet.2021.03.035
Dr. Patrick Krauss
Head of Cognitive Computational Neuroscience (CCN)
patrick.krauss@uk-erlangen.de