DeepMind Combines Logic and Neural Networks to Extract Rules from Noisy DataJesus RodriguezBlockedUnblockFollowFollowingFeb 27In his book “The Master Algorithm”, artificial intelligence researcher Pedro Domingos explores the idea of a single algorithm that can combine the major schools of machine learning.
The idea is, without a doubt, extremely ambitious but we are already seeing some iterations of it.
Last year, Google published a research paper under the catchy title of “One Model to Learn Them All” that combines heterogeneous learning techniques under a single machine learning model.
Last years, Alphabet’s subsidiary DeepMind took another step towards multi-model algorithms by introducing a new technique called Differentiable Inductive Logic Programming(∂ILP) that combines logic and neural networks into a single model to extract rules from noisy data.
∂ILP brings together two of the major machine learning schools.
Connectionists try to model knowledge by imitating representations of the brain in the form of neural networks and have been the driving force behind movements such as deep learning.
Symbolists rely on logic to model knowledge based on well-understood rules.
Both schools have well known advantages and drawbacks.
Symbolist systems based on inductive logic programming(ILP) tend to generalize knowledge efficiently and they are semi-immune to overfitting.
Also, ILP systems tend to be a great fit in transfer learning scenarios in which a trained model can be copied and reused in other models.
The main limitation of ILP systems is their struggle with noisy or ambiguous data which is so common in deep learning scenarios.
Connectionist systems tend to work well in environments with noisy data and can efficiently handle uncertainty and ambiguity.
However, they tend to be expensive to train and version.
Also, the knowledge learned from connectionist system is very hard to follow and understand which contrasts with the clarity of symbolist model.
For years, many experts have highlighted the theoretical value on combining robust connectionist learning with symbolic relational learning.
∂ILP is certainly a step on the right direction.
Conceptually, ∂ILP combines neural networks with ILP to provide a model that can process noisy and ambiguous data while also generalizing well and avoiding deterioration.
By combining the best of both worlds, ∂ILP is a technique that differs from connectionist models in the sense that can generalize knowledge symbolically while also differing from traditional symbolist models by generalizing knowledge visually.
The following matrix might help to illustrate the comparison between the three schools of thought.
To explain the capabilities of ∂ILP, let’s use a basic induction task in which a pair of images representing numbers, and has to output a label (0 or 1) indicating whether the number of the left image is less than the number of the right image.
Solving this problem involves both kinds of thinking: you need intuitive perceptual thinking to recognise the image as a representation of a particular digit, and you need conceptual thinking to understand the less-than relation in its full generality.
The task illustrated below is nothing particularly new and can be solved using a standard deep learning model such as convolutional neural networks(CNNs).
While a CNN model is perfectly capable of recognizing new images of digits it will most likely fail to identify the digits itself.
In other words, deep learning strategies such as CNNs are able to achieve visual generalization but not symbolic generalization.
∂ILP differs from standard neural nets because it is able to generalize symbolically, and it differs from standard symbolic programs because it is able to generalize visually.
It learns explicit programs from examples that are readable, interpretable, and verifiable.
∂ILP is given a partial set of examples and produces a program that satisfies them.
It searches through the space of programs using gradient descent.
If the outputs of the program conflict with the desired outputs from the reference data, the system revises the program to better match the data.
The results of the previous experiment show that ∂ILP is able to generalized both symbolically and visually outperforming a established baseline by other logic and deep learning models.
∂ILP is very creative approach to bring together two of the major tribes in machine learning.
Combining the intuitive knowledge of connectionist systems with the conceptual knowledge of symbolists is a step closer to emulate human cognition and, maybe, a step closer to Domingo’s master algorithm.
.