Trying to read this article is a complicated task from the neuroscientific standpoint.
At this time you are probably bombarded with emails, news, notifications on our phone, the usual annoying coworker interrupting and other distractions that cause your brain to spin on many directions.
In order to read this tiny article or perform many other cognitive tasks, you need to focus, you need attention.
Attention is a cognitive skill that is pivotal to the formation of knowledge.
However, the dynamics of attention have remained a mystery to neuroscientists for centuries and, just recently, that we have had major breakthroughs that help to explain how attention works.
In the context of deep learning programs, building attention dynamics seems to be an obvious step in order to improve the knowledge of models and adapt them to different scenarios.
Building attention mechanisms into deep learning systems is a very nascent and active area of research.
A few months ago, researchers from the Google Brain team published a paper that detailed some of the key models that can be used to simulate attention in deep neural networks.
In order to understand attention in deep learning systems it might be useful to take a look at how this cognitive phenomenon takes place in the human brain.
From the perspective of neuroscience, attention is the ability of the brain to selectively concentrate on one aspect of the environment while ignoring other things.
The current research identifies two main types of attention both related to different areas of the brain.
Object-based attention is often referred to the ability of the brain to focus on specific objects such as a picture of a section in this article.
Spatial-based attention is mostly related to the focus on specific locations.
Both types of attention are relevant in deep learning models.
While object-based attention can be used in systems such as image recognition or machine translation, spatial-attention is relevant in deep reinforcement learning scenarios such as self-driving vehicles.
When comes to deep learning systems, there are different techniques that have been created in order to simulate different types of attention.
The Google research paper focuses on four fundamental models that are relevant to recurrent neural networks(RNNs).
Why RNNs specifically?.Well, RNNs are a type of network that is mostly used to process sequential data and obtain higher-level knowledge.
As a result, RNNs are often used as a second step to refine the work of other neural network models such as convolutional neural networks(CNNs) or generative interfaces.
Building attention mechanisms into RNNs can help improve the knowledge of different deep neural models.
The Google Brain team identified the following four techniques for building attention into RNNs models:Attention is one of the most complicated cognitive abilities of the human brain.
Simulating attention mechanisms in neural networks can open a new set of fascinating possibilities for the future of deep learning.
Original.
Reposted with permission.
Related: var disqus_shortname = kdnuggets; (function() { var dsq = document.
createElement(script); dsq.
type = text/javascript; dsq.
async = true; dsq.
src = https://kdnuggets.
disqus.
com/embed.
js; (document.
getElementsByTagName(head)[0] || document.
getElementsByTagName(body)[0]).
appendChild(dsq); })();.