A Look Into Snorkel DryBellGoogle’s Machine Learning Model that Labels Data by Learning About Your OrganizationJesus RodriguezBlockedUnblockFollowFollowingMar 15The creation of high-quality training datasets is one of the main limitations of machine learning applications in the real world.
Effectively annotating and validating training datasets typically results on time-intensive exercises involving domain experts.
Additionally, considering that machine learning models often need to be retrained, labeled datasets need to be refactor on a regular basics.
Recently, Google partnered with Stanford and Brown University on a research paper that introduces Snorkel DryBell, a weak supervision model to label training datasets using existing knowledge from an organization.
Snorkel DryBell draws its inspirations from the emerging field of weak supervision models.
Conceptually, weak supervision(also known as noisier or higher-level supervision) techniques attempt to automatically generate supervision signals with minimum effort from domain experts.
A way to think about weak supervision techniques is as a tradeoff between accuracy and scalability.
Traditional data labeling approaches that rely on humans can result very accurate but really hard to scale.
Weak supervision models prioritize scalability and volume while sacrificing accuracy.
The idea of weak supervision models is nothing new but Snorkel DryBell adds a few major innovations.
For starters, Google’s Snorkel DryBell builds on weak supervision techniques to leverage organizational knowledge to effectively labeled training datasets.
To achieve that, DryBell builds on Stanford’s Snorkel framework which was designed to support different types of weak supervision models for data labeling.
SnorkelCreated by researchers at Stanford University, Snorkel is an implementation of data programming paradigms for weak supervision training models.
Snorkel uses a set of programmable labeling functions express different weak supervision strategies and then generates a model based on the effectiveness of the different strategies.
In that context, Snorkel streamlines the job of a data engineer by creating a weak supervision model that can be used to build an effective training dataset.
The Snorkel workflow is divided in three main stages:I.
Writing Labeling Functions: In this phase, data engineers author labeling functions that express various weak supervision sources such as patterns, heuristics, external knowledge bases, and more.
II.
Modeling Accuracies and Correlations: After the labeling functions are ready, Snorkel learns a generative model which estimates specific accuracies and correlations.
The generative model is essentially a re-weighted combination of the user-provided labeling functions.
III.
Training the Discriminative Model: In this stage, Snorkel produces a set of probabilistic labels that can be used to train a model machine learning model.
The key to an effective Snorkel solutions is to author effective label functions.
For DryBell, Google found an innovative approach for introducing these weak supervision methods.
Snorkel DryBellUsing the Snorkel framework as the foundation, Google incorporate labeling functions that leverage organizational knowledge in order to effectively label data sources.
Conceptually, Snorkel DryBell was based on three fundamental principles:1) Bringing All Resources to Bear: Creating weak supervision models that integrates with many resources across an organization.
DryBell supports integration with different resources such as topics, rules or heuristics available in an organization.
2) Non-Servable Knowledge to Servable Models: Organizational knowledge is often present in non-servable form factors, i.
e.
, too slow, expensive, or private to be used in production; instead, a weak supervision system can provide a way to use these to quickly train servable models suitable for deployment.
3) Decoupling User Interaction from Execution: A weak supervision system should cleanly decouple subject matter experts(SMEs), who should be able to rapidly and iteratively specify weak supervision, from the details of execution and model training over industrial scale datasets.
To enable the first principle, Snorkel DryBell provides an architecture that incorporate labeling functions based on a MapReduce template pipeline.
Each labeling function takes in a data point and either abstains, or outputs a label.
The result is a large set of programmatically-generated training labels.
However, many of these labels resulted to be very noisy (e.
g.
from the heuristics), conflicted with each other, or were far too coarse-grained (e.
g.
the topic models) for the target task, leading to the next stage of Snorkel DryBell, aimed at automatically cleaning and integrating the labels into a final training set.
The second step in the Snorkel DryBell workflow entails to resolve inaccuracies from the labeling functions.
To achieve that, DryBell combines the outputs from the labeling functions into a single, confidence-weighted training label for each data point.
The framework uses a generative modeling technique that learns the accuracy of each labeling function using only unlabeled data.
This technique learns by observing the matrix of agreements and disagreements between the labeling functions’ outputs, taking into account known correlation structures between them.
The final step of the DryBell pipeline entails transferring non-servable knowledge to servable models.
In many scenarios, the labeling functions might provide features that are too slow or expensive to use in production but still contain valuable information.
Snorkel DryBell uses an innovative cross-feature transfer that allow users to the labeling functions — i.
e.
express their organizational knowledge — over one feature set that was not servable, and then use the resulting training labels output by Snorkel DryBell to train a model defined over a different, servable feature set.
Putting all the pieces together, Snorkey DryBell provides an architecture that enables an end-to-end labeling workflow.
It all starts with a C++ library that defines a MapReduce pipeline for executing a labeling function with the necessary services, such as natural language processing (NLP).
Using that framework, engineers write methods for the MapReduce pipeline to determine a vote for each example’s label, using Google resources.
At runtime, Snorkel DryBell executes the labeling function binary on Google’s distributed compute environment.
The results of all labeling functions are loaded into its generative model, which combines them into probabilistic training labels that are consumed by production systemsGoogle tested Snorkel DryBell across different such as product, topics and event classifications.
The details can be found in the research paper but the results comparable with hand-labeled methods and much more scalable.
Weak supervision models is an emerging area of research in the deep learning ecosystem.
Frameworks like Snorkel provide the foundation for the implementation of these methods and efforts like DryBell validate its effectiveness at scale.
Certainly this is an area that will contribute a lot to the implementation of large scale machine learning solutions.
.. More details