For tasks that humans are good at, you can compare your system’s performance to those of humans, which gives you an estimate of the optimal error rate.
In other cases, it is often hard to define an optimal rate, which is the reasons why you should work on problems that humans can do well, which we will discuss at the next concept.
Throughout the book, he explains several times why it is recommended to work on machine learning problems that humans can do well themselves.
Examples are Speech Recognition, Image Classification, Object Detection, and so on.
This has several reasons.
First, it is easier to get or to create a labeled dataset, because it is straightforward for people to provide high accuracy labels for your learning algorithm if they can solve the problem by themselves.
Second, you can use human performance as the optimal error rate that you want to reach with your algorithm.
NG explains that having defined a reasonable and achievable optimal error helps to accelerate the team’s progress.
It also helps you to detect if your algorithm is suffering from high bias or variance.
Third, it enables you to do error analysis based on your human intuition.
If you are building, for example, a speech recognition system and your model misclassifies its input, you can try to understand what information a human would be using to get the correct transcription, and use this to modify the learning algorithm accordingly.
Although algorithms surpass humans at more and more of tasks that humans can’t do well themselves, you should try to avoid these problems.
To summarize, you should avoid these tasks because it makes it harder to obtain labels for your data, you can’t count on human intuition anymore, and it is hard to know what the optimal error rate is.
NG also proposes a way on how to split your dataset.
He recommends the following:Train Set: With it, you train your algorithm and nothing else.
Dev Set: This set is there to do hyperparameter tuning, to select and create proper features, and to do error analysis.
It is basically there to make decisions about your algorithm.
Test Set: The test set is used to evaluate the performance of your system, but not to make decisions.
It’s just there for evaluation, and nothing else.
The dev set and test set allow your team to quickly evaluate how well your algorithm is performing.
Their purpose is to guide you to the most important changes that you should make to your system.
He recommends choosing the dev and test set so that they reflect data which you want to do well on in the future once your system is deployed.
This is especially true if you expect that the data will be different than the data you are training it on right now.
For example, you are training on normal camera images but later on your system will only receive pictures taken by phones because it is part of a mobile app.
This can be the case if you don’t have access to enough mobile phone photos to train your system.
Therefore, you should pick test set examples that reflect what you want to perform well on later in reality, rather than the data that you used for training.
Also, you should choose dev and test sets that come from the same distribution.
Otherwise, there is a chance that your team will build something that does well on the dev set, only to find that it performs extremely poor on the test data, which you care about the most.
In this post, you’ve learned about 6 concepts of Machine Learning Yearning.
You now know why it is important to iterate quickly, why you should use a single-number evaluation metrics, and what errors analysis is about and why it is crucial.
Also, you’ve learned about the optimal error rate, why you should work on problems that humans can do well and how you should split your data.
Furthermore, you learned that you should pick the dev and test set data so that they reflect the data which you want to do well on in the future, and that dev and test sets should come from the same distribution.
I hope that this post gave you an introduction to some concepts of the book and I can definitely say that it is worth reading.
Bio: Niklas Donges is a Machine Learning and Data Science enthusiast, studying Software Engineering at CODE University of Applied Sciences in Berlin, with a deep focus on Machine Learning, and working part-time for the Machine Learning Foundation of SAP.
Original.
Reposted with permission.
Related: var disqus_shortname = kdnuggets; (function() { var dsq = document.
createElement(script); dsq.
type = text/javascript; dsq.
async = true; dsq.
src = https://kdnuggets.
disqus.
com/embed.
js; (document.
getElementsByTagName(head)[0] || document.
getElementsByTagName(body)[0]).
appendChild(dsq); })();.