And you’ve been asked (by your imaginary boss) to either:Describe how birds, in general, look like;Explain what it is that makes a bird, a bird; orPredict what the next bird would look like at any given place or time.You’d probably observe what a bird, or several birds look like, and make a few generalisations about the bird.Suppose next, that you come across a flock of crows and you conclude the following:All birds have two legsAll birds have beaksAll birds have feathersAll birds are blackCongratulations!.You’ve built your first model in the bird-verse.Of course, one doesn’t have to be a bird expert to know that “all birds are black” is an incorrect conclusion, but remember — you’ve never seen a bird before!.Your model is overfitted, where you’ve picked out too many details from the crow and tried to generalise it to an entire population of birds..You wanted to describe birds, but ended up describing crows.After getting a telling-off from your boss, you return to the bird-verse and build another model..“Keep It Simple, Stupid” came the advice..You then observe some other birds and conclude:All birds have feathersDame Edna has feathers..Is she a bird?That’s not quite right either..We’ve avoided the problem of making generalisations based on idiosyncratic features of some birds..However, we’ve gone too far by making a single declaration on what we presumed was a universal characteristic amongst birds.Not unreasonable by any stretch, but still wrong.That’s when we say a model is under-fitted..You’ve only taken one characteristic, but it doesn’t return a consistent prediction or explanation of what birds are or would be..In this case, you wanted to describe birds, but you ended up catching every one with feathers.Let’s try and bring this back to stats and machine learning parlance..In the first model, we’ve reduced the bias in explaining a local sample of birds, but there’s more variance when generalising to another sample.In the second model, we’ve gotten a low variance predictor that performs well when generalising to more bird samples, but faces more bias when confronted with a sample of stuff that has feathers, but is clearly not a bird!We build models everydayWe gravitate between over-fitted and under-fitted models, iterating through various models of how things function until we come to a satisfactory understanding of how things work.. More details