A different kind of (deep) learning: part 2Self Supervised learning: generative approachesGidi ShperberBlockedUnblockFollowFollowingDec 19IntroIn the previous post, we’ve discussed some self supervised learning articles, along with some attempts to strive towards the “holy grail”: exploiting the almost unlimited number of un-annotated images available wherever to generalize for other tasks..And hopefully, get closer to the currently unmet benchmark of ImageNet pre-training.Surprisingly, or perhaps not so surprisingly, we’ve got some extra tailwind from Yan Lecun, which devoted a few minutes of his NeurIPS talk (“The Next Step Towards Artificial Intelligence”) to self supervised learning..He described self supervised learning as “the body of the cake” when the topping is supervised learning, and the cherry is reinforcement learning (because the sparsity of the reward in RL)..Lecun also takes some slides from our favorite self supervised researcher, Aloysha Efros, so I’m happy we share interests.Additionally, a few readers have pointed out that there are also prominent self supervised body of work dealing with videos, which is evident..However, videos will be discussed in a later post, since we have another topic, which is the generative models.What’s generative models have to do with self supervisionIn his talks about self supervision, Efros (yes, he is going to be dominant in this post as well) frequently discusses the difficulty in finding the right loss function for self supervised tasks.In the previous post we examined the special classification loss used for the colorization task, and emphasized the difficulty of finding the right loss function for them.In the talk, Efros described a method for finding such loss functions..He called it: “graduate student descent”..In other words, there is a lot of trial and error in finding a good loss function for these models..So can we have some better way, more universal, of finding them?colorization, super resolution, etc: is there a universal self supervised loss function?Additionally, there is the colorization Turing test thing: to evaluate the results, researchers use mechanical Turks to tell between real and fake photos..So wishfully, we would like to have some kind of mechanism to tell between these two types of images.If you were into deep learning back in 2014, you probably remember that when Ian Goodfellow presented his groundbreaking GAN work for the first time, the community was very excited about the promising generational abilities, but many researchers were skeptical about the purpose of this work..To them, it was merely a toy, at least until some significant progress to be made.The self-supervised researchers had some different thoughts: The GAN, in their eyes, was potentially a custom loss function for the self-supervised tasks.This is how Generative Adversarial network worksLet’s think about it for a second: in the colorization work, we’ve used standard deep learning paradigm for predicting color for each pixel..Can we use the power of GAN discriminator as a custom loss?.if so, it will require structuring the problem in a different way.We know that GAN in its essence generates images from a completely random distribution.. More details