GANs Demystified — What the hell do they learn?

Stochastic gradient descent is then used to maximize the objective function (ACE) and minimize the number of units in the optimal set (By using L2 regularisation term for alpha).L2 regularisation term is added to minimize number of units in the optimal setThe figure below shows how ablating units that cause trees successfully removes trees from the generated image.Ablating units to remove treesResults in a nutshellDifferences between layers: Based on the results, first layer doesn’t match well object classes..This makes intuitive sense since early layers usually represent primitive features..Mid layers have many units that match semantic objects and object parts..Ending layers match local pixel patterns such as materials, edges and colors.Differences between GAN architectures: The paper also discusses how different architectures compare in internal representation by examining the units.Contextual Relationships via Insertion: The paper also describes insertion of features into specific locations of the generated image by forcing units..One interesting observation is that an object like door can only be inserted at specific locations where it visually makes sense (e.g buildings near windows, bricks)..This suggests that GANS infact capture relationships between objects, not simply pixel patterns.The above image illustrates how the generation of object classes is highly dependant on local context..The actual effect of insertion differs based on where it occurs.Diagnosing GANsBy using intervention technique, we can study the causation of erroneous / undesirable parts and artifacts in the generated images..Through manual annotation and ablating the causing units, results of your GAN can be considerably improved..The image below illustrates how ablating artifact causing unit produces visible results.Using this framework, you can manually tailor and influence the results of your GAN..It is relatively simple to enhance or reduce the presence of certain object classes (to an extent).Final ThoughtsThe paper makes an excellent attempt to demystify GANs, a relatively unexplored topic so far..Their framework enables researches and practitioners to better debug, improve and understand their models.The discussed debugging methods involves are mostly based on manual supervision..However, I believe the reasoning and techniques presented in the paper could potentially be used in developing more automated techniques to improve GANs (The paper discusses one such technique for automatic identification of artifact units).Credits: Ideas, formulas and images taken from the paper.ReferencesGAN Dissection: Visualizing and Understanding Generative Adversarial NetworksProject Official Website (With Github code, demo video, slides, e.t.c)Interactive GAN Paint Tool. More details

Leave a Reply