In this series, we will hear from visualization researchers on how to evaluate visualization quality. — MVHow do you know which visualization is better?.Mental effort is one cognitive process that we could use as an objective and generalizable measure of visualization quality..However, using speed and accuracy as proxies for mental effort is a very indirect method for measuring a mental process.As an alternative approach, I’ll describe methods that cognitive scientists have developed for more directly testing mental effort or working memory..The methods for measuring working memory, (detailed in the next section, have better construct validity than Vessey’s approach — meaning that they more closely measure what they claim to measure.Testing working memoryWithout getting too into the cognitive weeds, one function of working memory is as an indication of mental effort, which can be closely approximated with neuroimaging methods (increased blood flow to parts of the brain signals mental effort), pupillometry (our pupils dilate when we are thinking hard), individual differences (some people have a greater ability to sustain mental effort than others), and dual-task experimental designs..Working memory is the glass which is capacity limited..Visualization X requires some amount of mental effort to understand, depicted below as beer..Visualization Y is less intuitive and requires more mental effort (utilizing more of the glass’ capacity) than visualization X..We don’t know exactly how much mental effort either require but we hypothesize that visualization X is more intuitive than Y..To test if X requires less working memory capacity than Y, we have users complete a secondary task (such as mental arithmetic, denoted below as a bomb shot), which also requires mental effort..When doing a hard secondary task, we may find that people will exceed their capacity and their performance on the primary visualization task will tank when using visualization Y but not for X..Researchers have validated dual-task experiments with neuroscience methods and show that you can use dual-task findings to closely approximate working memory demand.Closing thoughtsThe moral of the story here is that one approach for deciding which visualization is better (with significant merit in terms of generalizability), is to compare how much mental effort is required to use each visualization effectively.. More details