We looked into two white papers this week.
Crowdsourcing Graphical Perception
The paper wants to see how viable crowdsourcing is to assess visualization design. They employ Amazon’s Mechanical Turk.
How they assess the viability is by:
- replicating prior laboratory results and see if the crowdsourcing results complied to the prior laboratory results
- figuring out what is the best optimizing display parameters
- analyze the performance and cost of its use by comparing the cost and speed of crowdsourcing versus lab
The paper concludes that it is viable to use crowdsourcing.
(We will write more notes about this paper in the future.)
Our Discussion
One key takeaway is that we do not believe that people who participate in Mechanical Turk is diverse. They are more diverse than the students used in lab experiments but they are not diverse enough that they can be used as a target audience for all visualization design evaluation.
They have intersting takeaways:
- Using crowdsourcing, many more subjects can participate for the same cost as the lab’s (cost reduction). You are also abe to run better experiments because you can run more since you can complete the experiment faster. You have wider access to populations than you do in the lab.
- It will be great if operating system and monitor details can be recorded so that we can better infer what the subjects are seeing.
- For treemaps, comparing rectangles with aspect ratios
of 1 led to higher estimation error than other aspect
ratio combinations. - Gridlines should be spaced at least 8 pixels apart.
- Increasing chart heights beyond 80 pixels provides little accuracy benefit on a 0-100 scale.
- Expect significant subject overlap and unreliable response time when using crowdsourcing.
- By using qualification tasks and verifiable questions, one can increase the likelihood of quality responses.
- Increase payment level to quicken completion time.
- To facilitate replication, state the qualification tasks and completion rate.
Exploring the Impact of Emotion on Visual Judgment
The paper applies emotion to the crowdsourced user study performed in the previous paper above. They come to the conclusion that affective-priming can significantly influence accuracy in visual judgment.
How they performed the experiment is that 1. they measure the Mechnical Turk user’s emotion 2. followed by asking the user to read an emotional article that is either positive or negative (aka affective-priming) 3. then they test the user how well they accurately read a chart from one of the given 5 4. followed up lastly by having the user’s emotion measured again
Our Discussion
The chart is used to rank visual variables. The experiment does not take into account how well the user can read the chart to begin with. It makes the assumption that the user already knows how to read the chart accurately, and the emotion affects how the user reads the chart.