Designing an integrated workspace for the ad quality review teams.

The Problem

I moved from Singapore to join Yahoo! Bangalore in early 2009 as Principal Visual Interaction Designer. This was a lead designer role in the Advertising Products Group (APG) at Yahoo! The APG was tasked with addressing ad quality management issues across the Yahoo ad networks. The 200-strong internal teams of quality reviewers handled ad quality using close to 16 different standalone legacy tools to complete a single review task. The tools allowed them to weed out objectionable ad material like porn, gambling etc. from the networks.

However, a reviewer had to constantly switch between tools to accomplish various aspects of a single review task. In addition, most of these tools were outdated, inefficient and challenging in terms of usability. This affected productivity and task completion rates suffered.

Users & audience

Ad quality reviews on the Yahoo! ad networks were managed by an organisation of 200+ ad reviewers in India and South East Asia. Each team had 10 to 12 reviewers led by a manager. Review managers allocated ad queues to reviewers and tracked progress against SLAs.

Team & role

As the lead designer on the project, I anchored user studies to identify issues with core review workflows and observe user behaviour. I led collaborative design sessions with the cross-functional product, engineering and reviewer teams to design a robust integrated digital workspace for ad reviewers. I developed finished designs for the end-to-end UX including visual and interaction design.

Design process & insights

To discover the core issues the ad reviewers were facing, we conducted contextual enquiries. Along with the product manager, I set up 1:1 sessions with reviewers and review managers to understand key pain points and define strategies to address them. We then formed focus groups to run participatory co-design sessions. The cross-functional teams became active participants in exploring possible solutions to the most common issue that surfaced during the interviews – the constant switching between multiple tools to complete a review task.

Apart from several tools for queue management of different ad types, a single review task involved flagging objectionable content with the right tagging and then pulling it off the network with subsequent actions based on the assigned task. The subsequent actions could be assigning the item to a different reviewer or escalating it to the line manager and so on. For the key task of tagging, reviewers needed precise inputs and they relied on web-based tools to look up tags and research applicable codes. They landed up opening browsers with multiple tabs, using handwritten notes in spreadsheets and switching between these tools.

During the co-design sessions, which I facilitated, we identified key sub-tasks that can be consolidated into a single view and converged on the idea of an integrated review workspace. The team sketched out ideas together and voted on the ones that resonated with the team.

Since the end-users (reviewers) were part of the sessions and easily accessible, it saved me turn-around time from iterations to validation and was able to converge on the designs faster.

Design solution

We wanted to build an ideal-state vision of what the workspace should allow reviewers to do, and how it will improve their productivity, be more efficient and reduce errors. The designs key components of the integrated workspace were

Outcome & learnings

The final designs were well-received by our target audience. The reviewers and review managers were completely onboard and wanted to know, “when can we have it”. We then ran reviews with product and engineering leadership. At a conceptual level, the designs generated a lot of excitement and the team was aligned with the vision of the integrated review workspace. They saw challenges in building out the experience in one go. The next steps for the product, design and engineering team were to plan out incremental development of the various components and launch in stages as part of an MVP. I was able to see parts of the tool being developed and tested until I left Yahoo! in 2012. One of the key learnings from this project was that having a robust plan of execution at the outset helps bring a product vision to life, even if incrementally, over time. In hindsight, bringing leadership reviews earlier into the design process would have helped the team move faster.