How might we make user research more efficient at ecobee?

 

Understand the pain points in the current research workflow and design an Insight Portal that enables better research insight documentation and communication across product lines


ResearchOps Intro@2x.png

Objective

Challenge

As the demand for user research increases at ecobee, the current research team found it challenging to consistently document, retrieve, and share research insights across the growing product team. When the product team asked “what do we know about [x]?”, it is difficult for researchers to find relevant research on existing platform.

My Role

  1. Built an Insight Portal where researchers, UX designers, and PMs can better store, retrieve, and share research findings across product lines

  2. Defined and operationalized the research workflow (spanning the entire research lifecycle) for the product team

  3. Automated the scheduling and recruitment process and set up resources for research best practices


About ecobee Inc.

Screen Shot 2018-10-27 at 6.28.01 PM.png

ecobee Inc. is a smart home product company with ~500 employees that creates smart thermostats and light switches with built-in voice assistant that help millions of consumers save money, conserve energy, and seamlessly bring home automation into their lives. ecobee just wrapped up a $61 million Series C in May 2018, bringing the Toronto-based company’s total funding to $146 million.

Screen Shot 2018-10-27 at 6.28.35 PM.png

Project Roadmap

Roadmap (Var)@2x.png

Timeline

Timeline (Var)@2x.png

Discovery & Frame

Stakeholder Interview

I spent the first month conducting user interviews to understand the needs, pain points, and challenges of the current research workflow. I spoke with 15 stakeholders, including various user researchers, UX designers, product managers, lab managers, content strategists, and VP product in the team. Moreover, I also sat in some of the research meetings and followed through research initiatives to closely observe their research workflow and experience from different stakeholders and product lines.

Jobs-to-be-done Statements

Screen Shot 2018-12-07 at 11.03.06 PM.png

After gathering data, I organized the notes as individual jobs-to-be-done statements for each research phase. I had also identified which stakeholder was involved in each phase along with corresponding pain points, opportunities, and context of the action. Based on the list of jobs-to-be-done statements, I then created a journey map that reflected the emotional ups and downs during the research workflow for various stakeholders.

This framework of organizing research data was very helpful because the jobs-to-be-done method allowed me to define different tasks each stakeholder needed to perform, while combining these jobs-to-be-done statements in a journey map allowed me to see an overview of the task flow for various roles within the product team. It became easier to identify which task or stakeholder required specific research support in the overall workflow, and it also revealed cross-functional relationships within the research process.

CurrentWorkflow_1.jpg
Current_Workflow2.jpg

Current-State Workflow Journey Map

Current-state Journey Map (Var).png

Reframe the Problem Space

Although the stakeholder interviews were initially conducted to understand the team’s need for a research insights database, the data actually revealed more underlying problems about the current research process and accountability. In collaboration with the user research team, we had a round table discussion to further investigate existing gaps in research operations from the jobs-to-be-done journey map.

As a result, we validated the need for a research database for product team and we have also identified two key insights that encompass the overarching theme of our data, which helped us reframe the initially abstract problem space and specify unmet needs of the product team.

Insights@2x.png

What is the purpose and value of creating an Insight Portal?

User Story@2x.png

What would be the evidence that prove its value?

Based on Google’s HEART framework of user-centered metrics and Sharon’s concept of Key Experience Indicators (KEIs), which provides a quantitative score of a specific, important, and actionable phenomenon related to using a product or service, it will be valuable to focus on measuring the happiness, adoption, and task success dimensions of the Insight Portal’s user experience in this case.

Why and how to focus on happiness, adoption, and task success dimensions of KEIs?

Metrics@2x.png

Concept & Design

What are the possible solutions of addressing these problems?

Inspired by Tomer Sharon’s Atomic Research and the concept of managing research knowledge that redefines the atomic unit of a research insight as an insight nugget, I begin to see the value of designing a research repository that documents insights as “insight nuggets” with proper tags to enable better discoverability, consistency, and connectivity between individual research findings. An insight nugget is a tagged observation supported by evidence.

First iteration: What if the Insight Portal looks like [x]?

In the first round of design, I experimented with a database structure that would gather all information needed for the product team. For research insights documentation, I had generative insights and usability issues placed under two sections as the type of findings are very different between generative and evaluative research. I have also designed the information architecture ranging from the most abstract (behavioural pattern and insights) to the most specific (user profile): with a descending order of generative insights, usability issues, studies, sessions, surveys, and users.

ArchivedPortal_Insight_Blurred.png
ArchivedPortal_Session_Blurred.png

Iteration & Evaluation

First Round of Usability Testing

After the design of the initial database structure is completed, I input current research data from existing documentation platform to experiment with its usability. I tested its user flow with researchers in the team and there were several major problems with the current database structure:

  1. It was difficult to navigate through the “Insight” section to find relevant research because the listing of insight nuggets was not organized by any theme. For a first time user who was not familiar with any key words to begin with, they did not know where to start.

  2. Many researchers indicated that it took too many steps to input data from the administrator’s (mostly researchers) side. Although Airtable could link content between sheets, when the database structure followed the “insight-study-session-userprofile” logic, it was easy for “visitors” (the product team) to browse through, but complicated for “administrator” (mostly researchers) to organize and link information between many spreadsheets.

  3. There was also privacy issues with directly linking certain insights with user profiles and relevant demographic information.

Second Iteration

Based on the insights and feedback from the first version, I reprioritized the database structure by conducting a participatory workshop with several researchers and designers that involved both mind mapping and brainstorming activities, which helped to understand how potential users may value different types of information in the portal. The mind mapping exercise and the participatory workshop helped me to refocus on the most critical component of a research repository. I also further probed into the field of information architecture and data science to understand how best to manage “many-to-many” data relationship.

Introducing “Session ID” to connect insights, user, and finance/legal data

Introducing “Session ID” to connect insights, user, and finance/legal data

Mindmap of the Essential/Prioritized Components of the Insight Database

Mindmap of the Essential/Prioritized Components of the Insight Database

The final design of the database “Session” that serves as a junction table

The final design of the database “Session” that serves as a junction table

What has changed between the first and the second iteration? Why?

New features/concepts added to the Insight Portal:

  1. Introduced “Top Keywords” as a tag system that organized insight nuggets with different themes and product lines for better discoverability and search experience, especially for first time users

  2. Positioned “Session” as a Junction Table that connects the “Insight”, “User” and “Finance/Legal” data into one centralized place

  3. Introduced “Session ID” to protect user privacy and it was used as a thread in the junction table

  4. Separated the compensation and consent information from “Session” and moved them to a new spread sheet called “Finance/Legal” because this portion can be automated in the future as the logistics part of the process

Features removed from the first version Insight Portal:

  1. Removed “Study” because it contains repetitive information with “Session” and creates extra steps for data input

  2. Removed “Usability Issue” and “Generative Insights” and combined them in the “Insight” section as different types of insight nuggets

  3. Removed “Surveys” and added survey information to “Session” as one type of research methods

Introducing “Top Keywords” section that serves as a tag system to provide better search experience for first time users

Introducing “Top Keywords” section that serves as a tag system to provide better search experience for first time users

“Insight” section of the  Insight Portal

“Insight” section of the Insight Portal

“Finance/Legal” section of the  Insight Portal

“Finance/Legal” section of the Insight Portal


Ideate Future-State Research Workflow

Together with the user research team, we had a round table discussion and defined an ideal research workflow that the ecobee product team aspire to achieve and adhere to. With this proposed research workflow, each role in the product team understand the overall research process and their respective role in the workflow.

Ideal-state Workflow.png

Automate Recruitment and Scheduling Process

Automation workflow.png