How might we make user research more efficient at ecobee?
Understand the pain points in the current research workflow and design an Insight Portal that enables better research insight documentation and communication across product lines
As the demand for user research increases at ecobee, the current research team found it challenging to consistently document, retrieve, and share research insights across the growing product team. When the product team asked “what do we know about [x]?”, it is difficult for researchers to find relevant research on existing platform.
Build an Insight Portal where researchers, UX designers, and PMs can better store, retrieve, and share research findings across product lines
Define and operationalize the research workflow (span the entire research lifecycle) for the product team
Automate the scheduling and recruitment process and set up resources for research best practices
About ecobee Inc.
ecobee Inc. is a smart home product company with ~500 employees that creates smart thermostats and light switches with built-in voice assistant that help millions of consumers save money, conserve energy, and seamlessly bring home automation into their lives. ecobee just wrapped up a $61 million Series C in May 2018, bringing the Toronto-based company’s total funding to $146 million.
Discovery & Frame
I spent the first month conducting user interviews to understand the needs, pain points, and challenges of the current research workflow. I spoke with 15 stakeholders, including various user researchers, UX designers, product managers, lab managers, content strategists, and VP product in the product team. Moreover, I also sat in some of the research meetings and followed through research initiatives to closely observe their research workflow and experience from different stakeholders and product lines.
After gathering data, I organized the notes as individual jobs-to-be-done statements for each research phase. I also also identified which stakeholder was involved in each phase along with corresponding pain points, opportunities, and context of the action. Based on the list of jobs-to-be-done statements, I then created a journey map that reflected the emotional ups and downs during the research workflow for the various stakeholders.
This framework of organizing research data was very helpful because the jobs-to-be-done method allowed me to define different tasks each stakeholder needed to perform, while combining these jobs-to-be-done statements in a journey map allowed me to see an overview of the task flow for various roles within the product team. It became easier to identify which task or stakeholder required specific research support in the overall workflow, and it also revealed cross-functional relationships within the research process.
current-state workflow journey map
Reframe the problem space
Although the stakeholder interviews were initially conducted to understand the team’s need for a research insights database, the research data actually revealed more underlying problems about the current research process and accountability. In collaboration with the user research team, we had a roundtable discussion to further investigate existing gaps in research operations from the jobs-to-be-done journey map.
As a result, we validated the need for a research database for product team and we have also identified two key insights that encompass the overarching theme of our data, which helped us reframe the initially abstract problem space and specify unmet needs of the product team.
What is the purpose and value of creating an Insight Portal?
What would be the evidence that prove its value?
Based on Google’s HEART framework of user-centered metrics and Sharon’s concept of Key Experience Indicators (KEIs), which provides a quantitative score of a specific, important, and actionable phenomenon related to using a product or service, it will be valuable to focus on measuring the happiness, adoption, and task success dimensions of the Insight Portal’s user experience in this case.
Why and how to focus on happiness, adoption, and task success dimensions of KEIs?
Concept & Design
What are the possible solutions of addressing these problems?
Inspired by Tomer Sharon’s Atomic Research and the concept of managing research knowledge that redefines the atomic unit of a research insight as an insight nugget, I begin to see the value of designing a research repository that documents insights as “insight nuggets” with proper tags to enable better discoverability, consistency, and connectivity between individual research findings. An insight nugget is a tagged observation supported by evidence.
First iteration: What if the Insight Portal looks like [x]?
In the first round of design, I experimented with a database structure that would gather all information needed for the product team. For research insights documentation, I had generative insights and usability issues placed under two sections as the type of findings are very different between generative and evaluative research. I have also designed the information architecture ranging from the most abstract (behavioural pattern and insights) to the most specific (user profile): with a descending order of generative insights, usability issues, studies, sessions, surveys, and users.
Iteration & Evaluation
First Round of Usability Testing
After the design of the initial database structure is completed, I input current research data from existing documentation platform to experiment with its usability. I tested its user flow with researchers in the team and there were several major problems with the current database structure:
It was difficult to navigate through the “Insight” section to find relevant research because the listing of insight nuggets was not organized by any theme. For a first time user who was not familiar with any key words to begin with, they did not know where to start.
Many researchers indicated that it took too many steps to input data from the administrator’s (mostly researchers) side. Although Airtable could link content between sheets, when the database structure followed the “insight-study-session-user profile” logic, it was easy for “visitors” (the product team) to browse through, but complicated for “administrator” (mostly researchers) to organize and link information between many spreadsheets.
There was also privacy issues with directly linking certain insights with user profiles and relevant demographic information.
Based on the insights and feedback from the first version, I reprioritized the database structure by conducting a participatory workshop with several researchers and designers that involves both mind mapping and brainstorming activities, which helped to understand how potential users may value different types of information on the portal. The mind mapping exercise and the participatory workshop helped me refocus on the most critical components of a research repository. I also further probed into the field of information architecture and data science to understand how best to manage “many-to-many” data relationship.
What has changed between the first and the second iteration? Why?
New features/concepts added to the Insight Portal:
Introduced “Top Keywords” as a tag system that organized insight nuggets with different themes and product lines for better discoverability and search experience, especially for first time users
Positioned “Session” as a Junction Table that connects the “Insight”, “User” and “Finance/Legal” data into one centralized place
Introduced “Session ID” to protect user privacy and it was used as a thread in the junction table
Separated the compensation and consent information from “Session” and moved them to a new spread sheet called “Finance/Legal” because this portion can be automated in the future as the logistics part of the process
Features removed from the first version Insight Portal:
Removed “Study” because it contains repetitive information with “Session” and creates extra steps for data input
Removed “Usability Issue” and “Generative Insights” and combined them in the “Insight” section as different types of insight nuggets
Removed “Surveys” and added survey information to “Session” as one type of research methods
Ideate future-state Research Workflow
Together with the user research team, we had a roundtable discussion and defined an ideal research workflow that the ecobee product team aspire to achieve and adhere to. With this proposed research workflow, each role in the product team understand the overall research process and their respective role in the workflow.