Hulu DevX Console
Role
Lead UX Researcher
+ UX Designer: Onboarding Flow
Tools
Figma
Mural
Validately
Optimal Workshop
Team
Randy Helmcamp
Hannah Jaffe
Andy Lin
Yu Tiffany Morimoto
Introduction
Behind the scenes, the DevX Team at Hulu plays a key role in building tools for developers and ensuring that everything runs smoothly. Some of the guiding principles of the DevX Team are to spark joy for customers, build on top of industry standards, and design for the many, not the few.
In order to complete their daily tasks, the DevX Team juggles numerous tools per day, and sifts through thousands upon thousands of documents related to those tools. The team was in need of a simplified, more efficient workflow—and that is where we came in. We sought to consolidate documentation, provide an agile way to switch between tools, and create an onboarding process for new hires, all in one unified web portal.
Methodology
Discover
Comparative Analysis
Tool Research
Surveys
Virtual Shadowing
User Interviews
Define
User Personas
User Journey Map
User Flows
Card Sort
"How Might We...."
Develop
Sketches
LoFi Wireframes
Questionnaire
Iterate
Deliver
LoFi Prototype
Usability Test #1
HiFi Prototype
Usability Test #2
More iterations
Final Prototype
Overview
Prior to beginning work on the DevX Console, we had very little insight into the workflow of developers at Hulu. Therefore, it was important for us to deep dive into the day-to-day lives of our users, discovering what they value and what was missing from their current work model. We explored other existing developer consoles, conducted surveys, facilitated user interviews, and virtually shadowed Hulu developers. With these research findings in mind, we defined user personas, user flows, and journey maps. We then conducted a card sort, and established “How Might We…” questions that served as guides as we moved into developing and delivering our final product.
Comparative Analysis
As we began the research phase, we sought to familiarize ourselves with consoles that developers already use and know, taking a closer look at Amazon’s AWS, Google Cloud Platform, and DigitalOcean. Among these three existing consoles, we observed that a strong search functionality was a prominent feature across the board. Other notable features include the service dropdown on AWS, and the ability to rate the helpfulness of guides on DigitalOcean.
Learning from Our Users
Surveys
We conducted a preliminary survey to get to know the DevX team, gain insight into which tools each team uses, and understand what our users expected to see in a developer console.
The survey results gave us a high-level overview of the workflows of members of DevX, which we used to inform our user interview script and approach virtual shadowing sessions moving forward.
.png)
Virtual Shadowing
We observed three members of the DevX team during one hour screen sharing sessions over Zoom. Participants were asked to think out loud, giving us a glimpse into their thought processes as they performed their daily tasks. We observed a software developer, an engineering manager, and a principle software development lead.
User Interviews
We conducted five one-hour user interviews with the DevX team over Zoom. Our sessions focused on their work lives, their positions at Hulu, the tools and resources they use, their memories of onboarding as new hires, and their experiences using developer consoles. We interviewed three software developers, one senior software developer, and a principle software development lead.
What did we discover?
The 15 responses revealed that members of DevX juggle multiple tools daily, and that the most commonly used tools vary greatly by team. Respondents noted that they hope to see a minimalist console with “not a lot of clutter”, and that supplemental reading and documentation would also be helpful to aid their time as a Nulugan (a new hire at Hulu).
All three developers heavily relied on search, whether this was through Apple’s spotlight search, CMD+F, or a web app’s universal search. Additionally, at least eight tools were used during each one hour session, and all participants switched frequently between tools. As each participant performed their tasks, all three would intermittently turn to Slack and documentation to troubleshoot.
Each participant had different ways of working and unique problems to solve. To organize their tools and documentation, participants took varying approaches: one relied on organized bookmark folders, another had numerous bookmarks without any sub-categories or folders, and the final participant memorized all of the URLs that they would need to access. Participants also diverged in the types of tasks they were completing and the arsenal of tools they were using during each one hour session. Moving forward, it was important for us to be mindful of the diverse workflows and needs of our users.
Many of the key findings and patterns learned from the survey results and shadowing sessions were confirmed and expanded upon during the five user testing sessions. Interviews revealed that a strong search functionality was still the most important feature for participants, followed by customization and the desire for a minimalist web portal that unifies content, tools, and documentation.
Sorting through internal documentation was a considerable pain point for our users. All participants mentioned that the Hulu Wiki (an internal reference that allows employees to manage and read documentation) has too much irrelevant documentation, requiring users to sift through thousands of documents to locate what they need. Moving forward, participants hoped for a developer console that would serve as a hub for organized documentation and guides.
Synthesizing Findings
After learning more about our users through independent research, shadowing sessions, surveys, and interviews, we began to synthesize our findings. We created a Mural board and worked together to find patterns, pinpoint who our users were, sort what they liked and did not like about their current tools, and uncover what an ideal developer console would look like for Hulu’s DevX team.
User Personas
From our research findings, we crafted three user personas: a Nulugan, a team lead, and a seasoned software developer.
User Journey Maps
We then drafted three user journey maps, which helped us to visualize the successes and struggles encountered by our personas while completing their daily tasks.
How Might We…
Keeping the challenges and needs of our users in mind, we brainstormed a list of “How Might We...” statements, which gave us direction as we ideated and moved forward through the design process.
Provide efficient tool switching to simplify developers' workflows?
Improve communication across dev teams and point people towards the right resources?
Organize and filter documentation so that it’s relevant and up-to-date?
Show how applications and connected and related to one another?
Decrease cognitive load when it comes to identifying the right resources?
Ideal Features
We clustered and organized the needs, desires, and pain points of our users, which provided us with insight into the features an ideal DevX Console should have.
A Central Hub
The console should serve as a hub for the most important and relevant documentation
Prominent Search
In response to the team's frequent use of search while completing their daily tasks
Customization
Adapts to the specific and varied needs of our users
Simple Design
A clutter-free, minimal homepage
Notifications and Statuses
Affords contextual insight into services and reported downtime. Keeps team members informed about deployments and updates
Documentation
The console should serve as a hub for the most important and relevant documentation
Transparent and Clear
Clearly categorizes all services, providing a bird’s eye view of all features the console offers
Tool Mapping
Helps to paint a picture of how tools interact with and are connected to one another
User Flows
Next, we created user flows for accessing documentation, tool switching, and onboarding.
Card Sort and Impact/Effort Matrix
After synthesizing our findings, we conducted a card sort, which gave us further insight into which features the DevX Team found most important. From there, we created an impact/effort matrix, which visualized and ranked the most impactful features, and how much effort it would take to implement each feature.
Overview
After we gained a deeper understanding of our users, we began wireframing and prototyping the console, conducting usability tests along the way. Our final deliverable implements a new onboarding process, streamlines documentation, and facilitates more efficient tool switching. Our goal was to create a DevX Console that serves as a unified web portal and answers the needs of a diverse team of individual developers at Hulu.
Sketching and Ideating
We began visualizing our ideal developer console features by drafting some preliminary sketches for our stakeholders, providing them with a more tangible glimpse into our direction moving forward. We illustrated a simple, minimalist homepage, prominent search functionality, tool shortcuts, and a dropdown that displays all available services offered within the console. Although our design evolved throughout the process, the core features and ideas represented in these preliminary sketches remained consistent.
Low-Fidelity Prototype, Usability Testing #1
Prototype Overview
Next, we moved from pencil and paper to Figma. For our first set of low-fidelity wireframes, we placed our focus on the onboarding process, homepage, service dropdown, tool shortcuts, and documentation. The service dropdown brought its own set of challenges as we explored how to organize a widely extensive list of tools and resources into intuitive categories. We knew that this particular feature would be crucial to explore further with our users as we moved into prototypes and usability testing sessions.
First Round of Usability Testing
Four developers from three different teams participated in our first round of usability testing. These testing sessions were 30 minutes each, and were conducted on Validately. Some of our key findings were:
During the onboarding process, all participants wanted to ensure that Nulugans were informed without feeling overwhelmed. Overall, onboarding received positive feedback. A Nulugan participant felt that the process was useful, easy, and did not take too long to complete. Two team leads mentioned that it would be helpful to include short descriptions of each tool so that new hires could be provided with more context and a high-level overview of their tools. One team lead noted that the onboarding process is “a great concept/aspiration from a UI perspective.”
Participants heavily relied on search. On the homepage, all participants went directly to search when tasked with locating a page within the console, which confirmed for us that placing it front-and-center was a good choice for our users.
All participants had varying, unique feedback and ways of approaching the service dropdown. Two participants specifically noted that there should be a distinction between “Guides” and “Documentation.” Additionally, the placement of “Recently Viewed” on the left was not intuitive for our users, as it placed the actual services too far right from the hamburger icon. We also must be mindful of the speed in which some users complete tasks and navigate through the console: for instance, one user only scans the first word of each navigational item (thus, for “Documentation by Tool” and “Documentation Guides”, all he saw was the word “Documentation”).
Participants are more likely to rate documentation positively. In this LoFi wireframe, we provided users with the ability to give feedback and rate each documentation out of five stars. Three out of four users noted that they would be more likely to only give positive ratings. Users who primarily give positive feedback mentioned that they would be more likely to use an alternate comment functionality for giving feedback and suggesting changes on specific sections.
High-Fidelity Prototype, Usability Testing #2
Prototype Overview
Next, we began working in high-fidelity, creating a vibrant, more expansive prototype with 27 screens. In addition to the previous features, we created a distinction between guides and documentation, added a history page, built out a bookmark feature, and implemented tool mapping and an org chart.
Second Round of Usability Testing
Five developers from four different teams participated in our second round of usability testing. These testing sessions were one hour each, and were conducted on Validately. Some of our key findings were:
The words “friendly” and “welcoming” were used frequently to describe the onboarding process. A few participants specifically noted that the addition of preferred pronouns is positive. Moving forward, we discovered it would be important to delineate which input fields are not changeable as they are part of Okta/Huluverse (Okta is an identity cloud that Hulu uses, and some of its user information will be automatically imported into the DevX Console).
The homepage was widely regarded as being both modern and informative by most participants. When opening the homepage for the first time, one user exclaimed, “Slick!” It is important to note that participants had different expectations regarding which sections should come first; one participant mentioned that it would be helpful if she could drag and drop sections and customize their order. Looking ahead, implementing more customization could bolster the homepage so that it better meets the needs of our unique, diverse group of users.
Users would like to see more granularity in refine your search (refining search by tools, team, code language, etc.), and they would also like to see the implementation of tags on documentation and guides (which would serve as another way to filter search results). One user specifically mentioned wanting to see pinned tools at the top of search suggestions because he “would use search first anyway.”
Our users appreciated having an overview of everything available in one place in the service dropdown, making it easy to utilize CMD + F to locate elements within the menu (as learned during shadowing sessions, our users heavily rely on search and frequently utilize CMD+F). A pain point for participants was completing tasks where they had to navigate through documentation by function; a recurring pattern among users was the request to sort documentation by team as a more intuitive alternative.
Users reacted positively to the changes we made to documentation and guides, and specifically welcomed the addition of in-line comments and suggestions. In more than one testing session, users emphasized Hulu culture of developers empowering and trusting fellow developers—with this in mind, giving users the option to edit smaller typos directly, in addition to in-line comments, would fit in with the principles of the DevX team.
Overview
After our final round of usability testing, we continued to iterate on our design. In the end, we streamlined the onboarding process, optimized the service dropdown to be more intuitive for users, improved the tool pinning process, developed features that facilitate further customization, and created channels for developers to discover and connect.
View Final PrototypeOur Users Said...
It seems like the dev console would make my life easier.
Maybe it’ll become my new homepage!
Key Screens
Onboarding, Welcome!
Onboarding, What Are Your Areas of Expertise?
Homepage, which features a prominent search to accommodate common user behavior discovered during the research phase
Service Dropdown, which provides access to all tools and services in one place
Guides and Documentation pages provide users with the ability to leave comments (qualitative feedback), and rate the usefulness of guides (quantitative feedback)
Search Results, including "Relevant Team Members", so that users can reach out to those who specialize in their search query for troubleshooting and help
Concluding Thoughts
Going into this project, we had a lot to learn about the day-to-day workflow of developers. After speaking with the DevX team, observing them at work, and listening to their input along the way, we fostered a sense of understanding of who our users were and what would make their lives easier. Our end product serves as a unified web portal that answers the needs of a diverse group of developers at Hulu.
