AI Assistant | Privacy & Security | Communication Tool
Hiya had recently launched their AI-powered calling app with 4 key features: call summarization, spam detection, calendar event detection, and incognito (private calling) mode. The challenge was to evaluate how effectively users could navigate and use these features, while identifying points of friction in the user experience.
As a UX researcher and designer, I designed and conducted moderated usability testing sessions, created test plans and research protocols, recruited and screened qualified participants through userinterviews.com. After collecting insights, I collaboratively analyzed qualitative and quantitative user feedback and developed actionable design recommendations to Hiya's UX team.
Mobile App
Contribution
Systems Thinking
Usability Testing
Qualitative Analysis
Quantitative Analysis
Affinity Mapping
High-fidelity prototypes
Team
Yuri Kawada
Shreya Saxena
Yiyang Li
Gily Yu
Tools
FigJam
Figma
userinterviews.com
Timeline
10 weeks
Key Outcomes & Impact
We synthesized 15 actionable insights from usability testing to create 15 actionable design recommendations to improve user satisfaction, including information architecture improvements and UI enhancement opportunities. We developed pain point visualizations to effectively communicate findings to Hiya's UX team, leading to prioritized design improvements.
Key outcomes from these usability tests include user results and findings, design recommendations, and pre- & post-task analysis, aimed at enhancing Hiya’s user experience and product strategy.
What is Hiya?
Hiya is a leading caller identification and spam-blocking mobile application that empowers users to manage incoming calls through AI-driven analysis, spam detection, and AI-generated call transcripts. By offering both free and premium subscription services, Hiya collaborates with mobile carriers and businesses to enhance call security and establish user trust.
Research Questions
To conduct the study, we analyzed the recently launched application, formulated research questions, and developed hypotheses to guide the usability test design.
Ease of Use
Can users successfully navigate through Hiya’s key features (onboarding process, spam call detection, event detection, and incognito mode)?
Effectiveness
Usefulness
What specific challenges do users face when completing tasks in the app?
Effectiveness
Usefulness
Can users easily navigate and manage incoming and outgoing calls using the dial screen?
Trust
How willing are users to trust and provide information and allow Hiya to manage their calls?
Methodology
To evaluate Hiya AI Phone's usability, we implemented six complementary methods to capture both quantitative metrics and qualitative insights across key features.
Moderated Usability Testing
Real-time observations and qualitative feedback during guided tasks.
Likert Scale Questions
Collected quantitative ratings on a 5-point scale for satisfaction, ease of use, usefulness, and trust.
Task Analysis
Broke down complex interactions into discrete steps to identify specific pain points and inefficiencies.
Think-Aloud Protocol
Asked participants to verbalize their thoughts while using the app to reveal mental models and expectations.
First-Click Testing
Tracked users' initial interaction points to measure feature discoverability and navigation patterns.
Task Success Metrics
We recorded pass/fail criteria to evaluate task success.
Our target participants
We utilized a platform called "user interviews" to screen for participants that had a 100% match with our target user criteria. Out of the 50 that applied, we moved forward with those who fit our description of being busy business professionals who are early adopters of tech and first time Hiya app users.
Busy business professionals
Including independent business owners, home service workers, and real estate agents.
Early adopters of technology
First time Hiya app users
Business professionals who rely on frequent phone communication
Key Features and Tests
In order to evaluate Hiya's AI Application, we focused on 4 features to conduct usability tests on our participants - the onboarding process, event detection and call transcript, incognito feature, and spam call detection.
Planning a Test Launch Before Participants Recruitment
Usability Test Details
In total 13 moderated usability tests were conducted with participants recruited through UserInterviews.com, among them, 8 participants successfully completed a 65-minute session, providing in-depth insights into their experiences with Hiya’s core functionalities.
Testing Objectives
Our usability study aimed to evaluate:
Ease of onboarding and account setup
User comfort with privacy permissions
Effectiveness of key features
(call summarization, event generation, spam detection)
Overall user experience and feature discoverability
Tasks
Task 1: Onboarding & First-time Setup
First-time user downloading and setting up the Hiya app.
Success Criteria
Participants complete the full onboarding process without assistance and understand the purpose of each permission requested.
Task 2: Calendar Event Generation & Call Summarization
Answer a call to plan a meeting and review the call summary and auto-generated calendar event.
Success Criteria
Participants discover the auto-generated calendar event, add it to their calendar, and locate the call summary with accurate details.
Task 3: Incognito Mode
Make a confidential business call with privacy features enabled.
Success Criteria
Participants find and enable the incognito mode before making a private call and verify that no summary was generated.
Task 4: Spam Call Detection
Receive a potentially fraudulent call.
Success Criteria
Participants notice the spam detection indicators and refuse to provide sensitive information during the call.
Results and Findings
After analyzing data from our 8 usability tests, we synthesized participant feedback through qualitative analysis of session recordings and observer notes. We compiled key quotes and observations while quantifying user sentiment through Likert scale responses to identify patterns and prioritize findings by severity level.
Severity Levels
In analyzing our usability findings, we have put them into severity levels based on Nielsen and Norman Group’s 0 to 4 rating scale.
Finding 1: Trust Increases with Feature Experience
3 - Major
Data Analysis
All 8 participants rated their initial trust on the onboarding 3.5 out of 5, but this increased significantly after experiencing the features.
Limited exposure to the Hiya brand created initial hesitation during the permission granting process.
7 out of 8 participants stated they were "Pretty Comfortable" (≥4) with the permissions requested, but needed to see features in action to fully trust the app.
Users expressed that experiencing the functions directly helped increase perceived usefulness and trust.
With only the variable of "using the product" changing, post-test trust ratings increased to 4.8/5.
Quotes
"3. I know nothing about the Hiya app except for the limited exposure. Through repetition, I would be more comfortable with it." - P6
"4, haven't use it yet, need to have a test try." - P8
"Trust enough to try it, but will build trust after using it." - P7
"I don't have any reasons to trust or distrust so would rate it 3." - P9
Recommendation
Use mock calls to accelerate feature familiarization for new users.
Introduce test calls as part of the onboarding to help users quickly understand Hiya's key features in action.
This provides basic usability assurance, demonstrates that calls are successfully forwarded, and offers an organic opportunity to explore key features such as call summaries and transcripts.
Finding 2: Calendar Event Detection Highly Valued for Professional Use
2 - Minor
Data Analysis
All 8 participants (100%) found this feature useful in their professional life, giving it an average rating of 4.87/5.
Users successfully located and added events to calendars but wanted improvements to the system.
Users felt that the event message did not capture the ending time of meetings.
Several mentioned the feature would be more useful if they could add events to third-party calendar apps beyond just the system's default.
Some expressed the feature wasn't as useful for casual discussions with friends versus professional contexts.
All participants found this feature easy to locate with an average rating of 5/5.
Quotes
"If it were business-related, I'd find it useful. However, if it were for a friend, I'd consider it a waste of time." - P2
"The downside is that it didn't capture the ending time." - P8
"This is better! I can edit the event while adding in the calendar." - P9
"Want to link this to my calendar app (Outlook, google...)" - P7
Recommendation
Incorporate functionality that allows users to select and integrate with multiple calendar applications when creating an event.
Consider developing in-app calendar integration rather than relying solely on the system's default calendar UI, providing a more seamless and customizable user experience.
Before saving a calendar event, display a preview screen that allows users to confirm or correct details like the caller's name, time, and purpose.
Finding 3: Lack of Transparency & Certainty in Private Calling
3 - Major
Data Analysis
6 out of 8 participants found the incognito feature in the dial pad during their call, but only 5 out of 8 trusted it after use.
User experience varied significantly based on individual call handling behavior.
Some users held their phone against their ear while others used speaker mode, creating inconsistent experiences in discovering the feature.
Users expressed uncertainty about whether recording had truly stopped.
There was a lack of reassurance to users that the recording had actually been disabled.
Participants wanted voice confirmation during calls to verify privacy mode was active.
Quotes
"I was not able to use it. Couldn't click on it because I couldn't see it." - P3
"There's no robot voice saying it's not being recorded anymore." - P8
"I don't trust it to stop recording like that. It's probably being recorded in some way or another. I don't trust it to not have my information." - P6
"Just to make sure the other person knows it's not being recorded and it's useful for private conversation." - P8
Recommendation
To increase transparency, add an onboarding screen introducing the incognito feature to build user trust and ease the learning curve around privacy protection.
When incognito mode is activated, implement a high-contrast notification that prominently alerts the user, confirming that both parties will hear a verbal confirmation that recording has been disabled.
This provides clear reassurance that the private call is truly private.
Finding 4: Spam Detection Alerts Lack Visibility and Engagement
3 - Major
Data Analysis
3 out of 8 participants couldn't detect the spam feature unless they were on speaker mode.
The current notification system wasn't sufficiently noticeable for many users.
4 out of 8 participants didn't notice the haptic feedback (vibration) meant to alert them.
Some participants expressed disinterest in engaging in conversation with a spam caller, preferring to end calls immediately.
The chatbot feature that engaged before the call was often considered more useful than the in-call spam detection.
Users valued the concept of AI detecting "new approaches" to scams they might not recognize themselves.
Quotes
"I always talk on speaker on phone so it was easy to see. If I was on my phone without the speaker I'm not sure if this would be noticeable." - P6
"The chatbot before call was more useful than the spam detection. She noticed it's a spam when seeing the transcript saying it's about 'urgent and financial' info." - P7
"Most of the time I can sniff it out, but you never know when they get more advanced, and trust AI in detecting the new approaches and things I never know." - P9
"Switching to the Hiya app, there's both vibration and pop-up. However, the vibration didn't feel like a warning. I only realized it when I picked up my phone and saw the pop-up" - P2
Recommendation
Simplify the process of ending spam calls by adjusting the call-to-action, making it feel like less of a commitment to answer. Introduce a pre-call pop-up to notify users before the call connects, saving time and effort.
Redesign the haptic feedback with increased intensity, sharpness, and frequency to effectively capture attention and convey urgency.
Highlight the 'Scam Tactics' detection feature to align with user expectations that AI can help identify evolving scam methods.
These 4 findings represent the major insights from our comprehensive usability testing of the Hiya application. Our research uncovered significant opportunities for feature enhancement while validating the core value proposition of the app.
For more detailed findings, design recommendations, and the full usability testing case study, please reach out to me directly!
Key Takeaways
Our approach to usability testing succeeded through quantifiable feedback scales, comprehensive participant screening, and structured note-taking forms that streamlined data analysis. However, the effectiveness was limited by having too broad participant criteria, insufficient understanding of the newly-launched app's business context, and an expansive testing scope that prevented deeper exploration of individual features.
Given more time, we would expand testing to include less tech-savvy users (particularly in older adults vulnerable to scam calls), conduct follow-up tests after resolving initial technical issues, and incorporate stakeholder interviews to better align our research with product goals and technical constraints.
