AI Assistant | Privacy & Security | Communication Tool
Hiya had recently launched their AI-powered calling app with 4 key features: call summarization, scam detection, calendar event detection, and incognito (private calling) mode. The challenge was to evaluate how effectively users could navigate and use these features, while identifying points of friction in the user experience.
As a UX researcher and designer, I designed and conducted moderated usability testing sessions, created test plans and research protocols, recruited and screened qualified participants through userinterviews.com. After collecting insights, I collaboratively analyzed qualitative and quantitative user feedback and developed actionable design recommendations to Hiya's UX team.
Mobile App
Contribution
Systems Thinking
Usability Testing
Qualitative Analysis
Quantitative Analysis
Affinity Mapping
High-fidelity prototypes
Team
Yuri Kawada
Shreya Saxena
Gily Yu
Yiyang Li
Tools
FigJam
Figma
userinterviews.com
Timeline
10 weeks
Key Outcomes & Impact
We synthesized 15 actionable insights from usability testing to create 15 actionable design recommendations to improve user satisfaction, including information architecture improvements and UI enhancement opportunities. We developed pain point visualizations to effectively communicate findings to Hiya's UX team, leading to prioritized design improvements.
Key outcomes from these usability tests include user results and findings, design recommendations, and pre- & post-task analysis, aimed at enhancing Hiya’s user experience and product strategy.
What is Hiya?
Hiya is a leading caller identification and scam-blocking mobile application that empowers users to manage incoming calls through AI-driven analysis, spam detection, and AI-generated call transcripts. By offering both free and premium subscription services, Hiya collaborates with mobile carriers and businesses to enhance call security and establish user trust.
Key Features
In order to evaluate Hiya's AI Application, we focused on 4 features to conduct usability tests on our participants - the onboarding process, event detection and call transcript, incognito feature, and spam call detection.
Our target participants
We utilized a platform called "user interviews" to screen for participants that had a 100% match with our target user criteria. Out of the 50 that applied, we moved forward with those who fit our description of being busy business professionals who are early adopters of tech and first time Hiya app users.
Busy business professionals
Including independent business owners, home service workers, and real estate agents.
Early adopters of technology
First time Hiya app users
Business professionals who rely on frequent phone communication
Research Questions
To conduct the study, we analyzed the recently launched application, formulated research questions, and developed hypotheses to guide the usability test design.
Ease of Use
Can users successfully navigate through Hiya’s key features (onboarding process, spam call detection, event detection, and incognito mode)?
Effectiveness & Usefulness
What specific challenges do users face when completing tasks in the app?
Effectiveness & Usefulness
Can users easily navigate and manage incoming and outgoing calls using the dial screen?
Trust
How willing are users to trust and provide information and allow Hiya to manage their calls?
Methodology
To evaluate Hiya AI Phone's usability, we implemented six complementary methods to capture both quantitative metrics and qualitative insights across key features.
Moderated Usability Testing
Real-time observations and qualitative feedback during guided tasks.
Likert Scale Questions
Collected quantitative ratings on a 5-point scale for satisfaction, ease of use, usefulness, and trust.
Task Analysis
Broke down complex interactions into discrete steps to identify specific pain points and inefficiencies.
Think-Aloud Protocol
Asked participants to verbalize their thoughts while using the app to reveal mental models and expectations.
First-Click Testing
Tracked users' initial interaction points to measure feature discoverability and navigation patterns.
Task Success Metrics
We recorded pass/fail criteria to evaluate task success.
Planning a Test Launch Before Participants Recruitment
Recruitment Criteria
Screener Questions
Consent Form
Usability Test Plan
Testing Objectives
Our usability study aimed to evaluate:
User trust in privacy settings
Feature discoverability and effectiveness
Tasks
Task 1: Onboarding & First-time Setup
First-time user downloading and setting up the Hiya app.
Success Criteria
Participants complete the full onboarding process without assistance and understand the purpose of each permission requested.
Task 2: Calendar Event Generation & Call Summarization
Answer a call to plan a meeting and review the call summary and auto-generated calendar event.
Success Criteria
Participants discover the auto-generated calendar event, add it to their calendar, and locate the call summary with accurate details.
Task 3: Incognito Mode
Make a confidential business call with privacy features enabled.
Success Criteria
Participants find and enable the incognito mode before making a private call and verify that no summary was generated.
Task 4: Spam Call Detection
Receive a potentially fraudulent call.
Success Criteria
Participants notice the spam detection indicators and refuse to provide sensitive information during the call.
Usability Test Details
In total 13 moderated usability tests were conducted with participants recruited through UserInterviews.com, among them, 8 participants successfully completed a 65-minute session, providing in-depth insights into their experiences with Hiya’s core functionalities.
Results and Findings
After analyzing data from our 8 usability tests, we synthesized participant feedback through qualitative analysis of session recordings and observer notes. We compiled key quotes and observations while quantifying user sentiment through Likert scale responses to identify patterns and prioritize findings by severity level.
Affinity Mapping User Feedback
Likert Scale Responses to Identify Patterns
Severity Levels
In analyzing our usability findings, we have put them into severity levels based on Nielsen and Norman Group’s 0 to 4 rating scale.
2 - Minor
All 8 participants rated the event-adding feature highly useful in their professional lives (average rating: 4.87/5) and found it easy to locate (5/5) but several limitations hindered its utility.
Users noted that event messages failed to include meeting end times, creating ambiguity. Additionally, many expressed frustration that events could only be added to the system’s default calendar, rather than to third-party apps like Google Calendar or Outlook. Some participants also noted that the feature felt less relevant in casual or social contexts, pointing to a need for greater contextual adaptability.
3 - Major
The incognito feature in the dial pad lacked clarity and consistency, which led to user distrust and uncertainty. While 6 out of 8 participants were able to find the feature during a call, only 4 out of 8 reported trusting it after use.
User experience varied significantly depending on user's call handling behavior. Some held the phone to their ear, while others used speaker mode. This resulted in inconsistent discovery and interaction with the feature. Participants also expressed doubt that recording had actually stopped, citing a lack of clear confirmation.
3 - Major
