AI Assistant | Privacy & Security | Communication Tool

Hiya

Hiya

Hiya

How does Hiya's core features impact user's trust, engagement, and adoption of mobile scam-protection solutions?

How does Hiya's core features impact user's trust, engagement, and adoption of mobile scam-protection solutions?

How does Hiya's core features impact user's trust, engagement, and adoption of mobile scam-protection solutions?

The Challenge

The Challenge

Hiya had recently launched their AI-powered calling app with 4 key features: call summarization, scam detection, calendar event detection, and incognito (private calling) mode. The challenge was to evaluate how effectively users could navigate and use these features, while identifying points of friction in the user experience.

My Role

My Role

As a UX researcher and designer, I designed and conducted moderated usability testing sessions, created test plans and research protocols, recruited and screened qualified participants through userinterviews.com. After collecting insights, I collaboratively analyzed qualitative and quantitative user feedback and developed actionable design recommendations to Hiya's UX team.

Mobile App

Contribution

Systems Thinking

Usability Testing

Qualitative Analysis

Quantitative Analysis

Affinity Mapping

High-fidelity prototypes

Team

Yuri Kawada

Shreya Saxena

Gily Yu

Yiyang Li

Tools

FigJam

Figma

userinterviews.com

Timeline

10 weeks

Key Outcomes & Impact

We synthesized 15 actionable insights from usability testing to create 15 actionable design recommendations to improve user satisfaction, including information architecture improvements and UI enhancement opportunities. We developed pain point visualizations to effectively communicate findings to Hiya's UX team, leading to prioritized design improvements.

Key outcomes from these usability tests include user results and findings, design recommendations, and pre- & post-task analysis, aimed at enhancing Hiya’s user experience and product strategy.

What is Hiya?

Hiya is a leading caller identification and scam-blocking mobile application that empowers users to manage incoming calls through AI-driven analysis, spam detection, and AI-generated call transcripts. By offering both free and premium subscription services, Hiya collaborates with mobile carriers and businesses to enhance call security and establish user trust.

Key Features

In order to evaluate Hiya's AI Application, we focused on 4 features to conduct usability tests on our participants - the onboarding process, event detection and call transcript, incognito feature, and spam call detection.

Our target participants

We utilized a platform called "user interviews" to screen for participants that had a 100% match with our target user criteria. Out of the 50 that applied, we moved forward with those who fit our description of being busy business professionals who are early adopters of tech and first time Hiya app users.

Busy business professionals

Including independent business owners, home service workers, and real estate agents.
Early adopters of technology
First time Hiya app users
Business professionals who rely on frequent phone communication

Research Questions

To conduct the study, we analyzed the recently launched application, formulated research questions, and developed hypotheses to guide the usability test design.

Ease of Use

Can users successfully navigate through Hiya’s key features (onboarding process, spam call detection, event detection, and incognito mode)?

Effectiveness & Usefulness

What specific challenges do users face when completing tasks in the app?

Effectiveness & Usefulness

Can users easily navigate and manage incoming and outgoing calls using the dial screen?

Trust

How willing are users to trust and provide information and allow Hiya to manage their calls?

Methodology

To evaluate Hiya AI Phone's usability, we implemented six complementary methods to capture both quantitative metrics and qualitative insights across key features.

Moderated Usability Testing

Real-time observations and qualitative feedback during guided tasks.

Likert Scale Questions

Collected quantitative ratings on a 5-point scale for satisfaction, ease of use, usefulness, and trust.

Task Analysis

Broke down complex interactions into discrete steps to identify specific pain points and inefficiencies.

Think-Aloud Protocol

Asked participants to verbalize their thoughts while using the app to reveal mental models and expectations.

First-Click Testing

Tracked users' initial interaction points to measure feature discoverability and navigation patterns.

Task Success Metrics

We recorded pass/fail criteria to evaluate task success.

Planning a Test Launch Before Participants Recruitment

Recruitment Criteria
Screener Questions
Consent Form
Usability Test Plan

Testing Objectives

Our usability study aimed to evaluate:
Ease of onboarding
Ease of onboarding and account setup
User trust in privacy settings
Feature discoverability and effectiveness

Tasks

Task 1: Onboarding & First-time Setup
First-time user downloading and setting up the Hiya app.
Success Criteria
Participants complete the full onboarding process without assistance and understand the purpose of each permission requested.
Task 2: Calendar Event Generation & Call Summarization
Answer a call to plan a meeting and review the call summary and auto-generated calendar event.
Success Criteria
Participants discover the auto-generated calendar event, add it to their calendar, and locate the call summary with accurate details.
Task 3: Incognito Mode
Make a confidential business call with privacy features enabled.
Success Criteria
Participants find and enable the incognito mode before making a private call and verify that no summary was generated.
Task 4: Spam Call Detection
Receive a potentially fraudulent call.
Success Criteria
Participants notice the spam detection indicators and refuse to provide sensitive information during the call.

Usability Test Details

In total 13 moderated usability tests were conducted with participants recruited through UserInterviews.com, among them, 8 participants successfully completed a 65-minute session, providing in-depth insights into their experiences with Hiya’s core functionalities.

Results and Findings

After analyzing data from our 8 usability tests, we synthesized participant feedback through qualitative analysis of session recordings and observer notes. We compiled key quotes and observations while quantifying user sentiment through Likert scale responses to identify patterns and prioritize findings by severity level.
Affinity Mapping User Feedback
Likert Scale Responses to Identify Patterns

Severity Levels

In analyzing our usability findings, we have put them into severity levels based on Nielsen and Norman Group’s 0 to 4 rating scale.
Finding 1: Calendar Event Detection Highly Valued for Professional Use
Finding 1: Calendar Event Detection Highly Valued for Professional Use

2 - Minor

Data Analysis

Data Analysis
All 8 participants rated the event-adding feature highly useful in their professional lives (average rating: 4.87/5) and found it easy to locate (5/5) but several limitations hindered its utility.

Users noted that event messages failed to include meeting end times, creating ambiguity. Additionally, many expressed frustration that events could only be added to the system’s default calendar, rather than to third-party apps like Google Calendar or Outlook. Some participants also noted that the feature felt less relevant in casual or social contexts, pointing to a need for greater contextual adaptability.

Before

Before

Design Recommendation

Design Recommendation
Finding 2: Lack of Transparency & Certainty in Private Calling
Finding 2: Lack of Transparency & Certainty in Private Calling

3 - Major

Data Analysis

Data Analysis
The incognito feature in the dial pad lacked clarity and consistency, which led to user distrust and uncertainty. While 6 out of 8 participants were able to find the feature during a call, only 4 out of 8 reported trusting it after use.

User experience varied significantly depending on user's call handling behavior. Some held the phone to their ear, while others used speaker mode. This resulted in inconsistent discovery and interaction with the feature. Participants also expressed doubt that recording had actually stopped, citing a lack of clear confirmation.

Before

Before

Design Recommendation

Design Recommendation
Finding 3: Spam Detection Alerts Lack Visibility and Engagement
Finding 3: Spam Detection Alerts Lack Visibility and Engagement

3 - Major

Data Analysis

Data Analysis
3 out of 8 participants couldn't detect the spam feature unless they were on speaker mode and 4 out of 8 didn’t notice the haptic feedback intended to alert them.

The current notification system wasn’t prominent enough to interrupt or inform users effectively during a call. Some users preferred to end suspected spam calls immediately, making ongoing call features less relevant.

In contrast, participants found the pre-call AI assistant more helpful, especially when it flagged suspicious activity. Users appreciated the potential for AI to recognize evolving scam tactics, but wanted clearer, earlier, and more interruptive signals.

Before

Before

Design Recommendation

Design Recommendation
These 3 findings represent the major insights from our comprehensive usability testing of the Hiya application. Our research uncovered significant opportunities for feature enhancement while validating the core value proposition of the app.

For more detailed findings, design recommendations, and the full usability testing case study, please reach out to me directly!
These 3 findings represent the major insights from our comprehensive usability testing of the Hiya application. Our research uncovered significant opportunities for feature enhancement while validating the core value proposition of the app.

For more detailed findings, design recommendations, and the full usability testing case study, please reach out to me directly!

Key Takeaways

Our approach to usability testing succeeded through quantifiable feedback scales, comprehensive participant screening, and structured note-taking forms that streamlined data analysis. However, the effectiveness was limited by having too broad participant criteria, insufficient understanding of the newly-launched app's business context, and an expansive testing scope that prevented deeper exploration of individual features.

Given more time, we would expand testing to include less tech-savvy users (particularly in older adults vulnerable to scam calls), conduct follow-up tests after resolving initial technical issues, and incorporate stakeholder interviews to better align our research with product goals and technical constraints.