Defining Success for a Pandemic-era Transit App
As a solo researcher, I designed and conducted an independent qualitative research study for RideCAT's transit app—highlighting the early wins, like real-time tracking, identifying unseen adoption barriers, and defining measurable paths forward to improve both rider and driver satisfaction and operational success.
Digital ad via Collier Area Transit
Project Details
Client
Collier Area Transit (CAT)
Team
Myself; a Spanish language translator for in-person interviews and survey translation
Role
User Researcher
Methodology
Evaluative Qualitative Research (Field Observations,
In-Person Interviews, Paper Surveys)
Timeline
Spring 2021, Four weeks
Overview
In August 2020, Collier Area Transit (CAT) launched the rideCAT mobile ticketing app as a contactless option in direct response to the COVID-19 pandemic, designed to make riding transit safer by eliminating the need for passengers to handle cash, physical tickets, or interact with ticketing infrastructure. This positioned the app as both a public-health intervention and a modernization of fare collection.
CAT had usage analytics from their technical vendor, Masabi, as well as written feedback submitted through the app and website. However, there was no plan to evaluate the rider or driver experience, app adoption factors, or the app's service impact. Without that, there was no meaningful way to set success metrics, prioritize improvements, or understand whether the communities most reliant on public transit were actually being served by this solution.
CAT was missing a research-driven strategy to define and measure what success should look like for the mobile ticketing app.
Alex Driehaus/Naples Daily News/USA TODAY - FLORIDA NETWORK
Impact
Riders who used the app reported high satisfaction and a growing reliance on it. But adoption was limited by awareness gaps, onboarding friction, and equity challenges that analytics alone couldn't surface.
This qualitative research allowed me to:
1.
2.
3.
...connect rider frustration to legacy hardware that was already outlined as an investment priority in CAT's Ten Year Development Plan.
...elevate bus drivers from passive observers to frontline adoption advocates.
...introduce new measurable success metrics including adoption benchmarks, route efficiency comparisons, driver satisfaction tracking, and “rider disappointment” as a behavioral indicator of the app's value to riders.
Together, these insights helped CAT move beyond analytics and reactive feedback toward a more intentional framework for understanding adoption, rider experience, and long-term success.
Process
Working as a solo researcher over nine days of field research and a total of four weeks, the study moved through six phases:
-
Review CAT's Ten Year Development Plan to ground findings in stated organizational goals.
-
Gather stakeholder insights and Masabi ridership data to identify where and when to intercept participants.
-
Analyze existing rider feedback from the in-app survey and pre-launch social media.
-
Prepare interview questions and surveys.
-
Conduct in-person interviews, distribute and collect paper surveys (in English and Spanish) with app users, non-app users, and drivers.
-
Synthesize findings into actionable recommendations with a forward-looking measurement framework.
Scope & Constraints
Interviews and surveys were done over nine days (between March 29-April 29, 2021) at a major transfer hub and onboard buses. I was the solo researcher and all work was done pro bono with no budget, but some printing and translation resources were provided by CAT.
Some other challenges of the study were:
Covid-19: PPE was required onboard as well as adherence to distancing and sanitizing, but some riders were still weary of talking or touching anything. Masks made it difficult to communicate with riders at the transfer station, as the buses were idling and it was hard to hear. This caused people to move closer to try to understand each other and that was not ideal for social distancing.
Language barrier: The population included Spanish and Creole speakers. CAT provided a translator for two intercept sessions and I was able to make a Spanish language version of the paper survey.
Sample size: The study was designed to be a snapshot in time, and with only one researcher
Recommendations
I recommended focusing on three areas to address immediately to boost engagement, user experience and operational goals:
Awareness & Adoption
Most non-app users hadn't heard of RideCAT despite existing marketing efforts. I recommended larger and more prominent signage at scanners and transfer stations, multilingual materials reflecting the communities CAT serves, messaging that addressed digital security concerns directly, and in-person onboarding workshops for riders with low technical literacy. For riders whose primary barrier was simply not knowing the app existed, these were low-cost, high-impact interventions.
Driver Safety & Advocacy
Drivers can become front-line advocates for the app if there are readily available resources to reference or hand out to riders. This would benefit the drivers by reducing the need to troubleshoot faulty machines, having to handle cash, and would limit physical interactions with riders. The ROI for this in terms of Driver job satisfaction, performance and retention could be something to track in conjunction.
Glitches & User Experience
The study further validates the need outlined in the Ten Year Development Plan for updates to hardware and examination of the systems currently in place. Fixing the cause of those glitches will address the user experience of the app. It would be ideal if CAT was able to work with Masabi to customize the app, however if it is not possible, then passing along rider feedback and feature requests is recommended.
I also presented some ideas on what can be measured going forward to keep tabs on CAT’s objectives.
-
Route Efficiency: Comparing pre- and post-app data on routes with high adoption of the app over time
-
Adoption Rate of the App: Setting a goal for a six month check in to assess progress
-
Driver Satisfaction: Could be measured to see if app adoption has any effect
-
Rider Satisfaction: This is a standard measurement, but requires an explanation to get actionable data
-
Rider Disappointment: Measuring loss is a good indicator of how essential a service is to someone’s life. Moving riders from “I don’t use the app” to “being very disappointed if they could no longer use it” will demonstrate more clearly how valuable it is to them.
Learnings
I learned how to streamline my interview and survey questions. For future projects, I will want to do more preliminary research to understand the population and to consider the constraints of field work.
Research Methodology
-
Edit questions
-
I originally had more demographic and trip questions for riders—where are you going today? What type of fare do you normally buy? How did you find out about the app? After the first hour of interviews, I realized that I had to be more direct and cut out these questions because the buses were loud, it was hard to communicate with masks on, and people were rushing to catch their bus.
I had to cut driver questions down as well. I went for a basic question that I elaborated on as I got answers. “What is your experience with riders using the rideCAT mobile fare app?” followup, “Do they ever ask for your help with the app?”, “Are there any technical problems with the app?”
-
-
Adjust timing
-
Changed the original time schedule based off of early tests, where the riders were tired or anxious to catch their bus for work. I thought adding more afternoon sessions would be better to catch crowds without those challenges. I had mixed results with this, but I think overall it was good to get that varying time spread of riders.
-
-
Understanding the population to get the best data
-
I had to rework the language on the survey and interviews to be extremely clear. Many people didn’t know the name of the app, so I had to refer to it very specifically as the 'phone app to buy your bus ticket." This was also to avoid confusion with CAT's other apps, planCAT for trip planning and maps, and MyStop for real-time updates.
-
Future research needs to include Spanish and Creole speakers for qualitative research. People were more likely to speak to someone in their native tongue face-to-face than to speak to me in English, their second language. I also think that the structure of the survey might have missed cultural or experiential cues. There were several participants who answered both the “NO” and “YES” questions, which made me wonder if I designed it in a confusing way or if they didn't have experience with that type of survey.
-
Interview Technique and Awareness
-
The language barrier and lack of knowledge of the app meant I ended up doing more marketing, handing out flyers and explaining how the app worked. This was helpful to see how much non-app users understood the purpose of the app and what their barriers might be.
-
I developed a rhythm of allowing passengers to get settled on the bus before approaching them about the study. Then trying my best to explain the app and the reason for the study, while the bus moved and while talking through a mask, a couple feet away from riders.
- Page 1