Defining Success for a Contactless Transit App
As a solo researcher, I designed and conducted an independent qualitative research study for RideCAT's transit app—highlighting the early wins, like real-time tracking, identifying unseen adoption barriers, and defining measurable paths forward to improve both rider and driver satisfaction and operational success.
Digital ad via Collier Area Transit
Project Details
Client
Collier Area Transit (CAT)
Team
Myself; a Spanish language translator for in-person interviews and survey translation
Role
User Researcher
Methodology
Evaluative Qualitative Research (Field Observations, In-Person Interviews, Paper Surveys)
Timeline
Spring 2021, Four weeks
Overview
In August 2020, Collier Area Transit (CAT) launched the rideCAT mobile ticketing app as a contactless option in direct response to the COVID-19 pandemic, designed to make riding transit safer by eliminating the need for passengers to handle cash, physical tickets, or interact with ticketing infrastructure. This positioned the app as both a public-health intervention and a modernization of fare collection.
CAT had usage analytics from their technical vendor, Masabi, as well as written feedback submitted through the app and website. However, there was no plan to evaluate the rider or driver experience, app adoption factors, or the app's service impact. Without that, there was no meaningful way to set success metrics, prioritize improvements, or understand whether the communities most reliant on public transit were actually being served by this solution.
THE PROBLEM
CAT was missing a research-driven strategy to define and measure what success should look like for the mobile app.
Alex Driehaus/Naples Daily News/USA TODAY - FLORIDA NETWORK
Impact
Riders who used the app reported high satisfaction and a growing reliance on it. But adoption was limited by awareness gaps, onboarding friction, and equity challenges that analytics alone couldn't surface.
This qualitative research allowed me to:
1.
2.
3.
...connect rider frustration to legacy hardware that was already outlined as an investment priority in CAT's Ten Year Development Plan.
...elevate bus drivers from passive observers to frontline adoption advocates.
...introduce new measurable success metrics including adoption benchmarks, route efficiency comparisons, driver satisfaction tracking, and “rider disappointment” as a behavioral indicator of the app's value to riders.
THE IMPACT
Together, these insights helped CAT move beyond analytics and reactive feedback toward a more intentional framework for understanding adoption, rider experience, and long-term success.
Process
Working as a solo researcher over nine days of field research and a total of four weeks, the study moved through six phases:
-
Review CAT's Ten Year Development Plan to ground findings in stated organizational goals.
-
Gather stakeholder insights and Masabi ridership data to identify where and when to intercept participants.
-
Analyze existing rider feedback from the in-app survey and pre-launch social media.
-
Prepare interview questions and surveys.
-
Conduct in-person interviews, distribute and collect paper surveys (in English and Spanish) with app users, non-app users, and drivers.
-
Synthesize findings into actionable recommendations with a forward-looking measurement framework.
Scope & Constraints
Interviews and surveys were done over nine days (between March 29-April 29, 2021) at a major transfer hub and onboard buses. I was the solo researcher and all work was done pro bono with no budget, but some printing and translation resources were provided by CAT.
Some other challenges of the study were:
Covid-19: PPE was required onboard as well as adherence to distancing and sanitizing, but some riders were still weary of talking or touching anything. Masks made it difficult to communicate with riders at the transfer station, as the buses were idling and it was hard to hear. This caused people to move closer to try to understand each other and that was not ideal for social distancing.
Language barrier: The population included Spanish and Creole speakers. CAT provided a translator for two intercept sessions and I was able to make a Spanish language version of the paper survey.
Sample size: The study was designed to be a snapshot in time, and with only one researcher
Recommendations
I recommended focusing on three areas to address immediately to boost engagement, user experience and operational goals:
Awareness & Adoption
Most non-app users hadn't heard of rideCAT despite existing marketing efforts. I recommended larger and more prominent signage at scanners and transfer stations, multilingual materials reflecting the communities CAT serves, messaging that addressed digital security concerns directly, and in-person onboarding workshops for riders with low technical literacy. For riders whose primary barrier was simply not knowing the app existed, these were low-cost, high-impact interventions.
Driver Safety & Advocacy
Drivers can become front-line advocates for the app if there are readily available resources to reference or hand out to riders. This would benefit the drivers by reducing the need to troubleshoot faulty machines, having to handle cash, and would limit physical interactions with riders. The ROI for this in terms of Driver job satisfaction, performance and retention could be something to track in conjunction.
Awareness & Adoption
Most non-app users hadn't heard of rideCAT despite existing marketing efforts. I recommended larger and more prominent signage at scanners and transfer stations, multilingual materials reflecting the communities CAT serves, messaging that addressed digital security concerns directly, and in-person onboarding workshops for riders with low technical literacy. For riders whose primary barrier was simply not knowing the app existed, these were low-cost, high-impact interventions.
Glitches & User Experience
The study further validates the need outlined in the Ten Year Development Plan for updates to hardware and examination of the systems currently in place. Fixing the cause of those glitches will address the user experience of the app. It would be ideal if CAT was able to work with Masabi to customize the app, however if it is not possible, then passing along rider feedback and feature requests is recommended.
I also presented some ideas on what can be measured going forward to keep tabs on CAT’s objectives.
-
Route Efficiency: Comparing pre- and post-app data on routes with high adoption of the app over time
-
Adoption Rate of the App: Setting a goal for a six month check in to assess progress
-
Driver Satisfaction: Could be measured to see if app adoption has any effect
-
Rider Satisfaction: This is a standard measurement, but requires an explanation to get actionable data
-
Rider Disappointment: Measuring loss is a good indicator of how essential a service is to someone’s life. Moving riders from “I don’t use the app” to “being very disappointed if they could no longer use it” will demonstrate more clearly how valuable it is to them.
Learnings
I learned how to streamline my interview and survey questions. For future projects, I will want to do more preliminary research to understand the population and to consider the constraints of field work.
Research Methodology
-
Edit questions
-
I originally had more demographic and trip questions for riders—where are you going today? What type of fare do you normally buy? How did you find out about the app? After the first hour of interviews, I realized that I had to be more direct and cut out these questions because the buses were loud, it was hard to communicate with masks on, and people were rushing to catch their bus.
I had to cut driver questions down as well. I went for a basic question that I elaborated on as I got answers. “What is your experience with riders using the rideCAT mobile fare app?” followup, “Do they ever ask for your help with the app?”, “Are there any technical problems with the app?”
-
-
Adjust timing
-
Changed the original time schedule based off of early tests, where the riders were tired or anxious to catch their bus for work. I thought adding more afternoon sessions would be better to catch crowds without those challenges. I had mixed results with this, but I think overall it was good to get that varying time spread of riders.
-
-
Understanding the population to get the best data
-
I had to rework the language on the survey and interviews to be extremely clear. Many people didn’t know the name of the app, so I had to refer to it very specifically as the 'phone app to buy your bus ticket." This was also to avoid confusion with CAT's other apps, planCAT for trip planning and maps, and MyStop for real-time updates.
-
Future research needs to include Spanish and Creole speakers for qualitative research. People were more likely to speak to someone in their native tongue face-to-face than to speak to me in English, their second language. I also think that the structure of the survey might have missed cultural or experiential cues. There were several participants who answered both the “NO” and “YES” questions, which made me wonder if I designed it in a confusing way or if they didn't have experience with that type of survey.
-
Interview Technique and Awareness
-
The language barrier and lack of knowledge of the app meant I ended up doing more marketing, handing out flyers and explaining how the app worked. This was helpful to see how much non-app users understood the purpose of the app and what their barriers might be.
-
I developed a rhythm of allowing passengers to get settled on the bus before approaching them about the study. Then trying my best to explain the app and the reason for the study, while the bus moved and while talking through a mask, a couple feet away from riders.
There were six steps in the process of developing and testing this product.
1. Review Ten Year Development Plan
A development plan was released in October 2020 detailing improvements in all areas of the transportation system over the next 10 years. I reviewed this for insights into CAT’s goals and ideas for what to focus on in the research.
CAT has technology goals outlined in section 7.2.7 Technology Trends that aligned with my research and I was able to validate some of their goals.
2. Gather Stakeholder Insights
I interviewed CAT marketing, operations, planners and the director of operations to get an idea of where they were focusing efforts for the app and what they had discovered so far about the app usage.
CAT provided their route data as well as Masabi’s app data, which included stops and number of tickets used at stops over a 3 month period. This steered the research locations and times to reach the most riders, and specifically riders who used the app.
3. Analyze Existing Rider Feedback
Check social media feedback, as well as customer service and in-app survey feedback for insights into pain points and concerns.
CAT has a Facebook and Instagram account that is not very active, however there were some good questions that were raised when the app was first released that were important to keep in mind throughout the test. The in-app survey feedback also had some thoughtful comments for the study.
4. Write Field Interview Script and Design Paper Surveys
Questions were chosen to get an idea of barriers for non-app users and pain points and satisfaction for app users. The questions were the same for in-person interviews and paper surveys.
A filtering question was asked first, “Do you use the phone app to buy your bus ticket?”
-
Based on answers, non-app users were asked two questions:
-
Does the phone app sound like something you would like to use? Why or why not?
-
Is there anything that would prevent you from using the phone app?
-
-
For app users, there were three questions:
-
How disappointed would you be if you could no longer use the phone app?
(Not at all Disappointed, Somewhat Disappointed, Very Disappointed)-
Please explain why you answered that way.
-
-
If you had a magic wand, what would you change about the phone app?
-
How satisfied are you with the phone app?
(Unsatisfied, Kind of Unsatisfied, Neutral, Kind of Satisfied, Satisfied)-
Please explain why you answered that way.
-
-
Paper Surveys in English (left) and Spanish (right)
5. Observe riders, Conduct Interviews, Administer Paper Surveys
Observations were performed of riders purchasing tickets and using different fare payment methods onboard. Paper surveys were handed out onboard and collected soon after. Interviews were performed at the Government transfer station and onboard various routes.
-
Observations were made of riders purchasing tickets from the customer service desk and onboard buses. As well as using existing ticketing methods such as cash, pass cards and the rideCAT mobile app.
-
Paper surveys in English and Spanish were distributed on busy routes to get as much data as possible.
-
Field interviews were performed at the Government Transfer Station and on various routes.
26 App Users
(20 in-person interviews,
6 paper surveys)
56 Non-App Users
(18 in-person interviews,
38 paper surveys)
16 Bus Drivers
across various routes
and times6. Synthesize data, compile a report and presentation
A report was presented to stakeholders of findings from the study and recommendations on how to address issues.
-
Gauge user satisfaction and uncover pain points of app users. Users were generally satisfied with the rideCAT app. They said that it was easy and convenient to use, however there were pain points. App users pain points can be broken down into 2 categories, User Experience and Technical. There is some overlap between the categories when it comes to getting real-time service alerts and accurate route times.They described this as a priority and wish that this information and the fare app was combined into a single app.
I prefer a physical card because I’m afraid of online scams
Nothing keeping me from using it, except would want to be able to put a few dollars in and not a whole $20.
The money doesn’t go in the farebox all of the time, and [I] hate to hold up the line of people waiting to board the bus.
When the farebox gets jammed, it’s not safe...especially in Immokalee, there is no supervisor out there to fix it, so you end up holding cash the whole time—it’s not safe!
Other user experience issues were not critical to the function of the app, such as a request for fewer screens to click through to get to the ticket, or having a longer window of time to purchase a single fare ticket. CAT will have to coordinate with Masabi to implement feature upgrades.
Technical problems were the bulk of pain points for app users. Riders mentioned the app freezing when they are trying to use it, or the validator not scanning. One rider mentioned having to restart the app after it glitched when he moved between apps on his phone. And as I mentioned before, service alerts were not updating, which caused a lot of frustration for app users.
-
Discover barriers for riders not currently using the app.
For Non-App users some of the obstacles were a lack of awareness of the app and assumptions about the specifics of the app. Some people I talked to didn’t know if they could use special fare passes on the app, like reduced fare tickets or if they could buy single fares.
Circumstantial barriers like not having a bank account or credit or debit cards can keep people from using the app. Insufficient devices such as using a non-smartphone, or currently using a smartphone that doesn’t support new apps, or a phone that doesn’t have enough memory or battery life present obstacles to app usage.
Behaviors and beliefs include riders who don’t use a cellphone at all, or who don’t use the bus often, and riders who are distrustful of digital devices for fear of scams, or they just prefer physical cards.
-
Assess how the app is performing in relation to stakeholders goals. All 16 drivers I spoke with liked the app. They said riders rarely had any issues and never asked for help. Compared to the farebox, where the driver has to intervene often when cash gets jammed and paper tickets won’t swipe. All of the operators were frustrated with these farebox malfunctions and some expressed that it put them in unsafe positions, one example being holding cash on their route. They also mentioned that it took up time from their route to troubleshoot those issues.
Drivers said that they didn’t know much about the app and when some non-app riders asked them about it, all they could do was direct them to the app store. They had no information to share with riders.
-