zeryabsial@gmail.com

The Digital Services Team within the Office of Innovation and Technology consists of Content Strategy, Digital Forms, and User Experience (UX) teams that each contribute to the City’s website and other digital products (applications, forms, etc.) by considering the needs of the residents, businesses, and City staff that use them. As the User Experience (UX) side of the Digital Services Team, we shape the strategy for digital services and how we build them. Our mission is to make these digital services as easy to use, reliable, and useful as we can for our users.  

However, direct user feedback was critically lacking from our current process leading to an incomplete understanding of our users’ needs or a mechanism to evaluate how well we are meeting them. This presented a significant barrier to fulfilling our mission, compromising the quality and accessibility of our services. While usability testing was previously conducted in isolation on other projects, we hadn’t conducted any form of testing at all for most of the City’s digital services. More broadly, we hadn’t determined best practices for reliable, regular, and ethical user research. 

Much of the existing knowledge our team had about our how our website performed was based on passive research like Google Analytics and Microsoft Clarity. This heatmap of the Phila.gov homepage reveals how it provides hints about what gets the most traffic but doesn't tell a complete story.

There are countless experiences people can have with a product that we often don't take into consideration. With user research, participants can share the unique aspects of their personal setup such as the assistive technology for color contrast and magnification seen here.

When I joined the City of Philadelphia in 2021, one of my goals as a UX strategist was to develop a user research practice within Digital Services for the UX maturity of the team and for my personal development as a researcher. I was given an opportunity to pursue this goal when the City launched the Operations Transformation Fund which supports projects across departments and agencies that innovate city processes with a focus on producing more equitable outcomes for Philadelphians. I developed a plan and pitched the case for usability testing on our digital products. It was one of 18 projects awarded during the Spring 2022 grant cycle with $155,000. 

I initiated this project with a question:

“How can we establish a consistent and reliable process for user feedback on our digital services to identify gaps in the user experience and ensure we address the unique needs of our remarkably diverse resident population?”

A City of Philadelphia researcher facilitates a session in Mandarin with an interpreter at a local library.

This primary goal aimed to transform the current process for delivering these services throughout the City by establishing usability testing as a key component in how we build and by providing guidelines for effective and efficient testing through the collation of learnings into a usability testing playbook. Secondarily we also looked to make sweeping improvements to user experience on our website by translating user feedback from testing into high impact, evidence based changes.

From inception to completion, I managed, led, and hired a team of two experienced Designer Researchers with the additional support of our Associate User Experience Designer fellow through a yearlong research project from July 2022 to June 2023. We conducted sixty-four, hour-long sessions designed to evaluate different testing modalities and reveal the pros and cons of each method. By including the following modalities in our testing, we were able to compare the procedures, effort, time, cost, and quality of results from each:  

A participant who is blind demonstrate for our team how they use braille screen input to type on their phone.

A participant describes their process for navigating a website using a screen reader.

Currently, this work continues through consultation to other teams on planning, preparing, and performing usability testing.