New York Philharmonic’s Shelby White & Leon Levy Digital Archives
A usability study to improve discoverability and accessibility for new and expert audiences.
My Role
UX Researcher. Focused on task design, findings synthesis, and findings and recommendation 1 and 2 development.
Methods
Moderated remote user tests, System Usability Scale (SUS)
Tools
Excel, Figma, Zoom
Team
Tk Cram, Anna Feldman, and Sayali Nikumbh @ Pratt Center for Digital Experiences
Timeline
6 weeks
(Nov.–Dec. 2022)
Our Project
The New York Philharmonic is one of the city's most prestigious cultural institutions with a history spanning over 180 years. In 2011, they launched the Shelby White & Leon Levy Digital Archives to catalog, digitize, and make publicly-available over 6 million pages of material related to their symphony orchestra. The platform serves a broad audience of researchers, music students, musicologists, historians, geneaologists, and casual visitors (“the general public”).
Our project together was prompted by a unique opportunity: a five-year NEH-funded grant project to redesign the Digital Archives website. To inform future redesign decisions, my team developed a usability study to:
Evaluate and improve the exploratory experience for new users landing on the site, and
Evaluate and improve the experience for experts/researchers seeking out specific archival information.
A focus on pathways to find highlighted content
Screenshot of the Digital Archives home page, including the carousel and the “Features” label in the top navigation.
In our kickoff meeting with the Digital Archives, a potential area of research that stood out strongly was how the platform guided users to exhibits and other special projects.
The Digital Archives’ uses two methods to guide users to this content: the “Features” label in the top navigation and the home page carousel. As this content is important for both new users (who want to understand what the Digital Archives has to offer) and expert users (who want to dive deeply into archival resources), my team and I decided early-on to test the usability of these two navigation methods.
A focus on accessibility
The Digital Archives expressed a strong interest in ensuring that their site complies with the Web Content Accessibility Guidelines (WCAG) at v2.1. While our usability study could not encompass a full accessibility audit, another team member and I conducted an ad-hoc accessibility audit of the pages most frequently visited during user tests with the WebAIM WAVE tool tool and a domain expert. Our findings informed the our team’s recommendations and present starting points for the Digital Archives’ future redesign work.
A focus on actionability
Knowing that we were testing the usability of a site in the midst of a larger redesign project, our team strove to provide actionable recommendations that could be quickly implemented and used as the base for future usability work. To further support future work, our team also included secondary findings-based recommendations and opportunities for further exploration in the final report.
My focus
To our team, I brought experience with information architecture design, interaction design, accessibility, and cultural and educational institutions. In our work together, some of the areas I was most heavily involved in included task design, designing the rainbow chart, synthesizing overall findings, and leading the design and writing of recommendations 1 and 2.
Timeline
Kickoff
Nov. 1, 2022
➜
Test Development
➜
User Tests
Nov. 17– 29, 2022
➜
Test Analysis
➜
Client Presentation
Dec. 13, 2022
Our Research
Our team ran a series of ten moderated user tests to analyze the usability of the Digital Archives for new and expert users: what they enjoyed, and where they ran into trouble. Each test involved participant navigation of five tasks, each replicating a user workflow our client and our team were most interested in understanding.
To recruit a broad range of users, we conducted our tests remotely over Zoom. Two team members facilitated each test: one moderator who, using a script, prompted users to complete tasks while sharing their screen and thinking through their processes out loud, and one observer who took detailed notes for later analysis.
Who are the Digital Archives’ target audiences?
Existing users are thought to be a combination of researchers regularly performing investigative searches and non-scholarly users performing casual research — for instance, seeking records on a relative, a specific performance they attended, or a piece of music they enjoy. In their redesign, the Digital Archives team seeks to continue to attract and serve these types of users.
With this in mind, we identified the following three user groups to inform our recruitment:
Amateurs
Students and other first-time visitors who are highly familiar with the internet and may be looking for a specific resource, such as a piece of music.
Explorers
Young professionals with some exposure to the New York Philharmonic who are exploring what the archives have to offer.
Professors
Researchers highly familiar with online archives and the Digital Archives site; specifically, those who want to do deep dive research into a subject or person.
Our team conducted a total of ten user tests with a diverse set of participants, ranging from those with expertise with the Digital Archives to users with little to no familiarity with the Philharmonic. Per the NN Group’s research, moderated users tests provide valid results when at least five user tests are used, but more are required when varied user groups need to be tested; when the tasks for two user groups are only slightly different, three to four users per group should be tested. Our recruitment involved reaching out to Pratt Institute’s user testing listservs and leveraging our partner’s outreach to connect with student groups that visited the New York Philharmonic during the recruitment period. Recruitment also included an incentive of a $10 Amazon gift card to compensate participants for their time.
What key tasks should new and expert users be able to complete?
We designed five tasks and corresponding user goals to cover the major workflows identified by the client and team. These tasks enabled the team to study and compare users’ thought processes as they navigated the site, as well as the usability of the platform for different users.
We presented the tasks alongside series of open ended questions to gain additional qualitative data: an initial question about first impressions of the homepage, and questions about self-perceived difficulty and success of a task after the user completed each.
The five tasks and user goals are as follows:
Task 1:
(a) Find a program that includes William Lincer, a member of the New York Philharmonic orchestra in the mid 1900s.
(b) Once you have found a program, find an image of William Lincer that is not included in this program.
User goal: I want to find a resource connected to my family member who I know was a member of the orchestra.
Task 2:
(a) Find the music for Tchaikovsky’s Nutcracker Ballet.
(b) Find the sheet music for a clarinet.
User goal: I want to find sheet music of a piece I want to play.
Task 3:
(a) Find the New York Philharmonic Archives’ digital exhibition about its first female musician.
(b) Once you have found the exhibition, find out where she was born.
User goal: I want to understand what types of resources are available through the NY Phil Archives website and find something that’s interesting to me.
Task 4:
(a) Find the New York Philharmonic Archives’ newly digitized set of press clipping scrapbooks, which are featured on their homepage.
(b) Once you have found the set, locate a press clipping scrapbook that documents one of the New York Philharmonic’s park concert series.
User goal: I want to explore newly digitized resources that are relevant to my research interests.
Task 5:
(a) Now, imagine you’re researching how orchestras like the New York Philharmonic reached audiences during the COVID-19 pandemic, which began around March of 2020. Find a recording of one of the virtual performances offered by the Philharmonic in the first few months of the pandemic.
(b) Going back to your search results, which soloist performed the most times during the first few months of the pandemic?
User goal: I want to find contemporary musical recordings and records for research.
Findings and Recommendations
Overall, participants had positive first impressions of the Digital Archives. They appreciated how clearly the Search was featured on the homepage, and often commented that the site felt welcoming in comparison to other archives sites they’ve visited in the past.
Impressions of the site as a whole, however, were mixed. The Digital Archives received a raw SUS score of 60.3, equivalent to a D grade (average=68). The score is higher than 29.29% of other websites, and correlates to an “OK” rating of usability on a scale of Best Imaginable to Worst Imaginable.
These results offer the Digital Archives a clear opportunity to improve. The full report and client presentation present five detailed sets of findings and recommendations to address the most frequent and severe usability concerns:
Improve navigation usability and accessibility by updating styling, spacing, and text.
Update style and navigation elements on the carousel to make it more intuitive and accessible.
Streamline search results page with consolidated sort and filter options and highlighted search results.
Move pop-up modal underneath each filter and introduce a search bar for the filter options when opened.
Minor findings and recommendations (Which broadly cover: Accessibility, Metadata, User Interface, Links and Labels, and Knowledge Organization.)
Here, I’ll summarize the findings and recommendations from the two I was most heavily involved in: 1 and 2.
1: Improve navigation usability and accessibility by updating styling, spacing, and text
Screenshot of main navigation and search bars on homepage.
First impressions of the main navigation were generally positive. Users appreciated that the search bar is featured prominently and recognized it as the main way of engaging with the Digital Archives’ collection.
“I definitely like that the first thing I see is the ‘Search the Digital Archives’ section”
In attempting to interact with the navigation, however, users ran into several issues. If solved, the revamped navigation could improve the user experience of the site as a whole.
Annotated mockup of recommendations to improve the usability of the navigation bar.
A summarized list of recommendations and issues from the full report are as follows:
Recommendation 1.1: Add link styling to the navigation labels to make it clear that they are interactive links and that each label on any given page signifies the active page. These changes meet WCAG v2.1 success criteria on Use of Color and Location.
Related finding: When visiting the homepage for the first time, several users experienced issues using the main navigation bar. Participants attempted to click the “Digital Archives” label when exploring what the site had to offer, which resulted in confusion — “Digital Archives” is the name of the homepage and is not designed to be interactive while the user is on that page. All navigation labels are styled the same regardless of whether or not they represent an active page. Their styling also matches the styling of body text and no changes occur to the labels when a user hovers over them — both signifiers that text is, or is not, interactive.
Recommendation 1.2: Change the “Features” label to “ Exhibits.”
Related finding: Users who interacted with the “Features” label also had difficulty understanding what it signified. Using the “Features” page was an option for completing multiple tasks in our tests, but only one user found and clicked the label to attempt to find task information. Suggestions for ensuring “Features” is the most intuitive label for Digital Archives users is detailed on page 13 of the report.
Recommendation 1.3: Add “…from the New York Philharmonic” to the explanatory text offered about the Digital Archives’ collections by the search bar.
Related finding: Users who were less experienced with searching archival sites, or with the New York Philharmonic, often did not understand that the Digital Archives specifically offers access to resources related to the New York Philharmonic, not traveling orchestras or other musical institutions. This confusion resulted in later issues with searching for and filtering through search results. Most users in our tests did read this text, and this simple addition would make it clear to them what the site contains.
Recommendation 1.4: Shrink the top navigation slightly to ensure the primary content on all pages is emphasized.
Related finding: On some browsers, such as Safari and Chrome, the size of the navigation bar pushed down the main content of the page — not just on the homepage, but also on Search Result pages — creating frustration for multiple users. One credited the size of the Navigation bar as the reason they missed the Document Type categories when searching for the part of a score.
Recommendation 1.5: Make additional adjustments to improve accessibility of main navigation and search. The mockup above addresses the following WCAG v2.1 success criteria:
Contrast: All instances of the use of Digital Archives’ brand-blue (#1276FF), which do not meet WCAG success criteria for contrast ratios against the site’s light gray, have been adjusted to brand-blue-ada (#0E6AE7), which does.
Target Size: The size of the form fields and the “Search” button have been adjusted to a height of 44 pixels, which meets WCAG’s success criteria.
Opportunities for additional analysis and work on the accessibility of the homepage are detailed in the full report (see page 14).
2: Update style and navigation elements on the carousel to make it more intuitive and accessible
Screenshot of a carousel slide demonstrating hard-to-see navigation buttons.
Like the search bar, first impressions of the carousel were generally positive. Many users commented positively on its’ inclusion and identified it as a place to find things that the Digital Archives wanted to highlight to its visitors. However multiple users experienced issues attempting to interact with and navigate through it. Adjusting the code of a few simple elements of the carousel could greatly increase its usability.
Annotated mockup of recommendations to improve the usability of the carousel.
A summarized list of recommendations and related issues from the full report are as follows:
Recommendation 2.1: Change the color of the navigation buttons. This mockup uses brand-blue-ada (#0E6AE7) at 100% for active squares and 50% for inactive squares. Note, this is not contrast accessible on all banners, so it must be used in combination with 2.2.
Related finding: If the user didn’t experience serendipitous discovery of the content they needed by the carousel happening to be on the correct slide when they looked, the user ran into trouble attempting to engage with and move forward through the carousel slides. There are two options to use the carousel on desktop, the primary of which are squares that signify how many slides there are and enable a user to navigate forward and backward, and a dragging functionality. The contrast is on these squares is too low and few users engaged with them.
Recommendation 2.2: Navigation arrows, which are available in the Flickity carousel code, should be unhidden.
Related finding: The second option for navigation through the carousel is a dragging functionality. No users attempted to use the dragging functionality.
Recommendation 2.3: Change the color of the “Explore” buttons that appear on each header to brand-blue-ada (#0E6AE7) and the height to 44px which is accessible by WCAG v2.1 and mirrors changes suggested in Recommendation 1.5.
Opportunities for additional analysis and work on the accessibility of the carousel, alongside other websites whose use or replacement for carousels may serve as inspiration for future work by the digital archives team, are detailed in the full report (see page 17).
Key Takeaways
Our recommendations were well-received by the Digital Archives team. While our work focused on evaluating the usability of the current site and providing reasy-to-implmement recommendations, the Digital Archives team was also interested in our personal observations and recommendations for a full redesign—such as, other websites we’ve enjoyed using that we’d recommend they reference. Based on this conversation, we met as a team afterwards to compile additional recommendations for their full redesign. We added these to the full report in Appendix F.
Key takeaways, including opportunities for further research, are as follows.
Accessibility impacts usability
Accessibility guidelines are vital to follow in ensuring that a website is usable by all, and the majority of the issues our users experienced on the homepage are also accessibility barriers: the navigation bar and the carousel as explained above. Their severity is strongest for new users, who are attempting to build a new mental model of how the site works, and their occurrence for all users could have been prevented by following WCAG. While the accessibility recommendations included in our report are a strong staritng point, the Digital Archives’ next steps should be to 1) conduct a full accessibility audit to identify other accessibility issues and 2) ensure that any new features designed for the website as part of their larger redesign work follow WCAG.
Look to other sources of inspiration
In envisioning how to solve user issues, pulling examples of how other platforms address and prevent those issues serve as useful inspiration for project teams, especially those who, like the Digital Archives, have the exciting opportunity to fund a larger redesign of their website. Based on what my team learned from the conversation with our client at the end of our project, I’d like to incorporate references to other platforms from the beginning of recommendation development.