Table of Contents
Overview
At the request of ArchivesSpace Development team member, Susan Pyzynski (Harvard University), Emilie Hardman (Houghton Library, Harvard University) and Amanda Strauss (Schlesinger Library, Radcliffe Institute) conducted a series of usability tests on three Archival Discovery Public User Interfaces (PUIs) with the goal of identifying features in these sites which might be integrated into the ASpace PUI. We also worked to note elements in the sites which serve to support or inhibit users in their work with the systems. Kate Donovan (Houghton Library) and Anna Speth (Simmons GSLIS student) aided in analysis. We were further assisted by Amy Deschenes (Harvard Library) and student workers at the Harvard Library User Research Center who computed time on task values for these tests.
...
Between September 16-20, 2016, 6 participants were recruited to test at Schlesinger and 8 at Houghton. For this test we recruited librarians (specifically avoiding manuscript catalogers and archivists), library assistants, and students as participants. We assigned 5 participants each to sites 1 and 2 and 4 participants to site 3. Participants ranged from advanced users to complete novices.
The Test
The three sites of interest were identified by ASpace Development team:
...
Site 1: NYPL - Archives Database http://archives.nypl.org
Site 2: Smithsonian http://siarchives.si.edu/collections
Site 3: Princeton http://findingaids.princeton.edu
A set of 5 unique but equivalent tasks for users to perform with their assigned site were designed in order to examine the following actions:
...
A script was developed and all participants signed consent forms to permit the screen-recording of their session with Morae. Notes were recorded in a grid concurrent with the tests and screen captures were reviewed to fill in detail after the fact.
Findings and Recommendations
Task 1: Locating specific collections/ finding aids
Summary
All participants entered collection creators’ names in a [first name] [last name] pattern. Only one participant used a qualifier for their search executed on Princeton’s site, which was the only site to offer the option. Quantitative results confirm what we observed: participants using the Princeton search were uniformly quickest to launch into their searches, spending the least time scanning the home/search page.
...
After executing their search, a majority of the participants scrolled around on the results page to view results, even when they acknowledged having seen their intended result as a first or second option on the page. Confusion was evident when collections and components were both returned and visible as results on the page. None of the icons used to differentiate results appeared helpful to participants; most simply did not see them and those who did did not understand what they were. One participant was unsuccessful in locating the result (NYPL), though it was the first result returned. Non-librarian participants assigned the Smithsonian site were unsure of where to click on their search results to view more details (the only linked item within the result is titled “Link: View Full Record.”)
Recommendations
On the search/main page: center search box on the page, employ generous padding around it and employ a neutral and uncluttered page background. Avoid or very carefully employ images.
...
Filters which would allow users to view only results at their desired level (collection, component, etc.) may also be helpful. A function that would allow users to sort/resort results also seems desirable.
Task 2: Locating series within finding aids
Summary
Across all sites, the majority of users utilized sidebar navigation to locate the series within the collection and generally appeared not to see search bars when present.
...
| Smithsonian | NYPL | Princeton | |
| Minimum | 15.73 | 33.23 | 34.09 |
| Maximum | 46.36 | 103.35 | 78.2 |
| Mean | 24.85 | 67.13 | 49.43 |
| Standard | 12.31 | 32.39 | 19.88 |
Recommendations
Offer a sidebar table of contents for users. Highlight their place within the table and maintain a view of the whole table. Context-specific refocusing of such a table that removes the larger view of a collection may not be useful or desirable to users. If a page has more content than fits within a screen, move the table along with the user as they scroll.
...
Create a priority place on the sidebar navigation to offer a search within collection function. Employ generous padding and a title which defines the element.
Task 3: Locating specific items within finding aid
Summary
4 out of 5 participants assigned to NYPL scrolled through finding aid to find answer, 1 participant used the search box available on the left side of the series description. This behavior may be linked particularly to the focused arrangement of this collection; participants had all noted during the previous task that there was a section which described Jones’ compositions and observed that it was arranged alphabetically, making it efficient in this instance to locate the item manually. This behavior may also be linked to the disorienting experience of moving into between different views participants had during completion of the previous task and a desire to remain with stability in the same view.
...
| Smithsonian | NYPL | Princeton | |
| Minimum | 7.85 | 34.09 | 15.76 |
| Maximum | 67.78 | 78.2 | 50.76 |
| Mean | 31.94 | 49.43 | 35.42 |
| Mean | 24 | 19.88 | 14.52 |
Recommendations
As above, offer a sidebar table of contents for users, a search box, and a table of contents.
...
Both of these recommendations go to the same emergent theme: users, even when executing fairly specific searches, demonstrate an affinity for orienting themselves toward the larger collection through scrolling through a full view or skimming a table of contents.
Task 4: Locating information contained in <bioghist> *
Summary
The development team was interested in how users would look for information on collection creators contained the in <bioghist> note, however, we were not able to specifically force participants to use only that source in their search and so not all participants landed there--some taking more creative approaches using the contents of the collections to extrapolate. This perhaps points to a sense that users do not know to expect meta-information unless they are specifically cued to it. This hypothesis is supported by the consistent behavior on this task seen in the NYPL site on which a header for “biographical/historical information” is visible without scrolling when a user is in the “Collection Overview” view. 4 out of 5 participants assigned to NYPL used this section to find the answer to their task. In contrast, on the Smithsonian site, only 1 out of 5 participants used the <bioghist> (titled “Historical Note”). The remaining 4 participants all employed variants of a similar logic to seek out collection materials which could be used to infer the information they were asked for. For the Princeton site, 3 out of 4 participants attempted to use collection materials to satisfy their task as well, but it was not possible in that instance to extrapolate from the item descriptions. 2 of these participants then went on to use the “Description” to successfully complete the task.
*Note that Princeton’s site did not offer a visible <bioghist> section (unless users were in the full finding aid view) so we drew on analogous information contained within the description, but the notably longer time on task observed for that site on this task may have more to do with the way this task was designed.
Recommendations
Clearly signal to users that biographical information is available to them with use of a heading in table of contents and an appropriately sized font header in the main window.
...
| Smithsonian | NYPL | Princeton | |
| Minimum | 13.6 | 22.44 | 48.17 |
| Maximum | 43.07 | 42.21 | 129.15 |
| Mean | 30.31 | 33.68 | 85.45 |
| Mean | 13.29 | 8.93 | 35.14 |
Task 5: Arranging for physical access/ site visit
Summary
For NYPL and Smithsonian sites, a majority of participants expressed some degree of uncertainty about how to request access to collection materials. The Smithsonian website in particular caused confusion with a “Reference Request” form which most participants indicated they would use to request access, despite a lack of confidence that this was the most appropriate method. All participants who opened this form except one even kept looking for another, better option to request access to the materials. Participants using the NYPL site were divided on looking at the left navigation “Using the Collection” and the button on the top of the screen for “Contact the Division.” “Using the Collection” assured participants that they could see the collection, but did not actually provide instruction on how they could do so. Participants using NYPL’s contact form noted and appreciated that they had an option to indicate they were writing about the collection whose finding aid they’d been looking at. None of the participants appeared to notice that the Smithsonian offered a similar service, perhaps because that site used a collection number and not the title of the collection.
...
| Smithsonian | NYPL | Princeton | |
| Minimum | 18.27 | 26.04 | 14.72 |
| Maximum | 379.79 | 74.09 | 37.68 |
| Mean | 116.06 | 49.15 | 26.37 |
| Mean | 150.22 | 19.67 | 9.4 |
Recommendations
The comparative ease with which participants were able to make confident requests from the Princeton site indicates that users should be given the ability to make online requests from within the finding aid.
...
Though forms for reference questions may be useful, they need to be clearly distinguished from a request function. Display an option for users to confirm they are writing about the collection title for the finding aid on which they open the form from.
Other observations
Students in particular expressed an interest in more description and context for the materials they were looking at. Series and collection level notes did not seem to feel available to them. They also expressed a lack of surety about the document itself, asking things list: Are these descriptions of things that are digitally available? Is this a description of material available to view in a Reading Room? Is this a document a sort of bibliography? Working throughout this design process to think about how information which offers both broad and specific detail and context without overwhelming users seems like an important area for further consideration through the design process.
...
In the course of our conversations about the results of these tests, we came back several times to a sense that it was important to consider that the PUI is likely to be most used for discovery by staff and users who have been initiated to the system. In our 2015 tests across 5 Harvard special collections and archives, we found that patrons, even advanced users, uniformly started with Google and so came into finding aids that way. This may mean that we could be well-served to test discovery with these user groups specifically and to respond to their needs, which may include more sophisticated and fine-tuned search options while emphasizing other user groups needs within the finding aid itself.
Submitted by: Emilie Hardman, September 25, 2016