AS01 - Archival Discovery Public User Interface Comparisons - Report



At the request of ArchivesSpace Development team member, Susan Pyzynski (Harvard University), Emilie Hardman (Houghton Library, Harvard University) and Amanda Strauss (Schlesinger Library, Radcliffe Institute) conducted a series of usability tests on three Archival Discovery Public User Interfaces (PUIs) with the goal of identifying features in these sites which might be integrated into the ASpace PUI. We also worked to note elements in the sites which serve to support or inhibit users in their work with the systems. Kate Donovan (Houghton Library) and Anna Speth (Simmons GSLIS student) aided in analysis. We were further assisted by Amy Deschenes (Harvard Library) and student workers at the Harvard Library User Research Center who computed time on task values for these tests. 

Between September 16-20, 2016, 6 participants were recruited to test at Schlesinger and 8 at Houghton. For this test we recruited librarians (specifically avoiding manuscript catalogers and archivists), library assistants, and students as participants. We assigned 5 participants each to sites 1 and 2 and 4 participants to site 3. Participants ranged from advanced users to complete novices.

The Test

The three sites of interest were identified by ASpace Development team:

Site 1: NYPL - Archives Database

Site 2: Smithsonian

Site 3: Princeton

A set of 5 unique but equivalent tasks for users to perform with their assigned site were designed in order to examine the following actions:

Task 1

Task 2

Task 3

Task 4

Task 5

Locate specific collections/ finding aids

Locate series within finding aids

Locate specific items within finding aid

Locate biographic information on collection creator in finding aid

Arranging for physical access/ site visit

A script was developed and all participants signed consent forms to permit the screen-recording of their session with Morae. Notes were recorded in a grid concurrent with the tests and screen captures were reviewed to fill in detail after the fact. 

Findings and Recommendations

Task 1: Locating specific collections/ finding aids


All participants entered collection creators’ names in a [first name] [last name] pattern. Only one participant used a qualifier for their search executed on Princeton’s site, which was the only site to offer the option. Quantitative results confirm what we observed: participants using the Princeton search were uniformly quickest to launch into their searches, spending the least time scanning the home/search page.

Mean time on task in seconds:

Values in seconds:

Smithsonian NYPL Princeton
Minimum 38.4834.4820.08
Maximum 109.0679.545.01
Mean 68.8 52.7928.19
Standard 26.88 19.211.35

After executing their search, a majority of the participants scrolled around on the results page to view results, even when they acknowledged having seen their intended result as a first or second option on the page. Confusion was evident when collections and components were both returned and visible as results on the page. None of the icons used to differentiate results appeared helpful to participants; most simply did not see them and those who did did not understand what they were. One participant was unsuccessful in locating the result (NYPL), though it was the first result returned. Non-librarian participants assigned the Smithsonian site were unsure of where to click on their search results to view more details (the only linked item within the result is titled “Link: View Full Record.”)


On the search/main page: center search box on the page, employ generous padding around it and employ a neutral and uncluttered page background. Avoid or very carefully employ images.

Consider the search algorithm carefully. Participants experience the search bar as equivalent to a Google search and expect results to return in a similar fashion. Not even our librarian/archivist participants employed [last name], [first name] patterns, used quotes, or, with one exception, appeared to look for any means of refining their type of search. This was true on the search/main page and the results page.

Appropriately-sized (something between Smithsonian and Princeton), colorful, and generally recognizable icons (unlike Smithsonian’s which did not appear to carry meaning for participants) may help alleviate confusion created by the equivalencies implied by the search results that include collections, components, digital items, etc. Use padding to differentiate results from other text on the search results page to visually clarify where results begin.

Identify and employ a variable font size and color scheme within the search results for titles, creators, summaries, etc., to help users identify and scan results. Particularly emphasize titles as links. Use zebra striping to differentiate the results.

Filters which would allow users to view only results at their desired level (collection, component, etc.) may also be helpful. A function that would allow users to sort/resort results also seems desirable.

Task 2: Locating series within finding aids


Across all sites, the majority of users utilized sidebar navigation to locate the series within the collection and generally appeared not to see search bars when present.  

Users easily manipulated the “Table of Contents” on the Smithsonian site to find the series. The Smithsonian presented the most uniform path to success on this task, with all but one user quickly identifying the series in this manner. Participants noted during this task and in general comments that they particularly liked being able to see the table of contents and that this feature was useful to their understanding of the collection as a whole. This table of contents was identified as being of particular value; though it was much the same as what Princeton offers, we did not hear similar comments regarding that site.

Users struggled most with NYPL’s site on this task, with three participants clicking on various buttons on the top navigation bar in an attempt to locate the series, bouncing in and out of the “detailed description” area without immediate recognition that they might find the series within that page. One NYPL participant was ultimately unsuccessful, though found the information while completing the next task. This may be because correspondence was not actually structured as a series or subseries in this collection and therefore was not visible in the table of contents; the lack of information in the table of contents seemed to prompt users to look elsewhere for answers. It should be noted that NYPL does offer a search within this collection feature, but no participants used it, perhaps because the element was not defined by a title on the page.

Princeton participants followed entirely non-uniform paths. Only one participant used the search within this collection function, others appeared not to see it as an option. One participant used the table of contents on the left, another (a novice user) moved into the “view entire finding aid” space and used control f. The other participant clicked into the “Contents and Arrangement” space and located the item from the center pane view on that page.

Mean time on task in seconds:

Values in seconds:

Smithsonian NYPL Princeton
Minimum 15.7333.2334.09
Maximum 46.36103.3578.2
Mean 24.8567.1349.43
Standard 12.3132.3919.88


Offer a sidebar table of contents for users. Highlight their place within the table and maintain a view of the whole table. Context-specific refocusing of such a table that removes the larger view of a collection may not be useful or desirable to users. If a page has more content than fits within a screen, move the table along with the user as they scroll.

Employ padding between the elements of the sidebar (like Smithsonian), avoid dividing lines and tight line spacing (as seen on Princeton site). This styling may account for the users’ successful employment and positive orientation toward the Smithsonian’s table of contents.

Create a priority place on the sidebar navigation to offer a search within collection function. Employ generous padding and a title which defines the element.

Task 3: Locating specific items within finding aid


4 out of 5 participants assigned to NYPL scrolled through finding aid to find answer, 1 participant used the search box available on the left side of the series description. This behavior may be linked particularly to the focused arrangement of this collection; participants had all noted during the previous task that there was a section which described Jones’ compositions and observed that it was arranged alphabetically, making it efficient in this instance to locate the item manually. This behavior may also be linked to the disorienting experience of moving into between different views participants had during completion of the previous task and a desire to remain with stability in the same view.

Behavior was notably different within the Smithsonian site where all participants utilized the same method: “control f.” 3 participants (at a variety of experience levels) immediately used control f to locate the specific item. 2 participants scrolled around the page, orienting themselves further to the material before concluding that all of the items were described in this page and they could therefore seek out the needle in the haystack by employing control f. One student participant indicated in the open comment period that not all users may be familiar with this technique and that the site might therefore employ a search box. It is interesting to note that these participants were highly enthusiastic about using this site’s table of contents for the previous task, but employed a different method here, indicating that they thought it would be most efficient.

The Princeton participants again had similar behaviors which were distinct from the other sites. 3 of the 4 participants used the left navigation table of contents. The table of contents on this site had been used by only one participant in the previous task, but a group they moved to it on this task, which represents a reversal of the pattern seen with the Smithsonian site. 1 participant on the Princeton site used the search bar to locate the material.

Mean time on task in seconds:

Values in seconds:

Smithsonian NYPL Princeton
Minimum 7.8534.0915.76
Maximum 67.7878.250.76
Mean 31.9449.4335.42
Mean 2419.8814.52


As above, offer a sidebar table of contents for users, a search box, and a table of contents.

Maintain a means for users to see the whole finding aid at once and to search within it.

Both of these recommendations go to the same emergent theme: users, even when executing fairly specific searches, demonstrate an affinity for orienting themselves toward the larger collection through scrolling through a full view or skimming a table of contents.

Task 4: Locating information contained in <bioghist> *


The development team was interested in how users would look for information on collection creators contained the in <bioghist> note, however, we were not able to specifically force participants to use only that source in their search and so not all participants landed there--some taking more creative approaches using the contents of the collections to extrapolate. This perhaps points to a sense that users do not know to expect meta-information unless they are specifically cued to it. This hypothesis is supported by the consistent behavior on this task seen in the NYPL site on which a header for “biographical/historical information” is visible without scrolling when a user is in the “Collection Overview” view. 4 out of 5 participants assigned to NYPL used this section to find the answer to their task. In contrast, on the Smithsonian site, only 1 out of 5 participants used the <bioghist> (titled “Historical Note”). The remaining 4 participants all employed variants of a similar logic to seek out collection materials which could be used to infer the information they were asked for. For the Princeton site, 3 out of 4 participants attempted to use collection materials to satisfy their task as well, but it was not possible in that instance to extrapolate from the item descriptions. 2 of these participants then went on to use the “Description” to successfully complete the task.

*Note that Princeton’s site did not offer a visible <bioghist> section (unless users were in the full finding aid view) so we drew on analogous information contained within the description, but the notably longer time on task observed for that site on this task may have more to do with the way this task was designed.


Clearly signal to users that biographical information is available to them with use of a heading in table of contents and an appropriately sized font header in the main window.

Pay careful attention to language; “Biographical” and  “Information” appear to be clearer and more actionable for users than “Historical” or “Note,” for example.

This observation goes beyond ASpace, but seems important to note. In this task participants sought specific information through examination of items and/or series. In the previous tasks as well, we heard participants reflect on the limited information that was present for the items/series they were examining. Where possible, it may create an improved user experience to leverage linkages to more data about an item, person, etc. “I could just Google it,” noted one participant after an unsuccessful search for the information, but we could also think about ways of connecting people to the information from our finding aids. A link from the creator’s name to a CPF record, for example, would have provided the necessary information for this participant, as would a Wikipedia entry.

Mean time on task in seconds:

Values in seconds:

Smithsonian NYPL Princeton
Minimum 13.622.4448.17
Maximum 43.0742.21129.15
Mean 30.3133.6885.45
Mean 13.298.9335.14

Task 5: Arranging for physical access/ site visit


For NYPL and Smithsonian sites, a majority of participants expressed some degree of uncertainty about how to request access to collection materials. The Smithsonian website in particular caused confusion with a “Reference Request” form which most participants indicated they would use to request access, despite a lack of confidence that this was the most appropriate method. All participants who opened this form except one even kept looking for another, better option to request access to the materials. Participants using the NYPL site were divided on looking at the left navigation “Using the Collection” and the button on the top of the screen for “Contact the Division.” “Using the Collection” assured participants that they could see the collection, but did not actually provide instruction on how they could do so. Participants using NYPL’s contact form noted and appreciated that they had an option to indicate they were writing about the collection whose finding aid they’d been looking at. None of the participants appeared to notice that the Smithsonian offered a similar service, perhaps because that site used a collection number and not the title of the collection.

Participants were most successful on the Princeton site, finding the request function with relative ease, though only one did so through the “shopping cart” feature. However, their success in using the “Request this Box” button meant that none of them read the information presented in the “Access and Use” section. Further, none of the participants noticed the information carried over from “Access and Use” to the request form, possibly because it was presented in a small and light gray font.

Across all sites, very few participants specifically noticed or looked for information regarding time frames for retrieval of materials. Request forms also offered datepickers which presented options that would not have been possible to fulfill the user’s request.

Mean time on task in seconds:

Values in seconds:

Smithsonian NYPL Princeton
Minimum 18.2726.0414.72
Maximum 379.7974.0937.68
Mean 116.0649.1526.37
Mean 150.2219.679.4


The comparative ease with which participants were able to make confident requests from the Princeton site indicates that users should be given the ability to make online requests from within the finding aid.

Provide a clear, i.e. “Using the Collection,” link to specific and detailed information on how users may access collections could be helpful, but the Princeton findings would seem to indicate that if users are able to request materials from within the finding aid, they will not look at such a section. A collapsable section with each request form may address this problem. Stock information about the repository, such as location, hours, and any other significant matters of policy, should also be integrated.

In the request form, inherit and make actionable as much information about the collection as possible, especially regarding any restrictions and the date on which the collection can be available.

Though forms for reference questions may be useful, they need to be clearly distinguished from a request function. Display an option for users to confirm they are writing about the collection title for the finding aid on which they open the form from.

Other observations

Students in particular expressed an interest in more description and context for the materials they were looking at. Series and collection level notes did not seem to feel available to them. They also expressed a lack of surety about the document itself, asking things list: Are these descriptions of things that are digitally available? Is this a description of material available to view in a Reading Room? Is this a document a sort of bibliography? Working throughout this design process to think about how information which offers both broad and specific detail and context without overwhelming users seems like an important area for further consideration through the design process.

In the course of our conversations about the results of these tests, we came back several times to a sense that it was important to consider that the PUI is likely to be most used for discovery by staff and users who have been initiated to the system. In our 2015 tests across 5 Harvard special collections and archives, we found that patrons, even advanced users, uniformly started with Google and so came into finding aids that way. This may mean that we could be well-served to test discovery with these user groups specifically and to respond to their needs, which may include more sophisticated and fine-tuned search options while emphasizing other user groups needs within the finding aid itself.

Submitted by: Emilie Hardman, September 25, 2016