AS02 - ASpace PUI Navigation/Search Usability - Report


Overview


At the request of ArchivesSpace Development team member, Susan Pyzynski (Harvard University), Emilie Hardman (Houghton Library, Harvard University) conducted a usability test on the ASpace PUI with focus on navigation and searching within a collection. Tests were conducted at the Harvard Library User Research Center with 5 participants recruited by the Harvard Student Agencies on Friday, October 7, 2016. Participants were sophomore, junior and senior students at Harvard University. 2 students were in the STEM field, one of whom had used archives out of personal interest in the past, while the other student had no experience. 3 students were arts and/or social sciences concentrators (majors) and 2 had used previously archives for class assignments.  Notes were recorded in a grid concurrent with the tests and screen captures were reviewed to fill in detail after the fact.


The Test


A script was developed to help structure the experience of each participant. The purpose of the site was briefly explained and a scenario was offered to guide the activities. Students were told that a professor had assigned a research paper on a person named John Jeffries and instructed them to make substantive use of Jeffries archive as a primary source of evidence. The professor told students that the Jeffries papers were held by Harvard’s Houghton Library and that they should look them up online to determine what parts of the archive they might want to use before going into the library. Students were told they’d gone to the website their professor supplied and entered “John Jeffries,” in a search bar and the first link returned opened to: http://aspace.hudmol.com:7310/repositories/5/archival_objects/8157  This page was preloaded for participants.


Two factors led to the decision to start participants at a component level. The first is that current ASpace search results return objects before collections. The second relates to a question about how participants would orient themselves to the collection as a whole when beginning on a specific object within it, a scenario which is possible and even likely in the future, even if search is reconfigured to supply collection level results first.


Participants were presented the object page without additional context from the test moderator. They asked for their initial impressions/sense-making and were then asked to complete 5 tasks designed to elicit various search behaviors within a specific collection. These tasks were also designed to be comparable to several of the tasks from the previous comparative PUI tests [link]. We did not test for how to arrange for physical access or make a site visit as currently no provision has been made to support a user in obtaining this information. Each session lasted between 20 and 30 minutes and was recorded with Morae.


Prior to testing we also calculated average and mean values for time on task with users who had some familiarity with the ASpace system. This value offers a opportunity to compare an efficient means of task completion with the time it takes for a new user to understand how to accomplish a task within the site. Due to a corruption of video files on transfer from Morae, however, it was not possible to extract all of the time on task values desired.



The Tasks

Task 1

Tasks 2 + 5

Task 3

Task 4


Locate overview of collection contents


Locate non-specific group of materials


Locate biographic information on collection creator in finding aid


Locate specific items within finding aid

Findings and Recommendations


Initial Impressions


Summary

Dropped into an archival object page, a majority of the participants expressed confusion and some frustration that there was not enough information on the page for them to determine what they were looking at and what it meant. Three participants commented on the large expanse (“excessive”) amount of white space on the page and wondered specifically if they were “looking at the right thing” because there didn’t seem to be enough on the page. “Maybe it didn’t load?” a participant asked, thinking the page display was an error. For another participant, the result felt “disembodied.” The minority opinion, expressed by one participant, was that the site had a “nice clean design” and felt useful and orderly.


40% of participants were able to express understanding within the first few minutes of observation that they were looking at a record for an item from within a larger Jeffries collection, but only one participant was able to make that determination directly from the item page, the other had to click around for context first before returning to the page and characterizing it correctly. Other participants were able to identify the page as some kind of record for a diary, but did not until future tasks, or not at all during the test, express understanding that it was linked to related to other materials in a collection.


Two participants indicated that their first impressions of this page was that they were not qualified to use it. One specified that it felt “esoteric” and alienating.  




“I feel off the bat that I’ve landed on something very esoteric.

Really something for people who specialize in this kind of research.”




More specific concerns and questions on the page were also raised. The “Archival Object” icon was seen as actionable by a majority of participants and most tried to click it, assuming it was a button and that it might produce a useful result, e.g. load digital images, etc.


icon_or_button.jpg

Confusion over identifiers/ids was also mentioned by all participants as an initial point of confusion. Two participants suggested that identifier was the most important thing and would be what they might copy to bring to a librarian. Another suggested that the component was probably important. No participants expressed confidence that they knew what any of the ids meant. The three different numbers on this page was called “overwhelming,” “confusing,” and something relevant probably only to “advanced scholars.” Comments on this issue led to two participants indicating the site had an awkward and contradictory mix of “too much unuseful stuff everywhere and too much white space.”


ids.jpg


Four participants struggled the prevalence of dates on this page as an additionally confusing element. While most participants were able to parse the title and express understanding that the first range of dates, “1745-1819,” were Jeffries’ birth and death dates and the second range of “May-December 1786” were the dates the diary was kept, some were confused about the breadcrumb date range of “January 1-July 21., 1779” (“what does that have to do with this diary?”) and the creation dates at bottom of the page. In fact, this particular example may have an heightened effect because it’s a nonsensical statement of “Creation: 1786 Creation 1786,” which may in fact be a repeated field error or an anticipation of date range(?). Regardless of the error though, participants were confused by any repetition of date data from the title field.


date.jpg


Recommendations


Rebalance the page layouts to make better use of available space. To contribute to a sense of completeness, include information on each page situating the material within the collection, e.g. present both higher and lower levels of description. Accounting for the fact that a user may enter the collection at any level, it may also be appropriate to consider making the collection’s abstract consistently visible. It may also be useful to consider restyling breadcrumbs to emphasize their existence, to provide users with a sense of their hierarchical arrangement, and to offer some insight into the meaning of this hierarchy.


Consider ways of making each page feel actionable to provide an answer to the user’s questions of “what can I do here?” Recognizing that request buttons may not be implementable across the board, users still need some sense of how they might act with this information, e.g.: What do I do to learn more? Can I request digital images? What do I need to do to get access in person? Etc.


Restyle icons (this surfaced on the initial impression, but carries through participants’ experience with the site). Giving users unique colors, shapes, and/or images may help orient them to different levels within the collection, but these should truly be unique (e.g. series, subseries, and items are all green “archival objects,” which was confusing to participants in this study) and should be styled separately from elements such as subject, people and organization, etc.


Add labels to information that clearly identify the nature and purpose of the data displayed. Make use of space on the page to present these data in orderly, easily scanable arrangements. Offer collapsable or mouse-over information for elements which may not make intuitive sense to users at all levels of familiarity with archival research, especially call number and component ids/box numbers/etc. Call numbers should also be visible at every level of the collection. Currently only display on Collection level page (bug?).


Hide the identifier; this information is available from the url and only serves to create confusion and visual clutter. Consider hiding or masking the ref. id. Relabel component information and concatenate it with the call number. Label/style to make clear the utility of this information.


Consider restyling creator dates to visually separate and/or de-emphasize dates. The Wikipedia parenthetical standard may be useful, e.g. Lucy Eldine Gonzalez Parsons (c. 1853 – March 7, 1942).  Date information is present in title of object and in a special “creation” field: presenting it in the creation field may be more effective and sensible to users and may obviate the need for it in the title.



Task 1: Locate overview of collection contents


Summary


In the first task participants were asked to use the site to find a way of getting an overview of all of the contents of the collection. The expectation was that participants would identify the Contents and Arrangement section and skim it to verify that this is an area where they will be able to see descriptions of the collection’s contents. Three people familiar with the ASpace interface also performed this task to provide a comparative measure that could serve as a baseline for efficient task completion.


The Contents and Arrangement section was only consulted in the course of this task by one of the participants. The participant read through the three top-level series displayed on the main collection page, clicked on the first series (trying first to click the arrow icon before realizing only the linked words were actionable) and scanned the items in it. He did not return to the collection page to see the contents of the other two series. Two participants made use of the Scope and Content note and indicated that it told them what they needed to know about the contents of the collection. The remaining 40% of participants could not find a way to obtain an overview of the collection. One indicated that he would “just go to a librarian” for assistance.




“I think I probably would have just Googled this is find out

what’s in the collection, but you told me to stay here.”




60% of the participants made use of the breadcrumbs in some way during this task. There was, however, no consistency with use: one participant clicked on “Houghton Library” (“because it’s logical to look at the first one in this list”), one on the next level up (a subseries of diaries), and one on the “John Jefferies Papers.” The other participants appeared not to see the breadcrumbs.


Getting to Overview of Collection



Participants

Experienced Users

Difference

Minimum

54.4

12.8

.43

Maximum

240.5

22.8

218

Mean

178.2

19.4

165


These data are from the three successful participants and are meant to help contextualize the experiences, not to serve as scientifically rigorous measures of a time on task. Certainly “success” this task was loosely defined, but it should have had a clear and quick path to obtaining a general overview of the collection. At the efficient end of the spectrum we see only about half a minute’s difference between the participant and the experienced user, but at the upper and middle ranges we see differences of almost three to more than three and half minutes. Outside of a testing environment a user would be highly unlikely to spend that much time trying to understand and navigate a site.


Recommendations


Provide a table of contents on every page view a user might encounter. Collapsed views may be sufficient, but very clearly identifiable icons should be employed to indicate more levels follow. These icons, if used for collapsable views, should either be incorporated into the textual link or themselves work to open the additional levels. Currently, the arrows used to indicate more content is available are not actionable and participants tried to click them with no result.


The contents should remain fixed in place (not adjust as a user clicks on different series, items, etc.) and should allow a user to see all levels of the collection on one page. Improved padding around the various titles in the contents may aid scanablity.


“Contents and Arrangement” may not be the most effective and intelligible label for this section. A future test will make further recommendations about labeling.


As above, this task points to the utility of restyling breadcrumbs to emphasize their existence, to provide users with a sense of their hierarchical arrangement, and to offer some insight into the meaning of this hierarchy. This seems particularly important for users who may come in at a component level and need a sense of how it relates to the collection in which it is situated.



Tasks 2 and 5: Locating non-specific materials within finding aids


Summary


Two different tasks were designed to see how participants would go about searching for materials with a broad theme or function. Two opportunities were provided with different subject matter to allow for some extra exposure to the user experiences in this core component of the research process. The tasks were spaced out through the test, one at the beginning and one at the end, to see if additional experience with the site would make it easier for participants to get results more efficiently. In the first task participants were asked to find any materials about Jeffries’ ballooning activities and in the second they were asked to find materials on Jeffries’ surgical activities.


A majority of participants were unable to complete the first task. Only two participant were successful, though they both also struggled to obtain viable search results. Persistence and creatively employed search term iterations helped those participants ultimately complete the task by getting a set of results. Three out of five participants used the search box to get results, although two expressed an initial lack of confidence that it was going to search only within the particular collection they were interested in. The reference to “Search Collection” within the box participants said they thought could refer to the whole site--a collection of the collections, so to speak. The participants who did not use the search box initially seemed not to see if as an option and spent time unsuccessfully clicking and scrolling until the moderator moved them to a new task.



bound_to_be_in_here.jpg

          mouse trail of a participant indicating that the items would be “in here somewhere…”



Those who searched tended to enter short strings like “balloon trip,” “ballooning trip,” “English Channel.” These strings landed them on results pages which indicated they were “Showing Results 1 - 0 of 0,” participants were confused by the language (“One of zero? Should I be seeing results here?”)



results.jpg

The lack of results and the language contributed to the sense of an error. On failed search pages, participants wanted to attempt a new search and they moved to enter a new term in the only box on the page, a date filter. Most realized the mistake, but one participant entered a new term in the filter and executed a search. Because the filter has no real data validation, the search goes through without specifically issuing an error, although it yields no results and is entirely ineffectual.  Participants had to make use of the back button to begin a new search.


Several participants were notably frustrated with the search and felt that they were using words that should yield results. When participants pared down to searches like “balloon,” they returned to this theme and wondered why “ballooning” wouldn’t have worked (“does it have to be that specific?”)




“It doesn't work like Google and I’m doubting the algorithm. It's a little bit frustrating that I'd have to waste time trying to figure out how this search algorithm works. I can almost guarantee then that there might be like twenty more results that I'm not seeing. So I don't feel confident."




Two participants also commented specifically on the number of search results returned and the fact that they could not create a more specific initial search to limit returns or search within their results. Though a filter option is offered for dates, the function does not appear work currently.  


In the second task on this theme (positioned at the end of the test) participants were asked to locate any of Jeffries’ writings or notes on surgical topics. In this case all were at least somewhat successful. This may have something to do with having a little more experience with the site, but may also be because the word “surgical,” which they were given, yielded immediate results when searched and there were no complicating terms or concepts to add as in “balloon travel,” a reasonable addition participants made to the first search.


Again in this task, not all participants used the search bar. Two participants scrolled through the collection contents to locate the material. Of those who searched, two also performed tests of their own on the search algorithm, testing it with “surgery,” “surgeon,” “surgical” and “medical.” They were disappointed to note that “surgery” and “surgeon” did not return any results (“It's kind of crazy to me that if I type in surgeon it's not going to give me the surgical results!”)


A participant who returned a list of surgical notes in the collection clicked on one of the results and said he thought he’d found something that would satisfy the task, but that it also felt like a “dead end.” With his mouse, he circled the white space on the page.




“This and other pages feel like dead ends. I want to know: like how do I get it? Can I see here at least one or two pages to see if it's useful? What do I even do from here?”




dead_ends.jpg

                mousetrack circling white space as participant describes feeling like he’s hit a dead end


Recommendations

Invest in the search algorithm and experiment with ways of returning results more in line with the expectations of users familiar with Google. Several users also recommended autofill or suggestions of other search terms, especially if a search is unsuccessful.


Provide means of executing more complex searches initially and implement a function that will allow users to refine a search they have already run.


Address date filter bug.


Consider new styling and language for the search results page when no results are returned to avoid the sense that it is an error.


Restyle Contents and Arrangements to avoid the block of text effect and make it more scanable for those who may use it to identify components of interest.


Address the sense that a page may be a “dead end” by adding information about how to access the material.



Task 3: Locating specific items within finding aid


Summary


Participants were asked to locate a specific item that Jeffries wrote reflecting on his balloon ascent. There is only one item within this collection that would fulfill the terms of this task, unlike previous tasks described, in which groups of materials would have served the hypothetical research interest. This task was also designed to have participants spend a little more time on an item/component level page because they were asked additional questions about the data presented on this page.


40% of participants were unable to complete this task. One attempted to use the search box, the other scrolled through the Contents and Arrangement list, but neither were successful. These methods were also both used by people who did successfully complete the task, in fact, one participant started by searching and moved to scrolling because he was not successful in executing a search that would return results.


In an attempt to help the participants make sense of the application, the initial introduction to this test included a statement that ArchivesSpace was “like a Hollis (Harvard Library catalog) for archives.” This meant that participants were more likely than they would have been otherwise to have the concept of descriptive surrogacy available to them as a framework for their encounter with ASpace. More confusion may have been experienced without this guidance, but it did not entirely serve to fix the anticipated issue. It is also likely to be the reason several participants compared this component page unfavorably to a record for a book. It was “like a Hollis page, but really blank” or “missing all of the detail I get to see for a book, especially like a link for it online or something about how I get it, like where it is in the library.” Others also expressed confusion over how to get the material and specifically asked where the link to digital images was. This task seems to have been a tipping point for some participants who, in the midst of it, expressed comprehensive frustration and confusion.




“From the moment I saw the site I was automatically like I’m going to be frustrated and, here, exactly, it’s just not giving me the right details.”



as03_white_space.jpg

screengrab illustrating mouse trail of the participant expressing frustration


When prompted to consider the details on the page, most participants seemed to struggle with the layout. One participant was immediately clear on the breadcrumbs and their purpose and was enthusiastic throughout about them. Others took longer to understand their function, if they did at all. The breadcrumbs may have visually interrupted the logical flow of information on this page, contributing to participants’ struggle to situate the information beneath breadcrumbs as descriptive data for the component described in the title. The lack of a label for the creation data and component information was also identified as a confusing element (“not sure what this is referring to…?”). Finally, “Immediate Source of Acquisition” caused three participants to initially think, with excitement, that this was a page element that would allow them to get access to the material. Reading the data under this label clarified the meaning, but they remarked on the rapid cycle of excitement-confusion-disappointment these terms (immediate and acquisition) provoked for them.


component.jpg



Comparing the time it takes to find a specific item in ASpace to the other PUIs tested in previous work, we find that similar users experiencing the Smithsonian or Princeton sites for the first time to locate a specific item in half the time.  These values are for the mean time on task in seconds.

Time_on_tasks__AS01.jpg


Recommendations


As previously, invest in search algorithm and restyle the collection search button to encourage use.


Consider ways of filling in the blanks. Offer users more data, perhaps from higher levels of description. Offer users a sense of purpose and action, especially in the form of requesting physical or digital access.


Consider moving the breadcrumbs (indent or move to new location under component data) to reunite the component data elements with the title/header.


Create labels for all data elements/groups to help contextualize them for users.


Seek new language to describe acquisition data.



Task 4: Locating information contained in <bioghist> / <scopecontent>


Summary


In this task, participants were asked to identify who Jeffries flew with in his famous ballooning trip across the English Channel. This information is prominent in the <bioghist> and <scopecontent> notes for this collection.


As in the previously conducted tests comparing a variety of archival discovery PUIs, this group of participants did not expect to be able to find, or did not expect to be able to easily find contextualizing historical details on this site.




"Honestly, I'd do a lot of things outside of this website because I don't expect it to tell me anything about the big historical context. So I'd look up a bunch of this stuff to get a better sense of dates and locations, and info about what things I'd really want to look at." [Later, reading Scope and Contents] “...actually it’s giving me good information."



60% of the participants were unable to complete this task. One participant indicated if not for the artificial circumstances of the test, he never would have looked for the information on this site. The non-successful participants looked in a variety of places for information, but did not extract the relevant information.One participant expressed surity that the information was there but that he just didn’t know how to find it because he lacked sufficient background knowledge, while another hypothesized that this site wouldn’t be the one to give that information. The third participant expressed a sense of futility and a clear desire to move to another task.


Of the two successful participants, one first looked at some of the component descriptions in Contents and Arrangement, before hypothesizing that Benjamin Joy Jeffries was the person who made the trip with Jeffries. When prompted to find support for this guess, the participant noted that Benjamin Joy was writing about the one hundredth anniversary of the flight and so actually was probably not a good guess. The participant spent just over two minutes seeking the correct information elsewhere before locating it in the Scope and Content note. The other successful participant’s quote is displayed above and his time to identify was 46.28 seconds. The fastest path to success with ASpace on this task maps to the slowest path to success experienced by first time users of the Smithsonian and NYPL sites, aligning more with the generally less-efficient user experience on Princeton’s site for this task.


Minimum time spent on task by a participant

biog.jpg


Maximum time spent on task by a participant

Time_on_tasks__AS02_max.jpg

Recommendations


Restyle <bioghist> and make it a feature on the component pages. Consider offering this information in the form of a sidebar populated with additional SNAC data and/or Google Knowledge Graphs.


Other observations


Two enhancements were requested during wrap-up by all but one of the participants. The most time was spent across all tests discussing participants wish for images of the material. This was sometimes phrased as a desire for digital access, which is of course not always possible to provide, a fact that was understood, and even preemptively articulated by participants. The function of an image or images seemed to be something that participants thought would help them vet and understand better what the material being described to them was. It was, they emphasized, not just decorative, it would be confirming and help them make decisions about whether or not the material could be useful to them, “even if it was just one or two pages.”


The other desire clearly articulated by participants was a “more robust search engine.” Participants wanted the site to provide suggestions for other search terms, especially if search results were not returned. They also were adamant about a more sophisticated search function; the current search eroded trust in the site and damaged participants’ confidence in their ability to make any positive use of the site.





Submitted by: Emilie Hardman, October 19, 2016