2021-2022 Metadata Standards Retrospective

Owned by Valerie Addonizio
Last updated: Jun 19, 2023 by Regine Heberlein
7 min read

2021-2022 Workplan

https://archivesspace.atlassian.net/wiki/spaces/AC/pages/2889777217

Accomplishments

  • Completed an update to the MARC importer with significant improvements:

  • Completed re-working and actual data modeling for the EAD2002 importer with thanks to Regine

  • Addressed default behavior for MARC 300 fields

  • Identified desired behavior for logging

  • Commented on tickets as appropriate

  • Presented at the 2022 Online Forum and solicited community feedback

On compliance versus interchangeability

  • Accounting for people who said both, roughly 10 votes for compliance/14 for interchangeability

  • 2 mentions of ISAD(G) and 1 mention of RAD

  • “You need compliance in order to exchange the data”

  • “I think it will ever be impossible to formally state AS is compliant to any standard, rather that its data model is informed by a variety of archival and library description and it 'supports compliance and interchange' with a variety of standards.”

  • Summary from Greg in the video: “Not all standards compliance encourages interoperability; might be the opposite”

On Elizabeth’s MARC topic

  • Beyond mappings, there are some concepts wholly missing (Main Entry in MARC does not have a clear equivalent in DACS [Kate argues choosing the creator for a collection name is a Main Entry]) which make mapping difficult

  • “59X would be one that may be problematic to leave out? We have a tricky catalog and end up using 59Xs a lot”

  • “I can see adding an additional job/config setting that allows you to list the excluded fields to import. So all real mappings would import and everything not in the excluded list would map to a note.” Conversation went on to agree with this, but switch it to an include list instead.

  • Piloted working meetings to help members get work done - it was acknowledged as a good idea but it didn’t work for everyone

  • Improved reactive strategies for tracking changes

    • Tested and implemented to be automatically notified of changes made to the import/exporter code

    • Tested and implemented a subscription to a Jira filter for the Metadata tag

  • Tiers of Support - Attempt at a more transparent process for how we do what we do

Priorities for next term

  • Continue work on the EAD importer in earnest. Plan work to accommodate those who may be able to read Ruby and those who cannot. As a preview for next term, I asked Jared to give his feedback on the work compared to the work just undertaken for the MARC importer. Please see below.

I think it will be a little more difficult for those folks who are not used to reading Ruby then it was for the MARC importer.  Unlike the MARC importer, The EAD importer uses a lot more generic templates (methods) for dealing with XML chunks.  For a newcomer I think it would be much more difficult to look at this code and make sense of how ASpace in pulling in EAD xml data and transforming into its local format. 

Because of the complex nature of the EAD xml people looking at the code will need to be able to navigate through several Ruby methods to understand how the data is being pulled and mapped  (this means skipping around the code depending on what chunk of EAD is being pulled).  

Documenting the EAD exporter will require folks who have a strong background in the EAD tag library (given), but I also think that it will require some people with a fairly high level of comfort reading Ruby.   I think an ideal subteam to work on such a project would be two people really familiar with the common EAD encoding practices (and the Tag library as a whole) working on importing test files and looking at the results and 1-2 folks who were comfortable reading Ruby to translate what the code is claiming to be doing.

  • Proposed strategies for the EAD work as discussed in the last meeting of term:

    • Invite Brian Hoffman (ASpace Dev) to present on the EAD importer. Key to this would be preparing a few concrete questions or asking for a demonstration of how he himself would fill out the spreadsheet. Early efforts to de-mystify the code would benefit the whole group.

    • Remember and embrace the fact that it took three years to complete the MARC importer (and there’s still some work to be done!) so take some pressure off: this work may not happen fast and it doesn’t need to. On that same note, embrace this by preparing for how this work could be spread out: would it be more like chunking the work (starting at the top and dividing into sections) or more like iterative processing, where there are multiples passes across the whole, each of which accomplishes something.

  • Revisit as a principles document instead of a statement on scope

  • Spec for dropping MARC import support:

    • Pull all the non-core fields (2xx, 3xx, 754, 856) except the 5xx’s; for the 5xx’s, everything that is a core field gets mapped, but everything that is a 5xx just gets a mapping to a local note, like a 590. We know this a relatively big change, maybe put this in the roadmap, so that it doesn’t happen quietly. Elizabeth volunteers to write that spec, including the justification. You’ll get a list of what was imported.

  •  Spec for , folder for spec documents here:

    • Update since retrospective: Nevermind! Looks like Princeton sponsored this

  • Bring up the retiring of MARC behaviors at the next joint UAC/TAC meeting, which will be in the fall. @Elizabeth Roke nominated to handle that communication

  • If we get Kevin’s ticket, we should do an email announcement to the listserv and Google Group

  • NEW Task Force idea - Suggest an ad hoc taskforce to update the ASpace data model

    • Continue working on the justification of that idea as recorded in this document (TAC-MD subteam members should have access to that link without a problem)

    • Note that there is an old (pre-container and location modeling) entity relationship model archived in the Wayback Machine:

    • Valerie has tested Whimsical, Regine suggests Mermaid

    • Next steps: Present this idea at an early TAC meeting and prepare for the follow through on determining leadership. Also formalizing the document linked above. Note that an ad hoc group can be made up of anyone, but leadership and structure should probably come from within TAC.

  • NEW Pilot a Rotating group of ex officio’s from Metadata who sit in on Dev Pri meetings

    • This excellent idea came from the acknowledgement that there’s only so much we can ask of our fellows to alert us to changes, and sometimes they may simply not be aware that these changes will impact TAC-MD

    • The idea was raised that we begin a rotating roaster of Metadata reps who sit in on Dev Pri meetings to listen and know when a ticket should be tagged for us to review. This is a strategy to get us involved further upstream.

    • We settled on the idea that one person volunteers each term, and one additional person volunteers as a backup in case the volunteer cannot make a meeting. Kevin volunteers for next term, with Elizabeth as a backup

    • Next step is: reaching out to Dev Pri with this idea and getting the volunteer(s) invited to their meetings

  • RiC review

    • Essentially we know we absolutely must render a verdict on this, but we decline to do so on a draft. We have doubts about the stated aspirations (transmission standard, descriptive standard, one ring to rule them all). We want to be included in the decision of how ASpace reacts, and not just the recipients of said decision.

Retrospective in Brief

Start doing

Stop doing

Keep doing

Start doing

Stop doing

Keep doing

  • Increase community engagement

    • Reports out to the community via the listserv (i.e. we’ve done this, what do you think?)

    • Engage ticket writers and commenters (invite to meetings, ask to work on specs)

  • Have Program Manger and/or Community Engagement Coordinator attend quarterly for questions

  • Ex Officio idea (see above)

  • Working meetings was acknowledged as a good idea but it didn’t work for everyone, so this isn’t so much a “stop doing” as a “continue if needed/wanted”

  • Presenting at the Online Forum

  • Looking for ways to be included upstream on relevant changes

  • Reminder of Action Items in email form

  • Explicit and helpful links in agenda and emails

  • GitHub File Watcher

  • Dedicated reading time in the meetings

Recommendations for finishing up this term

Recommendations for being included further upstream was one of our last kind-of unsettled tasks for term, so it was discussed as part of the retrospective. This discussion lead to the idea of the ex officio’s attending Dev Pri meetings.

A reminder on some background context:

  • Some of this feeling was generated after all the changes that came with 3.0, but we’ve moved forward on that concern forgetting that we were invited to participate and declined. So this is about aligning both our own expectations about our work and being included further upstream in changes.

  • This is a good example of where things that the metadata group should weigh in on are buried in specification documents and tickets.  Note that the specification includes a request to amend the EAD and EAD3 exports

  • Communicate it to Christine, the developers, but also TAC and Dev Pri.

  • More about a heads up than an approval process; fundamental to needing to track changes to documentation. Even if it becomes the primary job of this group to actively document and track those changes. We need to be tagged any time anything touches an importer/exporter

Actionable tasks and ideas

Mention tagging Metadata at the final meeting of this term
Mention tagging Metadata at the first meeting of next term
Focus on the EAD2002 importer
Ex Officio idea! Seek orange text in the sections above for more details; that idea came out of this last-minute discussion
Data model task force idea. Seek green text above. Mention this in an early TAC meeting to get the ball rolling.

Leadership for next term

Lead: Regine Heberlein

Vice Lead: To be identified from new members next term

Outgoing Lead: Valerie Addonizio