On 21 February, my Trove API keys were cancelled without warning. A week later, I met with NLA staff and was shocked to be told that downloading ‘content’, such as the text of digitised newspaper articles, was regarded as a breach of the API terms of use. Without API access I can’t continue my work helping researchers make use of Trove. More generally though, the NLA’s actions threaten innovative digital research. This post tries to answer some questions raised by my first two posts, and provides some updates on recent actions by the NLA.

What’s an API key?

You might be wondering what an API key is and why it’s important. At its heart, it’s all about access to data. The Trove API delivers information from Trove in a form that computers can understand and process. This allows researchers to compile datasets for detailed analysis or visualisation, and supports the development of innovative tools and interfaces that help all Trove users. But access to the API is controlled by keys ­– no key, no data.

I’ve had an API key since 2012, and have used my keys in a variety of ways to help people use and understand Trove. Some keys were linked to particular applications such as the Trove API Console, and Headline Roulette. Others are used in the development of tools and resources such as the GLAM Workbench, Trove Data Guide, and the Trove Newspaper Harvester.

The NLA suspended all my keys without warning or consideration as to how they were being used. Services like the Trove API Console which are dependent on keys for their operation simply stopped working.

No. warning. at. all.

However, the greater concern for me is that I can no longer develop or maintain things like the GLAM Workbench. I’ve spent a lot of time over the last 15 years responding to researcher inquiries, building tools to enable new research projects, and supporting researchers as they begin to explore the possibilities of Trove data. Most of this work has been unpaid. I do it because I think it’s important. But with no API keys my hands are tied. My ability to help researchers is severely limited.

The National Library of Australia chose to do this.

What are the terms of use?

I haven’t received a clear explanation as to why all of my API keys were cancelled. As I noted, discussion of this in my meeting with the NLA was confused and contradictory. But, in general terms, it seems to relate to the Trove API terms of use which were changed in 2020. In particular, the NLA now insists that accessing the ‘content’ of resources, rather than just the descriptive metadata, is a breach of the API terms of use. This includes the full text of digitised newspaper and journal articles that are included in API responses.

Through all of this it’s important to remember that the API terms of use are not imposed on the NLA by some external authority. They created them and can change them again at any time. If the NLA believes that work like mine has value, if they believe that researcher access to publicly-funded resources is important, they can change the rules to support these sorts of activities. To not do so is a choice.

The terms of use were changed back in 2020, so why has the NLA suddenly chosen to act? From my point of view nothing has changed. All my work is open. I’ve just been doing what I’ve been doing since 2010. No-one has reached out to me over the last five years with concerns about the terms of use. There must be something else going on here, but, given the NLA’s lack of transparency, it’s hard to know.

On 10 April, API users with keys dating back before 2020 started receiving emails that require them to explicitly accept the 2020 terms of use, or give up their keys. I suspect this is because the NLA has realised that API users weren’t informed of the changes at the time. Section 20 of the terms of use states:

The Library may change these API terms at any time at any time in its sole discretion. The Library will notify you of any changes to these API terms by adding a statement on Trove and the amended API terms will take effect 5 working days (in the Australian Capital Territory) after the date on which the statement was added to Trove. If you do not agree to the updated API terms, you must immediately cease using Trove.

If no such statement was added to Trove, then no agreement can be assumed.

The email sent to API users could have been an opportunity to explain why these changes were made, and support developers and researchers in making any necessary adjustments. But no. The major change in terms of access to content is not even mentioned. It seems the NLA just wants to slip this past quietly without anyone really noticing.

Is it all about AI?

Some have wondered whether the NLA’s actions were motivated by AI crawlers hoovering up vast quantities of online content. The Director General’s response to people who wrote to her expressing their concern over my treatment does highlight the challenges of AI, pointing to the NLA’s new Artificial intelligence framework. However, this framework is almost exclusively concerned with the NLA’s own use of AI technologies. The only reference to external AI use is the statement: ‘We will seek to protect our data from external AI systems where their use contravenes the access rights of publishers and authors’.

It’s also not clear how this relates to API use. Commercial use of the Trove API has always been handled differently to non-commercial use. Applications for commercial use are individually examined and typically involve a quid pro quo, such as access to paywalled services built using the API. So why impose new restrictions on non-commercial uses?

It’s also worth noting that the changes to the terms of use were made in 2020, before the impact of AI crawlers was well understood. Their impact, however, might explain why the terms of use are suddenly being enforced.

Of course, most AI crawlers are probably just going to scrape stuff from the NLA’s websites. To try and manage this, it would seem preferable to push bots towards the API, not away. That way their use could be monitored and better controlled. The Wikimedia Foundation recently published an interesting article on How crawlers impact the operations of the Wikimedia projects. Their planned responses to this challenge include improving their APIs and working with developers to manage their usage.

Perhaps the NLA’s new policing of their terms of use is some sort of hamfisted response to AI threats, but why go after me? As Seb Chan noted on LinkedIn:

Programmatic access to cultural collections has been vital to establishing new forms of practice, research, new scholarship and community discovery. Tim’s work has been globally recognized as important and vital for a very long time. Even if we acknowledge the increased cybersecurity risks (eg British Library), and concerns about AI bot content harvesting that likely lie behind this suspension of Tim’s generous work, this continues to feel like another own goal in the Australian cultural sector.

If there’s a problem, why isn’t the NLA discussing it with the research sector instead of picking on individuals?

Who does this affect?

The NLA’s actions have hit me pretty hard. Having 15 years of work discarded by an organisation that you’ve always sought to promote is disheartening to say the least. The impact on my work and life has been such that I’ve considered whether there might be legal recourse. I’m sad, anxious, and disappointed.

I’m also worried about the impact of the NLA’s actions on the research sector in general. In the Director General’s response she notes:

Where there is a clear need, the Library grants exemptions to the Terms of Use in order to support research and will continue to do so. We are working to make the process for applying for exemptions more transparent and to reach out to API users to clarify the process.

However, the email sent to API users simply says:

If your use of the Trove API does not align with the Trove API Terms of Use, please get in touch with Trove Support to discuss an exemption agreement.

No process. No transparency.

Any researcher wanting to analyse digitised newspaper articles, parliamentary papers, or other full text content will first need to obtain an ‘exemption agreement’. There is no information on what such an agreement involves, or how applications are assessed. The Trove API is no longer open. Research projects are now subject to the whims of NLA gatekeepers.

Established researchers, or large, well-resourced projects, might see no problem here. They can more easily, and more confidently, justify their needs. The greatest impact will be on experimental projects, where the boundaries are not yet charted. What worries me most are the HDR and ECR researchers who will be deterred by these new restrictions. What new questions will not be asked? What new approaches will be discarded because of these barriers erected by the NLA?

There will also be a significant impact on research training. How can you run a workshop on using the Trove API if participants are required not only to get an API key, but an individual ‘exemption agreement’ to work with full text?

Your support means a lot!

The one good thing to come from all of this is the support I’ve received from people around the world. Many have written to Trove, the Director General, and the Minister for the Arts. The responses received so far have not been very enlightening but, nonetheless, I think it’s really important to remind the NLA that their actions have an impact, and are being watched.

The Australian Historical Association has made a public statement, noting that:

The suspension of Dr Sherratt’s API keys, the result of recent changes to Trove’s API policy, marks a troubling restriction to the work of researchers in Australia and overseas.

The Centre for Contemporary Histories at Deakin University included a message from Professor David Lowe about the NLA’s actions in their recent newsletter. He wrote:

The best research stems from an environment that maximises the free flow of information and support for independent scholarship.

Kathryn Greenhill published an open letter that argued:

Reinstating Dr Tim Sherratt’s access to Trove API keys is essential to restore a set of cultural, educational and research tools that enhance access and usage for National Library resources.

There were also articles in Information Age (published by the Australian Computer Society) and The Sizzle.

Concerns are not just being expressed in Australia. Professor Shawn Graham from Carleton University in Canada published an open letter in which he stated:

Mr. Sherratt’s work is well known across the world in the galleries, libraries, archives, and museums sectors. His work developing the GLAM Workbench has promoted Australia’s cultural heritage world wide. Indeed, because of Mr. Sherratt’s work my own students in our Public History graduate program are more familiar with the National Library of Australia’s Trove service, and hence Australian culture, than with what our own Library and Archives Canada provides. By developing the GLAMWorbench with the Trove service, Mr. Sherratt has had a major impact in how cultural heritage materials are understood at scale, across the world. His work is complementary to your own, and enhances the prestige of the National Library of Australia

My battle with the NLA was also used as a case study in a paper on digital archives as infrastructure by Anne-Laure Donzel & Julien Benedetti presented at the forum de l’Association des archivistes Français.

Trove used to be recognised internationally as an beacon of open cultural heritage data, but that reputation has been tarnished. I’ve been contacted by a number of researchers and GLAM professionals around the world expressing their shock and disappointment at the NLA’s behaviour.

What comes next?

It’s now six weeks since my API keys were cancelled. The recent responses by the Director General and the email to API users indicate that something is happening at the NLA, but I’m not sure whether that’s going to help me. Anyone who asks the NLA about my situation is told that it can’t be discussed due to their ‘privacy’ obligations.

I’m seeking to find out more through my local member, Andrew Wilkie, who has contacted the Minister for the Arts. But now there’s an election, so I don’t expect any updates soon.

Beyond my personal situation, there are important issues that need to be considered by the research sector, and I’m hoping peak organisations such as the Australian Academy of the Humanities, the Academy of the Social Sciences in Australia, and the Australian Research Data Commons will take up the discussion.

It’s also worth noting that university libraries are Trove ‘partners’ and are represented on Trove’s Strategic Advisory Committee, so if you’re worried about the impact on your own research let your university librarian know, or tell the Council of Australasian University Librarians.

It seems the NLA is currently hunkered down, waiting for everything to blow over. As an institution they have a lot of cultural power, and people are often reluctant to criticise them in public. But I think it’s important to keep on the pressure.

What I want is pretty simple:

• my API keys back
• an apology for the way I’ve been treated
• more transparency from the NLA about API access
• an open discussion within the research sector about the problems and possibilities of working with Trove data