Tim Sherratt

Today version 1 of the Trove API was decommissioned. As I explained elsewhere, this meant that a number of Trove Twitter bots also died. The problem is that version 2 of the API provides no easy way to randomly select records. Bots, and other apps that share random content, require major reworking.

After a lot of experimentation, I’ve settled on a few methods for selecting random-ish results. They’re far from perfect, but they seem to work reliably. My major concern is that it’s hard to know how much of Trove is reachable by these methods. There may be resources that will never be found.

I’ve documented these methods in a series of Jupyter notebooks so you can try them out yourself.

• Get an random newspaper article from Trove
• Get a random work from Trove using work identifiers
• Get a random work from Trove using queries and facets

I’ve used these methods to rework the Trove Twitter Bot starter kits on Glitch that many people have used to create their own bots. This makes upgrading bots a pretty easy task. Each of the new starter kits provides instructions for current bot owners on how they can resurrect their creations.

• Trove Title Bot 2
• Trove Tag Bot 2
• Trove List Bot 2
• Trove Collection Bot 2

My feed of Trove bot tweets is looking a little sad tonight. Most of the 43 bots on my list seem to have stopped tweeting. I hope that their owners won’t feel discouraged, and will instead take the chance to upgrade their creations. And maybe a few new bots will be created as well.

#dhhacks

tl;dr

Version 1 of the Trove API will be discontinued soon so Trove Twitter bots need to be upgraded. Unfortunately, Version 2 of the Trove API doesn’t support the random selection of resources, so the current behaviour of many bots will change.

The problem

In January 2018, I created a series of templates on Glitch that made it easy for people to build their own Trove Twitter bots. And they did! It was very exciting to see the number of bots grow, and I was particularly pleased by the number of bot builders who professed to being ‘non-coders’. Yay!

Most of these bots share random selections from Trove. This is done by first finding the total number of results in a query, generating a random number between zero and the total, and using the s parameter to retrieve the selected record.

However, Version 2 of the Trove API changed the behaviour of the s parameter to support more efficient bulk harvesting. Using Version 2, it’s not possible to select random records without a lot of additional code, and perhaps not at all.

When Version 2 of the API was released, I let the Trove team know that this change would have a major impact of Twitter bots, and any other application that randomly selects records. I was told this would be looked at before Version 1 of the API was discontinued. As a result I held off updating the bots until the situation was clear.

Version 2.1 of the API was released a few weeks back with some great new features, but no added randomness. The announcement also noted that Version 1 would be switched off in November. I checked with the Trove team and was told that no changes were planned to the API to support the selection of random records.

So the Twitter bots need some major work, and soon…

Possible solutions

First the good news. Version 2 of the API is much faster and more reliable. Also, Trove List Bots won’t be affected by this change, as the list items aren’t paginated anyway.

Some random-ish selection of records will still be possible, but it’s going to take a bit of thinking, and the solution won’t be generally applicable across all the bots. The maximum number of results returned by a single API request is 100. So if the bots can use filters and facets to limit their query to less than 100 results, they could then select one at random.

For example, the Trove Title Bot (which tweets random articles from a specific newspaper) could be reconfigured to use the decade, year, and month facets to split up the result set. In fact, TroveNewsBot already does this to try and get around some of the inefficiencies of Version 1 of the API.

Another approach might be to use some sort of random ‘seed’ to limit the query. For example, the Title Bot could select a date at random, and search for articles published on that day.

Tag Bots and Collection Bots will create a different set of challenges. The types of facets and filters that will be most effective will depend, to some extent, on the nature of the collection itself. For example, if the items in the collection were all created at about the same time, then the date filter might not be very useful.

Narrowing down the result set in this way will probably require multiple API requests. And, in many cases, it’s likely that it will be impossible to get the total down below 100. Playing with the sort options might open up this window a bit, but I think that the range of results from which the bots draw their ‘randomly’ selected results will, in most cases, be limited. Some items will never be shared.

Does this matter? People who follow the bots on Twitter might not notice much difference. But the bots’ ability to surface unexpected content will be hampered. They’ll still be sharing records, but they’ll also be hiding some. I think this matters.

The plan

With Version 1 of the API being switched off in November, time is short. I’m also currently in the midst of preparing our house for sale ahead of a move to Tasmania. Eeeek!

I’m planning to create updated versions of all the bot templates on Glitch. They’ll be modified to use Version 2 of the Trove API and I’ll also take the chance to update them to Python 3. I’ll try implementing some of the random-ish strategies described above and see how they work. My aim at this stage is really just to try and keep people’s creations alive and tweeting.

For bot creators/maintainers, the upgrade process should be pretty painless. It should just be a matter of hitting the ‘remix’ button on the updated templates, copying across the configuration details from the current bot, and then adjusting your Cron service to point at the new bot. Of course, if you’ve made any customisations, you’ll have to copy these across to the new code.

As I noted, Collection and Tag bots might require some additional customisation to make sure as much of the collection as possible is exposed.

I’ll document the upgrade process and will be available on Twitter to help. I’m hoping to get this done in the next couple of weeks. Please bear with me…

TroveNewsBot and TroveBot

I’ll also have to make some changes to TroveNewsBot and TroveBot which have seen sharing Trove resources with the world since 2013. As with the Glitch bots, their ability to share random items and respond to the #luckydip hashtag will be limited, however, their ability to respond to other queries shouldn’t be affected.

Other apps

There are various other apps that I’ve created over the years that will be affected by this change — most notably the game Headline Roulette. People who have forked or remixed versions of Headline Roulette will also need to make some changes. I’ll let you know once I’ve worked out what needs to happen.

The value of randomness

I’ve written elsewhere about the way Twitter bots can mobilise collections, letting them loose in places where people congregate rather than waiting for visitors to find your website. I’ve also talked about the value of play in coming to grips with large cultural collections.

Collection bots are versions of ourselves, unable to contain our excitement at our latest discoveries. Look at what I found! They point to the play, to the fun, of research. But they also offer glimpses of a larger unknown.

The ability to select random items from a large cultural collection may seem trivial. But it opens up opportunities for creativity and engagement that I think are really important. If you agree, you might like to contact the Trove team and let them know — hopefully a bit of randomness might make it into future API upgrades.

NSW State Archives publishes a number of detailed indexes containing data manually extracted from their records. These provide additional entry points to the records, such as a person’s name, or a place. But they also provide useful data for analysis. However, to explore the index data we need to get it out of the web interface and into a form that can be easily downloaded and manipulated.

I’ve created a series of Jupyter notebooks to harvest the all the indexes and save the data in a series of CSV-formatted files. I’ve also updated my repository containing all the harvested CSV files. It’s available from the new NSW State Archives section of my GLAM Workbench. There are currently 64 different index datasets, containing 1,499,259 rows of data.

And to help you get a sense of what’s actually in all those CSV files, I’ve created an interactive Index Explorer. Just select an index from the list, and the Index Explorer will generate a series of tables and visualisations that provide an overview of the data. Try running it live on Binder.

Thanks to the State Archives staff and volunteers for preparing all this most excellent data. #dhhacks