glamworkbench on Tim Sherratt

Generosity in practice – a chat with Paula Bray at the State Library of Victoria

Mon, 16 Mar 2026 15:14:17 +1100

While I was in Melbourne during my time as Creative Technologist-in-Residence at the State Library of Victorian LAB, I had a conversation with Paula Bray for the LAB’s podcast series. Paula is the SLV’s Chief Digital Officer, and has long championed the importance of digital innovation in the GLAM sector. It was fun to chat about stuff that I’ve been doing for the last 30 years, any why openness and generosity is important in working with GLAM collections. You can listen to our conversation on the LAB site.

Zotero translator for Libraries Tasmania updated!

Tue, 10 Mar 2026 10:31:36 +1100

The Zotero translator for Libraries Tasmania has been updated, fixing a problem with attaching images of digitised resources. The fix is in the main Zotero repository now, so it should find its way to your computer automatically.

I created the first version of the Libraries Tasmania translator back in 2022 – this post describes what it does. It works across all three sections of the catalogue, including the archives, and names index. The translator captures metadata, PDFs, and images from records, including things like digitised pages from convict records. This makes it easy for researchers to assemble their own datasets of Tasmanian records in Zotero, where they can add notes and annotations, or share with colleagues.

Capture images and metadata from the Libraries Tasmania catalogue using Zotero

The update was necessary because Libraries Tasmania changed the way some digitised resources were displayed and downloaded. Keeping Zotero translators working across system updates can take a bit of work! I also took the opportunity to update the code to meet current Zotero guidelines and clean up a few lingering problems. If you notice any oddities, please let me know.

There are now at least 8 custom translators to help you work with Australian GLAM collections.

my place – exploring SLV collections through a street address

Mon, 02 Feb 2026 22:43:20 +1100

‘What can I find out about my house?' My work as Creative Technologist-in-Residence at the SLV LAB was inspired by questions like this that librarians at the SLV hear every day. I wanted to explore how the Library’s place-based collections could be used to provide new entry points for discovery and navigation – entry points based not on words, but locations.

At the end of my residency, I pulled all the different collections I’d been working with into a single interface – my place. It’s not polished or complete, but I think it’s a useful starting point to think about the possibilities. You just type in an address, street name, or place name and my place shows you maps, photos, newspapers, and even extracts from the Sands & MacDougall directories. Try it now!

Try my place! Just enter an address in the search box.

Search results in my place are bookmarkable. So save and share your discoveries!

The collections

my place draws its data from a number of different place-based collections that I’ve been working on during my residency.

OpenStreetMap

When you enter an address in the search box, my place looks it up in OpenStreetMap to get its geospatial coordinates. It then places a marker and re-centres the map at the top of the app.

Map centred on 149 Brunswick Street, Fitzroy

OpenStreetMap is also used to retrieve additional information about the suburb, including its boundaries.

Sands & MacDougall’s directories

my place queries the full-text searchable version of Sands & Mac for addresses. Results will vary based on the OCR quality and the nature of query, but it can give you a potted history of who has lived in your house. The search results are displayed in chronological order, and include an image snippet showing the actual printed entry as well as the text content and metadata.

Occupants of 149 Brunswick Street, Fitzroy from 1875 to 1925

Committee for Urban Action photographs

If you enter a full street address, my place will search the CUA collection for photos associated with the segment of road that includes the current address. It then displays the individual images from any matching photosets.

Photographs from CUA of the currently selected road

Otherwise my space will look for CUA photos that are near the current location, and display a randomly-selected image from each photoset.

Georeferenced maps

my place searches through digitised maps from the SLV collection that have been georeferenced by the public. It finds maps that either intersect with the currently selected location, or are nearby.

If you enter a full street address, the first 6 georeferenced maps will be positioned on a modern basemap with a marker indicating the currently selected point. This means you can see your address on a historical map. The number of georeferenced maps that can be displayed in this way is determined by the browser – so I’ve limited it to 6 to be safe.

Georeferenced maps positioned on a modern basemap, showing the location of the currently selected address

Parish maps

my place searches through parish maps in the SLV collection that have geospatial coordinates or approximate bounding boxes. It finds maps that either intersect with the currently selected location, or are nearby.

Newspapers

my place searches through my dataset of newspapers in the SLV collection that have a place of publication documented in the ‘Place newspaper published’ metadata field. It finds newspapers that are either associated with the current suburb/town, or a nearby suburb/town. This includes digitised and non-digitised titles. Digitised titles include a link to Trove.

Newspapers from the SLV collection published in Fitzroy

Photographs

I thought it would be cool to include a few photographs of the current suburb or town. To do this, I downloaded a list of place names from VicNames, then used the place names to search the SLV catalogue for photographs with relevant subject headings. A random selection of the harvested images is displayed in my place.

A few images of Fitzroy, displayed alongside a map of Fitzroy's current boundaries using data from OpenStreetMap

The interface

The interface is pretty simple. You type an address in the box and hit enter. If the geocoding process finds multiple matches, it’ll give you a list to choose from. Once the location is found, a marker is added and the main map re-centres. Then related resources are displayed below the map.

As you scroll down through the results you gradually zoom out from your initial starting point. This is reflected in the four bands or layers used to group resources: ‘my house’, ‘my street’, ‘my suburb’, and ‘nearby’. Each band contains a mix of resources from different collections.

When I started working on my place, I was thinking about a project from around 2010 called The History Wall. Like my place, The History Wall pulled many different types of resources together into a rich exploratory interface. As you scrolled through The History Wall you moved through time, with randomly selected items appearing from a range of sources including Trove newspapers, the ADB, and museum collections.

A version of The History Wall created for the National Museum of Australia's 'Irish in Australia' exhibition

I originally thought I’d inject some of the same randomness into my place, but I was worried it might just get too confusing. I thought it was important to keep the relationship between the starting point and the resources in focus even as you zoomed out. So my visual metaphor shifted to something more like a blast radius map, or a stratigraphic diagram, that displayed distinct groups and layers as you moved beyond the baseline. My limited CSS skills couldn’t make the vision in my head a reality, but there are lots of headings and colours instead to highlight the transitions!

The actual mix of groups and layers displayed depends on the nature of your query. If you’ve entered a complete street address, and there are results for that address in Sands & Mac, then you’ll see ‘my house’, ‘my suburb’, and ‘nearby’. If you’ve only entered a suburb or town, or your street address can’t be found, you’ll see two layers starting with ‘my suburb’.

Here’s an overview of what you might expect to see.

my house

Sands & MacDougall extracts (text search on full address)
georeferenced maps (search for maps that contain the base point)
parish maps (search for maps that contain the base point)

my street

CUA photos (search for matching street identifiers)

if there’s no ‘my house’ layer:

Sands & MacDougall extracts (text search on street name and suburb)
georeferenced maps (search for intersections between maps and street)
parish maps (search for intersections between maps and street)

my suburb/town

suburb boundaries from OSM
images (search for suburb name in metadata)
newspapers (search for suburb name in metadata)

if there’s no ‘my house’ or ‘my street’ layer:

georeferenced maps (search for intersections between maps and suburb boundaries)
parish maps (search for intersections between maps and suburb boundaries)

nearby

CUA photos (search for photosets within 5km of the base point, filtered to remove current street)
georeferenced maps (search for maps within 10km of base point, ordered by distance, max of 24 displayed)
parish maps (search for maps within 10km of base point, ordered by distance, max of 24 displayed)
newspapers (search for newspapers within 100km of base point, ordered by distance, max of 24 displayed)

The data

Most of the data used in my place is stored in two SQLite databases – one for Sands & Mac, and the other for CUA, georeferenced maps, parish maps, and newspapers. The metadata for the collection images is stored in a JSON file that is directly loaded by the interface.

I’ve published the SQLite databases online using Datasette and Spatialite. Spatialite makes it possible to find geospatial features that intersect, or are near, a given point. For example, you could find maps that include a specific set of coordinates.

Datasette has the ability to create ‘canned queries’ that feed url parameters into pre-defined SQL queries. This coupled with Datasette’s built-in JSON API makes it possible to construct query urls in my place and use them to retrieve JSON results data from my databases.

When you enter an address in my place, multiple queries are fired off to find intersecting or nearby resources. For example, this url finds georeferenced maps within 10km of a point at the centre of Fitzroy: slv-places-481615284700.australia-southeast1.run.app/georefere…&distance=10000&_shape=array.

In the case of Sands & Mac, my place uses a canned query that runs a full-text search across the OCRd content of a volume. Suburb names are often abbreviated in Sands & Mac, so the app first runs a query to find possible abbreviations, then adds them into the main query to inject a bit of fuzziness. This is repeated for all 24 digitised volumes.

Once the metadata is retrieved from the databases, images are loaded from the SLV’s IIIF service.

Next steps?

I’m not sure how much more work I’ll do on my place, but there are a few things I’d like to try. In particular, I’d like to help the user understand more about what data is being presented, or not presented, and why. Not all digitised maps have been georeferenced, not all parish maps have coordinates, street numbers have changed, and the OCR in Sands & Mac varies in quality. my place can only present a sample – a gesture towards the wealth of material available from the SLV. I feel that message needs to be made more explicit. Though I’m not sure how without overloading the interface.

There are additional data sources I’d like to play around with. my place already includes some code to query Wikidata for more information about a suburb. But I haven’t had a chance to do anything with it. I’d like to be able to provide additional contextual information from outside the SLV, such as electoral boundaries, populations, even election results. It would also be fun to display sightings of plants and animals from the Atlas of Living Australia.

What can I find out about my house? It would be great if my place could answer that question by taking the user on an open ended journey through our cultural, historical, and environmental landscape.

Geolocating photos from the SLV's Committee for Urban Action collection

Thu, 29 Jan 2026 18:08:06 +1100

Concerned about the loss of built heritage in the 1970s, the Committee for Urban Action photographed streetscapes across urban and regional Victoria. They compiled a remarkable collection of photographs that is now being digitised by the State Library of Victoria. More than 20,000 images are already available online!

The CUA worked systematically, capturing photos street by street, and recording the locations of each set of photographs. This information is used to prepare the title attached to each photo as it’s uploaded to the SLV catalogue. In general, titles include the name of the road where the photo was taken, the name of the suburb or town, and the names of two intersecting roads that define the boundaries of the current road segment. They can also tell you which side of the road the photo was taken on. For example, the title Gore Street, Fitzroy, from Gertrude Street to Webb Street - east side tells us the photo was taken on the east side of Gore Street, Fitzroy between the intersections with Gertrude Street and Webb Street.

Photos from Gore Street, Fitzroy displayed in the SLV image viewer

It’s great to have this sort of structured information linking photos to specific locations, but to navigate through the collection in space we need more. We need to link each photo to a set of geospatial coordinates by mapping each road segment. That was the challenge I took on as part of my residency in the SLV LAB.

When I started working on the collection I wasn’t really sure what was possible. I had to learn a lot, and ended up revising my processes multiple times as I got deeper into the data. But my aim was always to create some sort of map-based interface, that would allow users to click on a street and see any associated CUA photos. It’s still a bit buggy and incomplete, but here it is – explore the CUA collection street by street!

Gore Street, Fitzroy in the new CUA Browser

The process

My basic plan was to find the intersections using OpenStreetMap, then extract geospatial information about the segment of road between the two intersections. This involved much trial and error, but eventually I ended up with a process that:

parsed each item title to try and extract the names of the main road, the suburb, and the two intersecting roads
queried Nominatim for the suburb bounding box
for each intersecting road, queried OSM to find a node at, or around, its intersection with the main road, within the suburb bounding box
created a new bounding box from the coordinates of the two intersections
queried OSM for the main road within this bounding box
extracted the coordinates of the main road segment, removing any points outside of the bounding box

There’s more details below and in these notebooks: cua_finding_intersections.ipynb and cua_data_processing.ipynb.

Finding intersections

As described, the title of each photograph generally includes 4 pieces of information: the road, suburb, intersecting roads, and side. My plan was to find the intersections first to get the limits of the road segment. This is possible thanks to the awesome OpenStreetMap and its Overpass API. It took me a while to get my head around the Overpass query language, but there are lots of useful examples online. The query to find the intersection between Gore Street and Gertrude Street in Fitzroy looks like this:

[bbox:-37.8089071,144.9732006,-37.7929130,144.9851430];
way['highway'][name="Gore Street"];
node(w)->.n1;
way['highway'][name="Gertrude Street"];
node(w)->.n2;
node.n1.n2; 
out body;

You can try it out using Overpass Turbo’s web interface.

In OpenStreetMap, linear features, such as roads or rivers, are represented as ways. Each way is made up of a series of nodes or points with geospatial coordinates. Every way and node has its own unique identifier. Tags can be added to features to describe what type of things they are.

The query above looks for ways named ‘Gore Street’ and ‘Gertrude Street’ that are tagged as highway (a highway in OpenStreetMap is any road-like feature including things bike paths and foot trails).

way['highway'][name="Gore Street"];

It then extracts the nodes that make up each way and looks to see if there are any nodes in common between the two ways. A node shared between two ways indicates an intersection.

node(w)->.n2;
node.n1.n2;

The query is limited using a bounding box that encloses the suburb of Fitzroy. This avoids false positives and keeps down the query load.

[bbox:-37.8089071,144.9732006,-37.7929130,144.9851430];

The JSON result of this query gives as the latitude and longitude of the node at the intersection of the two roads.

{
  "version": 0.6,
  "generator": "Overpass API 0.7.62.10 2d4cfc48",
  "osm3s": {
    "timestamp_osm_base": "2026-01-27T03:11:45Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "node",
  "id": 224750459,
  "lat": -37.8062302,
  "lon": 144.9817848
}

  ]
}

After a bit of testing, I found this worked pretty well, except for roundabouts… In OpenStreetMap, roads don’t actually cross roundabouts – they end on one side, then begin anew on the other side. In cases like this, looking for shared nodes doesn’t work. Instead you have to look to see if the two roads have nodes that are less than a given distance apart. The query is similar to the one above, but uses around when comparing the nodes. In this case I’m looking for nodes that are within 20 metres of each other.

node(w.w2)(around.w1:20);

Finding road segments

Once I had the coordinates of the two intersections, I could look for the segment of road between between them. To do this I created a bounding box using the coordinates of the intersections, and then searched for ways by name within that defined area.

It’s important to note that there’s no one-to-one correspondence between roads and OSM ways. A single road might be represented in OSM as a series of separate, but connected, ways. For example, at a roundabout, or where a road divides, new ways might have been created to document the change. This means that when we query OSM for details of a road we often get back information about multiple ways. Some of these might be things like bike paths which we can filter using tags, but often they’ll be sections of the road that we want. For example, this query for Gore Street, within the bounds of its intersections with Gertrude Street and Webb Street, returns details of two ways.

way["highway"~"^(trunk|primary|secondary|tertiary|unclassified|residential|service|track|pedestrian|living_street)$"][name="Gore Street"](-37.8062302,144.98128480000003,-37.8040076,144.9826827);
out body;
>;
out body;

You can view the result in Overpass Turbo.

However, that doesn’t mean that the full extent of both ways is contained within the bounding box, just that some of the nodes of both ways are inside. Because of this, I filtered the results from all the ways and only kept nodes whose coordinates were within the desired region.

Problems finding intersections

The method described above works pretty well, and once I understood enough about the Overpass API to get out actual paths that I could display on a map, I fed all of the CUA photos through a script and got useful data for more than 80% of them.

One of my early tests.

Then I spent a lot of time trying to understand where the remainder were failing.

Some of them failed because the titles were missing information, or were formatted in a way I didn’t expect. For example, instead of a second intersecting road, some titles just said ‘to end’. This makes perfect sense to a human looking at a map, but it’s difficult to handle programmatically.

Some photos either recorded the wrong suburb, or the boundaries of the suburb had moved since the photos were taken. For example, many of the photos described as being from Eaglehawk are now in California Gully.

Similarly, some road names were wrong either because of documentation errors, or because the names have changed over time. There are also some variations in the way OSM records road names – in particular, I found that roads with hyphenated names sometimes had spaces around the hyphen and sometimes didn’t. There were also a couple of cases where names weren’t attached to the corresponding road segment in OSM, but I was able to edit these in OSM directly.

Other roads had multiple names, or change names along their path. I mean, what’s going on with Brunswick Street and St Georges Road in Fitzroy? Country towns seemed most prone to this – a highway might become ‘Main Road’ within the town boundaries, or the order of hyphenated places in road names might change. I found one road in Clunes that had four different names within the space of a few hundred metres.

One road, four names!

Finally, the routes of some roads had changed – intersections no longer intersected, roads were closed, or new parks had popped up to split a road in two.

My processing script logged the titles I couldn’t locate and I worked through the list manually, trying to identify what each problem was. I suppose there’s two ways I could’ve handled these problems – building more fuzziness into the process to check for things like alternative names, or by compiling a list of ‘corrected’ titles. I started off using the first approach, but as I worked through more and more anomalies, the checking logic became very complicated and inefficient. Just think about the knots you can tie yourself in trying to handle a title where the suburb is wrong and the main road changes names in between intersections.

I refactored the code multiple times, but it’s still pretty messy. In the end I created a list of ‘corrected’ titles as well, so it was a bit of a hybrid approach. I suspect I could have saved myself a lot of pain if I’d reversed the process – compiling ‘corrected’ titles first, then adapting the logic as patterns emerged.

There are still some photos I haven’t located. In some cases I just don’t have enough information. In others I need to manually record coordinates or way ids to feed into the process, and I haven’t worked out the best way to do this yet. You can see the titles that I’ve haven’t geolocated yet in the files: cua-not-found.txt and cua-not-parsed.txt.

In total, 18,603 out of 20,644 photos have been geolocated. That’s over 90%!

Assembling the data

I processed the data in a couple of phases to get it in the shape I wanted.

The first step was to group all the photos by title, so I could link each group to its location. But remember that titles often record which side of the road a photo was taken on. To bring all sides of a road segment together into a single group, I created a key from a normalised/slugified version of the title with the side value removed. I used this key to save information about each side within the same group.

I ended up with a dataset with this sort of structure (a truncated example):

"iffla-street-south-melbourne-coventry-street-normanby-street":
    {
        "title": "Iffla Street, South Melbourne, from Coventry Street to Normanby Street",
        "sides":
        {
            "east side":
            {
                "title": "Iffla Street, South Melbourne, from Coventry Street to Normanby Street - east side.",
                "images":
                [
                    {
                        "ie_id": "IE20321667",
                        "alma_id": 9939649155207636
                    }
                    more photos...
                ]
            },
            "west side":
            {
                "title": "Iffla Street, South Melbourne, from Normanby Street to Coventry Street - west side.",
                "images":
                [
                    {
                        "ie_id": "IE20320072",
                        "alma_id": 9939655629407636
                    },
					more photos...
                ]
            }
        },
        "ways":
        {
            "27631235":
            [
                [
                    144.9503379,
                    -37.835322
                ],
                more points...
            ]
        }
    },

You can see how the sides and matching ways have been brought together under the key value.

This structure was useful for grouping and processing the data, but to create a map interface I needed to bring the geospatial information to the surface. The first version of the interface used one big GeoJSON file in which the features were MultiLineStrings created from the paths of each road segment. The photo data was saved in the properties of each GeoJSON feature.

It sort of worked. The roads with photos were highlighted, and clicking on the roads displayed the photos. It was only when I changed the opacity of the lines that I realised that, in many cases, different road segments were being piled on top of each other. When the lines were opaque these piles were invisible, but add a bit of transparency and you could see that some lines were darker than others. Clicking on the lines only displayed the top layer, so some groups of photos were effectively invisible.

Version one of the interface showing how the colour of the highlighted roads varied once I decreased the opacity.

Why did this happen? I’d wrongly assumed that each segment of road would only have one group of photos associated with it. But it’s not hard to find cases where this is not true. Consider Moor Street, Fitzroy, between Nicholson Street and Brunswick Street. On the north side, there is a single group of photos that document the buildings between Nicholson Street and Brunswick Street. However, on the south side there’s two groups of photos. One covers the section between Nicholson Street and Fitzroy Street, the other covers Fitzroy Street to Brunswick Street. One section of road, three groups of photos…

Moor Street, Fitzroy, between Nicholson Street and Brunswick Street, in the new CUA Browser, showing the three photosets associated with the one section of road.

To make these layered groups more easily accessible through the interface I had to change the way the data was organised – separating the GeoJSON from the photosets so that multiple photosets could be associated with a single geospatial feature. I decided to create a GeoJSON feature for every OSM way in the dataset. However, I needed to prune the way’s coordinates to only include those that were part of the CUA road segments. To do this, I saved all the way data when I found the road segments. Then in the second processing phase, I grouped the way coordinates associated with the road segments and compared this list to the full way path. Any coordinate in the way path that wasn’t in the road segments was removed. It seems unnecessarily complex, but I wanted to make sure that only the parts of roads associated with photos were highlighted in the interface.

The result was two data files. The first, cua-ways.geojson, contains the pruned way paths and their OSM identifiers. The second, cua-photos.json, contains information about each photo set, including the sides, photos, paths, and associated way identifiers. The datasets are linked by the way identifiers.

Constructing the interface

My plan for the interface was pretty simple. There’d be a map on which all the road segments associated with CUA photos were highlighted. Clicking on a highlighted section would show the photos. I wanted to display the photos as if you were scanning the streetscape, so I decided to put them all side-by-side in a gallery that scrolled horizontally.

The first version used Leaflet to display the maps and, as noted above, had some problems where there were multiple photosets associated with a segment of road.

For the second version I decided to switch to MapLibre because it seems a bit more active and up-to-date. I’d already used MapLibre in the SLV Newspapers Explorer.

The interface first loads the cua-ways.geojson file to highlight the relevant roads. When you click on one of the roads, the way id is passed to a function that looks for associated photo sets in the cua-photos.json data. If there’s only one linked photoset, then the photos are displayed. However, if there’s more than one linked photoset, they’re displayed as a list. The user then selects from the list to display the related photos.

A couple of other things happen when you click on a way or select a photoset: the colour of the selected road segment changes, and the browser url is updated with the way or photoset identifier. You can bookmark or share these urls to go directly to a specific road or photoset. There’s also a button to reverse the order of the images – they scroll left to right, but sometimes they seem to have been photographed right to left.

Goodbye 2025! A brief summary of the highlights and lowlights…

Wed, 31 Dec 2025 17:14:14 +1100

My 2025 started badly and ended well. In the first few months of the year, battles with the gatekeepers at Trove sent me spiralling into a pretty dark place. But by year’s end I was having fun, working with the wonderful people at the State Library of Victoria. In between I caught up on some overdue project maintenance. Most of this is documented in the 37 blog posts I wrote this year, but here’s a quick summary.

Highlights

From September to December, I was Creative Technologist-in-Residence at the SLV LAB, exploring ways of opening up the Library’s place-based collections. There’s still a few things to finish off, but here’s list of the results so far.
As part of my SLV work, I created a fully searchable version of the 24 volumes of Sands & MacDougall directories digitised by the Library. This followed the pattern I’d used for 54 volumes of the NSW Post Office Directories, 44 volumes of the Sydney Telephone Directories, and 54 volumes of the Tasmanian Post Office Directories. So there’s now 176 volumes from the 1880s to the 1950s that can be easily explored for people and places— and all free to use of course.
In April, I added a new section to the GLAM Workbench documenting the Public Record Office Victoria’s collection API. I also used the API to create a Data Dashboard that provides an overview of PROV’s collection.
Also in April, I updated the GLAM Name Index Search to include an additional 6 million records from PROV. In total, the GLAM Name Index now includes more that 12 million records in 293 datasets from 10 Australian GLAM organisations — another free resource for Australian researchers.
In July I undertook some overdue maintenance on a variety of old apps and projects. In the process, I resurrected my old Wragge Labs domain and created a showcase of many of the websites, apps and experiments I’ve worked on over the past 30 years.
I was particularly pleased to get The Future of the Past working again, so once more you can create fridge magnet poetry from an odd collection of words harvested from Trove newspapers! I built FOTP back in 2012 when I was the Harold White Fellow at the NLA. Also this year I finally got around to transcribing my Harold White Lecture.
In June I wrote a short piece on the GLAM Workbench for the forthcoming publication Building User-Friendly Toolkits and Platforms for Digital Humanities. I think it provides a useful summary of what the GLAM Workbench is, and what I’d like it to be. I also wrote up the short but glorious history of Trove Twitter bots.

Lowlights

Saying goodbye to 15 years of work on Trove. It still hurts. And I still miss resources such as @TroveNewsBot and the Trove API Console which ran happily for more than a decade before being killed without warning by the NLA.
Saying goodbye to 17 years of work on the National Archives of Australia’s collections. This will be the first New Year’s Day in a decade when I haven’t updated my harvest of files with the access status of closed.

Next year

In 2026, I’m looking forward to starting work on the RAPID project, building on the work I’ve done on Commonwealth Hansard over the years to create new examples and documentation.
I’m honoured to be giving the closing keynote at the GLAM Labs Futures conference in Edinburgh in June — hoping we can pull together the funds to get there in person!

How you can help

Much of my work is unfunded, and keeping resources such as the GLAM Name Index running costs real money. I’ve been very grateful for the support of my GitHub sponsors over past years. Their contributions help cover a substantial proportion of my cloud hosting costs. But bidding farewell to Twitter and Trove has had an impact on my sponsorship income. If you use or value the things I build to help researchers make use of GLAM collections, you might like to sponsor me on GitHub, or Buy Me a Coffee. All contributions are greatly appreciated!

If you can’t afford a financial contribution, there are other ways you can help!

Let me know how you’re using my stuff! A bit of positive feedback does wonders when my enthusiasm is flagging. You can find my contact details at timsherratt.au.
Tell others how you use my stuff! Getting information about resources out to those who might benefit is really hard, so your help would be greatly appreciated.
The GLAM Workbench describes a few other ways you can get involved.

Goodbye 2025!

Exploring Victorian newspapers

Tue, 16 Dec 2025 13:03:48 +1100

Newspapers are a vital source for local history. That’s why, back in 2014, I created the Trove Places app – a map interface to help people find Trove’s digitised newspapers by their place of publication or distribution. Trove Places has proved very popular, and the State Libraries of South Australia, and Victoria, amongst others, point their users to it to help with their research. I’ve updated the data several times over the years, though the Trove’s new gatekeeping regime will make future updates difficult.

During my residency at the State Library of Victoria, one of the librarians noted how useful the app was, and asked whether it might be possible to include undigitised newspapers from the SLV catalogue as well as those in Trove. It was, and I did – here’s a brand new app to explore Victorian newspapers, both digitised and undigitised!

Just click on the map to find Victorian newspapers!

It’s pretty easy to use. You just click on the map in an area you’re interested in. The map will display the 20 nearest places where newspapers where published or distributed. The size of the markers indicates how many titles are associated with each place. In the sidebar, details of the newspapers are listed by place, ordered by their distance from your selected point.

You can also find local newspapers using the my place app. Once you enter an address, newspapers from your suburb or town will be displayed, as well as those from nearby locations.

newspapers from Geelong displayed in the my place app

Assembling the data

How do you find Victorian newspapers? The reference librarians at the SLV pointed me to the ‘Place newspaper published’ field in the catalogue. Searching this field for ‘Australia–Victoria’ returns 3,997 results, compared to the 460 digitised in Trove.

The first step in assembling the data was to harvest the newspaper records from the SLV catalogue. To do this I made use of the Primo JSON API. The method is documented in this notebook. The results was a newline-delimited JSON file, with one record per line.

The harvested metadata doesn’t include links to digitised versions of newspapers in Trove. To add these links I first looked in the 856 field of the newspaper’s MARC record. I also noticed that some Trove links were being loaded from an ‘edelivery’ JSON file, so I added these as well. I ended up with 344 unique links to Trove, but not all of these were to digitised newspapers as some more recent newspapers are available through eLegal deposit. In total there were 268 unique links to digitised newspapers. This is well short of the 460 Victorian newspapers in Trove. Why? It’s possible that the links haven’t been added into the SLV catalogue, or that the ‘place newspaper published’ field hasn’t been populated for records that include the links. It’s also possible that Trove links are hiding somewhere else in the SLV catalogue!

To try and fill this gap, I compared the catalogue metadata with my most recent harvest of Trove newspaper titles. If the Trove url was missing, I searched the catalogue data for the newspaper title. I then manually checked the results, making sure the dates and titles lined up, and added positive matches to a new CSV file which I merged back into the main dataset. This added another 152 Trove links.

The next step was to link the ‘place newspaper published’ values to places with known locations. The ‘place newspaper published’ information is included in the lds03 field of the harvested metadata. Records often contain references to multiple places, so I split all the newspaper/place combinations out into separate rows. I then matched the places against a list of Victorian place names and coordinates downloaded from the VicNames database. If there were no matches, I manually checked and adjusted the place names – for example, I changed ‘East Kew’ to ‘Kew East’, and ‘Bayside’ to ‘Bayside City’.

To add any Trove digitised newspapers that might still be missing, I made use of my existing Trove harvests. First I compared my Trove Places dataset with my latest harvest of newspaper titles. There were a few new titles, so I matched them to places using this notebook, based on my original Trove Places code. I then merged the Trove Places dataset with the new titles and checked it against the catalogue dataset. If any urls were missing, I added a record from the Trove data.

All of the processing steps are documented in this notebook.

Building the apps

To make the data easily searchable by its geospatial coordinates, I loaded all the data into an SQLite/Spatialite database and published it online using Datasette. The database contains linked tables for titles and places.

I also created a couple of canned queries which, together with Datasette’s built-in JSON API, made it possible to retrieve places and titles based on their distance from a given point. For example, this url retrieves places ordered by their distance from the point at latitude -36.815, longitude 144.965 : https://slv-places-481615284700.australia-southeast1.run.app/newspapers/places_from_point.json?longitude=144.965&latitude=-36.815&distance=100000&_shape=array

When you click on the map in the Victorian Newspapers Explorer, it fires off a request like this to find nearby places. It then makes a second request to find newspapers related to those places and displays the results.

The Victorian Newspapers Explorer was my first attempt at using MapLibre rather than Leaflet to display maps using Javascript. It’s more verbose, but more flexible, so I think I’ll gradually switch over my other apps, including Trove Places.

All the code of the Victorian Newspapers Explorer is in the slv-demo-apps repository.

Why bother?

Wed, 03 Dec 2025 16:18:22 +1100

This was the introduction to my talk on the results of my time as Creative Technologist-in-Residence at the State Library of Victoria. My slides, with my full notes are available online, but after a very strange year that has travelled from disappointment to exhilaration, I thought it was worth posting these words separately.

The work that I do, that I’ve been doing for the past 30 years, is focused on helping people find, use, and understand the wonderfully rich collections held by our libraries, archives, and museums – the GLAM sector. Much of it is quite practical, resulting in tools and applications that are used by a wide range of researchers. Some of it is playful, some of it is critical, and some of it is just weird.

You can browse through some of this history on wraggelabs.com. And if you’re interested in my current crop of tools you can head to the GLAM Workbench.

As some of you may know, I had a few setbacks at the beginning of this year which really made me wonder whether I wanted to continue doing this sort of work.

I mean, why bother?

I’m really grateful that this residency has given me a chance to refocus on the reasons why I do what I do.

I suppose my starting point is the fact that libraries can’t do everything themselves.

I’m thinking here specifically about the digital research space. There’s a lot that libraries, and other GLAM organisations, can do – provide search interfaces, APIs, downloadable datasets, documentation, and examples of how to access APIs and datasets using code. The sorts of things that the SLV LAB is doing.

I should pause here to unpack some acronyms. APIs deliver data in a form that machines can understand and process. Websites are for humans, APIs are for computers. APIs are also building blocks which can be connected up to create new applications – and I’ll be showing some examples of this later on.

So there is much that GLAM organisations can do to support digital research. But it will never be enough. Researchers – whether they be academics or family historians – will always want more. It is in the nature of research to ask new questions, to head off in new directions.

But rather than see this as a source of tension, I see it as an opportunity for collaboration. An opportunity to cultivate the in-between spaces where research methods, tools, and results can feed back into the contextual frameworks of GLAM collections. Where GLAM organisations can share and celebrate the work that’s done with their data. Where all can find inspiration, ideas and support.

We’ve tended to call this sort of stuff infrastructure, but I think that really downplays the human aspect. The research sector has started to develop the funding and career structures necessary to allow people to build and maintain these infrastructures, but we need more. We need to recognise that a single tool, developed by an individual without institutional support, can be just as important as a multi-million dollar platform. Passion is precious and needs to be protected.

Most of all, we need to keep a focus on the ethical imperatives – the reasons why we bother and why it matters. For me it boils down to openness and generosity. I have benefited greatly from the openness and generosity of others, and I want to pass that on. It’s the glue we need to hold those in-between spaces together; the sustenance we need to maintain our enthusiasm in the face of all the crap; the inspiration we need to try something new.

Initiatives like the SLV LAB are important, not just because they foster innovation, but because they invite new ideas in. They even give space for ageing hackers like me to spend some dedicated time doing what they love – crafting new pathways for people to explore our glorious GLAM collections.

Counting down... (to the end of my SLV residency)

Wed, 19 Nov 2025 16:01:22 +1100

My stint as Creative Technologist-in-Residence at the State Library of Victoria LAB comes to an end in a few weeks time and I’m frantically trying to pull things together. I’ll be back on-site at the Library from 1 to 5 December for a few events, and to report back to staff on what I’ve been doing.

On Tuesday 2 December, there’ll be a public workshop on using and contributing to the GLAM Workbench. Here’s the blurb:

More and more GLAM organisations are looking to share their data to foster creativity and support new types of research. But how can you help potential users understand the possibilities of your data? This workshop will explore how GLAM organisations can create and share resources that encourage experimentation.

The GLAM Workbench is a large collection of tools, hacks, and tutorials aimed at helping researchers make use of collection data. It uses platforms such as Jupyter notebooks to create live, working examples that run in your browser without additional software. Similar repositories of computational resources are being developed by GLAM organisations around the world.

This workshop will introduce the technologies and standards used in the GLAM Workbench, such as Jupyter notebooks. It will provide an overview of related activity around the world, including best practice guidelines for GLAM organisations developing computational resources. It will explain how organisations and individuals can contribute content to the GLAM Workbench, or use it as a model to create their own specialised workbenches.

Sharing data is important, but so is sharing skills, tools, and knowledge. Come along to find out how the GLAM Workbench can help.

It’s a free, hybrid event (in person and online) and will run from 1.00-3.00pm. A sign up page should be available soon.

On Wednesday 3 December I’m presenting the results of my residency in a ‘technologist’s talk’. It’s an internal event, but it’s in the public ‘Create quarter’ of the Library, so I think anyone can pop in. Hopefully there’ll be a video I can share.

To give you an idea of what I’ll be talking about, here’s some of the outcomes so far:

hacking the library workshop (slides, and blog post about urls)
bounding boxes for parish maps (blog post, code)
geolocating the Committee for Urban Action collection of photographs (prototype interface, still documenting the method)
a new fully-searchable version of the Sands & MacDougall’s directories (blog post, database, and another blog post)
georeferencing digitised maps – over 500 so far! (documentation, dashboard, data repository, blog post)
and as of yesterday, 3,000+ geolocated newspapers (documentation and interface coming!)

First attempt at mapping places of publication and distribution of Victorian newspapers

At the moment I’m trying to bring it all together in a new interface that let’s you type in an address and find collection materials relating to your home, your street, and your suburb. Only two weeks to go! Eeek!

Work in progress!

Some Sands & Mac tweaks thanks to ALTO and IIIF

Sun, 16 Nov 2025 15:17:26 +1100

I posted recently about my new fully-searchable version of the Sands & MacDougall directories. I’ve now moved on to try and pull together a number of the State Library of Victoria’s place-based collections into a new discovery interface. It’s going to be a busy couple of weeks as my residency ends in early December!

I wanted to incorporate Sands & Mac search results into the new interface. Getting the data was easy because Datasette has a JSON API baked in. But what about the images? I could just display a thumbnail of the whole page, but it would be better to show a snippet of the actual entry. Thanks to IIIF and ALTO, I now can.

IIIF makes it easy to cut small sections out of a larger image. You just put the coordinates of the desired section in the IIIF url. As I noted in my previous post, the ALTO files that contain the OCR data from Sands & Mac include the coordinates of every line, and every word. I just had to bring the two together.

All I did was update the code that extracts the data from the ALTO files to save the results as newline delimited JSON instead of a plain text file. Each line in each volume of Sands & Mac is now saved a JSON object that contains the text, as well as the height, width, vertical position, and horizontal position of the line within the page image. When I load up the SQLite database, I add the values for h, w, x, and y as well as the text for each line.

What does this make possible?

When you go to an individual entry, the page image now automatically pans and zooms so that the current entry is at the centre of the image viewer. I just updated the OpenSeadragon code to focus on the entry’s position.
If you share an entry on social media, a snipped out section of the page image showing the selected entry is displayed as there’s now an image META tag that points to an IIIF url.
You can retrieve entries via the API and use the coordinates to request snipped out images of them via IIIF.

Nice image snippets thanks to IIIF and ALTO (and a sneak prview of what's coming...)

A new way of searching Sands & Mac

Wed, 12 Nov 2025 22:25:21 +1100

In the fortnight I spent onsite at the State Library of Victoria, ‘Sands & Mac’ was mentioned many times. And no wonder. The Sands & McDougall’s directories are a goldmine for anyone researching family, local, or social history. They list thousands of names and addresses, enabling you to find individuals, and explore changing land use over time. When people ask the SLV’s librarians, ‘What can you tell me about the history of my house?’, Sands & Mac is one of the first resources consulted.

The SLV has digitised 24 volumes of Sands & Mac, one every five years from 1860 to 1974. You can browse the contents of each volume in the SLV image viewer, using the partial contents listing to help you find your way to sections of interest. To search the full text content you need to use the PDF version, either in the built-in viewer, or by downloading the PDF. There’s a handy guide to using Sands & Mac that explains the options.

However, there’s currently no way of searching across all 24 volumes, so as part of my residency at the SLV LAB, I thought I’d make one!

Try it now!

My new Sands & Mac database follows the pattern I’ve used previously to create fully-searchable versions of the NSW Post Office directories, Sydney telephone directories, and Tasmanian Post Office directories. Every line of text is saved to a database, so a single query searches for entries across all volumes. You can also use advanced search features like wildcards and boolean operators.

Search across all 24 volumes!

Once you’ve found a relevant entry you can view it in context, alongside a zoomable image of the page. You can even use Zotero to save individual entries to your own research database. This blog post from the Everyday Heritage project describes how the Tasmanian directories have been used to map Tasmania’s Chinese population.

View each entry in context! (Here's my Dad building his first house in Beaumaris in the 1950s.)

There’s still a few things I’d like to try, such as making use of the table of contents information for each volume. I’d also like to create some additional entry points to take users directly to listings for individual suburbs (maybe even streets!). Each volume has a directory of suburbs, so it would be a matter of extracting and cleaning the data and linking the entries to digitised pages. Certainly possible, but I don’t think I’ll have time to get it all done before the end of my residency. Perhaps I’ll try to get at least one volume done to demonstrate how it might work, and the value it would add. As I was writing this blog post I also realised there’s a dataset of businesses extracted from the Sands & Mac, so I need to think about how I can use that as well!

Technical information follows…

I’ve documented the process I used to create fully-searchable versions of the Tasmanian and NSW directories in the GLAM Workbench. I followed a similar method for Sands and Mac, though with a few dead-ends and discoveries along the way.

Downloading the PDFs

I assumed that it would be easiest to work from the PDF versions of each volume, as I’d done for Tasmania. So I set about finding a way to download them all. There’s only 24 volumes, so I could have downloaded them manually, but where’s the fun in that?

I started with a CSV file listing the Sands & Mac volumes that I downloaded from the catalogue. This gave me the Alma identifiers for each volume. To download the PDFs I needed two more identifiers, the IE identifier assigned to each digitised item, and a file identifier that points to the PDF version of the item. The IE identifier can be extracted from the item’s MARC record, as I described in my post on exploring urls. The PDF file identifier was a bit more difficult to track down. The PDF links in the image viewer are generated dynamically, so the data had to be coming from somewhere. Eventually I found that the viewer loaded a JSON file with all sorts of useful metadata in it!

The url to download the JSON file is: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. In the summary section I found identifiers for small_pdf and master_pdf.

I could then use these identifiers to construct urls to download the PDFs themselves: https://rosetta.slv.vic.gov.au/delivery/DeliveryManagerServlet?dps_func=stream&dps_pid=[PDF id]

Once I had the PDFs I used PyMuPDF to extract all the text and images. As I suspected the text wasn’t really fit for purpose. The OCR was ok, but the column structures were a mess. Because I wanted to index each entry individually, it was important to try and get the columns represented as accurately as possible. The images in the small PDFs were already bitonal, so I started feeding them to Tesseract to see if I could get better results. After a bit of tweaking, things were looking pretty good. But when I came to compile all the data, I realised there was a potential problem matching the PDF pages to the images available through IIIF. I found one case where some pages were missing from the PDF, and another couple where the page order was different.

As I was looking around for a solution, I realised that those JSON files I downloaded to get the PDF identifiers also included links to ALTO XML files that contain all the original OCR data (before it got mangled by the PDF formatting). There was one ALTO file for every page. Even better, the JSON linked the identifiers for the text and the image together – no more page mismatches!

Downloading the ALTO files

Let’s start this again shall we. After wasting several days futzing about with the PDFs, I decided to download all the ALTO files and extract the text from them. As I downloaded each XML file, I also grabbed the corresponding image identifier from the JSON and included both identifiers in the file name for safe keeping.

The ALTO files break the text down by block, line, and word. To extract the text, I just looped through every line, joining the words back together as a string, and writing the result to a new text file – one for each page.

It’s worth noting that the ALTO files include all the positional data generated by the OCR process, so you have the size and position of every word on every page. I just pulled out the text, but there are many more interesting things you could do…

Assembling and publishing the database

From here on everything pretty much followed the pattern of the NSW and Tasmanian directories. I looped through each volume, page, and line of text, adding the text and metadata to a SQLite database using sqlite_utils. I then indexed the text for full-text searching. At the same time I populated a metadata file with titles, urls, and few configuration details. The metadata file is used by Datasette to fill in parts of the interface.

I made some minor changes to the Datasette template I used for the other directories. In particular, I had to update the urls that loaded the IIIF images into the OpenSeadragon viewer. But it mostly just worked. It’s so nice to be able to reuse existing patterns!

Finally, I used Datasette’s publish command to push everything to Google Cloudrun. The final database contains details of more than 50,000 pages, and over 19 million lines of text! It weighs in at about 1.7gb. The Cloudrun service will ‘scale to zero’ when not in use. This saves some money and resources, but means it can take a little while to spin up. Once it’s loaded, it’s very fast. My original post on the Tasmanian directories included a little note on costs, if you’re interested.

More information

The notebooks I used are on GitHub:

Here are some posts about the NSW and Tasmanian directories:

Making NSW Postal Directories (and other digitised directories) easier to search with the GLAM Workbench and Datasette (September 2022)
From 48 PDFs to one searchable database – opening up the Tasmanian Post Office Directories with the GLAM Workbench (September 2022)
Where’s 1920? Missing volume added to Tasmanian Post Office Directories! (September 2024)
Six more volumes added to the searchable database of Tasmanian Post Office Directories! (November 2024)

Turning the SLV's maps into data with Allmaps and some GLAM plumbing

Tue, 04 Nov 2025 15:02:53 +1100

I often describe what I do as GLAM data plumbing. Most of the time I’m not creating new tools, I’m figuring out what data is available and how I can connect it up to existing tools. It’s rarely straightforward, but if I can get all the pipes connected and data flowing in the right direction, suddenly new things become possible.

Things like turning all the State Library of Victoria’s digitised maps into data.

I’ve just created a workflow that uses AllMaps and IIIF to georeference the SLV’s digitised maps. There’s some technical details below, but the idea is pretty simple. A userscript links the SLV image viewer to Allmaps – so you just click on a button, and the digitised map opens, ready for georeferencing.

Why is this useful? Georeferencing relates a digitised map to real world geography. It describes the map’s position and extent using geospatial coordinates – turning historic documents into geospatial data that can be indexed, visualised and manipulated. Georeferencing opens digitised maps to new research uses.

So, how many maps we can georeference before my residency finishes in December? Hundreds? Thousands? If you like maps and want to help, head to the documentation page to find out how to get started. And if you want to see how things are progressing, have a look at the project dashboard.

View the documentation to get started

A few technical details follow…

Early on in my time as Creative Technologist-in-Residence at the State Library of Victoria, I started playing around with Allmaps for georeferencing digitised maps. It’s a great tool (really a suite of tools and standards) because instead of constructing a whole new platform it integrates with existing IIIF services. The SLV provides digitised images through IIIF, so I thought it should be possible to use Allmaps to georeference the SLV’s map collection.

But I struck a problem that took some time to unravel. The IIIF urls in the SLV manifests include port numbers and that confused Allmaps. The manifests also sometimes contained references to image formats that weren’t actually accessible, generating errors when they were loaded. Hopefully these problems will be fixed by the SLV, but in the meantime I’ve created a proxy service that edits the manifest on the fly. The proxied urls can be loaded into the Allmaps Editor without errors. Pipes fixed, data flowing!

Using the manifest proxy

To generate a link to a proxied manifest, first grab the item's IE identifier from the url of the digitised item viewer. For example, the identifier in this url https://viewer.slv.vic.gov.au/?entity=IE15485265&mode=browse is IE15485265. Once you have the identifier, add it to the end of the url https://wraggelabs.com/slv_iiif/. For example, https://wraggelabs.com/slv_iiif/IE15485265. You can then supply this url to the Allmaps editor.

But having to fiddle around with proxies didn’t make a great user experience. I needed some way of integrating the two services, so that a user could just click a button in the SLV website and start editing in Allmaps. Userscripts to the rescue!

I wrote recently about hacking GLAM collection interfaces using userscripts. Since I started my residency at the SLV, I’ve also created a userscript to display the IIIF manifest url in the SLV image viewer, and run a Code Club workshop where we played around with an assortment of SLV website hacks.

As in a number of these examples, the georeferencing userscript adds new features to the SLV website, but there’s a fair bit more going on under the hood. It runs automatically every time you load the SLV image viewer, and then:

it checks the metadata of the digitised item to see it it’s a map (or something that contains maps, like an atlas or street directory)
if it looks like a map, it generates an Allmaps identifier using the item’s IIIF manifest url and checks with Allmaps to see whether the item has already been georeferenced
it adds a ‘Georeferencing’ section to the page, with a button to georeference the item (or edit the existing georeferencing)
if the item has already been georeferenced, it adds a button to view the item in the Allmaps Viewer, and embeds a live preview

Accessing metadata

The userscript gets the item metadata from a JSON file that's loaded by the image viewer. The JSON file includes a lot of extra, useful information about the digitised item. To access the JSON file, you just construct a url like this: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. The IE identifier is in the url of the image viewer.

Allmaps identifiers

Allmaps creates its identifiers by hash encoding the IIIF urls. The userscript borrows some code from the Allmaps id module to generate the ids, then sends a HEAD request to the Allmaps API to see whether an entry for the current manifest exists.

Example of an item that hasn't been georeferenced yet

Example of an item that has been georeferenced, displaying an embedded version of the Allmaps viewer

I’ve also created a GitHub repository to save copies of the data. Every two hours this notebook is run to query the Allmaps API for newly georeferenced maps. These are added to a dataset which is saved in three formats:

a CSV file
a CSV file that includes thumbnails and links for viewing in Datasette-Lite
a GeoJSON file, that can be viewed in services like geojson.io

At the same time, the data for each individual map is downloaded and saved as IIIF annotations (in JSON) and GeoJSON.

Finally, this notebook is run to generate a dashboard that provides an overview of the project’s progress.

The project dashboard is updated every two hours

One of the Allmaps developers described all my plumbing and workarounds as a ‘very cool lofi example of how you can set this up with little means’, and I think that’s pretty apt. It’s really just an experiment to demonstrate the possibilities, but by connecting up existing services it’s generating real data of long term value.

Me at 63...

Mon, 03 Nov 2025 17:55:17 +1100

Originally published in Pharos, newsletter of the Professional Historian’s Association (Vic & Tas), October-November 2025, in the ‘Member Profile’ section.

In the early 1990s I started working for a small self-funded organisation called the Australian Science Archives Project. Our mission was to preserve and raise awareness of Australia’s scientific past. When the web came along, we realised it provided an enormous opportunity to communicate history to the public. So I taught myself web development and created the first archives website in Australia. Since then my work has continued to explore what happens when we release GLAM collections into online spaces where people can see and use them differently.

What kind of work have you done? What are you working on now?

I’ve had a range of jobs in the GLAM and university sectors. While ‘history’ wasn’t often in my job title, I’ve always regarded myself as a historian first – whether I was coding, editing, writing, teaching, or managing, history was always the frame through which I understood my work. At the same time, I’ve maintained my own independent practice as a ‘historian and hacker’, developing tools and resources for other researchers, such as the GLAM Workbench. Much of this work is unfunded, but by sharing it openly I’ve created new opportunities for collaboration. For example, I’m currently the ‘Creative Technologist-in-Residence’ at the State Library of Victoria, bringing my years of GLAM hacking to bear on the Library’s place based collections.

Research or writing? (What do you enjoy more and why?)

Researching, or writing, or coding, or teaching, or outreaching (what is the correct verb?) – all have their joys and travails. For me, research is less about finding things in archives and libraries, and more about how we find things in archives and libraries. I poke about in online collections to try and understand how they work, what they reveal, and what they hide. This often leads to the development of new tools, the writing of documentation and blog posts, and sometimes even real, published articles. It’s a process that has consumed my life, for better or worse. Coding often slips into obsession when I have a gnarly problem to crack. Writing is a slog, but there’s nothing like the pleasure of a finely-turned sentence. Teaching is exhausting, but also exhilarating when you see the light bulb of understanding flick on.

What are the best and hardest things about the kind of work you do?

The best thing, the absolute hands-down best thing, is hearing from people who use, or have benefited from the tools and resources that I’ve created. I make things to help researchers see and use GLAM collections in new ways, so finding out what they’ve been doing with my stuff always provides a much-needed jolt of inspiration.

However, the flip side is that getting information about my tools and resources out to the people who might benefit most is hard and often frustrating work. I churn away in the social media mines, but people and organisations seem much more reluctant to share new work these days. There was a time (yeah, the good old days) when GLAM organisations actively engaged with researchers online, sharing the cool things people were doing with their collections. But not now. We all learn through the generosity of others, and I think its important that we find ways to support and enlarge the realm of generosity.

Creating bounding boxes for parish maps in the SLV collection

Mon, 06 Oct 2025 15:17:51 +1100

The State Library of Victoria holds a collection of 8,804 parish maps. As part of my residency at the SLV LAB, I’ve been poking around in the metadata.

SLV staff have geocoded many of the parish maps using the Composite Gazetteer of Australia, which provides coordinates for Victorian parishes and boroughs. These coordinates give us a point which should be roughly at the centre of each map, enabling us to visualise their locations and distribution. But how much area do they cover? To answer that question we need a bounding box that includes the coordinates of each corner of the map. We could create bounding boxes by using something like AllMaps or MapWarper to georeference each individual map, but that’s going to take a while! As a quick and dirty alternative, I wondered if it was possible to generate approximate bounding boxes from the available metadata. It seems we can!

The metadata

There are three pieces of metadata we need to construct bounding boxes:

the latitude and longitude of the centre point
the size of the physical map
the scale of the map (ie how the size of the map relates to real world)

The coordinates and scale can be included in a couple of different places in the map’s MARC record. The 034 field is specifically for ‘Coded Cartographic Mathematical Data’. The relevant subfields are:

$a: category of scale
$b: constant ratio linear horizontal scale (this is the most likely type of scale)
$d: westernmost longitude
$e: easternmost longitude
$f: northernmost latitude
$g: southernmost latitude

If the coordinates describe a point rather than a bounding box, then $d and $e will be the same, and $f and $g will be the same.

String representations of coordinates and scale can be found in the 255 field. The relevant subfields are:

$a: statement of scale, eg Scale [ca. 1:90,000].
$c: statement of coordinates, eg (E 142°18'/S 37°33')

The size of the map is recorded in 300 (physical description) field under the $c (dimensions) subfield. For example: on sheet 40 x 51 cm .

The method

I started with an existing dataset downloaded from the catalogue by SLV staff. This dataset included the scale and coordinate information in the 034 field, and the coordinate string in 255$c. At first I didn’t realise that the 034 held geo data, so I separately downloaded the scale information from 255:$a in each item’s MARC record (d’oh). If the maps were digitised, I also wanted their image identifiers so I could access them through the SLV’s IIIF service. The image id from the 956$e field of the MARC record can be used to construct an IIIF manifest url, so I extracted them as well.

Once I had all the catalogue data, I had to make sure everything was in a format I could work with. The coordinates in the MARC records are recorded as degrees/minutes/seconds, so I had to convert them to decimal values. The scale factor needed to be an integer, and I needed to extract the height and width as integers from the dimensions field.

I used lat_lon_parser to convert the coordinates to decimal, but needed a bit of regex string manipulation to get the values into a format that could be parsed. Regex also came to the rescue in getting the map dimensions. All the details are in this notebook.

Creating bounding boxes

After some searching I found this StackOverflow comment that described how to create a bounding box from a point, distance, and bearing. The point I already had, but the distance and bearing had to be calculated. Trigonometry to the rescue!

The distance from the point at the centre of the box to one of its corners is the hypotenuse of a right-angled triangle whose sides are equal to half the width and half the height of the map, and thanks to Pythagorus we know:

Once I had the distance in cm, I converted to inches, then multiplied by the scale factor, and finally converted the inches to miles. (It now occurs to me that there’s no need to convert to imperial measurements, but it doesn’t make any difference either way.)

The bearing that points to the corner of the box is the angle inside the same right-angled triangle, so can be calculated using:

With the point of origin, distance, and bearing I could use geopy to calculate the corners of the bounding box!

from geopy.distance import geodesic

destination = geodesic(miles=distance).destination(origin, bearing)
coords = destination.longitude, destination.latitude

See this notebook for the full details.

Limitations

Of course, this method is very rough and has a number of major limitations, in particular:

only about 38% of the maps have point coordinates
the point values don’t necessarily locate the centre of the map
not all the maps are oriented towards north
sometimes a parish includes multiple maps
the size of the margin around the map will affect the accuracy of the bounding box

But despite these problems the results seem pretty good. To test this I created a notebook to overlay the digitised maps on a modern basemap using the bounding boxes. Here’s an example.

You can see the map is slightly offset (presumably due to the second problem listed above). But the size seems about right. Certainly good enough to use the bounding boxes in some exploratory analyses!

Visualising the results

I’ve saved the processed data as a new dataset, and started playing around with a couple of ways of visualising the results. These are experiments, not discovery interfaces. But you can use them for a bit of exploration if you don’t mind a few bugs. They’re all in Jupyter notebooks that can be run using the Binder service.

The parish maps browser includes a dropdown list of parish maps with point coordinates. Select a map and:

if there’s a bounding box and an image identifier, the image of the parish map will be overlaid on the modern base map using the bounding box coordinates
if there’s a bounding box, but no image identifier, a rectangle will be drawn on the base map showing the dimensions of the bounding box
if there are point coordinates, but no bounding box, a marker will be placed on the base map

If the image of the map is displayed you can use the slider to adjust the opacity. Clicking on either the image, rectangle, or marker will display metadata about the parish map and a link to the SLV catalogue.

There’s also a visualisation of all the bounding boxes overlaid on a modern base map.

As you move your mouse over the bounding boxes the titles are displayed on the map, and if you click on a bounding box the metadata is displayed beneath the map, including a link to the SLV catalogue.

It’s obvious from the image above that some of the coordinates must be wrong! Visualisation is a great way of finding problems with your data. I now need to work through the results, documenting the problems, and thinking about how to make best use of the data. More to come!

Exploring SLV urls

Tue, 23 Sep 2025 17:22:45 +1100

I like urls. They take you places. And if you know how to read them, they can tell you things about the systems that created them. One of the first things I did when I started my residency at SLV LAB, was to try and understand how their collection urls work. There’s a couple of well-worn methods I use when digging into a new site.

The first is url hacking – this involves fiddling around with the parameters in a url and submitting the result to see what happens. The Trove Data Guide includes some examples of hacking Trove urls to change the delivery of search results.

The second method involves opening up the developer console in your web browser and watching the activity in the network tab as you click on links. This tells you where the information that gets loaded into your browser actually comes from – sometimes exposing handy urls that you can use to shortcut access to useful data.

Permalinks

The SLV uses Primo for its public-facing catalogue, as well as other systems such as Rosetta and IIIF to deliver digitised content. I’d noticed that Zotero gets some useful data from the catalogue using the default ‘Primo 2018’ translator, however, important things like the item url aren’t captured. The problem is that Primo’s ‘permalinks’ are generated as required by a browser click – they’re not embedded anywhere on the page. This makes it hard to Zotero to grab them. So I started wondering how Zotero could construct short, persistent(ish) links to items.

Here’s a link to an item in Primo: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636

It looks pretty long and messy, but if you start deleting parameters and resubmitting, you’ll find that only two parameters are essential, vid and docid. This means we can rewrite the url as: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&docid=alma9941325055707636 Much nicer.

The ‘permalink’ for the same item is: https://find.slv.vic.gov.au/permalink/61SLV_INST/1sev8ar/alma9941325055707636 If you look closely at the url path and compare it to the example above you’ll see the path is constructed from /vid/[some other id]/docid. One of the librarians explained to me that the other identifier in the permalink is an encoding of the view type, but given that the ‘fulldisplay’ view is the default, we don’t really need it. So the shortened url seems fine for use in Zotero and is easy to generate from the current url. Nice.

It’s also worth noting that the vid value doesn’t seem to change, so to construct catalogue urls in your code, all you really need is the ALMA identifier that’s in the docid parameter.

Structured data

Item pages in Primo include a link labelled ‘Display source record’. If you click on this you’re taken to a representation of the item’s metadata in MARC. Here’s what the urls look like: https://find.slv.vic.gov.au/discovery/sourceRecord?vid=61SLV_INST%3ASLV&docId=alma9941325055707636&recordOwner=61SLV_INST Notice that the ‘fulldisplay’ in the url path above has changed to ‘sourceRecord’. There’s also a new recordOwner parameter, but it seems you can delete this and still get the same result.

Having access to the MARC record is handy, because it delivers the metadata in a simple, structured plain text format. But while the ‘source record’ page looks like a plain text file, it’s actually a HTML page that embeds a plain text record. If you open up the network tab of your browser’s developer console and reload the ‘source record’ page, you’ll see a different url is loaded under the hood: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV&recordOwner=61SLV_INST&lang=en See how the url path has changed from /discovery/ to /primaws/rest/pub? This url does deliver a plain text version of the MARC record.

Once you have the plain text version you can parse the contents to extract the structured data. There are tools that can probably do this automatically, but it’s also pretty easy using regular expressions. Here’s an example of some code I used to parse map records.

def get_marc_value(marc, tag, subfield):
    """
    Gets the value of a tag/subfield from a text version of an item's MARC record.
    """
    try:
        tag = re.search(rf"^{tag}\t.+", marc, re.M).group(0)
        subfield = re.search(rf"\${subfield}([^\$]+)", tag).group(1)
    except AttributeError:
        return None
    return subfield.strip(" .,")

You can also access a JSON representation of the record by adding the parameter &showPnx=true to the catalogue url: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636&showPnx=true

Once again, this is a JSON representation embedded in a web page. Using the same developer console trick, you can identify the direct url is: https://find.slv.vic.gov.au/primaws/rest/pub/pnxs/L/alma9941325055707636?vid=61SLV_INST:SLV&lang=en&search_scope=slv_local&showPnx=true&lang=en You should be able to parse the response from this url as JSON and use it in your code. I think the Zotero translator makes use of this pnx data.

If you want to download the MARC or JSON representations in your code, all you really need is the alma identifier. Just use it to construct one of the direct urls, such as this: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV The recordOwner and lang parameters are not needed, and the vid parameter doesn’t change.

Librarians using Primo have documented a number of tricks like this and shared handy bookmarklets to rewrite urls and get catalogue data in different forms.

IIIF and images

SLV delivers digitised images using IIIF. The IIIF manifest urls are not directly exposed through the web interface, but you can construct your own.

IIIF manifest urls look like this: https://rosetta.slv.vic.gov.au/delivery/iiif/presentation/2.1/IE24074939/manifest.json All we need to construct them is the IE identifier, in this case IE24074939. But where do you find this identifier?

If you’re looking at an image in the SLV’s image viewer, the url will be something like this: https://viewer.slv.vic.gov.au/?entity=IE24074939&mode=browse Yep, the IE identifier is right there in the url. Just extract it from the viewer url, and plug it into the manifest url!

If you’re looking at a catalogue record, or starting with one of the alma identifiers, you can get the IE identifier from the 956$e field of the MARC record.

The IIIF manifest will, in turn, provide identifiers for individual images that can be requested using the standard IIIF syntax.

To save myself a bit of fiddling about, I created a userscript that exposes the IIIF manifest url within the image viewer. If you install it you’ll see something like this:

Handles

Links to digitised items sometimes come in the form of ‘handles’: http://handle.slv.vic.gov.au/10381/4338980 These urls are redirected to the image viewer.

If you want to construct one of these handles, the identifier can be found in 956$a field of the MARC record.

From old to new

I was looking at the datasets created about 8 years ago in the SLV open data repository and noticed they included urls from the previous catalogue. Fortunately, the old urls redirect to the new system.

For example, this url: http://search.slv.vic.gov.au/MAIN:Everything:SLV_VOYAGER1842440

Redirects to: https://find.slv.vic.gov.au/discovery/fulldisplay?context=L&vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&docid=alma9918424403607636

If you look closely at the urls you’ll see that the identifier from the old system is embedded in the new identifier: 1842440 is in 9918424403607636 – 99_1842440_3607636. This means if you have a lot of old urls, such as in the open datasets, you can easily rewrite them in your code.

The process of GLAM hacking

No doubt a lot of this is well-known to librarians, and there’s probably many subtleties or complexities that my poking about has missed. But I wanted to document the process as much as the results – to give an idea of what I do when I approach a new GLAM collection online. I suppose this is GLAM hacking 101.

Creative Technologist-in-Residence at the State Library of Victoria!

Tue, 23 Sep 2025 00:14:09 +1100

I’m very excited to be the new Creative Technologist-in-Residence at the SLV LAB. For the next few months I get to play around with metadata and images, think about online access, experiment with different technologies, and build things to help people to explore the State Library’s collections. In other words, I get to be in my happy place!

My group at the recent SLV WikiFest was thinking about ways of helping researchers find resources relating to particular locations – how do I find material about my suburb, or my street? Coincidentally, the main focus of my residency will also be place-based collections, so I get to really think through some of the possibilities. SLV staff have already pointed me to some amazing maps and photographs, such as the Committee for Urban Action collection, the Mahlstedt fire survey maps, the MMBW plans, and the Victorian parish maps.

At the same time, I’ll be using my usual GLAM hacking approach to poke around in the SLV website to try and understand what data is currently available, identify any roadblocks, and document opportunities for computational research.

The results of my residency will be shared on the SLV LAB site, in GitHub, in the SLV section of the GLAM Workbench, and of course here. As usual, I’ll be working in the open, documenting things as I go along, so please join me on the journey!

Although the residency was formally announced today, I’ve actually been working with SLV data for the last couple of weeks and I’ve already got a backlog of stuff I need to blog about. Here’s a taster – what happens when you generate bounding boxes for thousands of parish maps from the available metadata and throw them on a map…?

WikiFest at the State Library of Victoria

Fri, 29 Aug 2025 16:07:26 +1100

This week I was lucky enough to participate in WikiFest at the State Library of Victoria. Organised by the State Library’s new innovation LAB and Wikimedia Australia, Wikifest was a hands-on, participant-led workshop focused on the possibilities of connecting SLV’s collections to (and through!) Wikidata.

The day kicked off with a series of presentations demonstrating possible uses of Wikidata. I talked a bit about some of my recent GLAM/Wikidata experiments. My slides are online and contain plenty of links to code, demonstrations, and documentation. They’re openly-licensed, so feel free to take anything of use.

The rest of the day was spent in groups, working on particular projects and learning more about Wikidata in the process. My group was looking at providing placed-based entry points to SLV collections, and spent a lot of time exploring the representation of Victoria’s Local Government Areas (LGAs) in Wikidata. We realised there was quite a bit of work to do in adding things like dates and boundaries, but we could see some exciting future possibilities. We also made a start, adding an ‘inception’ date for the City of Moe, based on the Victorian Government Gazette, digitised by the SLV.

Bonus userscript

While I was preparing my presentation I was thinking about the the way entries for Australian people in Wikidata are linked to a range of different identifiers, such as DAAO, the Encyclopedia of Australian Science, and the Australian Dictionary of Biography (ADB). Often a single person can have multiple identifiers and this means that those identifiers themselves become connected through that person’s record. You can query Wikidata with one identifier, and get back links to a range of other information sources about that person.

To demonstrate this, I created a simple userscript that adds additional links to biographies in the ADB. The script grabs the ADB identifier from the url, queries Wikidata for additional identifiers, and writes the results into the page’s ‘Life Summary’. Basic, but useful!

For something more advanced, have a look at the Entity Explosion extension for Firefox.

GLAM hacking with userscripts

Thu, 17 Jul 2025 18:21:25 +1100

In teaching and workshops I used to get students to question the idea that websites are ‘published’. They’re not released into the world in a fixed, immutable form – they’re a set of blueprints which only reach their final form in your browser window. This makes it possible to change the way websites look and behave.

Mozilla used to have a nifty educational tool called X-Ray Googles. Using it you could explore the code underlying a web page and do fun things like inserting new text or images. I encouraged students to try hacking ASIO’s home page.

ASIO home page with some added First Dog on the Moon.

There are other ways you can fiddle with websites. For example, most browsers have a developer console that exposes the code and styling of a page. You can use the console to edit HTML elements or toggle styles, but your changes won’t be saved.

One way you can save and share your web site customisations is by creating userscripts. Userscripts are little bits of Javascript code that run in your browser after a web page loads. These scripts can change many aspects of a page – not just how it looks, but also how it works.

Some old userscripts

I’ve been playing around with userscripts for a long time. Back in 2008, I created a userscript that completely overhauled the way that digital files were presented in the National Archives of Australia’s online database, RecordSearch. My userscript added new options for navigating and printing the file, and even made it possible to view the complete file contents on a 3D zoomable wall.

This customised RecordSearch interface was created by a userscript.

The userscripts I’ve created over the years have tended to either be useful little hacks aimed at fixing annoying aspects of GLAM websites, or experiments in thinking about the sort of information that’s presented online by GLAM organisations, and how it might be different.

In the first category are hacks like my RecordSearch show pages userscript. In 2009, I got annoyed that there was no way of knowing how many pages were in a digitised file until you clicked on the link. So I fixed it. With my userscript running, the links to digitised files are rewritten to display the number of pages. I’ve updated the code numerous times over the years, adding new features, and dealing with changes to RecordSearch. The last update was just a few days ago.

In the second category is my userscript that inserts photos from The Real Face of White Australia into RecordSearch. The are many thousands of records in the National Archives of Australia that document the impact of the White Australia Policy on the lives of ordinary people. But it’s often hard to understand this from the file descriptions. The userscript displays portrait images extracted from the files alongside the metadata – it tells you there are people inside.

Amidst my recent self-archiving binge, I realised I’d never updated this userscript to work with the latest data from The Real Face of White Australia, so I spent some time getting it working again. In the process I realised that RecordSearch now included content security policies that made it a bit harder to insert new images. The solution was to use one of the special userscript functions, GM_addElement(), rather than plain old Javascript. But then I discovered that the if the show pages userscript ran after this one, it would trigger the security restrictions nonetheless! To avoid this I made sure that the two userscripts operated on separate elements. So now the show people userscript is working again!

And a new userscript to improve Trove lists

Fixing up the ‘people inside’ code reminded me of how much fun it was playing around with userscripts, so when David Coombe mentioned a problem he had using Trove lists on Mastodon last night, I had to have a go at fixing it.

The problem is that Trove lists display all the tags associated with each individual item. Some items have lots of tags, so this eats up the screen real estate, making it harder to browse the contents of a list. Notes attached to items can be hidden, but not tags. Why not?

My brand new userscript hides tags by default, and adds a new link to toggle their visibility for each individual item. The link also displays the number of tags attached to each item. This gives the user control over which tags are displayed and when.

The new userscript in action – toggle your tags!

The main difficulty in creating this userscript was knowing when the page had actually finished loading. The current version of Trove uses a lot of Javascript to load and manipulate content, so you have to tell the userscript to wait until everything has settled down. Otherwise the script could fire too soon and cause unexpected results. I tried a number of different approaches to handling this problem, but eventually settled on the waitForKeyElements script. (I just realised there’s a more recent version of this script that doesn’t require JQuery, so I might need to investigate this further.)

Another Trove problem fixed!

Using userscripts

In addition to the userscripts mentioned above, I’ve also created one that enables you to browse Trove newspaper pages using the arrows on your keyboard. Left and right arrows go to the next and previous pages, while up and down arrows jump between issues. Searching is great, but sometimes you just want to browse. Install this userscript for that old-time, authentic newspaper reading experience!

But how do you install userscripts? First of all you need a browser extension to manage your userscripts – I use TamperMonkey or ViolentMonkey. Just follow the instructions to add one of them to your browser.

To install one of my userscripts, you need to go to the script (saved as a GitHub Gist) and click on the ‘Raw’ button. Your userscript manager will then ask you if you want to add the userscript. Click install!

Click on the ‘Raw’ button to install.

Once they’re installed the userscripts will run automatically when specified pages are loaded. If you ever want to disable them, you can do that from your userscript manager’s dashboard.

For convenience, here are the Gist links to all the userscripts I’ve mentioned:

As with anything you install on your computer, you want to make sure that you trust the source of any userscripts you add.

The rebirth of Wragge Labs (and moving my Heroku apps)

Wed, 09 Jul 2025 17:48:23 +1100

It looks like some paid work I was counting on won’t be going ahead, so I’m trying to save a bit of money on cloud hosting. As I previously noted, this resulted in the resurrection of The future of the past, but I’ve also been continuing to slog away at migrating all my old Flask apps and experiments from Heroku to a single Digital Ocean droplet. As of today, I’ve migrated 11 apps. Here’s a few details…

A new (old) home

The first thing I had to figure out was how to group together a series of individual Flask apps so I could easily run and maintain them on a single server, without making major changes to the apps themselves. I decided to go with the application dispatching pattern described in the Flask documentation. This groups the apps within a single Python environment so I had to do some alignment of Python versions and packages, but it wasn’t too hard and having just one virtual environment to manage seems a lot easier in the long run.

The application dispatching pattern configures the server to run one application at the web root ('/'), with the other apps assigned individual sub-paths. This raised the question, what did I want sitting at the root address? Rather than selecting an existing application for the prime slot, I decided to take the opportunity to build a showcase that included details of many of the things I’ve created over the years.

The old Wragge Labs (circa 2012)

I also needed a new domain name. Or did I? Back in the old days, I had a site where I shared many of my tools and experiments – Wragge Labs. In the intervening years, I’d moved or migrated much of the content away and pointed the wraggelabs.com domain to my main site at timsherratt.au. But this seemed like a good opportunity to resurrect it. So if you’d like to have a play around with some of the things I’ve created over the last 30 years, head along to the all new wraggelabs.com!

The new Wragge Labs showcases websites, apps, and experiments from the past 30 years – some useful, some playful, and some creepy…

A surprising amount of the things I’ve built are still working. As I was compiling my list, I made a few running repairs – for example, fixing broken links in Linking history in place and Magic Squares to get them working again. However, some things only exist now in web archives, and others have been broken by the recent actions of the NLA and NAA.

To provide a bit of extra context, I’ve grouped together publications and presentations documenting many of the experiments. These are saved in Zotero, and tagged with the name of the app. When you click on a ‘Related’ link, the details of any linked resources are retrieved using the Zotero API and displayed on a new page. This means I can add new related resources simply by dropping them into Zotero.

Moving house

The process for moving the apps to their new home was pretty straightforward. On my local machine, I copied the code into the new aggregated structure, added any packages needed into a combined requirements file, and created a new top-level app to direct requests. And then I spun everything up and started fixing bugs…

All of the problems were easily resolved. Most involved fixing up paths to static assets, or in navigation links. The only significant changes to the Python code were caused by the deprecation of the .count() method in Pymongo.

To make life a little harder, I decided to take the opportunity to make sure that all the assets – javascript, css, and font files – were loaded from the local system, and not sitting in the cloud. Having everything local should make it easier to maintain the apps in the long term. It was a bit fiddly tracking down where everything was being loaded from, but not too hard.

The only other changes I made were to add some caching to most of the apps, particularly those that make calls to external databases or APIs. I used Flask-Caching with the local file system backend.

To get the new aggregated application working on a Digital Ocean droplet, I followed the instructions on how to serve Flask applications using uWSGI and Nginx. I think the only thing I did differently was to use pyenv to manage Python versions and the virtual environment. To update the app, I use rsync to copy across the code and systemctl to restart it. So far it’s all working pretty smoothly.

Redirecting Heroku

Once the apps were happy in their new home, I needed to redirect the Heroku addresses to wraggelabs.com. It was surprisingly hard to find good documentation on how to do this, so I’ll document my steps in detail in case its of use to others.

There are a few redirect apps for Heroku around, but I decided to use heroku-redirect because it basically just configures and runs Nginx without any additional processing. First I cloned heroku-redirect to my local system, and then for each app I wanted to migrate I followed these steps:

cd into the heroku-redirect directory
set the git remote for the app you want to redirect: heroku git:remote -a [app name]
if you’re not using the lastest Heroku stack, update it: heroku stack:set heroku-24 -a [app name]
set the url you want to redirect to (without trailing slash): heroku config:set LOCATION=[new url]
to add the full path to the redirected url: heroku config:set PRESERVE_PATH=true
I found I had to remove the Python buildpack from the app before updating, I did this using the Heroku dashboard, but no doubt there’s also a CLI command
I also used the dashboard to add a new nginx buildpack: https://github.com/heroku/heroku-buildpack-nginx.git
finally I pushed the new app, using --force to replace it completely: git push --force heroku master:main

Of course, you’ll need to have the Heroku CLI installed.

A number of my Heroku apps were using basic dynos (which cost US$7 a month), so once they were redirected, I changed them to used shared eco dynos. Yay – money saved! Hopefully, the redirects won’t push the eco dynos beyond their monthly limit.

More experiments to come?

One of the good things about all of this housekeeping is that it’s got me thinking about new experiments. I used to love Flask and Heroku because it they made it so easy to build and share things. Now I can do the same with wraggelabs.com!

The future of the past... in the present

Wed, 02 Jul 2025 13:26:45 +1100

I’ve been on a bit of a self-archiving binge lately. It started because I needed to cut back some of my web hosting costs, and was looking at ways of bringing together a group of separately hosted Heroku apps onto a single Digital Ocean droplet. While taking stock of my various apps and experiments, I remembered there were some that hadn’t survived earlier migrations – in particular, the future of the past.

The the future of the past was a weird little app built on top of a collection of 40,000 newspaper articles, harvested from Trove, that included the phrase ‘the future’. I created it as part of my Harold White Fellowship at the National Library of Australia in 2012, and told the story of its genesis in my fellowship lecture. In short, I extracted words with the highest TF-IDF values for each year in my dataset, and fell in love with them. The word groupings were so odd and evocative that I felt I had to find some way of sharing them.

The future of the past made words the primary means of navigating the collection of newspaper articles. At first you were presented with a random selection of words, sized according to their TF-IDF values. When you clicked on a word, you limited the results to years in which that word appeared. You kept clicking words until only one year matched. Then you were shown a random selection of words from that year, along with the words you’d followed to get to that point. Once you’d arrived at a year, you could click on words to display the content of articles that contained that word. But you could also make poetry.

The future of the past gestured towards fridge magnet poetry in its design and odd jumble of words. And when you finally landed on a single year, you could create your own poems by dragging words into the box at the bottom of the screen. Once you were happy with your poem you could share it on Twitter. And people did.

The most exciting and enjoyable part of the project was watching people create and share their poems. The future of the past even managed to win the ‘Best use of DH for fun’ in the 2012 DH Awards. As I said in my Harold White lecture, I wasn’t really sure what the future of the past was – a discovery interface? a game? a piece of art? But I suppose that’s one of the reasons why I liked it so much.

When I built the app, I was going through a stage of putting everything in Django. Only later did I realise that Flask was a much more suitable framework for the sort of small, experimental apps I was creating. Django was overkill, and the maintenance demands coupled with hosting issues made it difficult to keep things alive. At some point, the future of the past went dark and it just seemed too hard to get it going again…

But last week I had another look, and decided I could resurrect the app in a more maintenance-friendly form by converting it from Django to Flask, and migrating the data from MySQL to SQLite. Django and Flask are both Python frameworks, so it was mainly a matter of unpacking all the logic in Django’s views, models, and handlers and consolidating it into a couple of simple Flask functions. Fortunately, I managed to find an SQL dump of the original database in the backed-up downloads folder of an old laptop. It took a bit of fiddling, but I got the dumped data loaded into SQLite without too many problems.

I also realised I could use the new database to fix up another app I created during my fellowship – a word frequency browser. It’s just a static HTML page, so I added a couple of JSON APIs to the Flask app so it could access the data.

Both Django and Flask use Jinja2 templates, so I didn’t have to do anything much to the interface of the future of the past. I made sure that all the assets (fonts and javascript) were being loaded from local copies to avoid any future problems and, of course, I had to replace the Twitter integration. I decided to add options to share poems on both Mastodon and Bluesky. Mastodon was a little tricky because you need to know a user’s instance before you can post their toot. There are a number of solutions available, but I went with the pattern documented in this GitHub repository. It’s a little clunky because you need to enter your instance name each time you post, and you might also have to allow pop-ups for it to work properly, but it seems to do the job. I did think about updating some other aspects of the interface, but decided to preserve it in its original 2012 grey-toned glory.

So the future of the past lives again in the present!

Create your own poems!

Mining for meanings

Mon, 30 Jun 2025 18:36:42 +1100

In 2012, I was lucky enough to be awarded a Harold White Fellowship by the National Library of Australia. I used my time to explore ways of using Trove’s digitised newspapers as data, and presented my work at a public lecture in May 2012. I spoke from notes and never got round to writing it all up. The recording made by the NLA has disappeared from their website, but is still available in the Internet Archive. The text below is a transcription of the recording made in June 2025 with some minor editing.

You can also listen to the audio, browse the full set of slides, or download a PDF from Zenodo.

Photograph by Christopher Brothers, 2012, nla.gov.au/nla.obj-1…

I. Beyond discovery

Thanks Marie-Louise and thanks to the library for this great opportunity. And of course thanks to all of you for coming along on a night when I’m sure you’d rather be at home waiting for the budget speech. And this is the API working away here in the background. Okay, well, do I really need to introduce the newspaper database? I suspect I probably don’t for this sort of audience. You’re probably avid users of the digitized newspapers online. Are you? Yeah. I did my doctoral research back in the dark ages before Trove, and of course that meant spending many weeks, if not months, destroying my eyesight using microfilm readers. Using what are quite fragmentary printed indexes to try and find stuff which might be relevant to my study. But now of course more than 60 million newspaper articles online and most importantly, really the full text of these articles is searchable. It is something which we’re quite familiar with now, but it is something which is quite revolutionary in many ways.

This unprecedented access to a vast volume of material which documents the ordinary lives of Australians is already changing historical practice. We can now go beyond the well-known events, the big stories and explore the small stories, the fragments, the glimpses of lives which might not otherwise be recorded, but this access comes with a cost. What happens when we do a search and instead of getting 10 results or 100 results, we get 10,000 results or 100,000 results? How do we start to use or understand that sort of thing? What do we do when instead of the clarity and excitement of discovery, we end up with the anxiety and confusion that can come with overwhelming abundance?

Fortunately though, there are a growing number of digital tools which we can turn to. Tools and technologies which enable us to manage this deluge and to explore large volumes of text rather than sort of single search results. Tools that enable us to zoom out of our search results and have a look at the big picture to understand the trends and the patterns to see what’s going on. For example, perhaps we might want to try and track events over time. Have a look for example, this graph shows the prevalence of the words drought and floods in the newspaper database over time.

So we can actually look at that and use it as a way where we can map it to the specific events. And we can see here, of course, this is the federation drought at this point. We could also start to look for patterns that aren’t easy to see within your sort of normal list of search results.

I was interested in having a look at how the word decade might be used. So I searched for decade and I found, as you can see, that there’s these nice sort of regular peaks and I was wondering why have we got these such these regular peaks? And I did a bit more digging and I discovered why. That red line shows the usage of the word census. And you see here how the little peaks sit on top of each other?

So obviously the time we talk most about decades is when a census has come out. So again, this is a sort of pattern which would be very hard to find other ways by just sort of working through our list of search results. We can also use these sorts of technologies for exploring changes in language, the way we talk about things, the labels we use.

This is an example which I’ve taken from the National Library of New Zealand searching through New Zealand newspapers in this case. But what they’ve done is to look at the change in usage of the name for the south island, which was apparently, I didn’t know this originally called Middle Island and changed to the South Island. And so you can see here this sort of process of transition happening before South Island takes over completely.

We can also challenge our expectations. Now, I was always of the belief that the traditional name for people from English cultural background of that chap who wears a red suit and comes around at Christmas time was Father Christmas. And then in recent years that has been supplanted by the sort of Americanized Santa Claus, but it seems I’m wrong.

The red line here is Santa Claus, the blue line is Father Christmas. And so if we look here from the late 19th century to the early 20th century, Santa Claus is definitely winning. What’s interesting though, really interesting, is when we get the change over. Any guesses as to what’s going on there?

Audience: Coke advertising.

Pardon?

Audience: Coca-Cola advertising.

Well actually, I don’t know. My hypothesis, what’s happening, is this is around sort of 1914 and it seems that over the war period, Father Christmas starts to win over the top of Santa Claus. So whether, I mean this is pure hypothesis at this point, and it’s something which would be interesting to explore, whether it’s the Germanic sound of Santa Claus, it sort of lapses in popularity or perhaps there are other causes, completely other circumstances. But that’s the value of these sorts of things that they do allow you to ask some questions and to prompt you to do some other sorts of investigation.

Now these graphs which I’m showing you, were all created by a tool I developed called QueryPic. And we won’t just show you the slide, we’ll actually use it. I want a word. Anybody give a word?

Audience: Brooch.

Broach?

Audience: Because yours is nice.

I’m not sure it’s going to show anything. Yeah, brooch.

Audience: Know what we’ve done ‘automaton’, the one we talked about last week.

You have to spell it for me.

Audience: A-U-T-O-M-A-T-O-M. Actually, correct. That’s what you…

No, no, no, this is right. Okay. So what this is doing now, it’s actually going off to the Trove API and it’s getting the total results. I mean it’s actually a very simple tool. All it’s doing is it’s taking your query and it’s searching for each year across the span of the newspaper database, and it’s getting the total number of results for each year and it’s then presenting them in the form of the graph. As I say, it’s very simple, but it’s also quite effective as you can see. And it’s useful and it’s also quite fun. And what it gives you is the ability to quickly explore a hunch, to get a sort of sense of context or to start exploring, to start framing a more specific research question without spending… there we go… without spending days searching or tabulating as you would normally have to do. So you can see how easy it is to use and if you want to actually compare that to something else, you can just type in another word.

Okay, now there are obvious limitations to a tool like this. There’s a lot to unpack. I wouldn’t want to say that it’s evidence because there is so many assumptions built into the back end of it. Questions about what the search engine is actually giving you back, different usages of terms, obviously the contexts and things like the quality of the OCR itself. You know there are a whole lot of stuff. But despite all that, I think it is quite useful, as I said, in terms of allowing you to explore things quite quickly and to follow your hunches. I regard it as a starting point, not as an end.

Now, but there are some folks… let me see if it’s going to finish.. there are some folks who are a bit more confident about techniques such as this and who would suggest that not only can they provide evidence, but they can actually be used to develop mathematical representations of past behavior.

II. Finding formulas

You may have heard of the Culturomics project from Harvard University. These guys got access to the full corpus of Google’s digitized books. So 5 million books, the text of 5 million books. They pulled it all apart. They did a bit of cleaning up of the metadata, all sorts of stuff, and then they started searching it and they started to see what they, but they could pull out of it. And when they started searching, they noticed all sorts of patterns appearing and they argued that these patterns could actually form the basis for what they said was a new science of culture, hence culturomics.

There’s a lot I might say about that, but I just want to look at one example. Okay, this is an example which in their published paper in Science, they called ‘We forget’, and I generated it using an online tool called the Ngram Viewer. You can go and do this yourself if you like. And what it’s showing as you might be able to see is it’s searching for years used within the text. So 1883, 1910, 1950. It’s pulling out all the instances where those labels are used within the text, where those terms are used. And there does obviously seem to be some sort of pattern. And the research has noticed that the graphs have a characteristic shape, obviously rapid ascent and then a decline. But they also notice changes. Of course, the size of the peaks is changing over time, getting higher. They say that this is indicating a greater focus on the present and the rate of decay is increasing, so that the peak is actually dropping away faster. And they say from this, we are forgetting our past faster with each passing year.

I thought it would be interesting to repeat this experiment using QueryPic. So I did. It looks a bit different.

I mean, before we could interpret this difference, of course, there’s a lot that we would want to ask, a lot of, first of all, methodological questions. Again, exactly what are we searching in the two instances and how can we compare the searching, the books in one instance to the newspapers in others - dates obviously play a different role in newspapers than they do in books. But it was actually the conceptual issues, which really struck me in relation to this example and in particular the assumption that we can compare the past, present, and future uses of these labels as if we are talking about the same thing: as if the label 1950 means the same thing before 1950, in 1950 and after 1950. The names for events and periods that we assign, that we share, that we use are themselves the products of historical processes. They slip, they shift, they change.

We all know what we mean by the Great Depression. Where’s the Great Depression on this graph? So in terms of the usage at the time, the usage of the term ‘Great Depression’ was actually greater in the 1890s than in the 1930s.

We’re very familiar with the usage of the, we’re talking about ‘black’ days like Black Tuesdays. Black Fridays of course, is the one we’re most familiar with. And in Australia, these labels are generally attached to bushfires of course, and that’s the context where we generally understand them and use them and remember them. And over here, of course we have Black Friday. So what’s this big peak here? It’s not a bushfire. It refers to the Victorian government’s mass sackings of senior civil servants and judges in 1878. Obviously it was an extremely important event at the time, an extremely important event in government in Victoria, but it doesn’t quite figure in our collective memory in the same way as Black Friday does.

One of my early experiments with QueryPic was to look at the question of when did the Great War become the First World War? At what point did we stop thinking about the Great War as the war to end all wars and realize that it was one in a series of global conflicts? And the graph really does a nice job of confirming our expectations, I suppose, in that we see a nice crossover late in 1941, which if we were thinking about the passage of the war would be about when we probably would expect. But what’s missing from this, what’s missing of course is just the war.

Just as with the Great Depression and with the Black Wednesday, what’s hard is trying to recapture the moment as it was happening, the sense for want of a better word, of present-ness. Now, if we go back to ‘We forget’ what are we doing when we’re talking about one of these dates? I mean, if we think of the present as providing a line there - past, present, future - on the past side, what are we doing? We’re anticipating. We’re predicting. Perhaps, we’re dreading. And present, we’re experiencing, we’re enjoying, maybe suffering. In the future, we are remembering, we are regretting, perhaps reflecting. So instead of lumping all these together, it seems to me that we should be teasing them out and exploring their different interconnections.

We should be trying to give the past back its own sense of the present. And this in essence was the modest and thoroughly achievable goal of my Harold White Fellowship. I wanted to explore the possibilities of the digitized newspaper collection in supporting this sort of rich temporal contextualization using digital methods to recover the pasts, the presents and the futures of any moment in our history. I have to admit, I haven’t got very far yet, and Marie-Louise has been doing a good job of reassuring me that sometimes the fruits of these things take a while to develop. Now, there are a number of reasons why I haven’t gotten as far as I wanted, but I do have a few sort of sketches that I want to share with you.

III. The future of the past

Okay. What I decided to do is to try and create a sort of manageable sample set. So I decided to work with articles, which included the phrase, ‘the future’, in the heading or the first four lines of the newspaper articles, and that’s one of the facets you can use within Trove. Why did I limit it in this way? Well, I’ve been doing a lot of different work in Trove as Marie-Louise said. One project I’ve been working on was looking at ways of finding editorials within Trove and exploring the content of editorials over time. And in doing that, I discovered a number of frustrating things, one of which is sometimes the articles aren’t divided up as nicely as we want them to be. Particularly with editorials, editorials on different subjects are often joined together, so it’s difficult to separate out the specific ones that you want, but I thought by limiting my search in this way, that it increases my chance of relevance. And it also brought the number of matches down to what I thought was a reasonably manageable 60,000 or so.

So I started harvesting those 60,000 articles. I have over time been developing a number of tools working with Trove. One of which is a harvester, which enables you to get the data in bulk. And of course that’s necessary if you’re going to do this sort of large-scale analysis on it. I modified my existing harvesting tools to save the results directly into the database and when the API became available, I modified it to use the API, which makes a lot of things easier. Now, after about 40,000, I thought I probably had enough, and I decided I’d trust in Trove’s relevance ranking and just work with that set.

And then it was time to do some cleaning. Now, Trove’s crowdsourced OCR correction project has been a wonderful success, of course, but it’s worth noting that with the sample of articles that I harvested for this project, only 2% had any corrections at all. So 98% totally uncorrected, totally untouched. While I couldn’t hope to correct all of those articles myself, I could at least try to reduce some of the noise, which is created by these sorts of OCR errors. So I developed a series of scripts which would try to clean up some of that OCR output. First of all, it corrected some errors which are fairly consistent and hopefully fairly unambiguous OCR errors. And you can test yourself here. What’s that meant to be?

Audience: His.

Pardon?

Audience: His.

Nope. The.

What about this one? Nah, the. This one? No, of. Should get that one. Ah, yep. And you can check. There we go. Look of, ah, yep. So there are a series of these which I could through a script just fix up. I then checked each word in the text against a series of dictionaries and word lists, and this included a word list of places which I actually extracted from the places Gazetteer provided by Geoscience Australia. Anything which didn’t seem to match up, I marked up in a way that I could extract it later if I wanted to. And all of this, you’ve got to understand, went through a lot of trial and error, just trying stuff out, seeing what it produced, trying it again, fiddling with it lots and lots and lots of trial and error.

But after that, I could do some fun things. You’re all of course familiar with word clouds, but I bet you haven’t seen a non-word cloud. This is my non-word cloud.

Now, of course, the big question is what is ‘others’ doing there? I don’t know. For some reason my word list didn’t like the word others, but of course you can see here some more sort of consistent OCR errors. There’s another ‘the’ and another ‘the’, and that would be a ‘be’, in most cases. And we also see where words have been split up. We’ve got a ‘tralia’ down there. Oh, that’s a ‘which’ obviously. So it’s actually quite useful visualizing in this way because I can then feed that back into my process of cleaning. I can see where the common errors are, and I can start to feed that back into the process.

For each article that I processed in this way, I generated an accuracy score, which was simply the number of recognized words divided by the total number of words within the article. And I could use these scores to develop a couple of overviews.

So of my set, this is only just my sample of course, so this is OCR accuracy over time. There aren’t many articles in this earlier period, so it’s probably not worth worrying about too much. But what’s interesting is this decline here down to the 1920s where we’re going below the 80% mark. Why is that? I’ve got no idea. There are a whole lot of variables which could be certainly involved here, whether it’s the fonts, whether it’s the quality of the printing, the quality of the paper, the quality of the microfilming. I don’t know. It’s something which would be interesting to explore further and to and investigate.

We can also have a look at the poorest performing newspapers. So the Perth Gazette and West Australian Times didn’t do too well, and it got 58% in my scorecard. Again, this is only a select sample, so I’m not quite sure what you can read into any of this, but it’s sort of interesting. These figures weren’t particularly important for my work, but I do think that the general issue of OCR quality is really vitally important, particularly as we make more and more scholarly use of these sorts of collections in bulk. I mean, obviously we need to improve the quality, but we also need to expose our assumptions about the OCR quality that underlie our work so that when we are putting forward something, some sort of analysis of the text, we’ve got a way of communicating the quality of the material that we’re working with.

I then decided to make my sample set even more manageable by selecting just the first 10,000 articles, which had accuracy figures of over 80%. So I used my scores and I went through and I decided just to choose those ones which seemed to go pretty well. Of course, as any good digital humanities person does, I then started counting words.

As with most of this stuff that I’m showing you tonight, it’s online and you can go and play with it and use it yourself. So this shows the word frequencies over time, and there’s a time slider here, and you can just drag it along and see what’s happening in different years. Now, nothing really significant jumps out at me from looking at the word frequency clouds here. I mean, what is sort of interesting I suppose, is the preponderance of ‘would’ and ‘could’, which I suppose confirms the future orientation of the sample set that I’m working with. And there may well be other things within there that jump out at you. And so as I say, jump online and have a look and have a play with this and see what you can make of it.

I mean, word frequencies… Okay. So word frequencies can be interesting in getting a sort of overall picture of a large amount of text and starting to track some changes over time. But this sort of word frequency tells you what’s common. It doesn’t tell you what’s distinctive. It doesn’t tell you what’s interesting in an article. And another measure we can use to try and get at the distinctiveness of a piece of text is something called TF-IDF. It’s an acronym, term frequency, inverse document frequency. And what it does, it looks not just at the frequency of a word within a particular piece of text, it also looks at the frequency of that term across a collection of texts. So a word that is common in a particular article, but not very common in a collection of articles will appear as more significant, more heavily weighted in its TF-IDF value.

You use TF-IDF values all the time. They’re used by search engines in calculations of similarity. They can take the TF-IDF values, convert it into a sort of mathematical format and use it to calculate the similarity between two pieces of text. And the results of calculating TF-IDF values for collections like this are pretty interesting, and I’ll just show you a little comparison.

So on the left-hand side here, we have, this is 1939. The top 10 words on the left-hand side are just the plain frequency values, and here are the TF-IDF values. So you see we’re getting at something quite different and something quite interesting here. Obviously 1939, Hitler doesn’t figure in this list of terms. He’s at the top in this one. But we also get these really odd things like midget and roundabout. I found it really interesting producing these values, and I found them quite sort of evocative and interesting and wanted encouraging me to explore more, and I’m going to talk some more about this a bit later.

But I finally just wanted to show you one other way of understanding a collection of texts, and that’s through a thing called topic modeling. There’s a lot of topic modeling going on in the digital humanities at the moment, and there are a number of good blog posts, which I’ll put links to from here, which tell you about what topic modeling is. I’m just going to quickly race through it. Basically, I use a piece of software called Mallet. I pointed Mallet at my collection of texts, told that I wanted to define 10 topics, that is 10 clusters of articles within those texts, and it just did it.

And what it came back with is these lists of words which are grouped according to the topics which it believes existed. You can then go through and look at these lists of words and start to interpret them to try and understand what those topics are. And most of them are pretty clear. This of course, is the topic that tells me that I still didn’t clean up the OCR enough, but it’s interesting that it brought them all together.

We’ve got, I mean here we’ve got trade, here we’ve got technology, here we’ve got land/rural, here we’ve got international relations, here we’ve got government, and here we’ve got home and society. And it’s amazing once you run these things, how much sense the topics actually make to you.

And it also goes through and for each article in your collection, it weights them according to these topics. So you can then go and for each article you can see which is the most heavily weighted topic for that article, and you can calculate the number associated with each topic and you can do something like that. Okay, that’s not terribly instructive as it is, but you can, I won’t show you now, but if you click on that and go to the live version and you click on the legend down the bottom there, you can actually take away some of the lines so you can actually see what’s happening underneath, so you can just see the lines that you’re interested in.

But basically, I want to do a lot more work on these topics at this stage, and I haven’t really done a lot of interpretation of them. I want to see how I’m actually using those weightings and find better ways of actually looking at them. So anyway, here I am. Not a lot of interpretation at this stage. No great insights. I have some data. I have a data set, and I’m going to be continuing to play with it. And as I’ve said, all this stuff is available online, so you’re welcome to come and play with it too and see what you can make of it. Now, you may think that I’ve gone into a lot of tedious detail about what I did. Well, I’ve actually saved you from a lot of the gory details.

IV. Meanings for mining

The truth of much research in the digital humanities is that large amounts of time are spent yak shaving, and data munging. If you don’t know the term ‘yak shaving’, it’s that process that we’re all familiar with, when you start doing a particular task and you realize, in order to achieve that task, you have to actually do something else or research something else, and that actually continues into infinite regression until you find yourself doing something which seems totally unrelated to the task that you started with. I’ve had a lot of that recently. There were lots of issues just involved in using this data and starting to manipulate it. As I’ve said before, the issue of the OCR quality is crucial, and we have to be upfront about the problems and continue to look for the most effective solutions. We have to talk about questions of selection and completeness. What’s actually in Trove? How does it change and how does this sort of influence the results that we get?

One of my examples here is a thing called the Atomic Age Exhibition, which toured around Australia in 1948-49. It was a big thing. Many, many thousands of people visited. It was at the Easter Show in Sydney. If you search in Trove for Atomic Age Exhibition, you’ll find quite a lot of results coming from the Courier Mail in Brisbane. You’ll find virtually nothing from Sydney and Melbourne, and you might be inclined to think that the exhibition didn’t actually go to Sydney and Melbourne. Why is there nothing in Sydney and Melbourne? Because the exhibition was sponsored by the Herald in Melbourne and by the Daily Telegraph in Sydney, and both of those titles are currently not in the newspaper database.

So we’ve got to bring these sorts of questions and perspectives as we start to do this research. Another barrier, which I started to butt my head up against in doing this was that of computing power. Generating the TF-IDF values for my sample took about a day and a half on my laptop. And of course, then you realize, you did something stupid and you have to do the whole thing again. And I did wonder at various times whether I was reaching the limits of what’s practically possible for one bloke and his laptop and wondering whether my sort of piecemeal efforts will be blown away by academic teams with access to research funds, bright young graduate students, and time on a supercomputer.

Now, this list of problems and concerns might seem a bit depressing, and it might not be what you expected from this talk, but I want to reassure you, there are digital tools that make it easy to get started and start exploring the possibilities. QueryPic of course, there are other things like Voyant, which is a great online tool for starting to do text analysis, but sooner or later you’re going to have to confront some pretty hefty questions. But hey, that’s just history. The past is messy and it raises difficult questions about things like selection and interpretation. The issues aren’t necessarily new, it’s just that they’re raised in a bigger, more technically challenging context.

But what does really bug me is a nagging feeling that I should be taking statistics more seriously. That in constructing the sort of examples which I’ve been showing and the tools that I’ve been demonstrating, that I should actually be less impressionistic and more rigorous, as if I’m sort of not doing justice to the vast computing power that I have at my disposal. But I don’t want to do that. In January, I was at the American Historical Association meeting and I was actually able to see the culturomics guys live in person doing their spiel. And as they described their vision for a new science based on access to these huge cultural data sets, I tweeted, “Yeah, I want to use big data to tell better, more compelling, more human stories.”

The British historian Tim Hitchcock has similarly described his own unease that the demands of big data seem to be moving him towards a more positivist style of history. In the humanities, we’ve been really fortunate to make use of many decades of research into things like information retrieval. We’ve adopted many of their concepts, their tools, and their formulae, but we’ve also adopted some of their language. And so we talk about what we’re doing as mining. Mining is an extractive process. We dig stuff up, we pull it out of the ground. But this seems to be pretty much the opposite of what I want to do. I mean, I do want to find structures and separate them out for different types of analysis, but then again, I want to put them back together. I want to observe them in different contexts as rich and as complex as possible. How do we do that?

Well, first of all, we have to work out better ways of incorporating these sorts of big data perspectives into the narratives that we write. Just as QueryPic gives you that opportunity to sort of zoom out and get a big picture, I think we have to take control of the zoom and use it to our advantage. And this, by the way, probably means developing new forms of publication that allow easier and better integration of data and text. It’s challenging, but there’s not much point to dwelling on the dangers and problems of big data, and as Tim Hitchcock concludes, we simply need to get on with it.

V. Screwmeneutics and deformance

The second approach is to foster acts of creative subversion, to use digital tools in new ways. Literary scholars within the digital humanities talk about the possibilities of deformance, of using computational methods to change texts in ways that can open them up to new and new critical perspectives. Stephen Ramsey also talks about moving beyond traditional forms of search and browse and admitting ‘screwing around’ as a legitimate research methodology. Of course, historians don’t want to start deforming their sources or do they?

This is an experiment I created called The Real Face of White Australia. I always get a bit teary when I put this up. Well, what I’ve done here is actually use computer vision software to extract portrait photographs from certificates, which were used in the administration of the White Australia policy. These are records held by the National Archives of Australia. There’s several thousands of these, and this is just from one series, and you can just keep scrolling and scrolling forever, or almost forever. So by manipulating the sources in these ways, by extracting those photographs, I’ve created a new way of seeing these records and it’s quite powerful.

But we can also be playful. You may have seen this. This is a little game that I created using the newspaper database. Again, it’s very simple. It just picks a newspaper article at random from the database and asks you to try and guess the year in which it was published. So any guesses for this one? What would we say? Let’s say 1850… That’s a bit later than that… Let’s see, it’s earlier. Okay, so you can keep going like this. You can go and try it out yourself later. As I said, it’s very simple, but it’s also strangely addictive. And of course, it’s also a way of exploring the content of Trove by screwing around.

QueryPic, the Real Face of White Australia and newspaper roulette, my Headline Roulette, also have something else in common. They are public. I want people to use them. I want people to have fun. I want people to be moved. I want people to find things to be surprised and to do history.

Just yesterday I received an email from a self-confessed Australian history addict, oh no, Australian history fanatic, sorry. And she had become addicted to Headline Roulette. She wanted to know if I could add a facility for users to actually save their scores. So presumably they could go back and see if they’d improved or share them with their friends. So obviously the next step is the Facebook application. I mean, other people have described to me how scrolling through the Real Face of White Australia brought them to tears. And I’ve come to realize that these sorts of interactions really mean more to me than a footnote in an academic article. I’ve probably just killed my hopes of an academic career there. I want to use digital tools, I do want to use them to deform history. I want to use it to deform history in a way that makes it accessible to new audiences in new ways. And so I present to you in honor of my Howard White Fellowship, a new experiment.

Now, I described to you before the process involved in calculating the TF-IDF values. What I didn’t describe was the fun that I had when I was doing it. It was really quite exciting and amusing and funny and all sorts of things, watching the words fly past on the screen. And as I completed each year, I would had a little script which would show me the top 20 words for that year. And anybody who follows me on Twitter, we all have a good picture of what was going on there because I couldn’t help but share this. So I mean, it tells their own story. I mean, it was really like a sort of wonderful puzzle as I say there, as they all came up. And then I started tweeting some of them.

This was a nice one. I like the ‘hitler’ with ‘mudguards’, ‘duchess’, ‘opossum’, ‘hollywood’ and ‘canberra’.

And as I said here, of course, I mean there’s got to be a novel in ‘prince’, ‘pronunciation’, ‘keyboard’, ‘zulu’, ‘begged’, ‘unbent’, ‘diddle’, ‘candlesticks’, ‘virtuoso’, ‘highness’ and ‘pots’. This started me thinking, was there a way I could share this experience and use the TF-IDF values as a way of exploring my data set, a way of opening this experience to others, as creating a sort of shifting playful window on the future of the past. So this is my first attempt. Again, public, go play with it.

I’ve actually deliberately tried to keep most of the metadata away from this interface because I wanted the words to be the focus. And yeah, it does look a bit like that fridge poetry thing that you can get, and that’s quite deliberate. I mean, actually at some stage, I want to add a box down here where you can drag your words down here and make your own sort of collections and tweet them. What it’s showing you there is just a random selection of TF-IDF values from my sample, and you can click on any one of these, and it goes away and it sees, first of all, how many years have that value attached to it. If there’s only one year, then it’ll actually return that year. If there is, let’s see if we can find one that has more than one year. If it appears in more than one year, then it pulls out a random selection of those values from those years.

Okay. Oh no, we’ve got 1943. I’m not doing a good job of it this time. Anyway, you can have fun with it. And of course, if you want to actually see what’s going on, you can click on these and it will actually load the articles here, and you can explore the text of them there and see where the word’s popping up.

Okay, what is this? I’m not quite sure. It’s not really a discovery interface, although you can find interesting stuff. It’s not quite a game, but it is quite fun to explore. To me, I really, I’m sort of in love with it at the moment because it actually, I mean, you think about what I’m trying to do in terms of recapture the presentness of the past. Our experience is not about just the big stories of the day. Our experience of any moment includes a whole lot of trivial aspects. And I love the way that this sort of brings together Churchill and Corpuscle and Melvin, whoever Melvin is. I love the mix of words, and it just, to me is incredibly evocative. It makes you want to start imagining stories. It makes you want to explore, it makes you want to find out more, but it just has a wonderful, exciting aspect to it itself. So I’m not quite sure what I’m going to do with it or how I’m going to develop it, but really, as I say, I’m quite in love with it at the moment, and I hope you’ll have a play with it and see what you make of it.

Could it be a discovery interface? I don’t know. It does enable you to get into my dataset, but of course, it’s obviously from a rather indirect means and it includes lots of randomness as well. And I’m a big fan of randomness in actually developing new ways of discovery. So there you go. Please take it away, enjoy, play. I may not have conquered the meaning of time yet, but experiments like this actually make me think about the form in which I actually present those sorts of arguments and those sorts of ideas. How do we actually create resources which give that sort of sense of the disjunctions and the serendipity? So while I may not have achieved all I wanted to, I’ve come away with a better sense of what it is that I’m trying to do and what I want to do with this material. So thank you.

Questions

Marie-Louise Ayres:

Thanks very much, Tim. Just before we open up for questions, there was just a few things that I thought I wanted to say. One is, look, two Australians have corrected more than a million lines of text each. So if you think you couldn’t correct 40,000 editorials, you are not being ambitious enough. That’s the first thing. The second thing to say is that our own Trove team have found that the only surname that is not in Trove you could do things that are not is Kardashian. And the third is, I guess just thinking about how amazingly creative these visualizations are that Tim has been doing, and I hope you’ll ask him about them.

But the fourth thing I wanted to say is to pick you up on one of your early comments where you said, “I haven’t got as far as I wanted.” Now that’s a very interesting construction that includes the past, the present, the future, and a spatial term as well. So maybe you need to think about that. I think that we’d agree with the work that Tim’s has done. I don’t know where he wanted to be, but he’s gotten a long, long way and done things that the rest of us just probably haven’t even contemplated. So I’m hoping you’ll ask Tim some questions now and then we’ll have more opportunities afterwards. So it’s dark out there. So if you want to ask a question, can you just make sure you raise your hand and speak up? So don’t be shy. Yes.

Q&A 1:

How appropriate would your methodology be for spelling? Where I’m coming from is I know the Australian Labor Party and the British Labour Party spell different. And I can remember once going through microfilms that the Sydney Morning Herald in 1920s, and it dawned on me all those spellings were American. And so there must be things happening where you could compare how words are spelled or do all the correctors corrupt your data just by correcting.

Tim Sherratt:

I certainly think you could do that sort of analysis. And actually some of the nice examples that the culturomics guys used in the Google Books example was looking at, oh God, the name is… what irregular verbs, looking at changes in irregular verbs over time, which is quite interesting. But their data set goes back to quite a long way. And there are challenges. One of the challenges in working with Trove, I mean obviously the interface is geared towards discovery at the moment, making sure that people find what they’re after. But that means that sometimes if you want to find something exact, it can be a bit tricky. You’ve got to know sort of how to turn off the fuzziness in the searching. And sometimes you are foiled in that by the fact that people might’ve tagged something and the search by default also searches the tags and the comments.

So when I did my first, I don’t know whether you saw it in my World War I graph, you may have noted that there was a peak, little peak, for World War I actually during the First World War, which is sort of interesting if you think about it. And that’s because people had tagged those articles with World War I. So again, this is all about, and one thing which I would always emphasize as we start to do this research, we have to develop our literacy in terms of understanding search interfaces and how they work, and be prepared to go into the documentation and to look at the advanced searches and how they work and actually start experimenting a bit with what different searches bring back, so that you can actually have a good picture. And I think mean, obviously the institutions themselves have a role in communicating this and making it, exposing what’s going on behind the scenes. But I think it’s an important literacy for researchers going on into the future, being able to pull these things apart to understand exactly what’s going on. So you can do those sorts of quite detailed fine-grained comparisons.

Q&A 2:

I’m interested in what year was it the picture of White Australia and was that the people that were actually accepted or what?

Tim Sherratt:

Certainly not. Well, okay. I should say that this is part of a broader project called Invisible Australians. If you just go to InvisibleAustralians.org, there’s a lot more information about what we’re trying to do with these records. That particular set of records, as I think I said, those photographs were just pulled from one series within the National Archives of Australia. And there are many more series like that. They are a series of certificates. Basically, if a person deemed non-white was living in Australia, they wanted to travel overseas and get back into the country again…

Q&A 2:

Those people that lived in Australia rather than people that tried.

Tim Sherratt:

No, because… Yeah, there was no trying to it. So yes, it is people who are living in Australia. And this is what is particularly interesting about these records because what we want to do, and this isn’t Trove related, I’m sorry, but what we want to do with those records is to actually try and extract the biographical information which is contained within those certificates in order to sort of find out more about the community who was living under the White Australia policy, people who were living here, whose various activities were restricted in a number of ways by the White Australia policy in all its sort of legislative forms. And we’re bringing to bear a number of digital techniques to try and do that. As I said, with that particular case, it was a facial recognition script which pulled out as photographs, but we’re also harvesting material from the National Archives and doing some topic modeling. I’ll be doing some topic modeling as I showed there on some of the records to try and pull out clusters within those records. So anyway, check it out. It’s something I’m very passionate about.

Q&A 3:

I’m just wondering, it seems like the [inaudible 00:57:20] is a bit of issue as you talked about. And I guess there’s probably a couple of aspects to that. One is like a computer vision, technology is involved and the other part is what do you do after you’ve got that? Is there anything clever that you can do to guarantee IDF or whatever else to try and make better quality? I don’t know if you can describe how do you see the next few years panning out with that? Do you think that there’s a lot of improvement going on?

Tim Sherratt:

There’s certainly a lot of work going on and there are techniques which could be used. There are more developing specific language models that tell you that if you have a certain combination of letters, then after that combination you can expect a certain range of letters but not other letters, for example. And you can use those in sort of probabilistic techniques to sort of go across and see what’s likely to be at particular points. And specifically relating to digitized newspapers, there’s stuff going on. There’s a project in the US called Mapping Texts, which actually works with Chronicling America, the digitized newspapers from America. They actually went through and just actually very similar to what I did, but with a bigger budget and access to Stanford’s resources, did things like the topic modeling and word frequencies. It also did what’s called named entity recognition, which is pulling out people and places from the texts, but they also ran through their sample set and generated figures for OCR accuracy. And they’re fairly similar to my figures actually.

So there’s a lot of obviously recognition of this in Europe. There’s actually a particular research group which has been looking at methods of improving OCR. So I mean, and of course there are many more cases which are much more complex than this if you think about old Germanic scripts or something like that. And so there’s a lot of interest, a lot of concern, and a lot of work going on, I think, to see. And so I think it’s something that we have to be prepared to revisit over time that there are going to be more possibilities for doing stuff with computers as this comes online. And so we should need to assess constantly reassess what’s actually possible and see what we can do. But yeah, I, think, given the general awareness of the problem and the problems that it causes, I mean I think there’s certainly going to be a lot. And I think it’s really exciting the fact that we have now starting to get these collections all around the world of digitized newspapers and the possibilities that opens up for doing comparative stuff. What I didn’t mention with QueryPic is that you can also access New Zealand newspapers. It uses the DigitalNZ API and accesses papers past, so you can actually do graphs for New Zealand papers, but what you can’t do meaningfully is compare Australian and New Zealand results. And that’s because DigitalNZ, our research currently searches the titles of articles and not the full text. Wouldn’t it be really nice if we’re both searching the same things and we could do those sorts of comparisons and we could do it with the US and we could do it with Canada. I think there are some really interesting possibilities there.

Q&A 4:

I was just going to say ‘stine’ is very interesting there because in my opinion, it’s obviously a column break from Palestine. And so that’s a common sort OCR error, E and S being a sort of fragile combination. And not only that, but the rules of breaking, they tend to do chunks like that. And also it shows how the TF-IDF is working. Palestine itself as a whole doesn’t appear there because it’s not actually that important, but ‘stine’ got promoted because it’s extremely uncommon and ‘Pale’ got dropped off. And of course ‘Pale’ would be there… because it’s a very common word.

Tim Sherratt:

Yeah, that’s nice. But yes, I mean as you go through this, you will see other instances where the sort of OCR issue is coming up again. But that’s also another nice example of thinking about how using computational techniques we can start to improve some of the OCR because you are looking at the way words break and seeing if we can use that in some way. Thanks, that’s great.

Q&A 5:

As just before, you said the word evocative, Tim, I was saying to myself, evocative, and so I wanted to talk about that for a minute rather than talk about a technical thing. It seems to me this is a really interesting, I don’t know, I just want you to say more about is this a different kind of historical mode, this kind of desire to treat the past to evoke rather than to necessarily narratize or analyze, or define or pin down. What is it about, is there something distinctive about this evocative mode, which is to do with the digital techniques or, yeah, going on?

Tim Sherratt:

Yeah, I mean this is the thing I’ve actually just been trying to grapple with over the last few weeks as I started playing with this stuff. And I don’t know exactly what it is. All I know is what I feel as you do when you see it. And they do make you start to think in different ways and to imagine and make connections. I mean, I think with your work, with Cath’s work on Semble, there are possibilities for creating spaces which encourage people to make connections, to see relationships between things. And I think digital technologies do lend themselves to that because, I don’t know, as I said, I actually think randomness is something which is rather undervalued in terms of exploration and discovery. And as you know, there’s another project that we worked on called The History Wall at the National Museum of Australia, and that brought together material in quite a random fashion. And it was actually, again, quite evocative in terms of it being able to see the sort of possible relations between items there.

As I said, I don’t know what it lends itself to. What is the process? Is it discovery? Is it a prompt for research questions? I don’t know. But it just seems to me to be something which is worth exploring more and I find that it’s actually just something which I keep doing, so I must be interested in it in some way. And yeah, I mean obviously, I mean it’s something which would definitely be worth thinking about some more. I mean there are all sorts of ways in which you can develop a sort of evocative sense. I mean obviously historical photographs and they give you a different sort of feeling from seeing a text description. So yeah, I don’t know. Really interesting question.

Q&A 6:

Tim, I’d just like to congratulate you on your work. I just find this really interesting and I think it’s really great that researchers like yourself take our data and play around with it, as you said, because ultimately some of those ideas do lead to really useful actual applications. And I just wanted to say your OCR accuracy results are actually bang on because we did quite a bit of research on that five years ago before we launched, and it was 65% to 70%, which is of course low, which was why we said how can we change that and get the public to help?

But you’re quite right, as time moves on and as these big data sets are made more open and available, people develop technologies to improve that. So five years ago, that didn’t exist, an automated way to improve it. We now know of at least three other people like yourself have figured out how globally they could really increase that OCR accuracy rate and prove it. So I guess that’s questions that I would have, how some of this really fantastic research and ones you mentioned in Europe can be built back in to improve on services. But I just wanted to say, did you find it really useful having API being able to get that data? Because I know that was a dream we’d had for a long time and I know you waited a long time to go.

Tim Sherratt:

Well, I didn’t wait. I didn’t wait, did I? I actually just went ahead and did it myself.

Yeah, look, I mean background for those who didn’t know is that I built my own unofficial API at one point, which I used to do some experiments. But yeah, I mean it obviously makes a whole lot of things easier. First of all, the point of view of doing the large data dumps, getting that sort of material, and you’re not downloading the whole web page and all that stuff on it, but you’re just actually getting the data in a structured way. Great. And as anybody who also follows my work will know is that I had a number of frustrating experiences where things change on the web page and everything I created broke and I had to fix it.

So APIs do away with all that. It’s fantastic. But one of the really good things I like about having API access is how easy it makes it to do something like Headline Roulette. If you have an idea and you’ve got a bit of coding experience, you can act on it and you actually build something. And that to me is the most exciting aspect and encouraging people to actually experiment. That’s what it’s all about to me, is creating an environment where people do experiment with this stuff and build things.

A brief and biased history of Trove Twitter bots

Thu, 19 Jun 2025 12:08:35 +1100

The socials recently alerted me to an interesting article by Dominique Carlon, Jean Burgess, and Kateryna Kasianenko on the history of community-created Twitter bots. The article explores bot-making within the context of Twitter’s rise and fall, and provides a handy taxonomy of bot species. However, it doesn’t include any Australian bots amidst the examples. That’s a bit disappointing, as I remember the bot-building years as a time of great fun and creativity. My own contribution to the world of Twitter bots was mainly focused on Trove (what a surprise!), so I thought I might as well jot down a few incomplete and biased notes about the history of Trove Twitter bots.

Trove tweeting trends

It just so happens that I recently packaged up some data about Trove links shared on Twitter. Using this data, we can get a broad perspective on the activity of Trove Twitter bots between 2009 and 2020. The identification of bots is based on the Trove bots list I maintained on Twitter, so it’s possible I’ve missed some.

In total, 43 bots posted 318,767 tweets containing 270,474 unique Trove urls between June 2013 and December 2020.

Here’s a chart showing the total number of links to Trove shared by Twitter bots each year.

Most of the bots shared digitised newspaper articles, but some shared works from other Trove zones. This chart breaks the links down by the type of resource (‘article’ equals newspaper article).

And one final chart showing the number of active bots per year.

year	active bots
2013	4
2014	4
2015	6
2016	8
2017	12
2018	34
2019	38
2020	22

From the data above you can see that bot activity grew slowly between 2013 and 2017, before taking off dramatically in 2018. The peak year for Trove bots was 2019, when 38 individual bots shared more than 100,000 links to Trove. But a mass extinction event in 2020 almost halved the number of active bots. So what happened?

Build-a-bot begins

In June 2013, inspired by bot creators like Mark Sample, I hooked the Trove API up to Twitter to see what would happen when GLAM collections joined online social spaces. The result was @TroveNewsBot, sharing digitised newspaper articles from Trove.

Twitter bots started popping up around the world, sharing collection items from Europeana, the Digital Public Library of America, DigitalNZ, the Cooper Hewitt Museum, and the Brooklyn Museum, amongst others. But @TroveNewsBot was always a bit different. Instead of just sharing randomly selected resources, @TroveNewsBot helped people explore Trove without leaving Twitter. If you tweeted keywords at the bot, it would run a search using the API and tweet back the most relevant result. By adding hashtags, users could control a variety of search parameters – for example, if you included the hashtag #luckydip you’d get back a random article from your search results.

My favourite bot behaviour was its ‘opinionator’ mode. If you tweeted a url at @TroveNewsBot, it would retrieve the link, extract keywords from the text, and then search for those keywords in Trove’s newspapers. This enabled @TroveNewsBot to have conversations with other online resources – for example, it replied to tweets from DPLA and DigitalNZ, finding connections between different collections. I also used the ‘opinionator’ mode to set up a dialogue between past and present. Several times a day, the bot would grab keywords from the latest news items on the ABC (later the Guardian) website, search for historic newspaper articles, and then tweet both stories, old and new.

@TroveNewsBot’s opinionator mode in action – a slide from my keynote presentation ‘Life on the outside: connections, contexts, and the wild, wild web’ for the Annual Conference of the Japanese Association of Digital Humanities in 2014

As well as providing digitised content, such as the newspapers, Trove aggregates collection metadata from hundreds of organisations around Australia and makes it available through its own API. This meant that any organisation could use the Trove API to create a Twitter bot that shared items from their own collection. To encourage more of this sort of experimentation, I created the Build-a-Bot Workshop GitHub repository. This repository included instructions and code for anyone wanting to build their own collection bot on top of the Trove API. Like @TroveNewsBot, these collection bots could share random items and respond to user queries.

Before long, @CurtinLibBot was sharing photos from the Curtin University Library’s image collection, and @Kasparbot was tweeting about objects from the National Museum of Australia. By the end of 2013, I’d added to the family by creating @TroveBot. While @TroveNewsBot dug into the digitised newspaper articles, its younger sibling looked for inspiration amongst Trove’s other zones – sharing books, journals, photos, maps and more.

In 2015, Steve Leahy unleashed @TrovePenguinBot upon the world, searching for sardines amongst the digitised newspapers. In 2016, one of my students at the University of Canberra modified the NYPL Emoji Bot code to create @TroveEmojiBot – if you tweeted an emoji at the bot, it would respond with a suitably-themed newspaper article. In 2017, the Digitise the Dawn campaign bot-ified their Twitter account, posting an article each day from Louisa Lawson’s journal, The Dawn. Meanwhile, @astrove_bot started sharing newspaper articles relating to astronomy.

And then there was The Vintage Face Depot…

Things get weird

I’d been experimenting for a few years with faces as a way of connecting to GLAM collections –as alternative entry points, based not on metadata but the people inside. In 2015, this led me to create The Vintage Face Depot. If you tweeted a photo of yourself to @facedepot, the bot would select a face at random from a collection I’d compiled from Trove newspapers and superimpose that face over yours, tweeting you back the result and a link to the original article, so you could find out more about the person you’d been matched with.

@facedepot in action – a slide from my keynote presentation ‘Unremembering the forgotten’ for the Alliance of Digital Humanities Organizations Annual Conference in 2015

Now, in a time of deep fakes and AI generated images, @facedepot’s efforts seem quaint and kludgy. But that was always the point. I wanted to mess around with the barriers that put some people on the other side of this wall we call the past – to explore what historian Devon Elliot suggested on Twitter was an ‘uncanny temporal valley’. As I argued in The Perfect Face, a presentation at NDF2015:

The Vintage Face Depot tells you nothing about yourself. I built it at about the same time as Microsoft launched their How-Old bot that uses machine learning to estimate your age. Face Depot does nothing clever, and yet sometimes the results are uncanny, even unsettling. Microsoft might be able to tell you how old you are, but Face Depot asks who you are and pushes you in the direction of a past life, linked merely through chance.

Glitch bots for all

While I’d shared some bot-building code, rolling your own bot still required access to a web-connected server – a significant barrier for most would-be experimenters. This changed in 2017 with the arrival of Glitch, a platform that enabled anyone to build simple web apps for free. Perhaps most importantly, Glitch apps were remixable – simply by clicking a button, you could open an editor and create your own customised version of any app.

Glitch seemed like an ideal environment in which to experiment with bots, so I created four remixable Trove Twitter bot recipes:

trove-collection-bot – sharing resources from a partner collection
trove-list-bot – sharing items from a Trove list
trove-title-bot – sharing articles from specific newspapers
trove-tag-bot – sharing items with specific tags

These were supported by a detailed tutorial that walked through the process of customisation and suggested ways in which the basic recipes could be extended – for example, by adding a specific search query to a title bot.

This was the beginning of the bot explosion, with more than 30 Trove Twitter bots born between 2017 and 2019.

One of these, @NTTimesGazette, was created by curator and journalist Caddie Brain to tweet articles from the Northern Territory Times and Gazette. The bot was featured on ABC radio in Darwin under the headline: Twitter bot offers a rare look inside the Darwin’s forgotten first newspaper.

Historian Brett Holman created a series of bots related to aviation history. More than just a source of amusement, the bots became part of Brett’s research practice, as described in his History Australia article '@TroveAirRaidBot, a 24/7/365 research assistant'.

Perhaps the best part of this bot-making extravaganza was the number of self-professed ‘non coders’ who were able to take their first steps into the world of programming and actually create something. I have memories of sitting in the shade at Canberra’s now defunct Big Splash water park, troubleshooting someone’s Twitter bot on my phone, while the kids played on the water slides – it was fun, and it was exciting. Together, Trove, Twitter, and Glitch opened up new possibilities for learning and experimentation, and new ways of knowing Australia’s cultural heritage.

2019 bot roll call

As new bots emerged, I added them to my Trove bots Twitter list (here’s a partially archived copy).

You can get an idea of their diversity from the bot names – a mix of collections, subjects, and places. Here’s a list of Trove Twitter bots active in 2019:

astrove_bot
AustWWBot
BotCBR_QLD
CatsofTrove
digitisethedawn
DoSonTrove
facedepot
Kasparbot
KellyGangBot
LAAL_bot
NTTimesGazette
PenrithPictures
RemixHistorical
suthlib
TroveAirBot
TroveBot
TrovecakeBot
TroveCHIAbot
TroveDutchbot
TroveEmojiBot
trovefacesbot
TroveHoroscopes
Troveknitbot
Trovelandbot
trovelistbot
TroveMirrorBot
TroveNewsBot
TrovePenguinBot
TroveRefereeBot
trovesportsmel
trovetribunebot
TroveXmasBot
TsvBulletinBot
WomenAtWarBot

I also overhauled @TroveNewsBot in 2019, adding a number of new features, including article thumbnails.

Decline and fall

This golden age of bot-making came to an end late in 2019.

The first blow came when Trove updated its API. The bots needed some way of selecting random items from the millions available on Trove. This was fairly easy with version one of the API, but version two overhauled the way you accessed items within the result set, making random selections impossible. I eventually managed to hack together a random-ish method that added multiple facets to whittle down the results set until a selection could be made. Using this method, I created new versions of my Glitch bot recipes and updated the tutorial. But it seemed that the moment had passed, and many bot authors just let their creations die when version one of the API was switched off.

Surviving bots faced further challenges when Glitch started imposing limits on its free services. Glitch apps were designed to sleep when not in use, so to get your bot tweeting you had to fire regular web requests at it using a cron service. Glitch blocked access by these services and introduced a paid tier for ‘always on’ apps. More bots died as a result.

I was thinking about switching my recipes from Glitch to GitHub, making use of templates and scheduled actions. But while I prevaricated, Twitter started on its long, drawn out death spiral – first imposing new limits on API use, and later becoming the preferred networking site for nazis and transphobes. It was no place for creative bot-making.

The serious side of serendipity

Bot-making wasn’t just about fun – Trove Twitter bots had a serious purpose as well. In ‘Unremembering the forgotten’ I wrote:

Twitter bots can interrupt our social media meanderings with pinpoints of surprise, conflict, and meaning. And yet they are lightweight, almost disposable, in their development and implementation. No committees were formed, no grants were obtained—they are quick and creative: hacks in the best sense of the word. Bots are an example of how digital skills and tools allow us to try things, to build and play, without any expectation of significance or impact. We can experiment with the parameters of access.

A number of articles on the value of serendipity have considered how collection bots, like @TroveNewsBot, can puncture our research expectations. The random offerings of bots might offer new modes of discovery. In ‘Technologies of Serendipity’, Paul Fyfe argues:

For scholars or other readers, discovery results less from directed searching than from all the tangents encountered on the way. Thus, sources which are plural, redundant, and tangent-rich help promote discovery by the proliferating contingencies of their usage.

Similarly, Brett Holman notes that his own Trove bots help him make connections in his research:

By impinging on my consciousness when I am preoccupied by other things, @TroveAirRaidBot’s tweets draw my mind back to this research topic that is always sitting at the back of my mind somewhere, and it makes me make connections – randomly, haphazardly, but often very fruitfully leading me to think of something I hadn’t thought of before, or reminding me of something I’d forgotten, or juxtaposing some seemingly unrelated things. It’s a kind of directed serendipity.

Trove Twitter bots were also entry points and interventions – challenging our understanding of access. They offered playful demonstrations of how our experience of GLAM collections might be different. Mitchell Whitelaw suggested that such creations:

reflect an emerging interest in collections as active sites of meaning-making, and experimentation with how we might encounter such collections in an everyday digital environment.

In ‘Life on the outside’, I considered the lives that GLAM collections might lead beyond institutional confines:

These bots do not simply present collection items outside of the familiar context of discovery interfaces or online exhibitions, they move the encounter itself into a wholly new space. … Twitter bots loosen the institutional context of collections to allow them to participate in a space where people already congregate. They send collection items out into the wilds of the web, to find new meanings, new connections and perhaps even new love.

The promise of serendipitous discovery has now faded with the poisoning of social media spaces, and the retreat of many GLAM organisations from experimentation and openness. The need to control now carries more weight than the gift of creativity.

What remains

I migrated @TroveNewsBot to the Fediverse in May 2023, but sadly it was killed when NLA gatekeepers cancelled my Trove API keys without warning in January 2025.

A number of other Trove bots have survived the Twitter implosion and found their way to alternate platforms. @DigitiseTheDawn now shares articles on the Fediverse, while @TrovePenguinBot is pursuing sardines on Bluesky. Brett Holman has created new versions of his aviation-themed bots – @TroveAirBot, @TroveAirRaidBot, and @TroveUFOBot – on Bluesky. I’d be happy to add the details of any other survivors I might have missed.

In an odd coincidence, recent months have brought new restrictions on access to Trove API keys, and an announcement of the end of Glitch. There’s no going back.

ActivityPub and the Fediverse seem to offer new digital channels through which collections might flow and connect. See, for example, Aaron Straup Cope’s work at the SFO Museum. But how do we support and encourage this type of experimentation?

Personally speaking, this year’s been pretty shit so far, and I’ve been having trouble finding any motivation. But in pulling together these notes I found a section in ‘Unremembering the forgotten’ that reminded me of what’s at stake:

There is no open access to the past. There is no key we can enter to recall a life. I create these projects not because I want to contribute to some form of national memory, but because I want to unsettle what it means to remember: to go beyond the listing of names and the cataloging of files to develop modes of access that are confusing, challenging, inspiring, uncomfortable, and sometimes creepy.

There’s still plenty of work to do.

Some Archives Week goodies

Wed, 11 Jun 2025 17:40:11 +1100

It’s International Archives Week and I’m feeling a bit crook after being double-vaxxed yesterday, so instead of doing something productive, I’m just going to make a list of potentially handy archives-related resources from the Wonderful World of Wragge(TM).

The theme of Archives Week is #ArchivesAreAccessible, which you’d have to regard as rather aspirational given the various ways access is limited by law, policy, practice, technology, and history. But what the heck, discussions about the meaning of access are always welcome. It’s also a little jarring to see the #ArchivesAreAccessible theme being promoted by the National Archives of Australia just a few weeks after they implemented new restrictions that make it impossible to get machine-readable data out of their online database, RecordSearch. But I’m trying to move on, so…

Zotero

All Australian archives users should have Zotero installed. Through the magic of user-contributed ‘translators’, Zotero can capture structured data and digitised images from a variety of collections, including:

The first four are my work, so let me know if you have any suggestions or problems.

Indexes to records

Archives are well represented in the GLAM Workbench’s list of GLAM datasets shared through government open data portals. Many of these datasets are indexes that link records to people and places. They’re openly licensed and under used!

NSW State Archives has also compiled a lot of indexes. These aren’t shared through a portal, but you can harvest them from their website. To save you the effort, I’ve created a repository of the harvested indexes.

I’ve pulled many of these sources together to build a mega database of name indexes that lets you search for people across millions (yes millions) of records. As well as the sources described above, it also includes people-related records from the Public Record Office Victoria’s API. All together, the GLAM Name Index Search contains almost 13 million records in 293 datasets from 10 GLAM organisations. And unlike the commercial genealogical databases, it’s free! (If only Australian libraries and archives would link to it from their family history guides…)

Other datasets

Before my scrapers were scuppered by the NAA, I managed to compile a few datasets. Much of this data documents the way RecordSearch itself has changed, and while it might not be of use to researchers seeking particular records, it could help future researchers who are trying to understand the impact of online collections on the practice of history. This includes:

Summary data about all series in RecordSearch – a CSV file containing basic descriptive information about all the series currently registered on RecordSearch as well as the total number of items described, digitised, and in each access category. Harvests from May 2021 and April 2022 are currently available, and I’ll soon be adding May 2025.
Files digitised by the National Archives of Australia since 2021 – annual compilations of data harvested from RecordSearch’s list of recently digitised files. (The automated weekly harvests are now dead.)
Records held by the National Archives of Australia with the access status of ‘closed’ – a whole decade of annual harvests of records held by the NAA that have the access status of ‘closed’ (withheld from public access). The harvests were run on or about 1 January each year from 2016 to 2025. The aim in saving this data is to enable long-term analysis of the NAA’s access examination process.

Thanks to support from Wikimedia Australia, I’ve also added information about Australian government agencies from RecordSearch to WikiData. As a result you can get a list of Australian government departments since Federation using this Wikidata query. I’ve used the data to build this interactive visualisation of the relationships between government departments. There’s some more examples in the Wikidata section of the GLAM Workbench.

Also from the NAA is my collection of #redactionart found in ASIO surveillance files.

As part of the Real Face of White Australia project, we’ve been transcribing records created by the administration of the White Australia Policy, now held by the NAA. Some of the results are available in this data repository. (Note to self – I need to update this with the latest data!)

The ANU Archives section of the GLAM Workbench includes some datasets extracted from the Sydney Stock exchange stock and share lists. (Just noticed some CloudStor links in there that I need to fix…)

Public Record Office Victoria

PROV gets it’s own section because, as far as I know, they’re the only Australian archives with a functioning public API. (Brief moment of silence to remember the APIs that have come and gone over the years…).

There’s now a PROV section of the GLAM Workbench, that includes a ‘getting started’ notebook to document the basic functionality of the API. There’s some more information in this blog post.

I’ve also used the PROV API to create an automated data dashboard that provides an overview of their collection. It’s updated every Sunday.

Other things

RecordSearch users will understand the frustration of trying to share a url to a record, only to get an annoying error. There are a few ways around this (Zotero saves persistent links to things you save), but for a quick fix I created a simple tool to create persistent links in RecordSearch.

If you’re comfortable with a little browser hacking, you can also install this handy RecordSearch userscript (scroll to the bottom for installation instructions). It improves the functionality of RecordSearch in a few different ways, such as by indicating the number of pages in a digitised file.

Any ideas?

If you have any ideas for additional resources or datasets, or you’re having problems with an online collection, feel free to drop a note in the GLAM Workbench repository.

New dataset – Trove links shared on Twitter, 2009 to 2020

Tue, 10 Jun 2025 12:30:54 +1100

A few years ago, I harvested the details of tweets that included links to Trove. The data has just been sitting on my computer, so I thought I should package it up and share, in case it’s of use to anyone.

The story is that back in 2021, I was working on the article ‘More than newspapers’ for a special section of History Australia focusing on Trove. I was thinking that I might include something about the way Trove newspaper articles were mobilised within online discussions about history – a topic I first explored in ‘Life on the outside: connections, contexts, and the wild, wild web’, my keynote for the Annual Conference of the Japanese Association of Digital Humanities in 2014. In the end, the article went in another direction, so I didn’t use the data.

I remembered this recently and thought I should I should do something with it. I’ve now created a dataset and shared it on Zenodo. I’m not working on Trove any more, but I’m hoping that someone else might find the data useful!

The dataset contains information about tweets from 2009 to 2020 that include links to Trove. The tweet data was compiled using Twarc in May 2021, under Twitter’s academic access program. The search queries used were:

url:nla.gov.au/nla.news
url:trove.nla.gov.au
url:newspapers.nla.gov.au

Many of the tweets were produced by bots. Fortunately, I’d been maintaining a list of Trove bots on Twitter, so I used the list to separate the tweets into two files, one for bots and one for ordinary users.

To respect user intentions and comply with the Twitter API terms of use, I removed all the tweet information except for tweet_id and tweet_date from the files. If it hasn’t been deleted, the full data for each tweet can probably be obtained from the X API using the tweet_id, though you might need a paid subscription.

The two main files are:

trove_url_tweets.csv – links shared by human users (although it may include some unidentified bots)
trove_url_tweets_bots.csv – links shared by bots

I also created some additional data files:

trove_url_totals.csv – the number of times each Trove link was shared by users (not including bots)
active_users_per_year.csv – the number of unique users each year who shared a link to Trove
active_bots_per_year.csv – the number of active bots each year sharing links to Trove

There’s more information about the structure and contents of the data files in the Zenodo record.

Overview

I haven’t explored the data in detail, but here’s some quick summaries to give you a taste.

summary
number of unique users sharing Trove links	9,296
number of bots sharing Trove links	43
number of tweets by humans containing Trove links	48,323
number of tweets by bots containing Trove links	318,767
number of unique links shared by humans	36,906
number of unique links shared by bots	270,474

What types of links were people sharing?

types of link shared by humans	count
newspaper article	34,568
other (search queries, home page etc)	8,388
work (items other than newspapers – books, maps, photos etc)	4,856
newspaper page	1,378
newspaper title	406

How did the number of links shared by humans vary across time?

Which articles or pages were shared most often by humans? Here’s the top ten (click on the link to view).

trove_id	trove_type	tweets	retweets	quotes	total times shared
75869223	article	1,232	61	34	1,327
1298497	article	141	1,028	53	1,222
102074798	article	74	693	77	844
68141866	article	138	522	48	708
41602327	article	633	30	0	663
100645214	article	111	467	20	598
502650	page	1	513	12	526
60828173	article	48	444	19	511
4173156	article	53	321	10	384
79410604	article	2	303	69	374

The most shared article reports that PM Menzies had described Hitler as a ‘great man’ at a meeting in July 1939. However, most of the tweets sharing this link came from a single user. A number of the other articles relate to the weather, a reflection of the fact that Trove’s newspaper articles have been mobilised on both sides of the climate change debate.

How many Twitter users were sharing links to Trove each year?

I haven’t included any of the bot data in these summaries because I think I’ll write a second bot-themed post – coming soon!

Updates

I updated the data in this post on 19 June 2025, as I realised some Twitter accounts were originally run by humans before being bot-ified.

GLAM Workbench – preprint for 'Building User-Friendly Toolkits and Platforms for Digital Humanities'

Thu, 05 Jun 2025 16:16:53 +1100

This is a preprint of my contribution to the publication ‘Building User-Friendly Toolkits and Platforms for Digital Humanities’. It provides a brief overview of the GLAM Workbench. I had to leave a lot out, but hopefully it provides a useful summary of what the GLAM Workbench is, and what I’d like it to be.

The GLAM Workbench is a collection of tools and resources created to help researchers use and explore the digital collections of GLAM organisations (galleries, libraries, archives, and museums).¹ It’s mainly focused on collections from Australia and New Zealand, but some sections venture across international boundaries to explore topics such as web archives and Wikidata.

GLAM organisations make a lot of rich cultural data available online, but getting that data in a machine-readable form that can be aggregated and analysed is often difficult. The GLAM Workbench tries to fill this gap by providing code examples and API documentation, but data access alone is not enough. Researchers need to understand the history, structure, and extent of the data – both its limits and its possibilities. By sharing snapshots, building overviews, and exploring patterns and inconsistencies, the GLAM Workbench also attempts to contextualise GLAM collections and open them to new types of questions.²

History and motivation

I created the GLAM Workbench in 2017, but it incorporates the latest versions of tools, such as the Trove Newspaper Harvester, which I’ve been maintaining for more than 15 years.³ One of my motivations was simply to bring together useful snippets, notes, and doodles from a variety of blog posts, web applications, and code repositories, and make them available in a form that could be more easily navigated and maintained.

I was also keen to explore the way that Jupyter notebooks combine code and narrative. I wanted to find ways to support researchers as they developed their digital skills and confidence, not just dump them at the command line or point them to an app.

The ongoing development of the GLAM Workbench is also part of my own research. I’m interested in the meaning of access within the context of GLAM collections. What changes when you can download data and explore collections beyond the limitations of the web interface?⁴

Contents and technologies

At its heart, the GLAM Workbench comprises at least 171 Jupyter notebooks and 59 datasets shared through more than 70 GitHub repositories.⁵ Added to this are a number of web apps, online databases, and guides to related resources. Code from some notebooks has also been spun off into independent Python packages. All of this is brought together within a single documentation site, built using MkDocs Material.

The contents are mostly organised by institution, reflecting the idiosyncrasies of the data. I’ve partially implemented tags to draw together similar resources across institutions, but this needs to be made more consistent, ideally using the TaDiRAH taxonomy.⁶ Many of the notebooks describe methods for accessing data and building datasets. Others demonstrate techniques for visualisation and analysis, suggest workarounds for limits imposed by collection interfaces, or provide example-driven documentation for APIs and datasets.

There is no single platform or server underlying the GLAM Workbench. Instead, it follows a pattern described in the ARDC Community Data Lab’s architecture principles as ‘infrastructure at rest’.⁷ Notebooks can be run as required in a variety of contexts from cloud services to local computers. This is made possible by standardised configuration files and automated processes that build virtual computing environments from each GitHub repository.⁸

Impact and engagement

The GLAM Workbench has helped to expand understanding of the research possibilities of GLAM collection data. The list of publications citing the GLAM Workbench or one of its embedded tools now includes more than 100 entries.⁹ Some of these relate to individual research projects, while others survey the practices of GLAM organisations and the needs of research infrastructure around the world.

My work on the GLAM Workbench has helped inspire organisations such as the National Library of Scotland to explore new ways of supporting digital research.¹⁰ A recent report from the ‘Towards a National Collection’ project in the UK has mentioned the GLAM Workbench alongside a number of national libraries in Europe and the USA for ‘encouraging innovative research and expanding public engagement with heritage resources’.¹¹

And yet, there are disappointments. Most of the Australian GLAM organisations whose collections are featured in the GLAM Workbench have shown little interest in sharing or engaging with its resources. This makes it difficult to get tools to the people who could benefit from them. There’s some irony in the fact that the websites of the National Library of Scotland, the British Library, the UK National Archives, the V&A Museum, and DigitalNZ all include links to the GLAM Workbench, but the National Library of Australia (NLA) and the National Archives of Australia (NAA) do not.

Maintenance and sustainability

While a number of individuals have contributed notebooks and additions to the GLAM Workbench, it remains essentially a one man operation. Over the years, I’ve sought to ease the maintenance burden by automating processes, adding some basic testing frameworks, and generating machine-readable metadata that summarises the contents of each repository. For example, I created a GLAM Workbench repository template that makes it easy to start work on a new topic.¹²

Development of the web archives section of the GLAM Workbench was made possible by a grant from the International Internet Preservation Consortium, and the section’s ongoing maintenance is supported by the British Library.¹³ I’m grateful too for my GitHub sponsors who help cover some of my cloud hosting bills, and to the ARDC for funding to integrate RO-Crate metadata.¹⁴ But beyond this, the GLAM Workbench has received no dedicated funding or institutional support. It has, nonetheless, outlived some well-funded digital infrastructure projects in the HASS sector.

Sustainability means more than money, though. The GLAM Workbench doesn’t have to continue in its current form to have a long-term impact. My focus is on ensuring that its contents are open to future reuse and modification. Everything is openly licensed, published through GitHub, and preserved in Zenodo. If tools are useful they can live on, independent of me.

The future

I’m writing this at a difficult time. Changes wrought by the NLA and NAA in early 2025 have made it impossible for me to continue work on the Trove and RecordSearch sections of the GLAM Workbench.¹⁵ In the Trove section alone, there are more than 70 notebooks.

The GLAM Workbench is not my job, no-one pays me. I work on it because I think its useful and important, and because I enjoy the process of solving problems and helping researchers. The NLA’s actions, in particular, have robbed me of that joy, and made me consider whether I want to continue. Research infrastructure is people.

On the other hand, there are many more GLAM collections for me to explore. I’m also hoping to find new ways of collaborating with individuals and institutions. I’m often inspired to create new tools and resources by gnarly questions from researchers. While such questions continue, so the GLAM Workbench will grow.

References

Ames, Sarah, and Lucy Havens. “Exploring National Library of Scotland Datasets with Jupyter Notebooks.” IFLA Journal, December 27, 2021. doi.org/10.1177/0….

Bailey, Rebecca, Javier Pereda, Chris Michaels, and Tom Callahan. “Unlocking the Potential of Digital Collections. A Call to Action.” Towards a National Collection, November 21, 2024. doi.org/10.5281/z….

Candela, Gustavo, Sally Chambers, and Tim Sherratt. “An Approach to Assess the Quality of Jupyter Projects Published by GLAM Institutions.” Journal of the Association for Information Science and Technology 74, no. 13 (2023): 1550–64. doi.org/10.1002/a….

“GLAM Workbench (GitHub Organisation).” Accessed June 5, 2025. github.com/GLAM-Work….

IIPC. “Asking Questions with Web Archives – Introductory Notebooks for Historians.” Accessed June 5, 2025. netpreserve.org/projects/….

Jackson, Andy. “GLAM Workbench Update.” UK Web Archive Blog. Accessed June 2, 2025. blogs.bl.uk/webarchiv….

Sefton, Peter, Tom Honeyman, Tim Sherratt, and Conal Tuohy. “The ARDC Community Data Lab Architecture: Research Software Deployment Principles and Patterns for Integrity, Reproducibility and Sustainability,” May 10, 2024. zenodo.org/records/1….

Sherratt, Tim. “Develop a New GLAM Workbench Repository.” GLAM Workbench. Accessed June 5, 2025. glam-workbench.net/get-invol….

———. “Farewell Trove.” Tim Sherratt – Sharing Recent Updates and Work-in-Progress, May 7, 2025. updates.timsherratt.org/2025/05/0….

———. “GLAM Workbench.” Zenodo, June 5, 2025. doi.org/10.5281/z….

———. “GLAM Workbench.” Accessed June 5, 2025. glam-workbench.net/..

———. “GLAM Workbench Citations.” GLAM Workbench. Accessed June 5, 2025. glam-workbench.net/citations….

———. “Hacking Heritage: Understanding the Limits of Online Access.” In The Routledge International Handbook of New Digital Practices in Galleries, Libraries, Archives, Museums and Heritage Sites, edited by H Lewi, W Smith, S Cooke, and D vom Lehn, 116–30. London & New York: Routledge, 2020. doi.org/10.5281/z….

———. “No More Harvesting Data from the National Archives of Australia.” Tim Sherratt – Sharing Recent Updates and Work-in-Progress, May 19, 2025. updates.timsherratt.org/2025/05/1….

———. “Some Important Updates for the Trove Newspaper & Gazette Harvester.” Tim Sherratt – Sharing Recent Updates and Work-in-Progress, August 31, 2023. updates.timsherratt.org/2023/08/3….

———. “Supporters.” GLAM Workbench. Accessed June 5, 2025. glam-workbench.net/get-invol….

———. “Trove Newspapers: Data Dashboard.” Accessed June 5, 2025. wragge.github.io/trove-new….

———. “Trove-Newspaper-Harvester.” Python, October 23, 2023. doi.org/10.5281/z….

Sherratt, Tim, Harry Keightley, Ben Foley, and Michael Niemann. “GLAM-Workbench/Glam-Workbench-Template.” Python. GLAM Workbench, August 24, 2023. github.com/GLAM-Work….

“TaDiRAH The Taxonomy of Digital Research Activities in the Humanities.” Accessed June 5, 2025. tadirah.info/..

Talboom, Leontien, and Mark Bell. “Keeping It under Lock and Keywords: Exploring New Ways to Open up the Web Archives with Notebooks.” Archival Science, July 4, 2022. doi.org/10.1007/s….

“Trove Historical Data.” Accessed June 5, 2025. zenodo.org/communiti….

Sherratt, “GLAM Workbench.” ↩︎
See, for example: Sherratt, “Trove Newspapers: Data Dashboard.” and “Trove Historical Data.” ↩︎
Sherratt, “Trove-Newspaper-Harvester.” ↩︎
See, for example: Sherratt, “Hacking Heritage: Understanding the Limits of Online Access.” ↩︎
“GLAM Workbench (GitHub Organisation).” ↩︎
“TaDiRAH The Taxonomy of Digital Research Activities in the Humanities.” ↩︎
Sefton et al., “The ARDC Community Data Lab Architecture.” ↩︎
For more on best practices in sharing Jupyter projects, see: Candela, Chambers, and Sherratt, “An Approach to Assess the Quality of Jupyter Projects Published by GLAM Institutions.” ↩︎
Sherratt, “GLAM Workbench Citations.” ↩︎
Ames and Havens, “Exploring National Library of Scotland Datasets with Jupyter Notebooks.” For another example of the GLAM Workbench’s influence, see: Talboom and Bell, “Keeping It under Lock and Keywords.” ↩︎
Bailey et al., “Unlocking the Potential of Digital Collections. A Call to Action,” 58. ↩︎
Sherratt et al., “GLAM-Workbench/Glam-Workbench-Template.” For documentation see: Sherratt, “Develop a New Repository.” ↩︎
“Asking Questions with Web Archives – Introductory Notebooks for Historians.”; Jackson, “GLAM Workbench Update.” ↩︎
Sherratt, “Supporters.”; Sherratt, “Some Important Updates for the Trove Newspaper & Gazette Harvester.” ↩︎
Sherratt, “Farewell Trove.”; Sherratt, “No More Harvesting Data from the National Archives of Australia.” ↩︎

No more harvesting data from the National Archives of Australia

Mon, 19 May 2025 18:57:26 +1100

A couple of weeks ago I bid farewell to Trove due to the cancellation of my API keys and the NLA’s lack of transparency around changes to API access. Now it seems I have to wave goodbye to 16+ years of work on RecordSearch, the National Archives of Australia’s online database.

I noticed this morning that my weekly harvest of recently digitised files in RecordSearch had failed. A quick check showed that my harvester was being blocked by Cloudflare’s bot protection software. I wasn’t really surprised. Websites are using tools like this to protect themselves against AI scraper bots, and I’d already seen it in action on another Australian government site. In the war between content providers and AI scrapers, researchers and digital preservation efforts are copping collateral damage.

But while we can’t blame the NAA for safeguarding its systems, we can be critical of the fact that it still doesn’t provide its collection data in machine-readable form. There were a couple of datasets shared for a GovHack event many years ago, a short-lived API for the WWI service records in series B2455, and an API attached to a beta discovery service that never saw the light of day (despite many $$$ being spent on it). Without direct access to the data, researchers have had to scrape it from RecordSearch’s web interface. That’s no longer possible.

I started scraping data from RecordSearch back in 2008 when I was working at the NAA. Eventually I packaged up some Python code to help other researchers create datasets. This was completely rewritten as the RecordSearch Data Scraper a few years back, and you can find various tools and examples using it in the RecordSearch section of the GLAM Workbench. In theory, I might be able to modify the scraper to get around the bot protection, but with the bot wars escalating, it hardly seems worth it – I might get the scraper working, only for it fall foul of the latest bot detection rules. It’s really now up to the NAA to decide whether it will find other ways to give researchers access to its data.

So it seems like I’ll be archiving all my RecordSearch code. Unfortunately, many of the RecordSearch notebooks in the GLAM Workbench will no longer work, so I’ll be adding warnings and explanations over coming weeks.

While not entirely unexpected, it’s all pretty sad. The RecordSearch scrapers have powered some of my favourite research projects. They enabled Kate Bagnall and me to download the metadata and images behind The Real Face of White Australia – a process we described in our article ‘The people inside’. Using the scrapers I’ve been able to analyse the process of access examination, and extract thousands of redactions from digitised ASIO surveillance files. Without the scrapers I would never have discovered #redactionart!

But while I won’t be harvesting any new data, I have a few datasets that I’d like to explore further. Fortunately, I just finished compiling some summary data about every series in RecordSearch, and I want to compare this latest harvest with datasets from 2021 and 2022. I need to do some more analysis of ten years' worth of data capturing the details of files with the access status of ‘closed’. I’ve been working on an update to my redaction finder, which I think I should still be able to finish. And there’s also a lot of data that volunteers have transcribed from records relating to the Real Face of White Australia that I need to pull together.

While my life has been dominated by Trove in recent years, as a historian my heart has always been with the collections of the National Archives of Australia. I’m hoping this is just a temporary setback, and that new methods for data access will emerge.

Farewell Trove

Wed, 07 May 2025 14:51:34 +1100

Over the last few months I’ve been grappling with the cancellation of my Trove API keys by the National Library of Australia. It may seem like a minor technical hiccup from the outside, but it’s had a major personal impact. For the sake of my health, I’ve decided to stop work on Trove, archive all my code repositories related to Trove, and move on. Farewell Trove.

But don’t panic! All of my Trove tools and resources available through the GLAM Workbench and elsewhere will remain online. They just won’t be updated. I’ll be adding explanatory notices to the affected resources over coming weeks. All of my stuff is openly licensed, so feel free to take what’s useful and develop it further yourself.

I’ll also be adding warnings for researchers planning to use the Trove API in their projects. Given the fact that the NLA is willing to change the API terms of use to restrict access without any consultation, provides no transparency around acceptable use of full text content, and is willing to cancel API keys without warning, I can no longer recommend Trove as a reliable source for digital research. A PhD student could embark on a project in good faith, only to have the rules change mid-project.

I think this is a critical issue for the research sector, and hard questions need to be asked of the NLA. But I can’t be the one to do this any more. I’m sick of being the person calling the NLA out on its bad behaviour. I’m sick of their gaslighting.

I wanted to avoid making any dramatic gestures, but after talking it over with my partner last night, I realised my health is really suffering and I need to make a change. I also realised that even if my API keys were magically restored, I’d always be looking over my shoulder, wondering if I’d done something to offend the NLA gatekeepers. That’s not a good way to live. I’d rather spend my time working with organisations who value what I do.

Addendum, 22 May 2025

I noticed last night that the Trove API key application process had changed. Previously, non-commercial use was approved automatically. Now you have to fill in a two page form justifying your proposed use. Your application is then assessed against a complex four level review matrix. Responses are provided within 7 to 28 days. If you want to download the full text of resources, such as digitised newspapers, you additionally need to apply for an exemption to the terms of use.

The NLA has also changed the API terms of use, removing all reference to unauthenticated access. Limited unauthenticated (or keyless) access was introduced with version 3 of the API and was useful for quick experimentation, demonstrations, and teaching. Unauthenticated access has now been disabled and all attempts to access the API without a key return an error. This change together with the new and complex application process, makes it difficult, if not impossible, to use the Trove API in teaching and training contexts.

These changes will further discourage use of the Trove API, and that’s probably the point. I wouldn’t be surprised if further limits were imposed in the future, or if the NLA decommissioned the API entirely.

All posts on this topic:

25 February 2025: 15 years of work on Trove threatened by the NLA
2 March 2025: Trove API users beware! – the latest in the saga of my cancelled API keys
11 April 2025: Update on Trove data access and my suspended API keys
7 May 2025: Farewell Trove

SLV LAB and GLAM Workbench updates

Mon, 05 May 2025 14:26:15 +1100

Last week the State Library of Victoria launched SLV LAB, a prototyping and innovation lab that ‘experiment[s] with technology to open access to collections, data and spaces’. The SLV LAB encourages collaboration, and is sharing code, datasets, and tutorials. It’s an exciting development and I’m looking forward to seeing what they get up to. I’ve added SLV LAB to the GLAM data portals & repositories section of my Australian GLAM data list.

The launch prompted me to have a look at the SLV section of the GLAM Workbench, which I added about 5 years ago. There are currently two notebooks which both relate to the SLV’s use of IIIF to deliver their images. When I created them, there was an issue with IIIF image links needing to have a cookie set before you could access them, but that now seems to have been fixed, so I thought it was time for an update.

Download an image using the IIIF server and a Handle url – The SLV uses the Handle system to create persistent urls for images, and IIIF to deliver the images for use. But how do you get from one to the other? This notebook uses the Handle url to find an image’s IIIF identifier, and then uses IIIF to download the image. The Handle urls are aggregated into Trove, so you could use this method to download SLV images from Trove metadata harvests.
More fun with IIIF – This notebook demonstrates how you can use the standard IIIF API to manipulate images from the SLV collection.

I’ve removed the need for cookie consumption and simplified some of the code in the notebooks. I’ve also updated the repository to embed the GLAM Workbench’s latest systems and integrations. This means, among other things, that the repository is preserved in Zenodo with a DOI, and a Docker image is automatically built that makes it easy to run the notebooks in a variety of contexts – including the ARDC Binder service.

Perhaps most interestingly, I’ve also created a Jupyter Lite version of the ‘More fun with IIIF’ notebook that runs in your browser without any need for a cloud server. Unfortunately, I can’t do the same with the Handle/IIIF notebook because Jupyter Lite runs afoul of CORS permissions when requesting the Handle url.

I’m looking forward to adding additional notebooks and examples as the SLV LAB develops, and shares more data and code.

New PROV section added to the GLAM Workbench

Wed, 30 Apr 2025 15:00:09 +1100

There’s a brand new GLAM Workbench section to help you work with data from the Public Record Office Victoria!

Over the past couple of months, I’ve been poking around in the PROV’s collection API. The API provides data about PROV’s archival holdings in a machine readable format. This makes it possible to use, analyse, and visualise the collection in new ways.

I’ve already shared a few of the results of my explorations. There’s PROVbot sharing randomly-selected photos via the Fediverse; a data dashboard providing an overview of the PROV collection; and 6 million rows of PROV data added to the GLAM Name Index Search. At the same time I’ve been documenting how the API works, and the sorts of data it provides. I’ve now compiled this documentation into a Jupyter notebook – Getting started with the PROV API – and added it to the GLAM Workbench.

The PROV API provides a lot of rich, interconnected data, but there’s not much documentation on the PROV website. I’m hoping that this new section of the GLAM Workbench will encourage people to explore its possibilities. I’ll be adding more notebooks over time, examining the nature of the data in more depth, and probably creating a few useful tools and visualisations. Let me know if you have ideas for new notebooks!

I recently made one of the GLAM Workbench’s introductory notebooks available to run live using Jupyter Lite. This means the notebook loads everything it needs within your browser, rather than depending on a separate cloud service. The new PROV section comes with Jupyter Lite support baked in. If you go to the page describing the API notebook, you’ll notice there’s a brand new option to run the notebook using Jupyter Lite.

This is the first time I’ve integrated Jupyter Lite in this way, and I think it opens up some exciting possibilities. Not only does it make it easier and quicker to jump in and start playing, it means you can embed a live, working version of the notebook in any web page. Like this!

And a reminder – this work on the PROV API, like most of the GLAM Workbench, has received no direct funding. I do it because I want to help researchers use GLAM collections in new ways. If you’d like to support the GLAM Workbench you can sponsor me on GitHub or buy me a coffee. Most importantly, please share this information with anyone you think might find it useful.

The GLAM Workbench introduction to how notebooks work now runs in Jupyter Lite

Mon, 28 Apr 2025 12:51:42 +1100

I’ve just updated my introduction to using Jupyter notebooks in the GLAM Workbench so that it runs in Jupyter Lite – that means no more waiting for cloud services to spin up, it all happens in your browser!

All the Jupyter notebooks in GLAM Workbench can be run in the cloud using the free Binder service – either through the ARDC (requires authentication), or through the public, community-run service. While it’s usually just a matter of clicking a link, Binder can take a while to build the necessary computing environment, and sometimes it just fails. Jupyter Lite takes a different approach. Instead of building things in the cloud, it sets up everything it needs to run notebooks within your own browser.

I’ve been experimenting with Jupyter Lite a bit over the past couple of years, waiting for the technology to reach the point where I could integrate it into the GLAM Workbench without greatly multiplying the maintenance burden. The obvious place to start was my introductory notebook, which demonstrates how Jupyter notebooks themselves work. Using live data from the National Museum of Australia API, it describes the basic structure of notebooks, and shows you how to edit and run code within them. I’ve now set things up so this notebook runs in Jupyter Lite.

What does this mean? Previously, the link to the introductory notebook spun up a new Binder instance. Now, the link retrieves a static web page hosted on GitHub. As this page loads, it installs a Python kernel and everything else it needs to run the notebook within your browser. It’s a lot faster than waiting for Binder, and provides a smoother experience for new users. And because it’s just an ordinary web page, I can even embed a live, working version of the notebook within this blog post. Try it out!

Jupyter Lite won’t currently work with every notebook in the GLAM Workbench. Some Python packages are difficult to install, and some data sources can’t be accessed due to CORS problems. But I’m planning to add Jupyter Lite options where I can.

Update on Trove data access and my suspended API keys

Fri, 11 Apr 2025 16:27:16 +1100

On 21 February, my Trove API keys were cancelled without warning. A week later, I met with NLA staff and was shocked to be told that downloading ‘content’, such as the text of digitised newspaper articles, was regarded as a breach of the API terms of use. Without API access I can’t continue my work helping researchers make use of Trove. More generally though, the NLA’s actions threaten innovative digital research. This post tries to answer some questions raised by my first two posts, and provides some updates on recent actions by the NLA.

What’s an API key?

You might be wondering what an API key is and why it’s important. At its heart, it’s all about access to data. The Trove API delivers information from Trove in a form that computers can understand and process. This allows researchers to compile datasets for detailed analysis or visualisation, and supports the development of innovative tools and interfaces that help all Trove users. But access to the API is controlled by keys – no key, no data.

I’ve had an API key since 2012, and have used my keys in a variety of ways to help people use and understand Trove. Some keys were linked to particular applications such as the Trove API Console, and Headline Roulette. Others are used in the development of tools and resources such as the GLAM Workbench, Trove Data Guide, and the Trove Newspaper Harvester.

The NLA suspended all my keys without warning or consideration as to how they were being used. Services like the Trove API Console which are dependent on keys for their operation simply stopped working.

No. warning. at. all.

However, the greater concern for me is that I can no longer develop or maintain things like the GLAM Workbench. I’ve spent a lot of time over the last 15 years responding to researcher inquiries, building tools to enable new research projects, and supporting researchers as they begin to explore the possibilities of Trove data. Most of this work has been unpaid. I do it because I think it’s important. But with no API keys my hands are tied. My ability to help researchers is severely limited.

The National Library of Australia chose to do this.

What are the terms of use?

I haven’t received a clear explanation as to why all of my API keys were cancelled. As I noted, discussion of this in my meeting with the NLA was confused and contradictory. But, in general terms, it seems to relate to the Trove API terms of use which were changed in 2020. In particular, the NLA now insists that accessing the ‘content’ of resources, rather than just the descriptive metadata, is a breach of the API terms of use. This includes the full text of digitised newspaper and journal articles that are included in API responses.

Through all of this it’s important to remember that the API terms of use are not imposed on the NLA by some external authority. They created them and can change them again at any time. If the NLA believes that work like mine has value, if they believe that researcher access to publicly-funded resources is important, they can change the rules to support these sorts of activities. To not do so is a choice.

The terms of use were changed back in 2020, so why has the NLA suddenly chosen to act? From my point of view nothing has changed. All my work is open. I’ve just been doing what I’ve been doing since 2010. No-one has reached out to me over the last five years with concerns about the terms of use. There must be something else going on here, but, given the NLA’s lack of transparency, it’s hard to know.

On 10 April, API users with keys dating back before 2020 started receiving emails that require them to explicitly accept the 2020 terms of use, or give up their keys. I suspect this is because the NLA has realised that API users weren’t informed of the changes at the time. Section 20 of the terms of use states:

The Library may change these API terms at any time at any time in its sole discretion. The Library will notify you of any changes to these API terms by adding a statement on Trove and the amended API terms will take effect 5 working days (in the Australian Capital Territory) after the date on which the statement was added to Trove. If you do not agree to the updated API terms, you must immediately cease using Trove.

If no such statement was added to Trove, then no agreement can be assumed.

The email sent to API users could have been an opportunity to explain why these changes were made, and support developers and researchers in making any necessary adjustments. But no. The major change in terms of access to content is not even mentioned. It seems the NLA just wants to slip this past quietly without anyone really noticing.

Is it all about AI?

Some have wondered whether the NLA’s actions were motivated by AI crawlers hoovering up vast quantities of online content. The Director General’s response to people who wrote to her expressing their concern over my treatment does highlight the challenges of AI, pointing to the NLA’s new Artificial intelligence framework. However, this framework is almost exclusively concerned with the NLA’s own use of AI technologies. The only reference to external AI use is the statement: ‘We will seek to protect our data from external AI systems where their use contravenes the access rights of publishers and authors’.

It’s also not clear how this relates to API use. Commercial use of the Trove API has always been handled differently to non-commercial use. Applications for commercial use are individually examined and typically involve a quid pro quo, such as access to paywalled services built using the API. So why impose new restrictions on non-commercial uses?

It’s also worth noting that the changes to the terms of use were made in 2020, before the impact of AI crawlers was well understood. Their impact, however, might explain why the terms of use are suddenly being enforced.

Of course, most AI crawlers are probably just going to scrape stuff from the NLA’s websites. To try and manage this, it would seem preferable to push bots towards the API, not away. That way their use could be monitored and better controlled. The Wikimedia Foundation recently published an interesting article on How crawlers impact the operations of the Wikimedia projects. Their planned responses to this challenge include improving their APIs and working with developers to manage their usage.

Perhaps the NLA’s new policing of their terms of use is some sort of hamfisted response to AI threats, but why go after me? As Seb Chan noted on LinkedIn:

Programmatic access to cultural collections has been vital to establishing new forms of practice, research, new scholarship and community discovery. Tim’s work has been globally recognized as important and vital for a very long time. Even if we acknowledge the increased cybersecurity risks (eg British Library), and concerns about AI bot content harvesting that likely lie behind this suspension of Tim’s generous work, this continues to feel like another own goal in the Australian cultural sector.

If there’s a problem, why isn’t the NLA discussing it with the research sector instead of picking on individuals?

Who does this affect?

The NLA’s actions have hit me pretty hard. Having 15 years of work discarded by an organisation that you’ve always sought to promote is disheartening to say the least. The impact on my work and life has been such that I’ve considered whether there might be legal recourse. I’m sad, anxious, and disappointed.

I’m also worried about the impact of the NLA’s actions on the research sector in general. In the Director General’s response she notes:

Where there is a clear need, the Library grants exemptions to the Terms of Use in order to support research and will continue to do so. We are working to make the process for applying for exemptions more transparent and to reach out to API users to clarify the process.

However, the email sent to API users simply says:

If your use of the Trove API does not align with the Trove API Terms of Use, please get in touch with Trove Support to discuss an exemption agreement.

No process. No transparency.

Any researcher wanting to analyse digitised newspaper articles, parliamentary papers, or other full text content will first need to obtain an ‘exemption agreement’. There is no information on what such an agreement involves, or how applications are assessed. The Trove API is no longer open. Research projects are now subject to the whims of NLA gatekeepers.

Established researchers, or large, well-resourced projects, might see no problem here. They can more easily, and more confidently, justify their needs. The greatest impact will be on experimental projects, where the boundaries are not yet charted. What worries me most are the HDR and ECR researchers who will be deterred by these new restrictions. What new questions will not be asked? What new approaches will be discarded because of these barriers erected by the NLA?

There will also be a significant impact on research training. How can you run a workshop on using the Trove API if participants are required not only to get an API key, but an individual ‘exemption agreement’ to work with full text?

Your support means a lot!

The one good thing to come from all of this is the support I’ve received from people around the world. Many have written to Trove, the Director General, and the Minister for the Arts. The responses received so far have not been very enlightening but, nonetheless, I think it’s really important to remind the NLA that their actions have an impact, and are being watched.

The Australian Historical Association has made a public statement, noting that:

The suspension of Dr Sherratt’s API keys, the result of recent changes to Trove’s API policy, marks a troubling restriction to the work of researchers in Australia and overseas.

The Centre for Contemporary Histories at Deakin University included a message from Professor David Lowe about the NLA’s actions in their recent newsletter. He wrote:

The best research stems from an environment that maximises the free flow of information and support for independent scholarship.

Kathryn Greenhill published an open letter that argued:

Reinstating Dr Tim Sherratt’s access to Trove API keys is essential to restore a set of cultural, educational and research tools that enhance access and usage for National Library resources.

There were also articles in Information Age (published by the Australian Computer Society) and The Sizzle.

Concerns are not just being expressed in Australia. Professor Shawn Graham from Carleton University in Canada published an open letter in which he stated:

Mr. Sherratt’s work is well known across the world in the galleries, libraries, archives, and museums sectors. His work developing the GLAM Workbench has promoted Australia’s cultural heritage world wide. Indeed, because of Mr. Sherratt’s work my own students in our Public History graduate program are more familiar with the National Library of Australia’s Trove service, and hence Australian culture, than with what our own Library and Archives Canada provides. By developing the GLAMWorbench with the Trove service, Mr. Sherratt has had a major impact in how cultural heritage materials are understood at scale, across the world. His work is complementary to your own, and enhances the prestige of the National Library of Australia

My battle with the NLA was also used as a case study in a paper on digital archives as infrastructure by Anne-Laure Donzel & Julien Benedetti presented at the forum de l’Association des archivistes Français.

Trove used to be recognised internationally as an beacon of open cultural heritage data, but that reputation has been tarnished. I’ve been contacted by a number of researchers and GLAM professionals around the world expressing their shock and disappointment at the NLA’s behaviour.

What comes next?

It’s now six weeks since my API keys were cancelled. The recent responses by the Director General and the email to API users indicate that something is happening at the NLA, but I’m not sure whether that’s going to help me. Anyone who asks the NLA about my situation is told that it can’t be discussed due to their ‘privacy’ obligations.

I’m seeking to find out more through my local member, Andrew Wilkie, who has contacted the Minister for the Arts. But now there’s an election, so I don’t expect any updates soon.

Beyond my personal situation, there are important issues that need to be considered by the research sector, and I’m hoping peak organisations such as the Australian Academy of the Humanities, the Academy of the Social Sciences in Australia, and the Australian Research Data Commons will take up the discussion.

It’s also worth noting that university libraries are Trove ‘partners’ and are represented on Trove’s Strategic Advisory Committee, so if you’re worried about the impact on your own research let your university librarian know, or tell the Council of Australasian University Librarians.

It seems the NLA is currently hunkered down, waiting for everything to blow over. As an institution they have a lot of cultural power, and people are often reluctant to criticise them in public. But I think it’s important to keep on the pressure.

What I want is pretty simple:

my API keys back
an apology for the way I’ve been treated
more transparency from the NLA about API access
an open discussion within the research sector about the problems and possibilities of working with Trove data

All posts on this topic:

25 February 2025: 15 years of work on Trove threatened by the NLA
2 March 2025: Trove API users beware! – the latest in the saga of my cancelled API keys
11 April 2025: Update on Trove data access and my suspended API keys
7 May 2025: Farewell Trove

Using the Public Record Office Victoria's API to build an overview of their collection

Thu, 10 Apr 2025 14:02:03 +1100

Over the past few weeks I’ve been exploring the Public Record Office Victoria’s public API. There’s not a lot of documentation, but there is a lot of data!

What’s not immediately obvious is that the API includes information about a variety of different entities within the PROV’s model for archival description – not just items, but functions, agencies, series and more. You can limit your API requests to a particular entity using the category field. You can also request facet counts from the category field to tell you how many of each type of entity are available from the API.

I’ve been documenting this sort of information in notebooks for inclusion in the forthcoming PROV section of the GLAM Workbench. But I thought it might be useful to pull a few things together as a standalone dashboard, providing an overview of the PROV collection. So, here it is!

The dashboard tells you how many records are currently available through the API, and breaks down this count by entity, and category. It then works through the main entities – functions, agencies, series, items, and images – displaying a series of charts and tables that give you an idea what they’re actually made up of.

The dashboard is hosted on GitHub and is automatically updated every Sunday. In the future, I’ll do more to highlight changes over time.

More than 6 million rows of data from Public Record Office Victoria added to the GLAM Name Index Search

Wed, 09 Apr 2025 16:28:46 +1100

The GLAM Name Index Search now includes more than 6 million rows of data from the Public Record Office Victoria, downloaded using their public API.

The GLAM Name Index Search brings together records that include the names of people from 10 Australian GLAM organisations. With a single search, you can find information about individuals across millions of rows of data.

Previous versions of the GLAM Name Index Search included a few datasets from the Public Record Office Victoria that had been shared through government open data portals. However, as I was exploring PROV’s public API recently, I realised that there were many more records that included people’s names.

People’s names appear in a number of different fields in the PROV data – including family_name, description.name, and sams.description.name_of_person. They can also be attached to either ‘items’ or ‘images’. ‘Items’ are individual records, such as files, while ‘images’ relate metadata to a page from a digitised item. This means that an image record can tell you that a person is mentioned on a specific page. Both item and image records can point you to useful information.

By using the API to search for records with values in one of the possible name fields, I compiled a list of series that contain items or images that include people’s names in their metadata.

series_id	series_title	record_category	number_of_records
28	Probate and Administration Files	Item	2,578,652
948	Outward Passengers to Interstate, U.K. and Foreign Ports (Refer to Microfilm Copy VPRS 3506)	Image	1,661,181
947	Inward Overseas Passenger Lists (see Microfiche Copies: VPRS 7666 United Kingdom Ports; VPRS 7667 Foreign Ports; VPRS 13439 New Zealand Ports)	Image	1,608,515
7591	Wills	Item	933,110
17379	Probate and Administration Files (CourtView)	Item	294,882
24	Inquest Deposition Files	Item	216,768
14	Register of Assisted Immigrants from the United Kingdom [refer to microform copy, VPRS 3502]	Image	173,167
5357	Land Selection And Correspondence Files	Item	136,782
10010	Body Cards	Item	109,968
4527	Ward Register (known as Children’s Registers 1864 - 1887)	Image	55,429
13579	Teacher Record Books	Image	49,106
515	Central Register of Male Prisoners	Image	44,285
5714	Land Selection Files, Section 12 Closer Settlement Act 1938 [including obsolete and top numbered Closer Settlement and WW1 Discharged Soldier Settlement files]	Image	7,721
516	Central Register of Female Prisoners	Image	6,782
7933	Non-Issued Probate Applications	Item	3,301
5357	Land Selection And Correspondence Files	Image	644
7592	Wills and Probate and Administration Files	Item	606
18110	Clinical Notes and Patient Files (Receiving House)	Item	391
79	Articles of Clerkship Files	Item	374
266	Inward Registered Correspondence	Item	275

I then looped through each series in this list, saving details of each item or image. Finally, to prepare the data from the GLAM Name Index Search, I excluded some fields that weren’t useful and converted the data files into CSV format. All the details are in a notebook that I’ll add to the GLAM Workbench in coming weeks.

The result is that the GLAM Name Index Search now includes 20 datasets from the Public Record Office Victoria, containing 6,645,269 rows of data! All together, the GLAM Name Index Search includes around 13 million records, so almost half are from PROV.

Introducing PROVBot – sharing photos from Public Record Office Victoria

Wed, 09 Apr 2025 12:56:26 +1100

With poor old TroveNewsBot killed by the NLA, my Mastodon feed has had less GLAM goodness of late. To try and fill the void I’ve created PROVBot, sharing photos from the Public Record Office Victoria.

PROVBot makes use of the Public Record Office Victoria’s public API. At this stage it just selects and shares a random photograph once a day, but in the future I’ll probably add more features, such as the ability to respond to search queries.

You can find PROVBot on the Fediverse at https://wraggebots.net/@provbot. There’s also an RSS feed that you can pop into your preferred feed reader. The bot’s code is openly licensed and available on GitHub.

Trove API users beware! – the latest in the saga of my cancelled API keys

Sun, 02 Mar 2025 16:26:51 +1100

After my Trove API keys were cancelled without warning on 21 February, I reluctantly agreed to a meeting with the National Library of Australia. They had provided so little information in their emails, that it seemed to be the only way to find out what was really going on. I came out of the meeting shocked by the NLA’s change in attitude towards API use.

TL;DR – you’re probably breaching the API terms of use

All Trove API users need to be aware that the NLA now insists that accessing the ‘content’ of resources, rather than just the descriptive metadata, is a breach of the API terms of use. This includes the full text of digitised newspaper and journal articles that are included in API responses. Yes, that’s right, using the Trove API in the way that it has been designed and documented is a breach of its own terms of use. You can only download the full text of items using the API if you seek and obtain explicit permission from the NLA beforehand. Note also that the NLA is reviewing people’s use of the API and, as demonstrated by my case, they can and will suspend your API keys without warning.

Through the looking glass

I hate meetings and avoid conflict, but I didn’t see any alternative to meeting with the NLA to find the real reasons behind the cancellation of my API keys. As a classic introvert, I spent the days leading up to the meeting anxiously imagining every possible way the conversation might go. Or so I thought. The outcome was not one I predicted.

The meeting was attended by three Directors responsible for the delivery of ‘Trove business’: the Director of Trove Community Services, the Director of Trove Data and Platforms, and the Director of Strategy and Transformation. Feeling somewhat outnumbered, I took along a colleague. I’m glad I did, as afterwards we had to confirm with each other that what we thought had just happened, actually happened. We were both stunned.

The meeting started with a description of NLA’s change in API policy as described above. It was like stepping through the looking glass. I had not imagined a world where the NLA would set itself up as gatekeeper to every use of the digitised newspaper corpus through the API.

At one point we were told that this change coincided with the release of version 2 of the API. But this doesn’t seem right. Checking the web archive, the terms of use page seems to have been updated when the whole Trove web interface changed in 2020. (You can see how it changed between February and September 2020.) In fact, version 2.1 of the API, released in September 2019, was described by the NLA as opening up ‘access to richer data for API users, particularly the rapidly growing corpus of 1.6 million digitised articles from Australian journals, magazines and newsletters’. There was no indication then that access to this data required special permission.

But when the change happened is less important than how it was, or wasn’t, communicated, and what it means for researchers. At multiple points throughout the meeting I stressed that if this is how they are interpreting and enforcing the API terms of use, they need to be explaining this change to API users. I’m certain that I’m not alone in being totally blindsided.

The reasons for the change are not clear. There was some talk of ‘data governance’, and the fact that the online world had changed – though I fail to see how researchers downloading newspaper articles from the 1890s can be seen as a possible cyber threat. If there are particular problems or concerns, I suggested that it would be useful to have a broad-ranging conversation with the research sector, to see if it might be possible to carve out space for research uses within the terms of use. In response I was told there already is such a carve out – individual researchers can ask the NLA for permission.

Implications for my own work

I was so shaken by this turn away from open access that the question of my own API keys hardly seemed to matter. The immediate reason for the cancellation of my keys is still not clear. The NLA admitted that I hadn’t used the API to extract text from NED journals as their original email claimed. Then it was suggested I used the API to find NED journals to extract text from. This is also not true. Discussion then broadened to the whole of the GLAM Workbench, and, yes, I readily admit that under the new interpretation much of my work breaches the API terms of use. The Trove Newspaper Harvester makes it easy for researchers to create datasets from the full text of newspaper articles in a set of search results. There are notebooks within the GLAM Workbench to help users access the full text of journals and books. In most cases researchers want and need content, not just metadata, and I’ve developed a range of tools to help them access it. But as I explained in my last post, none of this is new. I’ve been helping researchers in this way for 15 years. It’s the NLA that has changed, not me.

It was suggested that if I wanted to regain my API keys, an additional series of meetings would be necessary to help me bring my work within the bounds of what is now permissible.

So it seems I have a choice. Either I try to get out of API jail by submitting to the NLA’s re-education program, or I work with others in the research sector and beyond to try and change the NLA’s policy. I’m inclined to do the latter.

What’s at stake

Ten years ago we celebrated the Trove API because of what it made possible. Everyone was free to explore and create, to analyse changes across 100 years of digitised newspapers, to shift scales and find new meanings. One of my most cited presentations from 2013 talks about how the API made Trove a platform we could all build upon. Openness was the key:

The more we become aware of the power of networked information, the more we become concerned with making and preserving its ‘openness’. To me open data is a process not a product – each visualisation, or interpretation can challenge our assumptions and help us to see things differently. Each use is an opening into new contexts.

Even if it does not directly impact their work, I think all researchers should be alarmed by the NLA’s turn away from openness. Experiments now will have to be approved, judged against criteria which are themselves not public. Is that what we want or expect from a major, publicly funded, cultural institution?

I also fear for the future of the API itself. This change makes it easier for the NLA to impose further limits over time. Perhaps researcher access to the API will be tied to future investment from the research sector. Perhaps it will be claimed the risks are too great and the API will be shut down completely. I’m no longer confident in the NLA’s commitment to providing researchers with long-term access to Trove data.

Where to now?

I want to thank everyone who has offered their support over the last couple of weeks. It’s been deeply encouraging to hear how my work over the past 15 years is valued, and how the GLAM Workbench has helped researchers and inspired new projects. Thanks too for making your views known to the NLA.

I’m now very doubtful that the Trove sections of the GLAM Workbench can be made acceptable to the NLA without major changes that would severely limit their usefulness. However, I’ll continue to maintain them as best I can without an API key, and I’ll continue to help researchers with their Trove questions.

To get myself back into a more positive frame of mind, I think I’ll also do some work with collections from organisations who value openness and are interested in new uses of their data. Suggestions are welcome!

But as I suggested above, the most important task ahead is to start talking about the implications of these changes at the NLA, particularly for the research sector.

Stay tuned…

All posts on this topic:

25 February 2025: 15 years of work on Trove threatened by the NLA
2 March 2025: Trove API users beware! – the latest in the saga of my cancelled API keys
11 April 2025: Update on Trove data access and my suspended API keys
7 May 2025: Farewell Trove

15 years of work on Trove threatened by the NLA

Mon, 24 Feb 2025 12:02:48 +1100

See my latest post for an update!

On Friday, without warning, I received an email from the National Library of Australia informing me that my Trove API keys had been suspended. This threatens the future of 15 years of work helping people use and understand the possibilities of Trove for new types of research.

What’s happened?

Here’s the full text of the email:

Your recently published work on the GLAM Workbench regarding extracting metadata and text from a National e-Deposit (NED) periodical has been brought to the Library’s attention.

Trove API Terms of Use specify that developers may access metadata only and do not provide extended rights. We consider the use of an API to extract and save full text as being in violation of the Terms of Use.

Effective immediately, the four API keys currently registered to you: glamworkbench, headlineroulette, troveconsole and wragge will be suspended.

Please feel free to get in touch for a more detailed conversation about this.

The reasons given for switching off my access don’t make any sense. While the API terms of use only mention metadata, the API, by design, delivers full text from newspapers, digitised periodicals and some books. If you interpret the terms of use as above, simply using the API as it has been designed and documented would be seen as a breach! Surely that’s nonsense.

In any case, the notebook they mention doesn’t even use the Trove API, so it’s hard to see how it could breach the API terms of use. I extracted the text from the periodicals simply by downloading the PDFs and using a standard PDF library. The notebook does scrape some metadata from the Trove website. This is necessary because the API has major limitations – you can’t, for example, get the members of a digitised collection. The NLA might want to argue that scraping breaches the website’s terms of use, but that’s a different point. I’d also note that I’ve been scraping data from the Trove website for 15 years without any objections (see below for more).

When I was Trove manager, I drafted a previous version of the API terms of use. It was a lot less legalistic back then, and I’ve always understood that the point of the API and website terms of use were to protect the NLA from exploitation by commercial interests, not to inhibit work done by researchers in good faith.

I developed the NED notebook in response to a request for help by a community group that uses the National eDeposit service to preserve its newsletter. I did it for free, and I documented the results in the GLAM Workbench in case it might be of use to other communities and researchers.

The ‘has been brought to the Library’s attention’ bit is also grimly amusing. Everything I do is open, and wherever possible I tag GLAM organisations on social media to let them know I’m making use of their collections. The email makes it sound like I was trying to hide what I was doing, when in fact I tagged them on Facebook and LinkedIn. I was thought they might be interested, and I suppose they were, just not in the way I hoped.

What’s at risk?

What’s the consequence of switching off my API keys? A few long-running services were broken immediately. Others will continue to work, but I’ll be unable to maintain them over time. Obviously I won’t be able to develop new Trove-related resources, and perhaps most importantly, my ability to help researchers with their Trove problems will be severely limited.

Tools and services that were broken immediately:

The Trove API Console – running since 2014, the API console has helped many people learn to use the API. I created it when I was Trove Manager, and it’s still linked from Trove’s Create something page.
The Trove Newspaper Data Dashboard (and related data harvests) – running since 2022, the data dashboard enables researchers to understand how the Trove newspaper corpus changes over time. The dashboard depends on weekly data harvests that can no longer run, so there will be no further updates. Similarly other automated data harvests capturing and sharing data about Trove categories and contributors are now broken.
@troveNewsBot – created back in 2013, the bot has survived changes to the Trove API, the demise of Twitter, and a recent forced change of Mastodon instances, but it can’t work without an API key. I’m already missing it’s regular posts.
Headline Roulette – just a simple game, but a fun way to start a workshop and get people thinking about the possibilities of Trove, it’s been running since 2010.

Tools and resources that I won’t be able to update or maintain:

The GLAM Workbench includes more than 80 Jupyter notebooks that demonstrate how to work with data from Trove. Most of these require an API key. Fortunately users will still be able to use them with their own keys, but I won’t be able to do any further development or testing. This includes tools like QueryPic that I’ve been maintaining since 2013 and is cited in the research literature.
The GLAM Workbench also includes more than 30 datasets capturing information about Trove. Some document changes in things like text correction, and the use of tags, while others provide alternative entry points to important collections of digitised resources, such as oral histories and maps. I won’t be to update any of these datasets.
The Trove Data Guide incorporates many visualisations and summaries generated from Trove data. I won’t be able to update these. There are also many API examples that link to the now broken API console.
trove-newspaper-harvester is a Python package that’s existed in different forms since 2010. It’s used by researchers to create datasets of newspaper articles and has been cited a number of times in the research literature. It won’t be immediately affected because users supply their own API keys, but I won’t be able to maintain it into the future.
Trove places is a map based interface to Trove’s newspapers. It’s dependent on data harvested using the API, so I won’t be able to update it.

Planned developments I won’t be able to undertake:

I had hoped this year to automate more data harvests so I could, for example, provide regular updates on digitised collections such as books, periodicals, and maps.
I was planning to add a number of new sections to the Trove Data Guide, including maps, photos, ephemera, and manuscripts.
I was intending to update the trove-newspaper-harvester to make it possible to identify and capture changes in a results set.

I’m very disappointed that the automated data harvests are now broken. As I suggested in this post, I think it’s important that we capture information about online collections so that future researchers will be able to investigate their impact. I’ve been working to streamline, standardise, and automate this data collection, both through the weekly harvests and the Trove historical data collection in Zenodo. But this will now stop.

Most disappointing of all, however, is that without an API key I won’t be able to help researchers who come to me asking how to get data out of Trove. In finding solutions to their problems I often end up creating new notebooks so that the knowledge can be shared and all researchers can benefit. I won’t be able to do this any more.

The GLAM Workbench includes a list of published research articles that cite the GLAM Workbench or one of its associated tools, such as QueryPic and the Trove Newspaper Harvester. Many of these publications have used my tools to work with data from Trove. This sort of research will suffer if the tools can’t be maintained.

Of course, all of my work is openly licensed and freely available through GitHub and Zenodo. If I can’t maintain the code, hopefully others will jump in and take over.

Trove and me

I started scraping data from the digitised newspapers in 2009, before they were even a part of Trove. In 2010, I created the first versions of QueryPic and the Trove Newspaper Harvester. There was no API then, so I built a library of screen scrapers to extract the data. I ended up publishing my own ‘unofficial’ API using the screen scrapers. I found out later that my ‘unofficial ' API was used in the design of the official version that became available in 2012.

The work I was doing analysing digitised newspapers won me the NLA’s Harold White Fellowship in 2012. In 2013, I was appointed Trove Manager. Throughout my time at the NLA I lived something of a double life – manager by day, hacker by night. I continued to build tools and demonstrations to help people understand what the API made possible. Talking about the API and the new types of research that Trove opened up was one of the favourite parts of my job.

Nothing much changed after I left the library. I continued to build tools, help researchers, and give talks and workshops on the possibilities of Trove data. In 2017, I started to bring a lot of this work together within the GLAM Workbench. In 2023-24, I worked with the Australian Research Data Commons to develop the Trove Data Guide, documenting what I knew about Trove’s intricacies and inconsistencies.

My point really is that I’ve been doing this for 15 years now. Everything has been in the open, my approach has never really changed, and some of the work was actually supported by the NLA. So what’s different now?

Certainly the NLA’s attitude has changed. When I was Trove manager we used to celebrate the interesting things that people did with the Trove API. In contrast, the NLA has never publicly acknowledged that the GLAM Workbench exists, and certainly hasn’t shared any links to it. This was taken to ludicrous extremes in 2021, when the NLA’s draft project plan for funding as part of the ARDC’s HASS Research Data Commons proposed to duplicate tools already available through the GLAM Workbench. Just a few months earlier in December 2020, the GLAM Workbench won the British Library Labs Research Award. It’s strange that there has been much more engagement with the GLAM Workbench from national libraries in Europe than Australia.

I don’t know why this is, but it has been immensely frustrating, even heart-breaking. I do the work I do to help people use and understand Trove. But how do they find out about it? You’d think that the NLA would be pleased to support researchers by pointing them to tools and resources that would help them make best use of Trove. You’d think that the NLA would be thrilled to have people spending their own time and money to build and maintain those resources. But no.

It seems to me that the NLA has become increasingly closed off and defensive in recent years. Perhaps that’s to be expected given the funding pressures they’ve faced. But in challenging times you’d think it was more important than ever to bring together your supporters.

Much of my work does involve criticism of Trove. It’s an unwieldy beast, with many problems and inconsistencies. It’s part of my job (mission? calling?) to expose these problems and help users work around them. It wouldn’t help anyone for me to ignore Trove’s shortcomings. My criticisms come with suggestions and solutions. My aim is not to undermine, but encourage – to guide people past the many pitfalls and challenges to find the treasure within.

Back in November 2016, on the day after Trump’s first election victory, I gave a short presentation at the ‘Digital Directions’ conference in Canberra. The main point of my talk, entitled ‘Caring about access’, was that GLAM organisations should embrace criticism. Here’s part of what I said:

Access is not something that cultural institutions bestow on a grateful public. It’s a struggle for understanding and meaning. Expect to be criticised, expect problems to be found, expect your prejudices to be exposed. That’s the point.

If cultural institutions want to celebrate their website hits, celebrity visits, or their latest glossy magazines – well that’s just fabulous. But I’d like them to celebrate every flaw that’s found in their data, every gap identified in their collection – that’s engagement, that’s access. We need to get beyond defensive posturing and embrace the risky, exciting possibilities that come from critical engagement with collection data – recognising hacking as a way of knowing.

In this new post-truth world it’s going to be more important than ever to challenge what is given, what is ‘natural’, what is ‘inevitable’. Our cultural heritage will be a crucially important resource to be mobilised in defence of complexity, nuance, and doubt – the rich and glorious reality of simply being human.

The early part of that 2016 was dominated by the #fundTrove campaign, when Trove users mobilised to make the government aware of its importance to the Australian community. It took over my life for a while, and while many were keen to claim credit for the campaign’s ultimate success, it left me thinking that GLAM organisations need to better understand who their real friends are – the people who actually give a shit. It seems that the NLA is still struggling with that.

So what now?

I have to admit that the NLA’s inability to acknowledge the existence of the GLAM Workbench has taken an emotional toll. At times I’ve considered giving up the work. Why bother if it’s not going to get to the people who might benefit most?

So at this moment I don’t feel like arguing with the NLA. If they think so little of my work that they’re happy to simply pull the plug and let it die, then what’s the point in trying to continue?

However, there’s a bigger issue. Whatever happens to my work, it’s important that this type of work be encouraged and supported. Trove offers immense possibilities for new types of research and we need to explore and document them together. Central to this is a well-supported API. I’m worried that this little battle is actually a sign of waning commitment to the API and what it represents. Earlier this year I was shocked when the NLA suddenly decommissioned version 2 of the API without fixing major bugs in version 3. I think we need to stress that easy access to Trove data is vitally important to the future of Australian HASS research.

So if you’ve used any of my tools or resources, or value the work I’ve been doing over the last 15 years, you might like to tell the NLA about it. I don’t know if it’ll make any difference, but at least they’ll be better informed about the sorts of things people are doing with Trove data, and the types of resources that are needed to support them.

Contact options:

Trove feedback form
Marie-Louise Ayres, Director-General of the NLA (directorgeneral@nla.gov.au)
Tony Burke, Minister for the Arts (tony.burke.mp@aph.gov.au)

Of course you can also share your thoughts on social media!

All posts on this topic:

25 February 2025: 15 years of work on Trove threatened by the NLA
2 March 2025: Trove API users beware! – the latest in the saga of my cancelled API keys
11 April 2025: Update on Trove data access and my suspended API keys
7 May 2025: Farewell Trove

The Primary Source – GLAM collection news and help

Thu, 20 Feb 2025 16:31:54 +1100

I’ve created a new site (or in fact, renovated an old site) to aggregate news from GLAM collections (that’s galleries, libraries, archives, and museums) and help researchers using those collections. It’s called The Primary Source which is a bit of a bad history pun.

Why is is needed?

Before the nazi takeover of the old bird site, I had a list of GLAM organisation accounts which made it pretty easy to follow what was going on in Australia’s galleries, libraries, archives, and museums. Things are more fragmented now and surviving social media accounts seem dominated by event promotion, cute videos, and cultural heritage clickbait. There are a few blogs (though apparently the fashion is to call them ‘stories’), but functioning RSS feeds are rare. How can researchers find out about new GLAM collections or resources across Australia? Hopefully The Primary Source can help by aggregating collection news from a variety of platforms.

At the same time, I wanted to provide a space where researchers can share their latest work using GLAM collections, and ask for help when needed. Unfortunately, GLAM social media accounts (with a few exceptions) rarely share the work of researchers outside their own fellowship and events programs. The Primary Source is built on top of the Discourse discussion platform, so anyone can create an account and contribute.

Discussion categories

At the moment there are four main categories:

Collection news: News about recent additions or updates to Australian GLAM collections
Useful resources: Resources that help people use and understand Australian GLAM collections – digital tools, finding aids, webinars etc.
Help needed: If you’re having a problem using Australian GLAM collections, ask for help here. No question is too basic.
Research updates: Share news from your own research projects.

I’m trying to keep things pretty simple, but ideas for new categories are welcome.

How to contribute

As mentioned, anyone can create an account and start contributing to The Primary Source. You can manually add posts using the Discourse interface, or you can take advantage of the site’s automated Zotero-bot. If you come across an interesting post or resource on the web, simply use Zotero to save it to the appropriate category in the Primary Source contributions Zotero group. Every 30 minutes, the Zotero-bot will check the group and add any new links to The Primary Source. If you add tags or comments to a link in Zotero, they’ll be attached to the new Discourse post. To use the Zotero-bot, all you need to do is join the Zotero group, then it’ll pop up in your list of group libraries. Once again, membership is open.

Obviously any content that is offensive or off-topic will be removed.

Staying in touch

To keep up-to-date with the latest posts from The Primary Source you can:

Create an account at The Primary Source. Once you’ve done that, you’ll receive a weekly email with posts you might have missed. (Of course you can disable this if you don’t want more emails!)
Add the The Primary Source RSS feed to your preferred feed reader: ozglam.chat/latest.rs…
Follow @primarysourcebot on Mastodon. The bot checks for new posts every hour and shares them through the fediverse.

Third time lucky?

The first version of The Primary Source was created way back in 1998. Then, as now, I wanted to help researchers find and use collections from Australia’s archives and libraries. There weren’t many content management systems around back then, so I rolled my own using PHP. It was a pain to maintain, and life went elsewhere, so it didn’t survive very long. I’ve always thought the name was pretty clever though…

The pandemic made me think again about ways of supporting researchers, so I created the OzGLAM help discussion board. Even though there wasn’t much activity, I’ve always believed something like this was needed. So with a bit of remodelling and renovation, OzGLAM help has been flipped to make The Primary Source. Will it survive this time? I suppose that’s a matter of whether you find it useful and valuable. If you do, please share with your friends and colleagues!

National Archives of Australia Digitisation Dashboard

Thu, 20 Feb 2025 14:41:46 +1100

Since March 2021, I’ve been harvesting details of newly-digitised files in the National Archives of Australia to help document long-term changes to online access. A few weeks ago, I summarised the data from 2024, and published annual compilations in Zenodo. I’ve now created an automatically-updated dashboard which displays digitisation progress in the past week, the current year, and since my harvests began.

Each week, after the latest data harvest, a GitHub action runs a Jupyter notebook that pulls in the data, generates some visualisations and summaries, and saves the results as an HTML page. It’s similar to the Trove newspaper data dashboard. Check in every Sunday afternoon to see what’s changed!

Search the content of periodicals uploaded to Trove through the National eDeposit service

Wed, 19 Feb 2025 15:12:41 +1100

I’ve added a notebook to the GLAM Workbench that walks through the steps involved in creating a fully searchable database of content extracted from a periodical uploaded to Trove through the National eDeposit service (NED).

Why is this needed?

I was contacted recently by a member of the team that publishes The Triangle, a community newsletter from the south coast of NSW. Issues of The Triangle from 2007 to the present have been uploaded to Trove through the National eDeposit service, but they were wondering whether it was possible to search across all their newsletters in Trove. Unfortunately, the answer is no.

Issues of The Triangle are saved in Trove as PDFs with a searchable text layer. Individual issues can be browsed and searched using the built-in PDF viewer, but there seems to be no way of searching across multiple issues in Trove. There’s a couple of reasons for this:

The text content doesn’t seem to have been indexed. Trove has the ability to index text from PDFs, though there’s a limit to how much text it consumes. Things like articles from institutional repositories, for example, will have some or all of their content indexed. It’s hard to be certain, but searches for content within The Triangle don’t work, so I’m assuming they’re not being indexed.
Even if they were indexed, there are no work level records for articles or issues within NED periodicals. So if there was a match on content within an issue, your search results would return the top-level title record, and you’d still have to search each issue individually to find the match. Compare this to the way Trove’s digitised periodicals are indexed at article level in the Magazines & Newsletters category.

On top of this, Trove’s inconsistency and lack of transparency means that you’re never really sure what you’re searching and why. Why do some PDFs get indexed, but others don’t? Why are community newsletters contributed through NED in the ‘Books & Libraries’ category rather than ‘Magazines & Newsletters’? Why are some periodicals searchable by article while others are not? I’m trying to document many of these inconsistencies in the Trove Data Guide as they cause confusion and uncertainty for users – if your search returns no results, is it because Trove has no relevant content, or is it because the relevant content isn’t fully searchable? You just don’t know.

Why is this important?

I recently updated my harvest of periodicals submitted to Trove through the National eDeposit Service. In total, there’s 8,572 different periodicals containing 179,510 issues! I used the l-format=Periodical facet to separate out the periodicals from other types of publications, and I don’t think it’s always accurate – some of the publications in the dataset look like one-off reports. Nonetheless, there are lots of periodicals. As I’ve noted previously, this includes a rich assortment of local and community newsletters – not just The Triangle, but The Apollo Bay News, Palm Island Voice, and many, many others. As local newspapers die out, these sorts of publications capture details of community life that might otherwise be missing from the historical record. Just as the diversity of Trove’s digitised newspapers have given historians new perspectives on the past, I believe these NED periodicals will provide researchers with an increasingly important source of information on daily life in Australia. Equally, having these publications accessible online through a free, national service like Trove ensures that communities themselves will have ongoing access to their own histories. But both for researchers and communities, the value of the publications will be affected by their accessibility – how can they be found, searched, and used?

One solution – a DIY search index

If you can’t search across a NED periodical within Trove, perhaps we need to develop alternative approaches outside of Trove. Using The Triangle as my test case, I’ve developed a workflow that creates a standalone, fully searchable database of content from a NED periodical. The database supports full text searches across the complete text content, including query options like wildcards and boolean operators that don’t work within standard PDF viewers. Have a go! You can try searching The Triangle here.

There are a number of different steps involved in creating databases like this from NED periodicals:

harvest basic metadata about all the issues
download the PDFs of every issue
extract the text for each page in the PDFs
create an SQLite database containing the metadata and text for each page
build a full-text index on the text content to allow easy searching

This is fully documented in the GLAM Workbench notebook Create a searchable database from issues of a NED periodical. Once you’ve created the database you can explore it using any SQLite client, but I like to use Datasette. The notebook creates a custom metadata file for configuring Datasette, and explains how to open the database using either Datasette or Datasette-Lite.

Customising Datasette for easy searching

The standard Datasette interface can look a little intimidating if you just want to run a full text search. To make it easier, I’ve developed a customised Datasette theme and canned query that generates a simple search page with a few extra features such as date facets, result snippets, and query highlighting. It’s basically designed to look like a standard search interface. Sometimes simple takes a bit of extra work!

The canned query defines the search parameters and constructs the SQL query called by the search box. It’s automatically included in the metadata.json file created by the notebook. To put everything together, you just need to point Datasette to your database, the metadata file, and the custom template.

I’ve embedded the template in a new Datasette-Lite repository. The notebook explains how to construct a url that will open your database using this repository. The Triangle search interface runs using this customised version of Datasette-Lite.

A new NED section for the GLAM Workbench

I’ve also created a new Trove NED section of the GLAM Workbench. The notebook that harvests metadata about NED periodicals was previously in the Trove periodicals section, but I think its better to keep the NED documentation separate. When I first started developing the GLAM Workbench, it made sense for the structure to mirror Trove’s zones. But over the years I’ve become very aware of the fact that the way content is treated in Trove has less to do with its format than the processing pipelines that get it into Trove. In Trove, digitised newspapers are very different to digitised journals, and digitised journals are very different to NED periodicals, even though they’re all really periodicals! There’s more of this sort of fun in the Trove Data Guide.

Trove and the National Library of Australia refuse to share links relating to the GLAM Workbench or the Trove Data Guide, so there’s a good chance that the people who might benefit most directly from this work will never know that it exists, and will instead be left struggling on their own. I think that’s bad for Australian HASS research. So if it seems this might be useful, please share amongst your networks!

Ten years of data! The files you're not allowed to see in the National Archives of Australia

Wed, 05 Feb 2025 13:26:30 +1100

I’ve created a new dataset containing 10 years of data that can be used to explore the workings of the National Archives of Australia’s access examination system.

Australian government records become available for public access after 20 years. But before being opened to the public, records go through a process known as access examination to determine whether they should be withheld, either partially or completely. The grounds for exemption are laid out in the Archives Act and include things like national security and personal privacy. If a record is completely withheld from access, the NAA’s database, RecordSearch, records its access status as ‘closed’.

On or about 1 January every year since 2016, I’ve harvested details of files in RecordSearch with the access status of ‘closed’. On the day when the media is full of revelations from the public release of the latest batch of cabinet records, I thought it was important to find out what we couldn’t see, as well as what we could. I’ve now published all the annual harvests as a dataset on Zenodo.

It’s important to note that records can be re-examined and their access status can change. Also some ‘closed’ files are actually ‘withheld pending advice’ – in these cases a final access decision hasn’t been made as the NAA has referred the files to their controlling agencies for advice. This means that this dataset should be treated as providing annual snapshots of an active system, not a cumulative record of closed files. Some of the complexities of the access examination system revealed by this data are discussed in the Inside Story article ‘Withheld pending advice’. I’m hoping to do some more analysis later this year.

A Community Data Lab (CDL) wishlist

Fri, 31 Jan 2025 12:40:52 +1100

The ARDC is holding an event on 18 February to begin shaping the next phase of the Community Data Lab. If you’re interested in the development of digital tools and resources to support HASS research, I’d suggest you go along.

I worked on the first phase of the Community Data Lab, developing the Trove Data Guide amongst other things. I’m very keen to see the CDL expand, working with researchers to create new possibilities for digital research, particularly using the rich collections of the GLAM sector (galleries, libraries, archives, and museums). As planning gets underway for the next phase of the CDL, I thought I’d pull together some rough ideas about what the CDL might be and might do. The ARDC needs co-investment in its projects, so new initiatives have to have some form of institutional support to become part of the CDL. Nonetheless, I think it’s useful to continue to think more broadly about HASS research infrastructure needs and possibilities – both long-term requirements and short-term interventions.

Extending co-design throughout the life of the CDL

Currently CDL co-design processes are focused on the initial design phase of a project and are structured around a plan developed from the institutional partners' interests and priorities. It can be difficult to relate specific research tasks and needs to these larger-scale initiatives. How can co-design processes be embedded in a project’s ongoing development?

One possibility is that CDL projects could have shortish development cycles with co-design processes before each cycle to refine the scope and priorities. This would increase opportunities for community participation and feedback, but the process would still be focused on the needs of the project rather than the needs of researchers.

I’d like to see an alternative (or additional) component that operates like a CDL ‘help desk’ or ‘triage’ service. For example:

a researcher outlines a specific problem or need
if useful tools/datasets/documentation already exist, CDL project staff point the researcher towards them
if there’s nothing to meet this specific need, some basic analysis is done to see where it might fit in the CDL universe and how much work might be required
for small-scale solutions within current capacities, simply create the new resource (eg. a piece of documentation, a single notebook, a slice of an existing dataset)
for larger-scale solutions, document the gap/need to feed into future CDL design processes
add all of the information – problem, requirements, solution – to a searchable knowledgebase
the knowledgebase should allow user comments/links and voting on unsolved problems to help identify future priorities

Some of the advantages to this approach would be:

direct engagement with CDL at any time when the researcher identifies a specific problem, not when allowed by the CDL’s co-design timetable and framework
recognises that many problems can be solved with existing tools and resources, both inside and outside of the CDL
encourages direct researcher engagement through the knowledgebase, bringing in broader community perspectives and suggestions, as well as information about related projects and tools
helps to put the ‘community’ in Community Data Lab

Connecting things up

There’s a lot of work to be done connecting up existing tools and data sources. This requires a combination of documentation and the development of small-scale tools or plugins to transform and move data as required. This sort of work fills in the gaps of existing research infrastructures helping to ensure that best use is made of existing investments, and encouraging re-use and collaboration instead of duplication or reinvention. It could be integrated with the ‘help desk’ function described above.

The pathways/tutorials in the Trove Data Guide are all examples of this, indeed, the development of additional Data Guides around particular collections or collection types could help fill a lot of these gaps.

Similarly, I think it’s important to put CDL projects within a broader context, so researchers are guided to solutions that best meet their needs.

For example, I’m a big fan of Omeka. I gave an Omeka workshop at THATCamp Melbourne way back in 2011, I created a Python client for interacting with the Omeka-S API, and I’ve used Omeka in a number of projects. But just as not every blog needs WordPress, not every online collection needs Omeka. There are alternative pathways that don’t require the full technology stack and might better suit the needs of individual researchers. It’s important to keep these alternatives in mind and provide documentation that will enable researchers to make decisions about what’s best for their project.

For example, if you want to share data (even relational databases) in a form that other users can search and explore, then Datasette might be a better option. Even within the Datasette ecosystem there are a variety of deployment possibilities. Datasette-Lite runs wholly within the browser, all you need is a GitHub repository (or similar) to point at. Most of the CSV files that I share through the GLAM Workbench have an option to explore using Datasette-Lite. All I do is create a url that points the CSV file at my existing Datasette-Lite repository. I’ve now created a simple tool to help others create these links. You can even add full text indexes and embed images. Because Datasette-Lite uses static technologies, the maintenance overheads are minimised.

If your dataset is too large or complex for Datasette-Lite, you can use a standard Datasette instance. It’s easy to scale. I have one Datasette instance that contains 11 million rows of data, in 279 datasets, from 10 different GLAM organisations running on Google Cloudrun. The Tasmanian Post Office Directories are also in Datasette and includes a IIIF viewer embedded in a customised theme.

Similarly, if a researcher’s main aim is to create a website that displays an annotated collection, then a static option, like CollectionBuilder, might be all they need.

Using static technologies was one of the patterns highlighted by the CDL architecture and principles documents, and I think it would be good to get into a ‘static first’ mindset, only implementing more complex solutions once the need had been established.

That said, I also think that investment in the development of additional, openly licensed plugins, themes, resource templates, and vocabularies for use with Omeka-S would also be very useful and welcome.

Zotero integration

An example of the small-scale interventions that can be made to mobilise existing data sources is the development of Zotero translators to extract metadata and images from GLAM collection databases. Using Zotero, researchers can collaborate in the creation and annotation of specialised datasets. Once they’ve saved a collection of resources to Zotero, they can use the public API to move the data into other tools for analysis or sharing. For example, at the moment I’m working on a collection of newspaper articles and extracts from Tasmanian Post Office Directories which have been saved and annotated in Zotero by members of the EveryDay Heritage team. Using the API I’m downloading the data, sorting the annotations, and populating an Omeka-S instance with details of sources, people, and places.

The problem is that support for Zotero by GLAM collection databases is patchy at best. I’ve created a spreadsheet to summarise the current situation. Few GLAM institutions embed anything beyond Facebook approved metadata, so translators (little bits of Javascript code) are often necessary to feed rich data to Zotero. There are currently 7 custom translators available for GLAM collections including the National Archives of Australia, PROV, and Trove.

Improving Zotero integration would open up new GLAM collections to digital research. Coordinated effort in the development of new translators, and best-practice documentation for supporting Zotero, would also help GLAM organisations understand what they can do to expand use of their collections.

The CDL could play a coordinating role in this, rather than do all the work. For example, it could share updates, compile documentation, and perhaps organise a GLAM hack style event to create/update as many translators as possible. It would be a low cost, but high impact initiative.

Collections as data

The Australian GLAM landscape is littered with decommissioned APIs, dead open data portals, datasets created for some hack event that never get updated, and disappeared labs. But at the same time, there are hundreds of open datasets shared by GLAM organisations through government data portals that are little acknowledged. In organisations with diminishing resources, shifting priorities, and internal jealousies, it’s difficult to maintain a persuasive argument around the importance of ‘collections as data’ for research. Perhaps the CDL can help.

Ultimately, it would be great for the CDL itself to incorporate some sort of national, collaborative GLAM Lab, but I think we should start by helping to develop the ammunition that institutions need to argue for their involvement in such an initiative. For example:

Documenting needs – What research might happen if particular datasets or tools were available? This is the sort of thing that could be captured through the ‘help desk’. These needs would be shared and discussed, and fed into co-design sessions with GLAM organisations.
Understanding use – What research is being done using GLAM collections in Australia and elsewhere? What types of data? What types of tools? Obviously there will be examples from CDL projects & elsewhere.
Developing skills – Providing opportunities for GLAM staff to develop their own digital research skills. Where CDL projects make use of GLAM collections, invite organisations to nominate staff for active participation in the development processes (like internships).
Creating examples – identify possible GLAM data related developments within CDL projects and pursue small-scale collaborations. For example in relation to government data.
Identifying opportunities – Which GLAM organisations are in a position to participate now? In particular, identify data assets (like PROV API or datasets in government portals) that are already available but not well documented or understood.
Sharing solutions – Many of the ‘connecting things up’ solutions are likely to involve moving or transforming GLAM data. Zotero integrations would benefit researchers and organisations.

Much of this would not be ‘extra’ work, but involve the coordination and packaging of existing CDL activities – a GLAM window onto CDL activities. Though I suppose there would need to be a coordinating role.

Obviously the GLAM Workbench would also have a role in this as a repository for tools and examples. It’d be great if the CDL could work with GLAM organisations to develop their own sections/repositories.

Files digitised by the National Archives of Australia in 2024

Mon, 27 Jan 2025 14:16:09 +1100

In 2024, the National Archives of Australia digitised 254,953 files (down from 416,602 in 2023). This chart shows the number of files digitised per day in 2024.

The decrease in the total number of files digitised is probably related to the completion of the NAA’s five year project to digitise Second World War service records. Thanks to $10 million in government funding, the NAA has digitised more than a million service records since 2019. In 2023, 81% of records digitised were from series containing service records. This has dropped to around 40% in 2024. Here’s the total number of files digitised per year since February 2021.

The files digitised in 2024 came from 1,439 different series. Here’s the top twenty series by number of items digitised in 2024. You’ll see that as well as war records, many photos, patents, and immigration records were digitised.

series	series_title	total
A9301	RAAF Personnel files of Non-Commissioned Officers (NCOs) and other ranks, 1921-1948	56,635
B883	Second Australian Imperial Force Personnel Dossiers, 1939-1947	39,163
A6135	Photographic colour transparencies positives, daily single number series with ‘K’ [Colour Transparencies] prefix	15,569
A13150	Specifications, examiners reports and correspondence relating to the Registration of Victorian Patents - Second system	13,519
A714	Books of duplicate certificates of naturalization A(1)[Individual person] series	13,384
A1500	Photographic colour transparencies [positives], single number series with ‘K’ [Colour] prefix	11,533
A7109	“Dead” card index of Registered Aliens	10,618
A705	Correspondence files, multiple number (Melbourne) series (Primary numbers 1-323)	9,568
A13149	Applications for Registration of Victorian Patents - Second system	8,843
A9300	RAAF Officers Personnel files, 1921-1948	6,371
D3481	Photographs (black and white, colour) of buildings, installations, sites, etc	5,054
A11708	Applications for Registration of Queensland Trade Marks	4,159
SP388/1	Personal documents of British Assisted Migrants	3,395
D879	Duplicates of negatives, annual single number series with D prefix (and progressive alpha infix A - K from 1948-1957)	3,253
B6286	Telstra Historical Collection	2,955
D1423	Original plans (negatives), single number series with alpha prefix denoting discipline	2,616
D1989	Application forms, medical examination documents and related papers of British and Foreign Immigrants (including Ex Service) in receipt of free and assisted passages, chronological order of ship arrival.	2,403
D1901	Loveday Internment Camp internees files, single number series with variable alpha prefix	2,305
PP9/5	Medical examination forms (form 47A) for non-British migrants, single number series	2,250
A6510	Classified prints of photographs relating mainly to Papua and New Guinea	2,242

This data was compiled from weekly harvests of RecordSearch’s list of recently digitised files. These weekly harvests are saved in the naa-recently-digitised GitHub repository. The harvesting method is documented in the GLAM Workbench.

I’ve also compiled these weekly harvests into annual datasets published through Zenodo.

Changes to Trove newspapers in 2024

Fri, 17 Jan 2025 17:42:59 +1100

Every Sunday I harvest information about the number of digitised newspaper articles in Trove. You can view the current results in the Trove Data Dashboard. By compiling all the data from 2024, you can find out what changed last year.

6,241,739 digitised newspaper articles were added to Trove in 2024. The rate of digitisation was pretty quick until the end of March when the processing of the Melbourne Sun ended, then things flattened out a bit.

While the number of articles with corrections, tags, and comments all increased steadily across 2024, there seems to have been a bit of glitch indexing tags and comments causing some jumps in the totals.

Most of the digitised newspaper articles were published in NSW (3,190,972), Victoria (2,680,855), and South Australia (363,483).

About 83% (5,211,532) of the digitised articles added to Trove in 2024 were published between 1930 and 1954. The publication years with the biggest changes were:

1953 (722,765 articles added)
1954 (695,104 articles added)
1952 (586,890 articles added)

About 7% of articles (450,712) were published after 1954 (the copyright cliff of death). Most of these were from South Australia.

Thirty-eight newspapers were added to Trove in 2024:

Newly digitised articles were added to 67 newspapers (including the new titles). The newspapers that had the largest increase in articles were:

The Sun News-Pictorial (Vic), 2,089,635 articles added
Daily Mirror (NSW), 1,533,759 articles added
Maitland Mercury (NSW), 658,331 articles added
Central Western Daily (NSW), 253,575 articles added
The Pastoral Times (NSW), 221,982 articles added
Sunraysia Daily (Vic), 210,759 articles added

If you want to drill down into the changes, have a look at the Trove Data Dashboard archive that has weekly snapshots. Or get the raw data for each weekly harvest from this repository.

@trovenewsbot has a new home

Thu, 12 Dec 2024 12:41:35 +1100

@trovenewsbot has been around for more than eleven years now – originally sharing Trove newspaper articles on Twitter, and now on the Fediverse. But with the imminent closure of the botsin.space Mastodon instance, I’ve had to find it a new home. Say hello to the latest version: @trovenewsbot@wraggebots.net!

Instead of just moving the bot to an existing instance, I decided to set up my own using GoToSocial. I thought this would give me more control, and encourage me to resurrect some more of my old Twitter bots. I installed GoToSocial on the smallest available DigitalOcean droplet, following the ‘bare metal’ instructions. Beyond the usual faffing around with permissions and DNS, I didn’t have any major problems. The GoToSocial documention is very comprehensive, and includes useful advice on things like setting up a firewall.

GoToSocial is a social network server based on the ActivityPub standard. It’s not the same as Mastodon, but because it supports the same standards, it interoperates with Mastodon. For example, once I had a @trovenewsbot account set up on wraggebots.net, I was able to use the standard migrate functions to move all of the bot’s existing followers to the new instance. Easy peasy!

The bot’s code makes use of the mastodon.py Python library, but again because of the similarities between the GoToSocial and Mastodon APIs, I was able to reuse the code with only minor changes. Specifically, I found that I had to add the parameter version_check_mode = "none" to the Mastodon client initialisation. The notifications_dismiss() method is not currently implemented in GoToSocial, so I also had to change the way I check for new notifications – saving the id of the last viewed notification, and then using it as the value for since_id when requesting a list of current notifications.

@trovenewsbot does a lot more than post random newspaper articles from Trove. By tooting keywords at it, you can search Trove from within the Fediverse. There are many options available for customising your queries, see the documentation for full details.

Six more volumes added to the searchable database of Tasmanian Post Office Directories!

Thu, 21 Nov 2024 16:05:37 +1100

A couple of months ago I realised my big, searchable database of Tasmanian Post Office Directories was missing the volume from 1920. It took a bit of work to add it in, as described in this post. Unfortunately, I’d barely finished when I realised that a number of other years were also missing! Argh! The good news is that I’ve been steadily working through these missing volumes, adding one a week, and now I’m finally, finally finished!

The new volumes are:

1920
1933-34
1941-42
1942-43
1943-44
1945-46

In total there are now 54 volumes from 1890 to 1948. Every line of every volume has been OCRd and indexed, so you can run fulltext searches across all 54 volumes to find matching entries. The fulltext search also supports advanced operators like wildcards and booleans.

As I mentioned in relation to 1920, while these volumes can be downloaded as PDFs from Libraries Tasmania, they don’t contain any OCRd text – they’re not searchable (despite what Libraries Tasmania says here). The quality of the scans is also quite variable – tight bindings cut off text, pages are skewed, and lighting is inconsistent. This means that the OCR processing is far from perfect. There will be names missing from the search index as a result of this. However, because you can search across all volumes at once, the database makes it easier to find people, as you can pick them up in one year and follow them through subsequent volumes, filling in any gaps.

It would be great if Libraries Tasmania would add a link to the database from their Directories and almanacs page. I’ve sent a couple of emails but haven’t received a reply. It seems odd that they’d link to commercial offerings like FindMyPast, but not to the free, community-developed version!

Where's 1920? Missing volume added to Tasmanian Post Office Directories!

Thu, 26 Sep 2024 13:58:54 +1100

Visualisation is a great way to find problems in your data.

As part of the Everyday Heritage project, I’m working with a team to document the lives of Tasmania’s Chinese residents in the 19th and early 20th centuries. We’re using a variety of sources such as Trove’s newspapers, the Tasmanian Names Index, and the Tasmanian Post Office Directories. To help with the research, I converted all the PDF volumes of the Post Office Directories into a public, online, searchable database. Or at least, I thought I had.

The Tasmanian Post Office Directories database embeds metadata about each line of text in its results, so it’s easy to save items of interest using Zotero. A member of our team has already saved hundreds of entries this way. The other day I started pulling these entries out of Zotero using its web API, and thought I’d get an overview by charting the number of results per year. It was then I noticed that 1920 was missing…

I checked the PDF volumes in Libraries Tasmania and the 1920 volume was there, so I worked back through my processing code to figure out why I’d missed it. It turns out the 1920 volume is named using a different pattern, and the regular expression I used to scrape the list of volumes was a little too specific. At least that was easy to rectify.

However, it wasn’t just a matter of feeding the 1920 volume through my processing notebooks, because the content of the 1920 PDF was also quite different to all the other volumes. Most of the PDFs made available from Libraries Tasmania have a single page per image, and the images have been pre-processed for OCR. The PDFs also include searchable, OCRd text. Here’s an example of one of the images from 1921:

The images in the 1920 volume are double page spreads, in colour, without any pre-processing. The PDF doesn’t include any OCRd text, so it’s not searchable. The quality of the images is also quite variable: tight bindings mean some text is cut off, pages are sometimes skewed, and bad lighting causes shadows across the right hand page – when converted to black and white for OCR, these shadows become black blobs that completely obscure the text. Here’s an example of one of the images from 1920:

All this meant I had to do a lot of additional processing of the images before I could extract useful text via OCR. Here’s a summary of the image pre-processing:

extracted all the images from the PDF
sliced all the images roughly in half and saved the results as separate files, so I had one image per page
manually cropped all 800 pages to get them as clean as possible
removed the shadows from the images thanks to this recipe in Stack Overflow
binarized and deskewed the images ready for OCR

From then on I could apply the processes in my existing notebooks:

OCRd the images using Tesseract
uploaded the cropped, colour images to the AWS bucket for delivery using serveless-IIIF
added the volume metadata and OCRd text to the Datasette database, generating a full-text index on the text column
updated the application on Google Cloudrun

I also took the chance to tweak the theme a bit, including a new dark mode.

The updated database is live, now containing 49 volumes from 1890 to 1948 including 1920!

Try it now!

UPDATE 27, September 2024

It seems I was too focused on the gap in 1920 and missed some other missing volumes from the 1930s and 40s. I’ve started processing these but it’s going to take a fair while to work through them all. I’ll add each volume to the database as it’s finished. Check here for regular updates.

Major update for the Trove Newspapers section of the GLAM Workbench

Mon, 23 Sep 2024 12:15:42 +1100

The Trove newspapers section of the GLAM Workbench was updated last week. Over the last year I’ve been gradually updating notebooks to use version 3 of the Trove API, but when version 2 suddenly disappeared a couple of weeks ago I had to hurriedly pull everything together. The Trove newspapers section includes 23 notebooks and 6 datasets, so it’s not a small job. The changes include:

updated all notebooks to use version 3 of the Trove API
removed remaining datasets from the code repository and created dedicated data repositories for them, integrating them with Zenodo where appropriate
added metadata to all the notebooks – this is used to build an RO-Crate metadata file for the code repository
updated all the Python packages
added a voila.json file to configure Voilá

None of the functionality of the notebooks should have changed. There’s a slight difference in the Finding non-English newspapers in Trove notebook because the language detection library I was using is no longer maintained. I’ve swapped in py3langid and it seems to work well, though the results are a little different. Interestingly, where the previous library thought that bad OCR was ‘Maltese’, the new one detects it as ‘Latin’! There’s no change to the list of newspapers with non-English language content detected by the notebook.

The documentation pages have also been updated. The notebook pages are now built using data from the code repository’s RO-Crate file. They also include embedded HTML previews of the notebooks. If a notebook generates visualisations, the visualisations are usually included in the HTML, so you can explore the outputs without running the notebook – see, for example, the charts in Visualise the total number of newspaper articles in Trove by year and state. Most of the dataset pages now include links to explore the contents using Datasette-Lite.

I still have to generate RO-Crate files for all the data repositories, but I wanted to get the code stuff finished first.

Preserving the history of online collections (my love letter to future historians)

Fri, 20 Sep 2024 16:18:34 +1100

It’s pretty obvious that access to digitised resources, like Trove’s newspapers, has changed the practice of history in Australia. But how? I’m certain that the historiographical implications of the growth and development of online collections will become a topic of increasing interest to historians, and that exploration of this topic will lead to important insights into the relationship between what we keep, what we value, and what we know. But for this to happen we need to have data documenting changes in online collections. What became available when? How was it delivered to users? How did the search indexing work?

In general, GLAM collection interfaces exist in an eternal present – they’re not good at explaining changes, or communicating their own histories. Australian GLAM organisations also share little statistical information. If you’re lucky, you might get something useful out of annual reports, but that’s about it. Trove, in fact, removed all their online collection statistics in the 2020 interface update. Web archives capture individual pages, but not complete systems. If we don’t document the shape and structure of online collections now, how will future historians understand their impact?

A couple of years ago I gave a short talk entitled ‘Living archaeologies of online collections’ for the Digital Preservation Coalition (video & slides).

In the talk, I described some of my piecemeal and inconsistent attempts to capture this history – starting back in 2011 when I first harvested data from Trove about the number of digitised newspaper articles. I’ve been continuing to create and update datasets, and have been trying to improve the way they’re organised and described – making them more FAIR – but I’ve still got a long way to go. At the moment there’s information spread across the GLAM Workbench, GitHub repositories, and Zenodo records, so I thought it might be useful if I pulled everything together into one big list. Hence this post. I often forget about what I’ve done in the past, so it’ll help me keep track of where the gaps are and what’s left to do. And hopefully it’ll encourage others to think about the significance and possibilities of this data, and perhaps share their own datasets.

There’s also a growing list of datasets in the Trove Historical Data community on Zenodo

I suppose I could also add the Trove Data Guide to this list. It’s not a dataset, but it is a snapshot of the current state of Trove. I’ll continue to update it, and these updates will themselves be saved as versions in GitHub and Zenodo, allowing future researchers to dig back through the layers.

A lot of what I do is focused on the present – building tools and resources that help researchers make use of GLAM collections right now. Those tools and resources will eventually decay as I shuffle off, as collections evolve, and as technologies change. But I’m hoping that these datasets will grow in value over time. I think it was Jason Scott who coined the phrase ‘metadata is a love letter to the future’. I suppose this is my love letter to future historians.

1. Trove zones, categories, and formats

trove-zone-totals

https://github.com/wragge/trove-zone-totals

This repository contains an automated git scraper that uses the Trove API to save data about the contents of Trove’s zones and categories. It runs every week and updates the following data files:

Total number of resources by Trove category, weekly updates, 13 June 2023 to present
Total number of resources by Trove category and format, weekly updates, 13 June 2023 to present

In the web interface, ‘zones’ were replaced by ‘categories’ in 2020. However, categories were not available through the Trove API until the release of version 3 in June 2023. To try and document the differences between zones and categories, totals from both were captured until version 2 of the API was switched off in September 2024.

Total number of resources by Trove zone, weekly updates, 9 March 2023 to 1 September 2024
Total number of resources by Trove zone and format, weekly updates, 9 March 2023 to 1 September 2024

2. Trove newspapers

trove-newspaper-totals-historical

https://github.com/wragge/trove-newspaper-totals-historical/

The files in this dataset were created at irregular intervals between 2011 and 2022 for use in visualising Trove’s newspaper corpus. Harvests from 2011 were screen scraped from the Trove website. Harvests after 2012 make use of the year and state facets from the Trove API. There are 9 versions:

12 April 2011
4 August 2011
12 September 2014
29 November 2015
14 December 2016
28 July 2019
10 July 2020
27 April 2021
21 January 2022

Each version includes two data files:

Total number of newspaper articles by year
Total number of newspaper articles by year and state

trove-newspaper-totals

https://github.com/wragge/trove-newspaper-totals

This repository contains an automated git scraper that uses the Trove API to save information about the number of digitised newspaper articles currently available through Trove. It runs every week and updates four data files:

Total number of newspaper articles by year, weekly updates, 19 April 2022 to present
Total number of newspaper articles by year and state, weekly updates, 19 April 2022 to present
Total number of articles by newspaper, weekly updates, 20 April 2022 to present
Total number of newspaper articles by category, weekly updates, 20 April 2022 to present

By retrieving all versions of these files from the commit history, you can analyse changes in Trove over time.

A weekly summary of the harvested data is presented in the Trove Newspapers Data Dashboard.

trove-newspaper-titles-web-archives

https://github.com/GLAM-Workbench/trove-newspaper-titles-web-archives

These datasets were created by harvesting information about newspaper titles in Trove from web archives. The harvesting method is documented by Gathering historical data about the addition of newspaper titles to Trove in the GLAM Workbench.

trove-newspapers-corrections

https://github.com/GLAM-Workbench/trove-newspapers-corrections

OCR errors in Trove’s digitised newspapers can be corrected by users. To help understand patterns in newspaper correction, this dataset has been created to record information about the number of articles with corrections. The data was extracted from the Trove API using this notebook.

Number of corrections by newspaper, 7 versions:
- 13 August 2019
- 10 July 2020
- 27 April 2021
- 21 January 2022
- 24 June 2022
- 14 September 2024
Number of corrections by year, 2 versions:
- 24 June 2024
- 14 September 2024
Number of corrections by category, 2 versions:
- 24 June 2024
- 14 September 2024

trove-newspapers-data-post-54

https://github.com/GLAM-Workbench/trove-newspapers-data-post-54/

Due to copyright restrictions, most of the digitised newspaper articles on Trove were published before 1955. However, some articles published after 1954 have been made available. This repository provides data about digitised newspapers in Trove that have articles available from after 1954 (the ‘copyright cliff of death’). The data was extracted from the Trove API using this notebook.

There are 8 versions of this dataset:

7 June 2019
12 August 2019
10 July 2020
11 November 2020
27 April 2021
21 January 2022
27 June 2024
14 September 2024

trove-newspapers-non-english

https://github.com/GLAM-Workbench/trove-newspapers-non-english

This dataset contains information about newspapers published in languages other than English that have been digitised and made available through Trove. Data about the languages present in newspapers was generated by harvesting a sample of articles from each newspaper using the Trove API, and then using language detection software on the OCRd text of each article. The method is documented in this notebook in the GLAM Workbench.

There are two data files:

There are two versions of this dataset:

9 July 2024
14 September 2024

trove-newspaper-issues

This dataset contains information about the published issues of newspapers digitised and made available through Trove. The data was harvested from the Trove API, using this notebook in the GLAM Workbench.

There are two data files in this dataset:

Total number of newspaper issues per year for each digitised newspaper
A complete list of newspaper issues available from Trove

There are 5 versions of this dataset:

18 October 2021
20 January 2022
3 August 2023
26 June 2024
13 September 2024

3. Trove lists and tags

trove-lists-metadata

https://github.com/GLAM-Workbench/trove-lists-metadata/

Trove users can create collections of resources using Trove’s ‘lists’. This dataset contains metadata describing all public lists, harvested via the Trove API. To reduce file size, the details of the resources collected by each list are not included, just the total number of resources. The data was extracted using this notebook from the Trove lists and tags section of the GLAM Workbench.

There are 4 versions of this dataset:

20 September 2018
22 September 2020
5 July 2022
6 June 2024

Public tags added to resources in Trove, 2008 to 2024

This dataset contains details of 2,495,958 unique public tags added to 10,403,650 resources in Trove between August 2008 and June 2024.

There are 3 versions of this dataset:

10 July 2021
6 July 2022
6 June 2024

Tag counts

This dataset was derived from the full harvest of Trove public tags. It contains a list of unique tags and the total number of resources in Trove each tag is attached to.

There are 3 versions of this dataset:

10 July 2021
6 July 2022
6 June 2024

4. Trove contributors

https://github.com/wragge/trove-contributor-totals

This repository contains an automated git scraper that uses the Trove API to save details of organisations and projects that contribute metadata to Trove. As well as counts of total resources by contributor, this dataset includes counts of resources from each contributor by format and category. It runs every week and updates the following data files:

Number of resources by contributor (unmodified JSON from API), weekly updates, 9 March 2023 to present
Number of resources by contributor (flattened data as CSV), weekly updates, 9 March 2023 to present
Number of resources by contributor and category, weekly updates, 21 June 2024 to present
Number of resources by contributor, category, and format, weekly updates, 21 June 2024 to present

These data files were generated using version 2 of the Trove API:

Number of resources by contributor and zone, weekly updates, 9 March 2023 to 12 May 2024
Number of resources by contributor, zone, and format, weekly updates, 9 March 2023 to 12 May 2024

5. Trove digitised resources (other than newspapers)

To help people find and use digitised resources other than newspapers in Trove, I’ve been harvesting, sharing, and visualising metadata relating to specific formats, such as books and periodicals. The methods I’ve used have changed over time, and there are some earlier versions that I still need to extract from the Git repositories, but these are the current datasets. I’m planning to set up automatic re-harvests for some or all of these, so there’ll be a better record of change over time.

There’s more information about these datasets in both the GLAM Workbench and the Trove Data Guide.

Books

https://github.com/GLAM-Workbench/trove-books-data

There are 2 versions of this dataset:

20 November 2023
14 February 2024

Periodicals

https://github.com/GLAM-Workbench/trove-periodicals-data

This dataset was created by checking, correcting, and enriching data about digitised periodicals obtained from the Trove API. Additional metadata describing periodical titles and issues was extracted from the Trove website and used to check the API results. Where titles were wrongly described as issues, and vice versa, the records were corrected. Additional descriptive metadata was also added into the records. Separate CSV formatted data files were created for titles and issues. Finally, the titles and issues data was loaded into an SQLite database for use with Datasette.

There are 4 data files in this repository:

There are two versions of this dataset:

29 February 2024
12 March 2024

Parliamentary Papers

https://github.com/GLAM-Workbench/trove-parliamentary-papers-data

This dataset contains metadata describing Commonwealth Parliamentary Papers that have been digitised and are made available through Trove.

There’s one version of this dataset:

23 February 2024

Maps

https://github.com/GLAM-Workbench/trove-maps-data

This dataset contains metadata describing digitised maps in Trove, harvested from the Trove API and other sources.

There are 2 data files in this dataset:

There are 2 versions of this dataset:

1 February 2023
8 June 2024

Oral histories

https://github.com/GLAM-Workbench/trove-oral-histories-data

There are 2 data files in this dataset:

There are 2 versions of this dataset:

16 November 2023
15 December 2024

Images

https://github.com/GLAM-Workbench/trove-images-rights-data/

This dataset includes information about the application of licences and rights statements to images by Trove contributors.

There are 2 data files in this repository:

There are 3 versions of this dataset:

17 February 2020
9 March 2022
24 April 2024

Finding aids

https://github.com/GLAM-Workbench/nla-finding-aids-data/

This repository contains data about the National Library of Australia’s digitised manuscript finding aids, harvested from Trove.

This dataset contains 2 data files:

There is one version of this dataset:

1 March 2023

6. Trove born digital resources

Pandora web archive collections

https://github.com/GLAM-Workbench/trove-web-archives-collections

This dataset contains details of the subject and collection groupings used by Pandora to organise archived web resource titles.

There are two data files in this dataset:

There are 2 versions of this dataset:

2 May 2024
7 May 2024

NED periodicals

https://github.com/GLAM-Workbench/trove-ned-periodicals-data

This dataset contains details of periodical titles and issues submitted to the Trove through the NLA’s National edeposit scheme.

There are 3 data files in this dataset:

There is one version of this dataset:

10 April 2024

7. National Archives of Australia

The NAA datasets are all over the place at present and I need to do a lot of work to get them standardised and organised. These are the main datasets, but there are others I need to add.

Records with the access status ‘Closed’

https://github.com/wragge/closed_access

https://github.com/GLAM-Workbench/recordsearch

Versions in Figshare:

Versions in GitHub:

January 2016
January 2017
January 2018
January 2019
January 2020
January 2021
January 2022
January 2023 (harvested but not in repo yet)
January 2024 (harvested but not in repo yet)

Summary data about all series in RecordSearch

https://github.com/GLAM-Workbench/recordsearch

CSV file containing basic descriptive information about all the series currently registered on RecordSearch as well as the total number of items described, digitised, and in each access category.

Recently digitised files

https://github.com/GLAM-Workbench/recordsearch

Details of files digitised between 25 February and 26 March 2020

Recently digitised files – weekly snapshots

https://github.com/wragge/naa-recently-digitised

This dataset contains weekly harvests of newly digitised files in RecordSearch. The automated scraper is currently scheduled to run each Sunday, saving a list of files that have been digitised in the previous week.

There are 177 data files, created between 28 March 2021 and the present.

Saving Trove's digitised periodicals as PDFs

Thu, 19 Sep 2024 13:46:22 +1100

I was recently contacted by a researcher who wanted to be able to automatically download the issues of a digitised periodical in Trove as PDFs. There was already a notebook in the GLAM Workbench that downloads the issues of a digitised newspaper as PDFs, but newspapers work differently to other digitised periodicals in Trove. While there was no corresponding notebook for other types of periodicals, all the necessary steps were documented in the Trove Data Guide, so it was just a matter of pulling together a few blocks of code.

There are three main steps:

get the nla.obj identifiers for all the periodical’s issues
get the number of pages in each issue
construct a url to download each issue as a PDF using the nla.obj identifier and the number of pages

Get issue identifiers

Version 3 of the Trove API added a new endpoint to provide information about periodical titles and issues. However, the issues data provided by the API is incomplete. A more reliable alternative is to scrape the list of issues from the browse window in the digitised object viewer – see HOW TO: Get a list of items from a digitised collection in the Trove Data Guide.

Get number of pages in each issue

It’s possible to scrape the number of pages along with the identifiers in the previous step. However, I’m not certain that the information is displayed consistently across all periodicals. To play it safe, you can extract embedded metadata from the digitised object viewer and get the number of pages, issue dates, and publication details (if available). See HOW TO: Extract additional metadata from the digitised resource viewer in the Trove Data Guide.

Download PDFs

Once you have an issue’s identifier and number of pages you can construct a url to download it as a PDF. See: HOW TO: Get text, images, and PDFs using Trove’s download link in the Trove Data Guide.

Putting it all together

It seemed like this would be useful to other researchers as well, so I’ve created a new notebook in the Trove Periodicals section of the GLAM Workbench that puts all of this together, see: Download issues of a periodical as PDFs.

The future (and past) of Historic Hansard

Thu, 29 Aug 2024 14:59:45 +1100

Don’t panic! Historic Hansard is not closing down – on the contrary, I’m planning a major update in the next few months. But as I look to the future, I thought it was a good time to pull together a few threads documenting my adventures with Commonwealth Hansard.

The past

Commonwealth Hansard is made available online through ParlInfo (there’s an alternative search interface here). The Parliamentary Library has invested a lot of time and effort in converting the printed volumes into nicely-structured XML files which break up the sitting day into debates and speeches, and identify individual speakers. For the most part, there’s one XML file for each sitting day in each house. However, there’s currently a gap between 1981 and 1997 where no XML files are available.

I started pulling data out of ParlInfo around 2011, but in 2016 I decided it would be more efficient to harvest all of the XML files into a dedicated repository. I started with the House of Representatives debates to 1965, and gradually expanded the coverage. The repository now contains all the Hansard XML files for both houses from 1901 to 1980, and 1998 to 2005. I stopped in 2005 because Open Australia provides access to the Hansard XML files from 2006 onwards.

The harvesting process revealed some interesting anomalies. In particular, I discovered that Parlinfo was missing Hansard from 94 sitting days. Most of the gaps were in the Senate between 1910 and 1920.

Fortunately the Parliamentary Library was quick to investigate the problem and fill the gaps. Over the years since, they’ve continued to improve the quality and accuracy of the XML files. Earlier this year I noticed that lots of new versions of the XML files were appearing and so I reharvested them all. It looks like the accuracy of the OCRd text has been improved and some structural issues fixed. This is one reason why a new version of Historic Hansard is needed!

If you’ve ever used ParlInfo you’ll know that while you can find things in Hansard, browsing and reading are difficult. There’s no easy way of just perusing the proceedings of a single day. A few months after I created the XML repository I decided to use the files to build Historic Hansard – ‘Commonwealth of Australia parliamentary debates presented in an easy-to-read format for historians and other lovers of political speech’. Historic Hansard is mostly just a static site, with one web page for each sitting day. Unlike ParlInfo, the focus is on reading, and you can view each speech within the context of the complete day’s proceedings.

You can browse Historic Hansard by house, parliament, year, and day. There are also indexes to the bills presented in the House of Representatives and the Senate, and pages for every person in the House of Representatives and the Senate with a complete list of their speeches. Because the focus was on browsing, Historic Hansard didn’t originally include a full text search function, but I eventually succumbed to user demand and added one in 2017. You can search for either debates or speeches, and download your results as a CSV file.

I also integrated Historic Hansard with Hypothes.is and Voyant Tools. Using Hypothes.is you can add notes and annotations to the speeches. You can even create deep links to fragments of text within a speech. I’ve often suggested that you could structure a whole undergraduate history unit around the annotation of a year of Hansard – identifying people and events, and finding and linking to related information. The Voyant Tools integration allowed you to analyse and visualise the language of a complete year of Hansard. Unfortunately I broke this at some point, so it’s on my list of things to fix in the new version!

In 2017, I did a bit of work with David Lowe at Deakin University to analyse the language of Hansard. Most of it was focused on the 1970s, but I did create word clouds using the top 200 TF-IDF weighted words for each decade and each parliament. In 2019, I compared the usage of the term ‘aliens’ in Hansard, newspapers, and The Bulletin.

One interesting feature of the Hansard XML is that it identifies interjections as well as formal speeches. In 2017, I extracted almost a million interjections from Hansard into a separate dataset. One of my favourite experiments used the interjections to reimagine political communication in the pre-internet era by transforming interjections into (fake) tweets. Of course I also started feeding the interjections through a text-to-speech processor, creating RoboHansard.

During the Real Face of White Australia transcribe-a-thon at Old Parliament House in 2017, I had the chance to take some of the interjections back to the place they were uttered. I set up speakers around the House of Representatives chamber and then started them hurling interjections about the White Australia Policy at each other. The drama (and spookiness) was heightened when a power failure turned off all the lights!

I summarised my adventures with Historic Hansard for a conference in South Africa in 2020.

I also wrote a piece on ‘The language of Parliament’ for the Museum of Australian Democracy which, unfortunately, seems to have disappeared during their latest site update. You can find it on Zenodo or in the Australian Web Archive.

In more recent times, I’ve integrated the XML harvesting code into the GLAM Workbench and added some Jupyter notebooks that demonstrate how you can access and analyse files from the repository.

The future

A number of historians have told me how much Historic Hansard has helped their research. A quick search in Google Scholar for “historichansard.net” returns 78 publications, many of which seem to include multiple links to specific sitting days. It seems I accidentally created a key piece of digital research infrastructure for Australian historians.

Given that, I’m not intending to change very much. My current plans include:

update all the content to use the latest versions of the XML
extend the date range to include files from 1998 to 2005
get the Voyant integration working again
make some back-end changes to the search application
fix various outdated links
I’d also like to see if I can clean up the Bills indexes a bit by merging together some of the titles, but that’ll require a bit of experimentation

Are there any improvements you’d like to see? If you have any suggestions, feel free to add an issue to the GitHub repository.

Historic Hansard is one of my passion projects. I haven’t received any funding to create or maintain it. Currently I estimate it costs me about AU$400 a year to run. It’s not much in the world of research infrastructure but, from a personal perspective, it all adds up. I’m committed to keeping Historic Hansard going, and it’s already outlasted some similar, well-funded projects, but if you’d like to help with the costs you can sponsor me on GitHub, or just buy me a coffee.

There seems to be a lot of interest in Hansard amongst digital humanities researchers at the moment, and there’s a couple of new projects starting up. It’ll be interesting to see where they go, but whatever happens, Historic Hansard will continue to serve lovers of political speech.

Join the Research Data Alliance's new Collections as Data Interest Group!

Tue, 27 Aug 2024 14:24:10 +1100

If you’re interested in opening up GLAM collections for use in research, you might like to join the new Collections as Data Interest Group, part of the Research Data Alliance.

According to the group description:

This group is aimed at collections professionals such as archivists, librarians, records managers and museum curators, as well as related professions such as IT professionals, knowledge scientists, and those involved in standards development, who serve in a range of critical roles: as experts in ensuring access, preservation, and reuse of digital records, objects, data, and collections; as provocateurs for good collections curation practices; and as advocates for the construction of responsible and sustainable infrastructures for information sharing.

The group has been running for a few months now, but communication has been difficult due to the upgrade of the RDA website. Things now seem to be working ok, so it’s time to spread the word!

The group has monthly online meetings. To try and cover a variety of timezones, we’re rotating the meeting times. The next meeting will be held in an Australia/Asia/Pacific timeslot – 3pm AEST (5.00am UTC), on Wednesday, 18 September. Further meeting details, including a registration link and agenda are available here.

To become part of the Collections as Data Interest Group, first register for membership of the Research Data Alliance. Once you’re logged into the RDA site, visit the Collections as Data IG page and click on the Join Group button. You’ll then receive notifications of new group posts and activities. You’ll also be able to access archived group meeting minutes under the ‘Wiki’ tab.

Current activities include planning for a session on ‘Gathering Metrics and Setting Boundaries: Reusing Collections as Data and the impacts of AI’ at the RDA’s 23rd Plenary Meeting in November. There’s also a Zotero group where we’ve started to capture useful resources.

Given my work on the GLAM Workbench and the Trove Data Guide, I’m particularly interested in ways we can collaborate to engage researchers, build tools, and share resources. But what are your interests? I’m the Australian-based co-chair, so let me know if there are topics you’d like to discuss in future meetings.

More datasets added to GLAM Name Index Search – now almost 12 million rows of data!

Mon, 26 Aug 2024 18:54:48 +1100

The GLAM Name Index Search brings datasets from 10 Australian GLAM organisations together into a single search interface. All these datasets index collections by people’s names, so with one search you can find information about individuals across a broad range of records, locations, and periods. It was created as an experiment during Family History Week in 2021, so I thought I’d update it for Family History Week 2024.

The update added 18 new datasets, so the GLAM Name Index Search now includes 279 datasets from 10 organisations – almost 12 million rows of data!

Start exploring now!

Most of the datasets come from government open data portals, so the GLAM Name Index Search is also a demonstration the value of data sharing. By making these datasets openly available, GLAM organisations support the development of new tools and resources that work across institutional boundaries.

As well as updating the datasets (details below), I took the chance to tweak the interface a bit. In particular, there’s now a new, svelte dark mode for people (like me) who struggle with screen glare.

The GLAM Name Index Search is created using Datasette with a custom theme. For more information on the how I compiled the data, see the GLAM data portals section of the GLAM Workbench.

New datasets

Queensland State Archives

Book of Trials 1835-1842 (created 2024-01-08)
Brisbane Gaol Prisoners 1850-1898 (created 2022-08-30)
Brisbane Passenger Arrivals 1913-1915 (created 2023-11-14)
Gympie Mining Claims 1868-1901 (created 2023-01-30)
Immigrant Remittances, IMA Maryborough 1864-1905. (S13055, S13060 & S13061) (created 2022-08-30)
Immigrant Remittances, Nanango 1863-1901 (created 2024-08-13)
Index to Deed Polls (Change of Name) 1889 - 1920.csv (created 2024-01-08)
Index to Immigrants and Missing Immigrants cards 1920-1945 (created 2024-01-08)
Index to Inquests 1859-1905 (digital) (created 2023-11-20)
Index to Register of Immigrants, Maryborough 1871-1915 (created 2023-11-17)
Land Orders 1861 - 1878 (created 2022-11-30)
Records on Frontier Wars and violence in Queensland (created 2023-05-29)
Reformatory School for Boys 1871-1906 (created 2022-09-05)
Teachers in the Education Office Gazettes 1926-1952 (created 2022-08-30)

NSW State Archives

Most of the NSW State Archives indexes are harvested from their web site. I last updated them about a year ago, but there have been some additions since then.

Assisted immigrants digitised shipping lists 1828-1896
Governor’s Court Case Papers
Small Debts Registers Index
Wages paid to orphans

Updated datasets

Libraries Tasmania

Bankruptcy - CSV (modified 2024-08-14)
Colonial Secretary correspondence - CSV (modified 2024-08-14)
Court - CSV (modified 2024-08-14)
Education - CSV (modified 2024-08-14)
Employment - CSV (modified 2024-08-14)
Hotels & Properties - CSV (modified 2024-08-14)
Land Records - CSV (modified 2024-08-14)
Miscellaneous - CSV (modified 2024-08-14)
Tasmanian Arrivals - CSV (modified 2024-08-14)
Tasmanian Births - CSV (modified 2024-08-14)
Tasmanian Census - CSV (modified 2024-08-14)
Tasmanian Convicts - CSV (modified 2024-08-14)
Tasmanian Convicts - permission to marry - CSV (modified 2024-08-14)
Tasmanian Deaths - CSV (modified 2024-08-14)
Tasmanian Departures - CSV (modified 2024-08-14)
Tasmanian Divorces - CSV (modified 2024-08-14)
Tasmanian Health & Welfare Records - CSV (modified 2024-08-14)
Tasmanian Immigration - CSV (modified 2024-08-14)
Tasmanian Inquests - CSV (modified 2024-08-14)
Tasmanian Marriages - CSV (modified 2024-08-14)
Tasmanian Naturalisations - CSV (modified 2024-08-14)
Tasmanian Prisoners - CSV (modified 2024-08-14)
Tasmanian Wills - CSV (modified 2024-08-14)
World War One Tasmanian Photographs - CSV (modified 2024-08-14)

NSW State Archives

Inquest index 1851, 1916-1963

Queensland State Archives

Assisted immigration 1848 to 1912 - combined (modified 2023-08-28)
Australian South Sea Islanders 1867 - 1948 (modified 2023-08-11)
Book of Trials 1835-1842 (modified 2024-07-12)
Brisbane Gaol Prisoners 1850-1898 (modified 2024-05-28)
Brisbane Passenger Arrivals 1913-1915 (modified 2024-07-12)
Civil servants 1851 - 1867 (modified 2023-02-10)
Consumptive patients 1897 to 1903 (modified 2023-07-24)
Criminal Depositions 1861-1900 (modified 2024-08-02)
Farm Lads 1922-1940 (modified 2023-10-24)
Female prisoners admitted, Toowoomba 1887-1891 (modified 2023-08-14)
Gympie Mining Claims 1868-1901 (modified 2023-01-30)
Immigrant Remittances, IMA Maryborough 1864-1905. (S13055, S13060 & S13061) (modified 2022-08-30)
Immigrant Remittances, Nanango 1863-1901 (modified 2024-08-20)
Immigrants 1909-1932 (modified 2024-06-13)
Immigrants and Crew 1860 - 1864 (modified 2024-06-12)
Immigrants landed Bowen 1888-1896 (modified 2024-06-13)
Immigrants nominated for passage, Maryborough 1884 to 1907 (modified 2023-02-10)
Immigration 1922 to 1940 (modified 2023-05-30)
Index to Deed Polls (Change of Name) 1889 - 1920.csv (modified 2024-01-08)
Index to Immigrants and Missing Immigrants cards 1920-1945 (modified 2024-02-15)
Index to Inquests 1859-1905 (digital) (modified 2024-07-09)
Index to Outdoor Relief 1892-1920 (modified 2023-08-07)
Index to Register of Cases and treatment at Moreton Bay Hospital 1830-1862 (modified 2023-02-13)
Index to Register of Aliens 1922-1923 - Sugar Exemptions (modified 2023-02-13)
Index to Register of Immigrants, Maryborough 1871-1915 (modified 2024-07-04)
Inquests 1859 to 1905 (non-digital) (modified 2024-07-09)
Justices of the Peace 1857 to 1957 (modified 2023-10-26)
Land Orders 1861 - 1878 (modified 2022-11-30)
Land Selections 1885-1981 (modified 2024-06-13)
Land orders 1861 to 1874 (modified 2022-11-30)
Land orders 1862 to 1878 (modified 2022-11-30)
Land orders 1865 to 1866 (Lands Dept) (modified 2023-02-10)
Mackay Hospital admissions 1891 to 1908 (modified 2023-07-24)
Miners rights 1874 to 1880 A-Z (modified 2022-09-06)
Nominated immigrants 1905-1928 (modified 2024-07-05)
Nurses examinations 1912 to 1925 (modified 2023-08-14)
Passport clearances 1923-1940 (modified 2024-02-16)
Prisoners admitted, Toowoomba 1895-1906 (modified 2023-08-11)
Prisoners discharged, Toowoomba 1869-1879 (modified 2023-07-31)
Prisoners tried, Toowoomba 1864-1903 (modified 2023-07-31)
Records on Frontier Wars and violence in Queensland (modified 2023-05-29)
Redeemed Land Orders 1860-1906 (modified 2024-06-13)
Reformatory School for Boys 1871-1906 (modified 2022-09-05)
Register of Court Fees Marburg 1885 to 1908 (modified 2023-09-05)
Register of Lands Sold 1842-1868 (modified 2022-08-30)
Register of immigrants 1864 to 1878 (modified 2023-09-01)
Register of immigrants, Toowoomba 1880 to 1888 (modified 2023-08-30)
Register of the Engagement of Immigrants at the Immigration Depot - Bowen 1873-1912 (modified 2023-09-05)
Registers of immigrants 1882 to 1938 combined (modified 2023-08-11)
Seamen 1882 to 1919 (modified 2022-11-30)
South African (Boer War) service records 1899-1902 (modified 2023-05-30)
South African (Boer) War Paybooks 1899-1902 (modified 2023-08-14)
Teachers in the Education Office Gazettes 1926-1952 (modified 2022-08-30)
Toowoomba Prisoners 1864-1906 (modified 2023-07-31)

State Library of Queensland

Queensland railway appointees 1890-1915 (modified 2024-07-25)
Queensland railway removals 1890-1915 (modified 2024-07-25)
Southern and Western Railway appointees 1866-1876 (modified 2024-07-25)
Southern and Western Railway removals 1866-1876 (modified 2024-07-25)

2023-05-08: Updated harvest of NSW State Archives indexes – more than 2 million rows of data!
2021-10-18: More GLAM Name Index updates from Queensland State Archives and SLWA
2021-09-29: More records for the GLAM Name Index Search
2021-08-23: A Family History Month experiment – search millions of name records from GLAM organisations

New Zotero translators for PROV and Queensland State Archives

Thu, 22 Aug 2024 14:35:55 +1100

Good news for Australian archives users – you can now use Zotero to capture item details and digitised files from the collections of the Public Record Office Victoria and the Queensland State Archives!

What is Zotero?

According to the Zotero website:

Zotero is a free, easy-to-use tool to help you collect, organize, annotate, cite, and share research.

While you can use it instead of commercial reference managers like EndNote, Zotero is much, much more. I use Zotero as my personal research database, capturing publications, websites, PDFs, as well as records from a growing number of GLAM collections. You can add tags and notes to items, organise them into collections, and annotate PDFs and website captures. You can also share your collections, create groups to collaborate with others, and access all your Zotero data via an API. And on top of all that, Zotero has just been completely revamped with a cool new interface (yay, dark mode!).

Extending Zotero

One of my favourite features is the ability for users to extend Zotero by creating new ‘translators’. Translators are little bits of Javascript code that capture information from a website and load it into Zotero. If there’s a online collection or database that doesn’t currently work with Zotero, you can write a translator for it and have it added into the main application. The Zotero translators repository currently contains more than 700 translators created by more than 200 contributors. That’s a pretty significant community effort!

And as of today, the repository includes translators for the PROV and Queensland State Archives collection databases.

Capturing PROV records

The PROV translator makes use of PROV’s collection API. It will capture item records from search results or individual item pages – just click on the Zotero icon in your browser’s toolbar. Here’s an example of the metadata Zotero captures from this record.

As well as expected details like ‘title’ and ‘date’, the translator saves the agencies that created the file as ‘contributors’. Some additional archival metadata such as the series details are included in the ‘Extra’ field.

If a file has been digitised, the translator also captures a number of ‘attachments’:

an image of the first page (particularly handy if the item is a photograph)
a link to a PDF version of the file
a link to the IIIF manifest describing this file

Why doesn’t it attach the complete PDF rather than just a link? The download PDF links in the PROV interface behave differently depending on the size of the file. If the file is small, the link loads the PDF as expected. But if it’s large you’re redirected to a web page that tells you that the PDF is being generated and could take up to an hour. There wasn’t a straightforward way for Zotero to handle these two possible outcomes, so instead of trying to attach the PDFs, I’ve saved the link. You can then click on the link to view/generate the PDF and add it manually to the record if you desire.

What’s an IIIF manifest? PROV delivers digitised images using the International Image Interoperability Framework (IIIF). An IIIF manifest is a standard way of describing a group of related images. PROV uses IIIF manifests to describe the contents of each digitised file. The link to the file manifest isn’t available through the PROV web interface, but the translator picks it up from the API and attaches it to the Zotero record.

Because manifests are based on the IIIF standard, they can be reused across different applications. For example, you could use the PROV manifest links to load a digitised file into Tropy, a desktop tool for managing and annotating images from the creators of Zotero. There’s an example of doing this with an IIIF manifest generated from Trove in the Trove Data Guide. You can also use the manifest links to download all the images from a file using a tool like IIIF Download.

Capturing Queensland State Archives records

The QSA translator will capture item records from search results or individual item pages – just click on the Zotero icon in your browser’s toolbar. Here’s an example of the metadata Zotero captures from this record.

As far as I can see, digitised files in the QSA collection will either be a PDF or an image. Either way, the translator captures the file and attaches it to the record.

Zotero and Australian GLAMs

Unfortunately there’s still a lot of work to be done before all Australian GLAM organisations are integrated with Zotero. Ideally, GLAM collection databases would support Zotero directly by embedding metadata in their web pages. This would make custom translators unnecessary (and support other types of integration as well). But few actually do this. When it comes to embedded metadata we seem to have gone backwards in recent years as systems have been ‘upgraded’.

The good news is that there are now custom translators for Libraries Tasmania, National Archives of Australia, Trove, National Gallery of Australia, State Records Office of WA, PROV, and Queensland State Archives. But translators need maintenance, and they’re often broken by site upgrades. Also some systems (particularly the large commercial ones) make it very difficult (if not impossible) to write useful translators.

If you’re interested in the current situation, I’ve created a spreadsheet documenting GLAM Zotero support. As I’ve suggested previously, a project aimed at improving Zotero integration across Australian GLAM organisations would make a big (and relatively cheap) contribution to Australia’s HASS research infrastructure.

Explore Trove's digitised maps

Fri, 16 Aug 2024 14:27:20 +1100

Trove contains thousands of digitised maps from the collections of the National Library of Australia, but they’re not always easy to find because of the way they’re arranged and described. To help you explore these maps I’ve created a new database and published it using Datasette.

Try it now!

To get started, head to the map sheets table and search for some keywords. The results are displayed both as a cluster map using Leaflet, and as a table. To view the details of an individual map sheet, click on the id value.

The individual record displays a zoomable version of the map image. If the record includes geospatial coordinates, these are also displayed on a modern basemap. There are also options to download the map image as a JPEG or a high-resolution TIFF (if available). Where possible I’ve also tried to link the Trove records to the NLA MapSearch interface.

To find map collections, you can either search in the collections table, or click on the collection link in a map sheet record to move up the descriptive hierarchy. A collection record includes links to browse the sub-collections and map sheets it contains.

About the interface

I usually share GLAM Workbench datasets using Datasette-Lite, but in this case I’ve created a full Datasette instance running on Google Cloudrun. This is because of the size of the database, and because I wanted to try out some plugins that don’t work in Datasette-Lite. In particular:

datasette-cluster-map – displays results on a cluster map using Leaflet
datasette-leaflet-geojson – converts GeoJSON strings in cells into Leaflet maps, this creates the location previews in each row and item view

About the data

The data was compiled using the method outlined in the Trove Data Guide. This involves:

harvesting all the work records from a search for maps
unpacking any versions grouped within the work records
using metadata embedded in the digitised object viewers to identify collections, harvest collection items, and enrich the records
merging duplicate values and records

In addition, I’ve added geospatial information if it’s available. If there’s a corresponding record in NLA’s MapSearch interface, I’ve linked the records and added the geospatial data from MapSearch. Otherwise I’ve looked for geospatial information in the item metadata and attempted to parse the coordinates.

The result is a flat structure where every row in the database represents a record with a unique nla.obj identifier. I’ve divided the rows into three tables according to their position in Trove’s descriptive hierarchy – collections, sub-collections, and map sheets. Collections have child records, but no parents as they’re at the top of the hierarchy. Sub-collections have parent collections as well as child records. Map sheets have no child records but can belong to collections or sub-collections. There are links within each record to help you navigate up and down this hierarchy.

You’ll notice some problems and limitations with the data, in particular, maps within collections sometimes inherit their metadata from their parent. This means that their geospatial coordinates refer to the whole series, rather than the individual sheet. There are also numerous errors in the geospatial coordinates parsed from metadata.

Coming soon

I originally harvested this data to try and help me understand the scope of the digitised maps in Trove. I’m currently adding a ‘Maps’ section to the Trove Data Guide in which I’ll provide an overview of what’s available, and document methods for accessing and using the data.

The notebooks used to create the dataset will be added soon to the Maps section of the GLAM Workbench. I’ll also be sharing CSV-formatted versions of all the data.

Share your spreadsheet as a searchable online database using Datasette-Lite

Fri, 19 Jul 2024 12:13:36 +1100

HASS researchers often compile data in spreadsheets. Sometimes they want to ‘publish’ this data online in a form that encourages others to use and explore – but how? I’ve just added a simple tool to the GLAM Workbench that helps you construct a url that will open a CSV file as a searchable database using Datasette-Lite.

What’s Datasette?

Datasette is a fantastic tool that helps you publish your data as an interactive website. There’s a few different varieties of Datasette, but Datasette-Lite is probably the easiest, as you don’t need to install any software. Datasette-Lite runs completely within your web browser, converting your data into into a searchable database on demand.

I’m using Datasette-Lite throughout the GLAM Workbench. For example, try clicking the Explore in Datasette buttons on any of these pages:

What do the buttons do? They’re simply links to my customised version of Datasette-Lite in GitHub. This links retrieve the Datasette-Lite web page, which instructs your browser to install the necessary software and run the Datasette application. Parameters in the url point Datasette to a specific data file in another GitHub repository. Datasette loads the data file and builds the interface.

Create your own Datasette-Lite links

All you need to publish your spreadsheet using Datasette-Lite is:

your spreadsheet saved in CSV format in a public GitHub repository
a url that points Datasette-Lite to the CSV file

To make this as easy as possible, I’ve created a tool that generates the necessary url. It’s just a simple web form – paste in the link to your CSV file, add some optional parameters, click the button, and the url to open your CSV file in Datasette-Lite will be displayed. Click on the url to open Datasette, or copy it to share your data with others.

The optional parameters let you index full text columns for easy keyword searching, hide unwanted columns, and add links to more information about your dataset. They’re described more fully in the documentation.

Limitations and alternatives

This tool makes it easy to share single CSV files as searchable databases. But if your CSV has millions of rows it’ll probably make your web browser unhappy if you try and load it in Datasette-Lite. Instead you can make use of Datasette’s ‘publish’ option to push your data to a dedicated Datasette instance running in the cloud. You can also customise your instance by changing the theme or using canned queries.

For example, the GLAM Name Index Search contains more than 10 millions rows of data from multiple datasets, all aggregated through a single search interface. The Tasmanian Post Office Directories provides full-text search across 48 digitised volumes and has a heavily customised interface that displays page images as well as the OCRd content. Both of these Datasette instances are hosted on Google Cloudrun.

Similarly, the method described here is designed to work with single CSV files. If you have multiple, interconnected tables you’ll probably want to generate your own SQLite database to use with Datasette. There’s an example of how you can do this in the Enrich the list of periodicals from the Trove API notebook in the GLAM Workbench.

Updated datasets describing Trove's digitised newspapers

Tue, 09 Jul 2024 17:05:03 +1100

The Trove newspapers section of the GLAM Workbench includes a number of notebooks and datasets that document the context and content of the newspaper corpus. I’ve just updated a few of these datasets:

I’ve also used the issues data to update my visualisation of the number of digitised newspaper issues in Trove published every day from 1803 to 2021 (there’s a lot of data so it can take a little while to load!).

The notebooks in the Trove newspapers section still need to be updated to work with version 3 of the Trove API. I’m part way through and should get it finished in the next few weeks. I’ll also be adding some more of this data into the Understanding the digitised newspapers section of the Trove Data Guide.

Understanding Trove at the AHA annual conference

Mon, 01 Jul 2024 21:36:41 +1100

A fairly intensive period of work came to an end today as I delivered a workshop on ‘Understanding Trove’ at the Australian Historical Association’s annual conference in Adelaide. In effect, the workshop was also the launch of the Trove Data Guide, which I’ve been developing as part of the ARDC’s Community Data Lab. The ARDC sponsored today’s workshop and has provided bursaries to help five ECRs and HDRs participate in the conference’s digital history stream.

Thanks to everyone who came to the workshop. It was great to have so much interest in developing a critical understanding of Trove and thinking about new research uses for Trove data. If you couldn’t make it, the slides are available below. Like the Trove Data Guide, the GLAM Workbench and pretty much everything else I do, the slides are openly licensed so feel free to share and reuse if any of it is useful to you.

Now I think I need a day off before I start thinking about the topics I’d still like to add to the Trove Data Guide…

Who is the Trove Data Guide for?

Fri, 21 Jun 2024 16:32:29 +1100

The Trove Data Guide aims to help researchers understand, access, and use data from Trove. But just because it’s about ‘data’ doesn’t mean you need to be able to code. To understand Trove data and its possibilities for research, you first need to understand Trove itself – its history, its structure, its assumptions, and its limits. This knowledge is useful to any Trove user.

For example, all Trove users would benefit from knowing more about works and versions, or how to use the ‘simple’ search box for complex queries. There’s also an introduction to what’s in (and not in) the digitised newspapers, and similar overviews for other digitised content such as books, parliamentary papers, and oral histories.

Number of newspapers articles in Trove by publication year, showing the change from 2011 to 2022

Several sections document the way the web interface works (or doesn’t work). There’s a field guide to the various interfaces and identifiers you might come across, and details of options for downloading data. The Trove Data Guide fills many gaps in the official Trove documentation, so check here if you run into problems, or can’t figure out how to achieve a particular task. Perhaps you were wondering how to download digitised images at their highest available resolution?

For people who are more comfortable with code there are plenty of useful snippets and complete working examples. For example there are sections that document how to get metadata, text, and images from newspapers, and other digitised resources. There are also a series of ‘HOW TO’ pages that describe more complex data access methods.

But what can you do with Trove data? The Trove Data Guide’s Pathways provide detailed tutorials that lead you step by step through examples of packaging Trove data for use with other digital tools. Use these examples as a starting point in planning your own projects.

Loading locations of Trove's digitised maps into the Gazetteer of Historical Australian Placenames

Tue, 18 Jun 2024 17:57:16 +1100

For this part of the ARDC’s Community Data Lab project, I’ve been focusing in particular on adding a series of researcher pathways to the Trove Data Guide. These pathways link data from Trove to a variety of tools and approaches and include five detailed tutorials. The first four were:

I’ve now added the fifth and final (for now) tutorial:

Create a layer in the Gazetteer of Historical Australian Placenames using metadata from Trove’s digitised maps

As has been the way in a lot of TDG development, this tutorial builds on and extends resources available through the GLAM Workbench. The Trove maps section of the GLAM Workbench already included a dataset of digitised maps and their coordinates, but for this tutorial I added a notebook that lets you create a subset of maps relating to a particular region. It does this by putting all the available map locations on world map. You then draw a rectangle on the map to select a region and display details of all the maps whose centre points fall within that region. It also displays links to download your new dataset as either a CSV or GeoJSON file.

The tutorial walks you through this process, then demonstrates how you can upload data from the CSV file to create a new layer in the Gazetteer of Historical Australian Placenames (GHAP).

This part of the Trove Data Guide project is now finished, but I’ll be continuing to add and refine content. If you have any suggestions for additional tutorials, feel free to add them to the ideas board (no promises though!).

Instant exhibitions with Trove and CollectionBuilder

Mon, 10 Jun 2024 22:38:04 +1100

You’ve been collecting and annotating items relating to your research project in a Trove List. You’d like to display the contents of your list as an online exhibition for others to explore. But how? One possible approach is now documented in the Trove Data Guide. I’ve added a tutorial which walks through the process of using a GLAM Workbench notebook to extract and process data from a Trove List, before uploading it to CollectionBuilder to create an instant exhibition.

CollectionBuilder creates online exhibitions using static web technologies. It provides a GitHub Pages template repository, so all you need to do to create an exhibition is upload your metadata and images to GitHub. The GLAM Workbench notebook gets your list data from the Trove API, and enriches it a bit to take advantage of CollectionBuilder’s built-in visualisations. For example, if there’s any digitised maps in your list, the notebook will try and extract their coordinates from the digitised map viewer and add them to the metadata so that CollectionViewer can display the location on a map. The notebook also downloads images of newspaper articles and other digitsed resources, and links them to the metadata, ready for upload.

Check out the tutorial: Sharing a Trove List as a CollectionBuilder exhibition

Keyword analysis of Trove newspapers with the GLAM Workbench & ATAP

Mon, 03 Jun 2024 16:06:44 +1100

There’s a new draft tutorial in the development version of the Trove Data Guide. It walks through the process of harvesting a collection of digitised newspaper articles from Trove, reshaping the harvest to create sub-collections, and then loading the data into the Keyword Analysis Tool provided by the Australian Text Analytics Platform (ATAP).

Along the way it goes into a fair bit of detail about constructing searches, using the Trove Newspaper Harvester, and thinking about your data. Much of the information on creating and reshaping datasets would apply to using the digitised newspapers with other analysis tools as well.

Running Mirador on GitHub Pages

Sun, 02 Jun 2024 22:27:40 +1100

I’ve just created a GitHub repository template that you can use to get your own Mirador version 3 installation running in minutes. You can also configure it to display local or remote IIIF manifests. I was thinking that it could be useful for researchers who want to create their own customised Mirador workspaces to examine a particular set of documents, but don’t want to install any software or fiddle about on the command-line.

I’ve been doing a lot with IIIF lately. First a GLAM Workbench notebook to save a collection of images from Trove as an IIIF manifest. Then a tutorial for the Trove Data Guide that walks through the whole process of generating an IIIF manifest from Trove, then loading the manifest into Tropy for analysis and annotation. I’ve also just about finished another tutorial that saves parts of a manuscript collection as IIIF manifests, then loads them into Mirador for side-by-side comparison.

In the new tutorial I was planning to use GitHub to put the manifests online so that they could then be displayed using the Mirador demo site. But then I thought if we’re saving the manifests to GitHub, why not use GitHub Pages to run a customised version of Mirador that can read the manifests from its own repository? There are various examples around of getting Mirador running on services like Netlify, but I couldn’t find a GitHub Pages example. So I decided to make my own.

The Mirador integration examples were useful in getting a basic set-up working. Then it was a matter of creating a GitHub action to generate the Mirador site any time the repository changed. As mentioned, I wanted users to be able to load local manifests into their Mirador workspace, so I wrote a little Python script to check the manifests directory for files, and copy the paths into the Mirador config. So all a researcher has to do is upload a manifest to the repository, then the deploy scripts rewrite the config and create a new version of the Mirador site automatically.

Similarly, if you want to include some remote manifests in your default workspace, it’s just a matter of adding the urls to the manifest_urls.txt file. The deploy scripts read the file and add the urls to the config.

I’ve included the Mirador Image Tools plugin in the default installation, but you can modify the template to add additional plugins if you want. With Mirador version 4 currently being tested, I’ll no doubt create a version 4 template before too long. Let me know if you find this useful!

Commonwealth Hansard XML repository updates

Sun, 26 May 2024 14:45:51 +1100

Hey Australian Hansard fans, I’ve done a complete reharvest of all of the Commonwealth Hansard XML files from 1901 to 1980 from ParlInfo. There’s been lots of improvements/corrections, and most of the file names have changed (they now have a version flag). The improvements seem to be ongoing, so I’ll try to harvest more regularly from now on. You can download the lot from the GitHub repository.

I still need to load the updated XML into the Historic Hansard site, but that’s going to have to wait for a month or two…

Trove to Tropy via IIIF – documenting data pathways in the Trove Data Guide

Tue, 21 May 2024 23:16:16 +1100

Last week I added a notebook to the GLAM Workbench that saves a collection of images from Trove as an IIIF manifest. This week I’ve written a tutorial that shows how you can use the notebook to load the collection data in Tropy – a desktop tool for managing and annotating images for research.

This is the first tutorial in the Trove Data Guide’s Research Pathways section. While most of the TDG documents the types of data available in Trove and how you can access it, the pathways aim to connect Trove data with other tools and platforms – to point at possibilities for analysis, enrichment, and sharing. For example, I’m planning tutorials on packaging OCRd text from Trove for use with text analysis tools, as well as ways of sharing selected data through tools like CollectionBuilder, and Datasette.

If you have any ideas for additional tutorials, feel free to add them to the ideas board in GitHub.

Using IIIF to explore Trove's digitised images

Thu, 16 May 2024 00:24:24 +1100

I’ve just added a new notebook to the Trove images section of the GLAM Workbench. It helps you save a collection of digitised images as an IIIF manifest. But what does that mean? It means the notebook packages up all the metadata describing the images in a standard form that can be used with a variety of IIIF-compliant tools. These tools let you do things with the collections that you can’t do in Trove’s own interface.

Perhaps you’d like to browse the complete digitised contents of Sir Edmund Barton’s manuscript collection, without all the back and forth and up and down navigation imposed by the Trove web interface. Here you go, thanks to IIIF!

What’s IIIF?

The International Image Interoperability Framework, more conveniently known as IIIF, develops open standards for sharing digital objects, such as images. IIIF platforms and standards are used by GLAM organisations around the world to deliver their image collections online.

Once you have standards for sharing image metadata, people can build tools that work across collections. For example, Universal Viewer and Mirador are both richly featured, open source, community developed image viewing platforms.

IIIF manifests are JSON files that describe a set of digital objects. They include technical information about the images and how to access them, as well as metadata describing their content and context. Everything you need to explore the images is packaged up in a single, standards based file. This means that if you point a manifest at tools like Universal Viewer and Mirador, they can present the images to users without any special configuration.

Why is the new notebook needed?

Unfortunately Trove doesn’t provide data using IIIF standards. Indeed, it doesn’t really supply any machine-readable data about the contents of digital collections. The notebook scrapes metadata from Trove’s digital collection viewer, reassembling it as a standard IIIF manifest.

What’s possible?

Here are a few more examples of manifests created with the new notebook.

Trove collection	manifest
B.A.N.Z. Antarctic Research Expedition photographs	nla.obj-141170265-v3-manifest.json	view in UV3	view in Mirador
The Papers of Sir Edmund Barton	nla.obj-224441684-v3-manifest.json	view in UV3	view in Mirador
Papers relating to the Federation Campaign (a single series from the Barton papers)	nla.obj-224441858-v3-manifest.json	view in UV3	view in Mirador
Postcard portraits of actresses, and of Australian towns, 1900s	nla.obj-224441858-v3-manifest.json	view in UV3	view in Mirador

I’m just linking to the standard demonstration versions of Universal Viewer and Mirador, but they can be extended with plugins that add more functionality, such as annotation. As demonstrated above, it’s also easy to embed the viewers in your own website.

Manifests can also be used to move data between programs. Tropy, for example, is a desktop tool for managing collections of images for research. It can import images and metadata from an IIIF manifest. So if you want to add your own notes and transcriptions to a digitised manuscript collection in Trove, you could save it as an IIIF manifest, then load it in Tropy.

I’m just about to start documenting some of these possibilities and pathways for the Trove Data Guide.

One more thing

While I was developing the notebook, I noticed another inconsistency in the way Trove’s digitised collections are arranged. This meant that in some cases my notebook to download all the images from a collection might not get every image. I’ve made some changes that should fix this and uploaded a new version of the notebook.

Using Pandora's collection of archived websites

Tue, 07 May 2024 17:23:39 +1100

There’s a brand new section of the GLAM Workbench to help you use data from Pandora’s collection of archived websites.

What’s Pandora?

Pandora is an initiative of the National Library of Australia which has been selecting web sites and online resources for preservation since 1996. It’s assembled a collection of more than 80,000 archived website titles, organised into subjects and collections. The archived websites are now part of the Australian Web Archive (AWA), which combines the selected titles with broader domain harvests, and is searchable through Trove.

Why is this needed?

The GLAM Workbench already has a Web Archives section that provides documentation, tools, and examples to help you work with data from a range of web archives, including the Australian Web Archive. But Pandora is unique to the NLA, and its curated collections offer a useful entry point for researchers trying to find web sites relating to particular topics or events.

Imagine you’re a researcher examining Australian election campaigns over the last decade. One thing you might want to do is analyse the language of campaign web sites. But how would you find them? You can search across the full text content of the Australian Web Archive in Trove, but you’d need some way of filtering the results to find sites of interest. Or you could just go the Elections category in Pandora.

Full-text search is great for some tasks, but carefully-curated collections with good-quality metadata can save us a lot of time and effort. Unfortunately, the Trove web interface prioritises search over Pandora’s collection metadata. If you head to Trove’s ‘Categories’ tab, you’ll find a link to Archived Webpage Collections. This collection hierarchy is basically the same as Pandora’s – combining Pandora’s subjects, subcategories, and collections into a single hierarchical structure. However, it only includes links to titles that are part of collections. This is important, as less than half of Pandora’s selected titles seem to be assigned to collections. Even stranger is the fact that I can’t find any link in Trove to the main Pandora site. This means that most researchers using the Australian Web Archive through Trove probably don’t even know that Pandora’s subject groupings exist!

For more on Pandora’s approach to describing collections see Creating Structure in Web Archives With Collections: Different Concepts From Web Archivists.

What’s in the new section?

The new Trove web archive collections (Pandora) section of the GLAM Workbench includes three notebooks and three datasets.

The main aim is help researchers assemble datasets of archived website urls based on Pandora’s subject groupings. So if, as described above, you want a list of websites associated with election campaigns you can go the Create title datasets from collections and subjects notebook and generate a CSV file containing 9,304 urls. Easy! The notebook also generates an RO-Crate metadata file capturing the context in which your dataset was created.

To make all that possible, I’ve harvested Pandora’s complete subject hierarchy and list of titles. The code to do this is in these notebooks:

And pre-harvested subject, collections, and titles datasets are here:

To give an overview of Pandora’s subject organisation, I’ve also created a single-page view of the complete hierarchy.

What do you do with a dataset of Archived website urls?

Once you have your own dataset of archived urls you can make use of the tools in the Web Archives section to gather more data for analysis. For example:

I’m hoping to explore some more of these possibilities in the Trove Data Guide, part of the ARDC’s Community Data Lab.

How to download all the images from a digitised collection in Trove (& learn some cool Trove tricks)

Thu, 25 Apr 2024 00:35:50 +1100

Digitised resources in Trove are sometimes grouped into collections – an album of photographs, a set of posters, a bundle of letters. I’ve just added a notebook to the GLAM Workbench that downloads all the images in a collection at the highest available resolution.

A sample of the 3,048 posters download from nla.obj-2590804313

Why is it necessary?

Trove’s digitised collection viewer includes a download option. But in most cases that seems to be limited to downloading 20 images at a time. Part of the reason for that is probably because the images are zipped up into a single file, which could get very large if 100, or 200 images were added.

Another limitation of the built-in download option is that the images are often fairly low resolution copies (many have a maximum width of 1000px). The quality of the images limits how you can use them.

For many research purposes, you’ll want a complete collection at the highest resolution possible. Trove makes that difficult. The new notebook makes it easy.

How does it work?

The Trove API is not much help. The only collections it knows about are articles in newspapers, and issues in periodicals. However, when you browse through a collection using the digitised collection viewer, a little internal API is called that delivers an HTML list of the next 20 items. By stepping through the collection page by page, you can eventually harvest details of all the items in a collection.

Once you have the nla.obj identifiers for each image, you can download a high-resolution version simply by adding /image to the url. Downloaded this way, the images generally seem to have a longest dimension equal to 5000px – a considerable improvement!

As well as downloading all the images, the notebook also generates an RO-Crate metadata file that describes the context of your harvest – when it was run, what collection you downloaded, what notebook you used, as well as the details of each image. This’ll help you in the future when you’ve forgotten where all the images came from!

Where can I learn all these Trove tricks?

This new notebook came about because I was documenting the method for harvesting collections in the Trove Data Guide. I realised that I needed to adapt my original code to work with complex collections that included multiple layers of nested sub-collections (like manuscript finding aids). Having done that, I thought it would be useful to create a working example in the GLAM Workbench.

The Trove Data Guide is being developed as part of the ARDC’s Community Lab. In it I’m trying to document as many of these problems and workarounds as I can to open Trove data to new research uses. Here, for example, is the page that discusses how to get a list of collection items, and here are some suggestions for downloading high-resolution images. One of my favourite discoveries from recent weeks was the internal API that delivers OCR layout information about book and periodical pages – find out how to save illustrations from books, visualise the layout of a periodical page, and even create some #redactionart poetry.

The content of the Trove Data Guide is changing and developing all the time. It’s all happening in the open, so feel free to explore the bleeding-edge development version or the latest published version. If there’s something you’d like to see, please post it on the GitHub ideas board.

Where is the new notebook?

The new notebook is part of the Trove images section in the GLAM Workbench. Because I was adding a new notebook, I also took the chance to update the whole repository. Changes include:

now using Trove API v3
now using Python 3.10
updated Python packages
now includes an ro-crate-metadata.json containing machine-readable metadata describing this repository

I also updated the datasets that provide information about the application of licences and rights statements to images by Trove contributors and moved them to their own GitHub repository.

What do you want to do with Trove data?

Thu, 18 Apr 2024 17:22:39 +1100

In my work on the Trove Data Guide I’ve started sketching out a series of research pathways. These are intended as ways of connecting Trove data to tools and questions – providing examples of the steps involved in gathering, preparing, and using data to explore particular research topics.

I’ve currently defined six pathways, roughly based on different types of data that you can get from Trove:

‘Creating collections’ is a bit different I suppose, as it’s meant to relate to the work of assembling research collections from data in Trove – for example, creating a collection of annotated newspaper articles in Omeka.

I have some ideas, of course, about the types of tutorials and examples to include in each pathway, but I’m wondering what you would like to see. What would you like to be able to do with Trove data?

You might get some inspiration by browsing through what’s already in the Trove Data Guide and the GLAM Workbench, or perhaps you have a research question that’s foundered because you couldn’t get the data you needed out of Trove. If you have any ideas please share them via the TDG’s ideas board. This is a chance to get some of your gnarly Trove data problems solved!

Note that the TDG links in this post go to the development version, which changes frequently. There is also a published version that doesn’t include the latest content.

Update! Saving Trove newspaper articles and pages as images

Thu, 18 Apr 2024 13:32:50 +1100

You probably know that when you select the Download as Image option for a digitised newspaper article in Trove what you get back is not actually an image – it’s an HTML document, in which the original image has been sliced up to try and fit on an A4 page when printed. So this article:

Ends up looking like this!!

So what do you do when you just want an image of an article as it appeared in the newspaper? Some years ago I figured out a workaround that involves scraping the OCR positional data that’s embedded in Trove’s newspaper viewer and cropping the article from a high-resolution image of the page. The method is documented in the GLAM Workbench and the Trove Data Guide, and I’ve packaged up the code in trove-newspapers-images so you can embed it in your own projects.

I also created a web app (using Jupyter and Voilá) to make it as simple as possible for people to download images from articles. Unlike most of the other notebooks in the GLAM Workbench which are spun up on demand, this web app was hosted on a constantly-running server. This made it faster to start and use, but it was relatively expensive, wasteful, and difficult to maintain. So I decided to make a change!

The new version of the Save Trove newspaper article as image web app is actually embedded with the GLAM Workbench. Behind the scenes, the page calls a AWS Lambda function which uses trove-newspapers-images to generate the image. So far it seems to be working pretty well. Try it now!

Even better, I’ve made some changes to image generation code to give users the option of masking the articles. The original version crops a rectangle from the page using the article coordinates. If an article extends over multiple columns with different lengths, the image will include content from neighbouring articles. It’s not a big problem, but it always annoyed me. Recently I realised that the solution was quite simple – instead of cropping one big box from the page, you can crop each individual OCR ‘zone’ and paste them into a new empty image with the same dimensions as the original. Once you’ve pasted all the zones, you crop the new page image using the article coordinates. Here’s an example of an article without masking:

And the same article with masking:

This enhancement has been pushed to the trove-newspapers-images package, and is available through the web app by simply checking the ‘mask image’ option.

Another frustrating feature of the Trove web interface is that there’s no way of saving a newspaper page as an image, only as a PDF. In this case the workaround is pretty simple, you just have to know the url pattern used to download page images. This is documented in the Trove Data Guide. Once again, I’ve been providing a web app to make this easy for users, and once again I’ve just updated it so that it’s embedded with the GLAM Workbench itself. Try it now!

Getting to know NED – born-digital periodicals in Trove

Thu, 11 Apr 2024 00:05:01 +1100

I spend a lot of my time trying to highlight the wealth of resources available through Trove – whether that’s 25,000 digitised Parliamentary Papers, 6,000 oral histories you can listen to online, or 3,471 full-page editorial cartoons from The Bulletin. Most recently I’ve been working on digitised periodicals, developing a new section for the Trove Data Guide. But as I was harvesting data about the 900 periodicals and 37,000 issues that had so far been digitised, I wondered about periodicals that were born digital – in particular, those that had been submitted to the National Library by publishers and authors through the National eDeposit Scheme (NED). It turns out, there’s a lot more than I realised.

I’ve added a new notebook to the Trove Periodicals section of the GLAM Workbench that harvests data about NED periodicals, and created a new dataset with lists of titles and issues. You can also explore the harvested data using Datasette-Lite. But here’s a quick overview.

There are at least 7,973 born-digital periodicals contributed through NED, comprising a total of 156,151 issues!

One of the 428 issues of the Palm Island Voice

What are they? Here’s the twenty titles with the most issues.

title_id	title	issues
nla.obj-1916881555	Western Australian government gazette.	1869
nla.obj-2940864261	The Australian Jewish News.	1067
nla.obj-2692666983	APSjobs-vacancies daily … daily gazette.	1043
nla.obj-2945379691	Tweed link	825
nla.obj-2541626239	Weekly notice	798
nla.obj-2940863963	The Australian Jewish News.	726
nla.obj-1252109725	Queensland Health services bulletin	700
nla.obj-1247944368	Hyden Karlgarin Householder News.	642
nla.obj-1775015332	E-record : your news from across the Archdioce…	640
nla.obj-638303044	Class ruling	580
nla.obj-2536144595	Plantagenet news.	574
nla.obj-1252305285	Clermont rag : Community newspaper.	514
nla.obj-2815835489	The Apollo Bay news.	513
nla.obj-1908935587	Assessment reports and exam papers	512
nla.obj-3125539859	The Peninsula community access news.	506
nla.obj-2859788676	Council news : weekly information from us to you	469
nla.obj-1252119874	Rot-Ayr-Ian [electronic resource] : the offici…	467
nla.obj-2994765231	Townsville Orchid Society Inc. bulletin.	442
nla.obj-1252246096	Palm Island Voice.	428
nla.obj-3267060622	News & views from George Cochrane.	399

These are born-digital, so they’re not images and OCRd text like the digitised periodicals and newspapers. Most of them are PDFs as we can see from the metadata.

format	number of issues
application/pdf	154,976
not specified	1,075
application/epub+zip	100

Not all NED periodicals can be viewed online. Publishers submitting periodicals through NED can place restrictions on access, specifying that that the publications can only be viewed on-site in a library. The three access categories are:

Unrestricted – you can view online and download
View Only – you can view online but not download
Onsite Only – you can only view when onsite at the designated libraries

Fortunately, the vast majority are Unrestricted.

Access status	Number of issues
Unrestricted	138,557
View Only	12,937
Onsite Only	4,657

One of the most important things about the digitised newspaper corpus is its diversity – it’s not just the metropolitan dailies, but many local, community, political, and religious newspapers as well. While local newspapers might be dying out in their traditional form, electronic publications are popping up. Look at the titles in the list above – the Apollo Bay News, the Palm Island Voice – while current historians mine the digitised newspapers for fragments of everyday life, future historians will be grateful for what’s being captured and preserved by NED.

But wait there’s more! Since 1996, the Australian Web Archive (previously Pandora) has been capturing online periodicals. My next task is to harvest some details of these as well.

More tools and data for working with Trove's digitised periodicals

Tue, 26 Mar 2024 15:35:02 +1100

The Trove Periodicals section of the GLAM Workbench has been updated! Some changes were necessary to make use of version 3 of the Trove API, but I’ve also taken the chance to reorganise things a bit – starting with the name. This section used to be called ‘Trove journals’, reflecting the naming of Trove’s ‘Journals’ zone. But zones have gone, and periodicals are now spread across multiple categories, so I thought a name change was necessary to better reflect the type of content being examined.

What periodicals have been digitised?

It’s surprising difficult to find out what periodicals have actually been digitised in Trove. There’s no straightforward list of titles as there is in the newspapers category. Over the years I’ve created a variety of lists and tools to try and overcome this. I’m now trying to consolidate these efforts into a single dataset which you can explore using Datasette-Lite. I’ve made a few improvements to this in recent weeks, in particular, title records now include a link to download all the OCRd text from periodical.

New notebooks

The notebook pages in the GLAM Workbench now include previews of the notebook’s content. There are a number of new notebooks:

Get details of periodicals from the /magazine/titles API endpoint – shows how you can get a list of titles from version 3 of the Trove API and explores some of the problems with the data
Enrich the list of periodicals from the Trove API – shows how to work around some of the problems with the titles data, adds some extra metadata, and generates the database described above
Harvest illustrations from periodicals – extract illustrations for periodical pages, issues, articles and searches using a OCR layout data

If you’d like an example of the sorts of illustrations you can extract from the digitised periodicals, here’s a collection of photos found by searching for periodical articles with cat or kitten in their titles.

Updated and reorganised datasets

I’ve moved all the datasets out of the main GitHub repository into their own separate repositories. Some large collections that were previously stored on the sadly-deceased Cloudstor service are now sitting in an Amazon s3 bucket. These include:

Details of digitised periodicals from the /magazine/titles API endpoint – these are the datasets created by harvesting and enriching titles and issues data from the Trove API
CSV formatted list of journals available from Trove in digital form – this is an update of an older dataset of titles created by searching for digitised works with the format Periodical
Editorial cartoons from The Bulletin, 1886 to 1952 – the cartoons haven’t been updated, but I’ve created a new metadata file and fixed up some problems with page numbering
OCRd text from Trove digitised journals – I’ve reharvested all of the OCRd text and made it available as individual zip files for each title, and one big zip file with everything!

As previously noted, I’ve also made the Bulletin cartoons available through Datasette-Lite for easy exploration.

A new way to explore editorial cartoons from The Bulletin

Tue, 19 Mar 2024 15:46:22 +1100

About five years ago I created a collection of full-page editorial cartoons from The Bulletin, harvested from Trove. Through a process that might be politely described as ‘iterative’, I fiddled with an assortment of queries and methods until I had at least one cartoon from every issue published between 4 September 1886 and 17 September 1952 – 3,471 cartoons in total. The details of the collection and how I created it are available in the Trove periodicals section of the GLAM Workbench.

Last night, as I was tidying up a new release of the Trove periodicals repository, I had a thought – why not put all of the details of the cartoons in a little database and make it available using Datasette-Lite for easy exploration? So I did.

Try it now!

One of the coolest new features is that I’ve harvested the OCRd text from each page containing a cartoon and created a full-text index. This means you can find cartoons by searching for words in their captions! Other features include embedded thumbnail images and links to download high-resolution versions of each page image.

In creating the database, I realised there were a few problems with the original metadata (dodgy page numbers), so I’ve fixed that up as well. I’ve also moved the mega zip download of every image (over 60gb) from the unfortunately deceased CloudStor service to AWS.

New GLAM Workbench section for working with government publications in Trove

Tue, 27 Feb 2024 14:34:18 +1100

The GLAM Workbench has a brand new section aimed at helping you find and use government publications in Trove. Most of the GLAM Workbench’s existing sections focus on a particular resource format, or are related to one of Trove’s top-level categories. This didn’t quite work for government publications, as things like Parliamentary Papers are spread across multiple categories, and can encompass a variety of formats. So I thought a new section was the best way of bringing it all together.

At the moment the Trove Government section includes two notebooks and three pre-harvested datasets.

It took a bit longer than I was originally expecting because I also made some changes in the way I store and display information about GLAM Workbench resources. You might notice, for example, that each of the datasets lives in its own separate GitHub repository, rather than being rolled together with the notebooks into one big repository. This makes it easier to manage and share information about individual datasets, and also trims down the size of the Docker images built from the code repository.

Each of these data and code repositories have their own machine-readable metadata following the RO-Crate standard. This continues work I’ve been doing with the ARDC Community Data Lab to describe GLAM Workbench resources and outputs using RO-Crate. Having this metadata in a standard format creates new possibilities for integration and automation. I’m now using the RO-Crate files to produce different, public-facing views of the resources they describe. The README files in each repository and all the GLAM Workbench pages in the Trove Government section are automatically generated from the RO-Crate data. In the latter case, I’ve extended my MkDocs setup using macros to pull in the RO-Crate JSON files and make the data available to the page templates. Connecting all the bits up took a lot of time, but I’m pretty happy with the result and will eventually extend this approach to the rest of the GLAM Workbench.

I also fiddled a bit with the way Jupyter notebooks are presented in the GLAM Workbench. The Trove Government pages include a notebook preview – basically an HTML rendering of the notebook in an iframe. This means you can browse the content of the notebook without having to do anything extra, or go anywhere else. In other sections you can view notebook content by following links to GitHub or NBViewer, but the embedded previews seem cleaner and more useful.

I’ve also changed the way options to run the notebook are presented. In the Trove Government section, these options are displayed as tabs beneath the preview – allowing you to choose, for example, between ARDC Binder and the public MyBinder service. In other sections I have a big blue button to launch the notebook using a specific service, with other options listed below. This new approach means I don’t have to prioritise one particular service – it’s left to the user to choose. It’s also expandable. In the future, I’m hoping to make some of the GLAM Workbench’s notebooks available using JupyterLite. As I do this, I can just add the JupyterLite option under another tab.

As with some other sections of the GLAM Workbench, the dataset pages are integrated with Datasette-Lite. If there’s a CSV file in the dataset, you’ll see a button to explore it using Datasette. For example, this link leads to a searchable database with details of 24,997 digitised Parliamentary Papers. That same dataset has also been used in the Trove Data Guide, to visualise Trove’s holdings of Parliamentary Papers. Yay for integration and reuse!

Digital history stream at AHA annual conference in July

Fri, 16 Feb 2024 11:44:32 +1100

This year the annual conference of the Australian Historical Association will include a digital history stream, sponsored by the Australian Research Data Commons (ARDC), and convened by me!

The call for papers is available here or through the Conference website. The list of possible topics is deliberately broad and inclusive – if you’re using digital tools or methods in the organisation, analysis, and visualisation of historical data we’d love to hear from you. Proposals are due on ~~23 February~~ 4 March and can be submitted through the Conference website.

We’re particularly keen for HDR and ECR scholars to be involved. To help meet registration and travel costs, the ARDC is funding up to four $1000 bursaries. More details are available here. Bursary applications close on 31 March.

There’s also likely to be a digital history workshop, as well as updates on the work of the HASS & Indigenous Research Data Commons and ARDC Community Data Lab, including the Trove Data Guide.

Time is short! Get your proposals in now! Contact me at tim@timsherratt.au if you have any questions.

Some recent presentations on the GLAM Workbench and Trove Data Guide

Tue, 13 Feb 2024 09:11:17 +1100

Last week I attended the ARDC Workshop on Repositories & Workspaces where I gave a quick intro to the GLAM Workbench and the Community Data Lab.

Then it was off to the ARDC HASS&I Research Data Commons Summer School where I explored some of the mysteries of Trove in a walk-through of the Trove Data Guide.

Exploring Trove’s digitised periodicals

Tue, 30 Jan 2024 22:53:14 +1100

While Trove’s digitised newspapers get all the attention, there are many other digitised periodicals to explore. But it’s not easy to find them from the Trove web interface – unlike the newspapers, there’s no list of digitised titles. So to help researchers find and use Trove’s digitised periodicals, I’ve created a searchable database using Datasette-Lite. Try it out!

Search for the titles of digitised periodicals.

View the details of an individual title (note the link to available issues at the bottom.

Browse a list of available issues.

The database currently contains details of 923 different titles, and over 37,000 individual issues. You can search for titles by keyword, then click through to view a full list of issues from a periodical. As well as basic descriptive metadata and links back to Trove, there’s a couple of other handy inclusions:

Titles include a ‘Search for articles in Trove’ link that opens up the Trove interface and pre-populates the search box with the title’s identifier. By adding some keywords you can search for articles within the publication.
Issues include a text_download_url link that downloads all the OCRd text from the issue.

Regular viewers might be thinking – wasn’t there already something like this? Yes indeed, for several years I’ve been maintaining the Trove Titles app, which provides a similar list. I’ve also provided harvests of OCRd text. So why the new database? First of all, I’ve harvested the data in a different way – making use of the new /magazine/titles API endpoint. This had several problems (see below), but I’m hoping that in the long term it will make updates easier.

Second, I’m exploring ways to make these sorts of resources available in a more sustainable way. The current Trove Titles app runs on the Heroku platform and there are costs associated with the app and the databases it uses. It just seems a bit silly for a relatively small amount of data. Datasette-Lite takes a very different approach – there’s no constantly running server, just a static site pointing at a dataset. All the magic happens within your browser!

I’ve written previously about how I’ve been customising Datasette-Lite for use within the GLAM Workbench, but I had to handle the periodicals data a bit differently. Because there’s a foreign key relationship between the titles and the issues (each issue is linked back to a title), I loaded the harvested data into a SQLite database (using sqlite-utils), defined the foreign key, and built a fulltext index on the periodical titles. Then I just saved the whole SQLite database to a GitHub repository and pointed Datasette-Lite at it.

I had to modify the GLAM Workbench template a bit to insert links back to the title when you view an individual issue. This happens automatically when you view a list of results, but not when you view an individual item. First I used the install parameter to tell Datasette-Lite to install the datasette-template-sql plugin. This plugin lets you run SQL queries within a template. Then I could run a query to see if there was a foreign key associated with the current item:

{% set fk = sql("SELECT * FROM pragma_foreign_key_list(?)", [table]) %}

If there is a foreign key, I run another query to get the title of the linked title:

{% if fk %}
    {% set flinks = sql("select title, " + fk.0.to + " from " + fk.0.table + " where id = ?", [display_rows.0[fk.0.from]]) %}
    {% set ftitle = flinks.0.title %}
{% endif %}

Then when rendering the column containing the linked value I can insert the title and a link:

{% if fk and cell.column == fk.0.from %}
	<a href="/{{database}}/{{fk.0.table}}/{{cell.value}}">{{ftitle}}</a> <em>{{cell.value}}</em>
{% else %}
	{{ row.display(cell.column) }}
{% endif %}

It seems to work ok, and doesn’t cause problems on databases where there’s no foreign keys.

I’m also using the datasette-json-html plugin to render the thumbnails. I’m also using the metadata parameter to point Datasette-Lite at a custom metadata file – this was primarily to define a custom sort order for the tables.

The data

I’ll write up more about the data and the harvesting process in coming weeks. There’ll also be a new section in the Trove Data Guide and some updated notebooks in the journals section of the GLAM Workbench. But a few notes about the /magazine/titles API endpoint:

there are a few hundred duplicate records – I’ve removed these from the dataset
the API doesn’t provide full information about issues, in particular undated issues are not returned – I’ve tried to fill these gaps
the data includes a thousand or more Parliamentary Papers – I’ve harvested these separately and thought it was best to exclude them from this dataset
some titles are actually nested collections, so the ‘issues’ are actually another level of title, while alternatively some titles are actually issues – I’ve tried to sort as much of this out as I can, but it gets confusing!

So I’m not confident that I’ve got everything, but I think it’s a useful start. I’ve reported the API problems to Trove but haven’t heard anything back yet.

The Trove Newspaper Data Dashboard now has an archive!

Mon, 15 Jan 2024 10:51:58 +1100

Since July 2022 I’ve been generating weekly snapshots of the contents of the Trove newspaper corpus. Every Sunday a new version of the Trove Newspaper Data Dashboard is created, highlighting what’s changed over the previous week, and visualising trends since April 2022 (when I first started regular data harvests).

All of the past versions of the dashboard are preserved in GitHub, but there wasn’t an easy way to browse them, until now. If you want to find out what changed in any week since July 2022, you can now visit the Trove Data Dashboard Archive and select a date from the list!

I created the archive by pulling all the versions from GitHub and saving them as individual files. I’ve also added some code to the weekly process that should automatically archive the past week from now on – we’ll see if it works next Sunday…

Customising Datasette-Lite to explore datasets in the GLAM Workbench

Fri, 12 Jan 2024 16:09:41 +1100

As well as tools and code, the GLAM Workbench includes a number of pre-harvested datasets for researchers to play with. But just including a link to a CSV file in GitHub or Zenodo isn’t very useful – it doesn’t help researchers understand what’s in the dataset, and why it might be useful. That’s why I’ve also started including links that open the CSV files in Datasette-Lite, enabling the contents to be searched, filtered, and faceted. Just look for the Explore in Datasette buttons!

Datasette is an excellent tool for sharing and exploring data. I’ve used it in a number of projects such as the GLAM Name Index Search and the Tasmanian Post Office Directories. Datasette-Lite is a version of Datasette that runs completely in the user’s web browser – no need for separate servers! All you do is point a Datasette-Lite Github repository at a publicly available CSV file, and it builds a searchable database in your browser. So instead of having to configure and maintain a series of servers running Datasette, I just have one static GitHub repository that only springs into action when needed.

For example, click this link to explore metadata describing oral history collections in Trove using Datasette-Lite.

I’ve made a few changes to the standard Datasette-Lite application for use with the GLAM Workbench. These are all included in the GLAM Workbench’s Datasette-Lite fork and described below.

Custom theme

I’d already created a custom Datasette theme for use in other projects. The question was how do I get it to work with Datasette-Lite? Just putting a templates folder in the repository wasn’t enough, as the virtual environment created within the browser doesn’t have direct access to all the files. I eventually figured out that I could zip up the templates folder, fetch the zip file using Javascript, and then unzip the folder into the browser’s virtual environment. This is the code in webworker.js that does all that:

let templateResponse = await fetch("templates.zip");
let templateBinary = await templateResponse.arrayBuffer();
pyodide.unpackArchive(templateBinary, "zip");

Then it’s just a matter of changing the Datasette initialisation to point to the templates directory:

ds = Datasette(
    names, 
    settings={
    	"num_sql_threads": 0, 
    	"truncate_cells_html": 100 # truncate cells
    }, 
    metadata=metadata, 
    template_dir="templates", # point to custom templates directory
    plugins_dir="plugins", # point to custom plugins directory
    memory=${settings.memory ? 'True' : 'False'}
)

As you can see, I’ve also added the "truncate_cells_html": 100 setting to truncate the contents of cells in the table view.

Custom plugins

Sometimes fields can contain multiple urls. While Datasette will make single urls clickable, multiple urls are just left as plain text. The datasette-multiline-links plugin fixes this for urls separated by line breaks, but I generally separate multiple values in CSV fields using the | character. It wasn’t hard to modify the plugin, but again it wasn’t clear how to make the modified plugin work with Datasette-Lite. You can use the install parameter to load plugins, but the plugins have to either be published in PyPI or available in GitHub as a Python wheel. That all seemed like overkill for my tiny plugin modification, but then I realised that I could use the same method as I was using for the custom template – zip, fetch, unzip, then point Datasette to the new plugins directory.

It also took me a while to figure out how to get the plugin to work nicely with the truncate_cells_html setting. Unless a cell-formatting plugin returns None, other cell format operations, such as truncation, aren’t applied. So I had to make sure that the plugin returned None if there were no urls in a cell.

Custom metadata

You can use the metadata parameter in Datasette-Lite to point to a metadata file in either JSON or YAML. I’ve added a custom metadata.json file to the GLAM Workbench repository, and adjusted the webworker.js code to load it by default.

Full text indexing

One really cool things about Datasette is the ability to run full text searches across specified columns. If Datasette detects a full text index, it automatically adds a keyword search box.

There wasn’t a way of adding full text indexes to CSV datasets in Datasette-Lite, so I added a new fts url parameter and used the value in webworker.js to modify the database using SQLite-utils.

fts_cols = ${JSON.stringify(settings.ftsCols || "")} 
try:
    db[bit].enable_fts(fts_cols.split(",")) # add full text indexes to columns
except sqlite3.OperationalError:
    print("Column not found")
    pass

For example, adding fts=title to a Datasette-Lite url will automatically add a full text index to the title column. You can also index multiple columns – just separate the column names with commas.

This url opens a CSV dataset with oral history metadata harvested from Trove and indexes the title and contributor columns: https://glam-workbench.net/datasette-lite/?csv=https://github.com/GLAM-Workbench/trove-oral-histories-data/blob/main/trove-oral-histories.csv&fts=title,contributor

Datasette converts your CSV file to a SQLite database, and SQLite supports a number of advanced search options. These options aren’t enabled by default in Datasette – you need to set searchmode to raw in the table metadata. To enable advanced searches, I’ve added a line in webworker.js to modify the default metadata:

metadata["databases"]["data"]["tables"][bit] = {"searchmode": "raw"}

Drop unwanted columns

Not all the columns in pre-harvested datasets are useful or interesting. To remove selected columns from Datasette-Lite, I added a drop url parameter. Once again, you can submit multiple values separated by commas.

In webworker.js the drop values are used with the SQLite-utils transform() function to remove the columns from the database.

drop_cols = ${JSON.stringify(settings.dropCols || "")}
db[bit].transform(drop=set(drop_cols.split(",")))

This url opens a CSV dataset with oral history metadata harvested from Trove and drops the publisher and work_type columns: https://glam-workbench.net/datasette-lite/?csv=https://github.com/GLAM-Workbench/trove-oral-histories-data/blob/main/trove-oral-histories.csv&drop=publisher,work_type

What’s going on?

Thu, 04 Jan 2024 18:21:41 +1100

The hardest part of developing tools and resources like the GLAM Workbench is getting information about them to the people who might benefit. The collapse of Twitter has only added to the difficulty, as has the reluctance of GLAM organisations to share new resources with their users. I’d rather spend my time making new tools, but what’s the point if no-one knows they exist?

Anyway, I thought I’d do a bit of a communications refresh for the new year. If you’re interested in GLAM Workbench and Trove Data Guide updates you can:

Keep an eye on the GLAM Workbench channel of my microblog (or add the feed to your RSS reader)
Follow the GLAM Workbench Facebook page for cross-posted updates from the RSS feed
Follow the GLAM Workbench LinkedIn page for cross-posted updates from the RSS feed

I’m also working on an email newsletter thing that’ll compile the updates at regular intervals.

For more social socials, as well as questions, requests and problems, you can always find me on Mastodon: @wragge@hcommons.social. My email address is not too hard to find, but, honestly, your chances of getting a reply are slim.

If you’ve got a bug report, or a suggestion for a new notebook or data source, feel free to create an issue on GitHub.

And, of course, everything I do is openly-licensed, so you are very welcome to modify and share! See the GLAM Workbench for other ways to get involved.

Exploring oral histories in Trove

Thu, 04 Jan 2024 12:20:30 +1100

The National Library of Australia holds over 55,000 hours of oral history and folklore recordings dating back to the 1950s. This collection is being made available online, and many recordings can now be listened to using Trove’s audio player.

However, the oral history collection is not easy to find in Trove. You need to go the ‘Music, Audio, & Video’ category and check the ‘Sound/Interview, lecture, talk’ format facet. To limit results to oral histories that have been digitised, you can add “nla.obj” to your query and set the ‘Access’ facet to ‘Online’. But what’s actually in the oral history collection and what can you do with it?

To help researchers explore and analyse the NLA’s oral history collection, I’ve added some notebooks to the Music, sound, and oral histories section of the GLAM Workbench:

Harvest oral histories metadata – harvests metadata describing the NLA’s oral history collection from Trove and saves the results as a CSV file
Save a list of oral history collections and projects – extracts a list of series from metadata describing oral histories held by the NLA and described in Trove
Download summaries and transcripts from oral histories – download all the available transcripts and summaries from digitised oral histories available in Trove

There’s also a couple of associated datasets:

The Trove Data Guide uses these datasets to create an overview of the collection. For example, here’s how the oral histories are distributed over time.

And here’s the top ten subjects of digitised oral histories.

subject	count
Painters – Australia – Interviews	193
Politicians – Australia	192
Prime ministers – Australia – Quotations	188
Older people – New South Wales – Biography	187
Menzies, Robert, Sir, 1894-1978. Speeches	185
Federal politicians	184
Politicians – Australia – Quotations	183
Australia – Politics and government – 1945-1965	172
Politicians – Australia – Interviews	171
Academics	126

The Trove Data Guide also includes information on the types of data from the oral histories and how you can access it.

Mapping MARC Geographic Area codes to Wikidata

Wed, 03 Jan 2024 18:03:48 +1100

Trove uses codes from the MARC Geographic Areas list to identify locations in metadata records. I couldn’t find any mappings of these codes to other sources of geospatial information, so I fired up OpenRefine and reconciled the geographic area names against Wikidata. Once I’d linked as many as possible, I copied additional information from Wikidata, such as ISO country codes, GeoNames identifiers, and geographic coordinates.

I’ve saved the resulting dataset in two formats – as a flattened CSV file (handy for loading as a dataframe), and as a JSON file that uses the geographic area codes as keys (handy for looking up values). You can download the datasets from this GitHub repository.

I’ve also written the codes back into the Wikidata records, so you can now find them with a SPARQL query like this.

The columns in the CSV file are:

code – MARC geographic areas code (without any trailing dashes)
place – name of geographic area from the MARC list
wikidata_label – name of geographic area from Wikidata
wikidata_id – Wikidata identifier
coordinates – pair of decimal coordinates in the form latitude,longitude (multiple values are pipe | separated)
iso_country_code – ISO two letter country code (multiple values are pipe | separated)
iso_numeric_country_code – ISO numeric country code (multiple values are pipe | separated)
geonames_id – GeoNames identifier (multiple values are pipe | separated)

Note that some fields can contain multiple values. For example the area Mediterranean Region is linked to 22 countries, so there will be multiple values in the ISO code fields.

For an example of this dataset in use, see Which countries do the oral histories relate to? in the Trove Data Guide.

National Archives of Australia in 2023 – digitisation of files

Wed, 03 Jan 2024 11:18:21 +1100

In 2023 the National Archives of Australia digitised 416,602 files (down from 575,597 in 2022). This chart shows the number of files digitised per day in 2023.

These files were drawn from 1,423 different series, but the vast bulk (81%) were from 4 series of World War Two service records. (This media release includes some details about the funding of the WW2 digitisation.)

Here’s the top twenty series by number of items digitised in 2023.

series	series_title	total
B883	Second Australian Imperial Force Personnel Dossiers, 1939-1947	201,511
A9301	RAAF Personnel files of Non-Commissioned Officers (NCOs) and other ranks, 1921-1948	111,673
A9300	RAAF Officers Personnel files, 1921-1948	14,125
B884	Citizen Military Forces Personnel Dossiers, 1939-1947	11,265
A14435	Stanley Fowler photographs showing the Australian fishing industry and coastline, numerical series with ‘LA’prefix	10,512
D3481	Photographs (black and white, colour) of buildings, installations, sites, etc	8,295
A1	Correspondence files, annual single number series [Main correspondence files series of the agency]	7,571
K1145	Certificates of Exemption from Dictation Test, annual certificate number order	4,169
A13150	Specifications, examiners reports and correspondence relating to the Registration of Victorian Patents - Second system	3,941
J853	Architectural plans, annual single number series with alpha (denoting Papua New Guinea and discipline) prefix and/or alpha/numeric (denoting size and amendment) suffix	3,322
A2571	Name Index Cards, Migrants Registration [Bonegilla]	2,204
D1423	Original plans (negatives), single number series with alpha prefix denoting discipline	2,102
AP67/1	Personal documents of British migrants (including ex-service) in receipt of free and assisted passages	2,058
E1652	Northern Territory Pastoral Applications (Pastoral Claims)	1,822
D5440	Photographs of post office buildings, personnel and equipment, single number series (with variations)	1,488
C609	Payment cards for employees' entitlements claims, alphabetical series	1,317
B3104	Photographs, Trans-Australian Railway, single number series	1,222
MP1117/2	Microfilm reels of RAAF Engineering Drawings	1,168
A2572	Name Index Cards, Migrants Registration [Bonegilla]	1,130
B6295	Photographs and negatives of Commonwealth building sites and Works departmental activities, single number series	1,113

For more data, see the naa-recently-digitised GitHub repository which runs a process every Sunday to save details of files digitised in the previous week.

Trove newspapers in 2023

Tue, 02 Jan 2024 17:56:58 +1100

I’ve been capturing weekly snapshots of the Trove newspaper corpus for the last couple of years. You can see the latest results in the Trove Newspaper Data Dashboard. Using this data I’ve compiled a quick summary of changes over the last year.

7,518,764 digitised newspaper articles were added to Trove in 2023. The total number of articles increased from 236,530,127 to 244,048,891. The chart below shows how the number of articles varied across the year. You’ll notice that the rate of digitisation increased about the same time the government announced new funding for Trove. Were more articles digitised because of the funding, or were articles in the digitisation pipeline held back until the funding was announced? Or both?

Most of the new articles were published in either Victoria or NSW – both these states had an increase of more than 3 million articles each! There were smaller increases for WA and SA. This chart shows the distribution of articles by state.

Fifty-seven new newspaper titles were added to Trove in 2023:

Trove Data Guide update – accessing data from newspapers and gazettes

Fri, 15 Sep 2023 15:53:20 +1100

I’m continuing to slog away at the Trove Data Guide (part of the ARDC’s HASS Community Data Lab) – dumping everything I know about Trove into a format that I hope will be useful for researchers.

I’ve just finished a first pass through the section on accessing data from newspapers and gazettes, and it’s online if you want to have a look. There’s still lots of things to add, update, and reorganise, but getting the basic content of the section defined is a bit of a milestone, so I’ll allow myself a little moment of celebration. Yay!

Of course it took longer than I expected, but that’s largely due to the fact that I was sketching out related sections as I went along. You’ll see lots of pages in the navigation that only contain a list of dot points, but they’ll get filled out over the next couple of months.

The ‘Accessing data’ section is going to be the most code heavy, as it’s focused on using the API to develop reusable methods for harvesting machine-readable data. Other sections, such as ‘Understanding search’ and ‘Collections and contexts’ will be more discursive, aimed at helping all Trove users better understand what Trove is and how it works.

Comments and suggestions are welcome! You can add issues on GitHub, or use Hypothes.is to annotate the text.

Some important updates for the Trove Newspaper & Gazette Harvester

Thu, 31 Aug 2023 18:00:54 +1100

Version 3 of the Trove API is out, and version 2 is scheduled to be decommissioned in early 2023 – that means I have a lot of code to update! First cab of the rank is the Trove Newspaper & Gazette Harvester with version 0.7.1 now available.

The Harvester is a Python package that can be used as either a library or a command-line tool. It’s been around in some form for more than 10 years. The latest updates include:

support for version 3 of the Trove API
automatic creation of a metadata file describing each harvest according to the RO-Crate format
automatic creation of a harvester config file, capturing the query parameters sent to Trove as well as the Harvester options
the ability to initiate a harvest from an existing config file
more memory-friendly generation of CSV result files (no loading everything into Pandas)

The RO-Crate integration was part of my work for the ARDC’s HASS Community Data Lab. The Harvester was already generating a simple metadata file that captured some of the harvest parameters, but now it documents the context of the harvest in much more detail, and saves it in a standard, Linked Open Data based, format.

Every harvest now creates an ro-crate-metadata.json file. This file includes details of the datasets created by the Harvester, such as the results.csv file that includes article metadata, and the text directory that contains the OCRd text. It also captures contextual information about the Harvester itself. The Harvester and the datasets are linked through a CreateAction that describes the harvesting process. The harvester_config.json file that saves the query parameters and Harvester options is also linked as an input to this process. In this way, all the components of the harvest are described and linked.

Here’s an example RO-Crate file.

Trove is changing all the time. By capturing information such as the query, the harvester version, the date, and the number of results, the RO-Crate file will help researchers document, manage, and share their research. And now that you can start a new harvest with an existing config file, it’s easy for researchers to re-run a harvest to see what changes over time.

As well as updating the Python package, I’ve also updated the Trove Newspaper & Gazette Harvester section of the GLAM Workbench. Here you’ll find examples of the Harvester in action, as well as some ways of exploring the harvested data. If you’d like to take the Harvester for a spin, the easiest way to start is with web app version – no software to install, no code to navigate! If you’re an Australian university researcher you can spin it up on the new ARDC Binder service in seconds.

Run GLAM Workbench notebooks on the ARDC’s new Binder service

Thu, 31 Aug 2023 13:52:11 +1100

There are a number of different ways to run the Jupyter notebooks in the GLAM Workbench depending on your needs and technical skills. But the easiest and quickest has always been the public, international Binder service, based in Europe. One click in the GLAM Workbench and Binder prepares a customised computing environment and loads up the Jupyter notebooks ready for you to explore. Unfortunately, the public Binder service has been having some capacity issues in the last few months, and sometimes repositories fail to run. The good news is that Australian university researchers now have another option with the launch of the Australian Research Data Commons Binder service!

The big difference between the ARDC’s Binder service and the international version is that you need to log in using your university credentials. While that’s an extra hassle, the service itself should be faster and more reliable for Australian researchers. For this reason, I’ve started making ARDC Binder links the default in a number of GLAM Workbench sections. Of course, not all GLAM Workbench users are attached to Australian universities, so the international Binder links remain – it’s just a matter of emphasis.

For example, near the top of many GLAM Workbench pages you’ll see Explore live on Binder buttons that launch the current repository. I’ve now added an Explore live on ARDC Binder option.

Most notebooks in the GLAM Workbench now have their own documentation page with a big blue button to launch the notebook on Binder. I’ve started changing these buttons to use the ARDC Binder service but, as you can see in the screenshot below, there’s also a link to run the notebook on the original Binder service, with no authentication required.

I’ve added some information on using the ARDC Binder service to the GLAM Workbench help pages.

I’ll be continuing to explore new options for running GLAM Workbench notebooks (I’m particularly interested in the possibilities of Jupyter Lite). Also the ARDC’s HASS Community Data Lab project is currently investigating ways of adding more authentication options to the Binder service to open it up to researchers outside of universities.

Trove Query Parser updated!

Sat, 26 Aug 2023 18:21:23 +1100

I’ve just updated the Trove Query Parser to work with version 3 of the Trove API. You just give it the url of a search in Trove’s newspapers, and it translates the search into a set of parameters that the API will understand. So this:

parse_query("https://trove.nla.gov.au/search/category/newspapers?keyword=wragge&l-artType=newspapers&l-state=Queensland&l-category=Article&l-illustrationType=Cartoon", 3)

Produces this:

{'q': 'wragge', 'l-artType': 'newspapers', 'l-state': ['Queensland'], 'l-category': ['Article'], 'l-illustrated': 'true', 'l-illustrationType': ['Cartoon'], 'category': 'newspaper'}

You can then feed the parameters to the Trove API with your API key and you’ll get data back. Easy! It’s simple but handy – I use the Query Parser in other tools like the Trove Newspaper Harvester and QueryPic.

This version adds a second parameter to the parse_query() function so you can specify the version of the Trove API you’re using. The default value is 2 for backwards compatibility. See the documentation for more information.

Family history resources in the GLAM Workbench

Fri, 18 Aug 2023 15:06:21 +1100

It’s Family History Month, so I thought a brief post was in order describing some of the family history related resources in the GLAM Workbench.

GLAM Name Index Search

This is the biggie (in more ways than one). I’ve brought 263 datasets from 10 Australian GLAM organisations together into a single search interface. All these datasets index collections by people’s names, so with one search you can find information about individuals across a broad range of records, locations, and periods. There’s more than 10 million rows of data to explore!

Try it now!

NSW Post Office and Sydney Telephone Directories

Many volumes of the NSW Post Office and Sydney Telephone Directories have been digitised and made available through Trove. However, they’re not easy to search. I’ve taken the text from these volumes and indexed it by line to make it easier to find people and places. There’s two databases to explore:

New South Wales Post Office Directories (54 volumes from 1886 to 1950)
Sydney Telephone Directories (44 volumes from 1926 to 1954)

Tasmanian Post Office Directories

The Tasmanian Post Office Directories have been digitised by Libraries Tasmania, but each volume is available as a separate PDF, making it difficult to search across the collection. I’ve downloaded all the PDFs, extracted the text and images, and indexed the content by line. Now you can search all 48 volumes, from 1890 to 1948, at once!

Try it now!

Trove Places

If your family history research is focused on a particular region, it can be useful to know what newspapers from that region are digitised in Trove. Trove Places is a map interface to Trove’s digitised newspapers, just click on a place to find newspapers published or distributed nearby.

Try it now!

Save a Trove newspaper article as an image

You might have noticed that Trove’s download option for digitised newspaper articles can slice articles up in unfortunate ways. This simple web app saves the complete article as a single image (or multiple images if the article is split across different pages). Simple, but very useful.

Try it now!

Trove Newspapers Data Dashboard

Just about every week more digitised newspaper articles are added to Trove. The search you tried a couple of months ago might now produce different results. How can you keep track of these changes? The Trove Data Dashboard uses weekly snapshots of the digitised newspapers to visualise changes over time. It’s updated every Sunday!

Try it now!

And more!

The GLAM Workbench provides a wide range of tools and examples to help you work with data from libraries, archives, and museums.

The resources above cost money to keep online. If you find them useful, you might like to sponsor me on GitHub or buy me a coffee.

2024 update!

If you want to know more about using Trove, head to the new Trove Data Guide. The Guide explores the different types of data available from Trove – what it is, how you can access it, and what you can do with it. It aims to give users a critical understanding of Trove data, both its limits and its possibilities.

Exploring the front pages of newspapers (10 years on)

Tue, 08 Aug 2023 17:28:18 +1100

Way back in 2012, I used the brand new Trove API to download the details of 4 million articles published on the front pages of newspapers. I did it for two reasons: first, I wanted to see how the content of front pages changed over time; and second, I wanted to show that large-scale data wrangling was entirely possible using nothing more than a laptop and a home broadband connection. I described my adventures in this blog post, but if you look at it now you’ll see lots of sad, empty boxes where live charts used to be. This is because I shared my results though a custom web application which, 10 years later, seems like a really, really dumb idea. Needless to say the app fell foul of web hosting changes and is no more. Nowadays I use GitHub and Jupyter notebooks to share my data noodling, so I thought it was time to revisit the topic of newspaper front pages, and create something a bit more robust.

If you just want the data and code, head over to the GitHub repository. I’ve shared the notebooks used to harvest, convert, and explore the data, as well as two parquet formatted datasets.

As you might expect, there are a lot more articles now, as new articles and newspapers are added to Trove every week (see the Trove Newspaper Data Dashboard for details). Instead of 4 million articles, I ended up downloading details of more than 19 million articles! I used my trusty Trove Newspaper Harvester as a library so I could more easily manage the way the data was saved. After several days I had a 14.6gb newline-delimited JSON file. I pared the data down to only the necessary fields, and used DuckDB to create two parquet files – one with the article data, and another that added up the number of words on each page in the different article categories.

One nifty thing about the Trove API is that it tells you the number of words in each newspaper article. Articles are also assigned to a series of categories, such as ‘Article’ (your standard news-type piece) and ‘Advertising’. By adding up the number of words in each category I could explore how the front page’s mix of articles and advertising changed over time. Here, for example, is what happened to the front page of the Sydney Morning Herald.

The first chart shows the average number of words per page in the ‘Article’ and ‘Advertising’ categories across the full span of the SMH’s publication run digitised in Trove (advertising is blue, and articles orange). The second focuses in on the point where the number of words in articles overtakes the number of words in advertisements. You can see that for most of the publication run, the front page was dominated by advertising. But when change came it was quite abrupt. Sydney-siders awoke on 15 April 1944 to a new look newspaper. Here’s what the front page looked like at the beginning and end of the period represented by the second chart.

Different newspapers have different histories. I’ve created a notebook that generates more of these visualisations for a range of newspapers, and tries to provide a bit more context around the changes taking place.

As I continue my work on the Trove Data Guide for the HASS Community Data Lab, I’m discovering more and more inconsistencies, oddities, and undocumented possibilities. As I was finishing up the work on front pages, I realised there was another way of exploring the same topic – by looking at the space articles take up. This data isn’t in the API, but it can be scraped from the web site. Hmmm, interesting…

Trove API Console updates

Tue, 18 Jul 2023 12:27:11 +1100

The Trove API Console provides examples of the Trove API in action that you can run, edit, and share. It’s been online for 9 years now, and I’ve just updated it to use version 3 of the Trove API by default. I’ve also added a new ‘Share’ button that makes it easier to share and embed examples.

If you click on the ‘Share’ button, a box will pop up.

If you add a comment, this will appear above the example query when users follow the shared link. You can use this to provide them with some context or a description.

There are two buttons providing different share options:

Copy share url – copies the full url to the shared example
Copy Markdown button – copies a Markdown snippet that embeds a button like this linked to your example. Just paste it into your Markdown-formatted documentation!

Other improvements include:

The parameters used in any request are now displayed in a table for easy reference.
The Console includes a link to an API Status page. This page runs all the example queries in the Console and checks the results to make sure the API is working as expected. It’s updated every 6 hours.

Version 3 of the Trove API includes a standard Swagger UI that you can use to build queries, and provides limited anonymous access (without the need for an API key). But if, like me, you learn best by looking at examples, then you should find the API Console handy. In particular, the API Console makes it easy to share live examples which can be very useful in training, troubleshooting, and writing documentation. The Trove Data Guide, which I’m working on at the moment, includes lots of ‘Try it!’ buttons.

Updated harvest of NSW State Archives indexes – more than 2 million rows of data!

Mon, 08 May 2023 13:25:09 +1100

The NSW State Archives (now part of Museums of History NSW) publishes a series of useful indexes to its collections. The indexes include basic data transcribed from the records, such as names, dates, and places, providing fine-grained access to the collections. But when they’re explored as data, the indexes also suggest new ways of analysing, visualising, and linking sets of records. (For some of the possibilities and challenges of using this sort of data see Missing Links: Data Stories from the Archive of British Settler Colonial Citizenship).

In 2016, I started harvesting the index data from the NSW State Archives website to make each dataset available as an easily downloadable CSV file. In 2019, changes to the website made it impossible to access the complete indexes, so I was unable to update the CSV files. Fortunately the website has changed again, and I’ve been able to reharvest all the indexes, capturing any changes since 2019. There are currently 75 indexes containing 2,481,881 rows of data.

Title	Number of rows	Download data	View at State Archives
Aboriginal People in the Register of Aboriginal Reserves 1875-1904	78	CSV file	Browse index
Assisted Immigrants Index 1839-1896	200,000	CSV file	Browse index
Australian Railway Supply Detachment 1914	65	CSV file	Browse index
Bankruptcy index 1888-1929	30,000	CSV file	Browse index
Bench of Magistrates Index 1788-1820	4,442	CSV file	Browse index
Botanic Gardens and government domains employees	916	CSV file	Browse index
Bubonic plague index 1900-1908	567	CSV file	Browse index
Census - 1841	9,355	CSV file	Browse index
Chemists, druggists and pharmacists index 1876-1920	2,967	CSV file	Browse index
Child care and protection index 1817-1942	21,292	CSV file	Browse index
Colonial (Government) Architect index 1837-1970	2,373	CSV file	Browse index
Colonial Secretary Letters Received, 1826-1896	205,863	CSV file	Browse index
Colonial Secretary’s Papers 1788-1825	144,572	CSV file	Browse index
Colonial Secretary’s letters relating to land 1826-1856	20,000	CSV file	Browse index
Colonial Secretary’s main series of letters received	7,638	CSV file	Browse index
Convict assignments index 1821-1825	6,156	CSV file	Browse index
Convict exiles index 1849-1850	3,004	CSV file	Browse index
Convict indents (digitised) index 1788-1801	20,000	CSV file	Browse index
Convicts applications to marry 1825-1851	14,327	CSV file	Browse index
Convicts index 1791-1873	150,000	CSV file	Browse index
Coroners' inquests index 1796-1824	808	CSV file	Browse index
Court of Civil Jurisdiction index 1799-1814	2,876	CSV file	Browse index
Court of Claims (Land) index 1833-1922	2,966	CSV file	Browse index
Crew and passengers 1828-1841	2,560	CSV file	Browse index
Criminal court records index 1788-1833	5,028	CSV file	Browse index
Criminal depositions (Deposition Books) index 1849-1949	117,508	CSV file	Browse index
Criminal indictments index 1863-1919	20,000	CSV file	Browse index
Deceased estates index 1880-1958	577,891	CSV file	Browse index
Depasturing licenses index 1837-1851	7,449	CSV file	Browse index
Dependent children registers 1883-1923	28,910	CSV file	Browse index
Devonshire Street Cemetery reinterment index	9,559	CSV file	Browse index
Divorce records index 1873-1923	21,239	CSV file	Browse index
Fire Commissioners Personnel	3,767	CSV file	Browse index
Gaol inmates & prisoners photos index 1870-1930	52,055	CSV file	Browse index
Gold (auriferous) lease registers 1874-1953	60,000	CSV file	Browse index
Indigenous colonial court cases 1788-1838	65	CSV file	Browse index
Infirm & destitute (Government) asylums index 1880-1896	20,000	CSV file	Browse index
Inquest index 1942-1963	45,547	CSV file	Browse index
Insolvency index 1842-1887	23,108	CSV file	Browse index
Intestate estates index 1821-1913	30,000	CSV file	Browse index
Land grants and leases (registers) 1792-1865	5,627	CSV file	Browse index
Letters re migration to NSW 1838-1857	22,771	CSV file	Browse index
Maintenance registers - Metropolitan Children’s Court 1915-1917	1,372	CSV file	Browse index
Miscellaneous immigrants index 1828-1843	8,821	CSV file	Browse index
NSW Government employees granted military leave	20,000	CSV file	Browse index
NSW King’s / Queen’s Counsel appointment correspondence	2,083	CSV file	Browse index
Naturalization index 1834-1903	9,860	CSV file	Browse index
Nominal Roll of the First Railway Section (AIF)	416	CSV file	Browse index
Norfolk Island special bundles index 1794-1813	216	CSV file	Browse index
Nurses index 1926-1954	46,499	CSV file	Browse index
Police service registers 1852-1913	20,000	CSV file	Browse index
Port Macquarie Small Debts Register, 1845-1887	2,036	CSV file	Browse index
Probate records - supplementary index 1790-1875	1,626	CSV file	Browse index
Public Works Salary Registers	523	CSV file	Browse index
Publicans' licenses index 1830-1861	20,000	CSV file	Browse index
Quarter sessions cases 1824-1837	6,232	CSV file	Browse index
Railway employment records 1856-1917	763	CSV file	Browse index
Railways and Tramways Roll of Honour	1,214	CSV file	Browse index
Register of Firms index 1903-1922	50,000	CSV file	Browse index
School teachers' rolls 1869-1908	20,000	CSV file	Browse index
Schools and related records 1876-1979	30,181	CSV file	Browse index
Soldier (Closer) Settlement - Returned Soldiers Transfer files 1907-1951	9,656	CSV file	Browse index
Soldier (Closer) Settlement transfer registers 1919-1925	4,957	CSV file	Browse index
Soldier (Closer) settlement promotion files index 1913-1958	4,354	CSV file	Browse index
Soldier Settlement loan files index 1906-1960	7,642	CSV file	Browse index
Soldier Settlement miscellaneous files index 1916	1,050	CSV file	Browse index
Soldier Settlement purchases index 1905-1937	9,776	CSV file	Browse index
Squatters and graziers index 1837-1849	9,003	CSV file	Browse index
Surveyor General’s crown plans 1792-1886	5,455	CSV file	Browse index
Surveyors' field books 1794-1860	813	CSV file	Browse index
Surveyors’ letters 1822-1855	157	CSV file	Browse index
Tramway employees 1879-1911	10,606	CSV file	Browse index
Unassisted immigrants index 1842-1855	140,000	CSV file	Browse index
Unemployed in Sydney 1866	3,222	CSV file	Browse index
Vessels arrived in Sydney 1837-1925	129,999	CSV file	Browse index

The list below is also available as a CSV file. The complete repository of CSV files is available on GitHub, and the methods used to harvest the data are documented in the GLAM Workbench.

But what’s in all these wonderfully-rich datasets? You can find out using the Index Explorer in the GLAM Workbench. Just select an index from the dropdown list, and the Index Explorer will analyse every column, summarising the contents and building visualisations to help you understand the range of values.

Data from the NSW State Archives Indexes is also included in the GLAM Name Index Search, which helps you find people in 263 different indexes from 10 Australian GLAM organisations.

A big milestone, Trove contributor data, and the coming of API v3 – recent GLAM Workbench updates

Fri, 24 Mar 2023 12:40:58 +1100

There have been quite a few GLAM Workbench updates over the last month, here’s some notes. (See February’s update for more recent changes…)

General developments

After many months of work, all thirteen Trove repositories within the GLAM Workbench have been updated to include standard configurations, integrations, and basic tests. This will make ongoing development and maintenance much easier. Docker images of every repository are now built automatically whenever the code changes. These images can be used across multiple computing environments, including cloud services such as Binder, Nectar, and Reclaim Cloud, as well on a local computer. This means users have more options for running the notebooks within a consistent, pre-configured, and tested environment.
With all the Trove repositories now Docker-ised, I worked with the Nectar Cloud team to update the GLAM Workbench app. Now when you install the app, you can select from any of the Trove repositories (excluding API intro and Random items), as well as the NAA RecordSearch, Web Archives, Digital NZ and Te Papa repositories.
Datasette Lite integration! Datasette Lite turns CSV and JSON files into fully searchable databases running within your browser (no server required). I spent some time creating a customised version of Datasette Lite for the GLAM Workbench. Now all I can just point the urls for CSV datasets to my Datasette Lite repository to open them up for quick exploration – like this! I’ve started adding Explore in Datasette buttons to dataset pages in the GLAM Workbench. Some examples are mentioned below.

New sections

Trove contributors

Trove aggregates metadata from thousands of organisations and projects around Australia. Data about contributors is available from the /contributor endpoint of the Trove API. This section includes examples of harvesting and exploring this data.

Notebooks: One notebook converts the nested data available from the /contributor endpoint into a single flat list of contributors. Another uses this list to to find out the number of records contributed by each organisation, aggregated by zone and format.
Datasets: Three datasets, generated by the code in the notebooks above, are being updated weekly – a list of organisations contributing metadata to Trove, a count of records by contributor and zone, and a count of records by contributor, zone, and format. You can explore them using Datasette Lite.

Trove API v3

This is a temporary section of the the GLAM Workbench created to bring together information and examples relating to version 3 beta of the Trove API. It will probably disappear once the new version is officially released and I reorganise the Trove sections of the GLAM Workbench accordingly. It currently includes:

A link to the new v3 beta section of the Trove API Console
A summary of breaking changes in the new version – these are the things you’ll need to change in your code before v2 of the API is switched off in early 2024.

Over the coming months I’ll be adding updated notebooks and examples from other Trove sections.

Updated sections

Trove newspaper & gazette harvester

New notebook! When I overhauled the trove-newspaper-harvester Python package last year, I made it possible to use the harvester as a library as well as a command line tool. This means you can integrate the harvester into your own tools or workflows. This new notebook, Harvesting articles that mention “Anzac Day” on Anzac Day, gives an example of a complex search query where it might be easier to use the harvester as a library – in this case, finding newspaper articles with particular keywords, published on a particular day, over a span of years!

Trove unpublished works (diaries, letters, and archives)

Repository upgraded: Python packages updated, configuration files standardised, basic tests added, automated Docker builds configured, integrations with Zenodo and Reclaim Cloud added.
New notebooks! I’ve added a series of notebooks related to finding, using, & analysing the NLA’s digitised manuscript finding aids (which are somewhat submerged beneath other content). One notebook, finds the finding aids, another one gathers some summary information about all 2,337 finding aids, and a third helps you reconstruct the hierarchy from the HTML of a single finding aid & saves the content as JSON.
New datasets! I used the new finding aids notebooks to generate a couple of datasets. One is just a list of Trove urls pointing to finding aids. The other dataset provides some summary information about each finding aid – this includes the number of items described, digitised, and searchable. You can explore the summary data using Datasette Lite.

Trove music & sound

Repository upgraded: Python packages updated, configuration files standardised, basic tests added, automated Docker builds configured, integrations with Zenodo and Reclaim Cloud added.
Updated harvest of ABC Radio National program metadata: there’s now 421,277 records from about 163 programs, though there doesn’t seem to have been any additions since early 2022.

Random items from Trove

This section documents some ways of retrieving random-ish works and newspaper articles from Trove.

Repository upgraded: Python packages updated, configuration files standardised, basic tests added, automated Docker builds configured, integrations with Zenodo and Reclaim Cloud added.

I’ve also created a couple of new ‘git scrapers’ to capture information about Trove. These are just bits of code that run on a schedule using GitHub actions and save their results into a GitHub repository. Along with the Trove Newspaper Data Dashboard and other historical datasets, these help researchers understand how the contents of Trove changes over time.

trove-zone-totals: saves data about the contents of Trove’s zones (this will need to be changed to categories with the release of v3 of the Trove API)
trove-contributor-totals: saves details of organisations and projects that contribute metadata to Trove

Maps, people, lists & more – recent updates to Trove resources in the GLAM Workbench

Fri, 17 Feb 2023 16:53:07 +1100

Once again I’ve gotten a bit behind in noting GLAM Workbench updates, so here’s a quick catch up on some Trove-related changes from the last couple of months.

Trove API introduction

The section that introduces the Trove API (or APIs!) hasn’t had much love over recent years. I’m hoping to add some more content in the coming months, but for now I just did a bit of maintenance – updating Python packages and config files, including tests, and setting up automated builds of Docker containers. The documentation pages have also had a bit of a refresh. More soon!

Trove lists and tags

For the Everyday Heritage digital workshop last November, I added a new notebook to convert a Trove list into a CollectionBuilder exhibition. The notebook harvests metadata and images from the items in a Trove list, then packages everything up in a form that can be uploaded to a CollectionBuilder-GH repository. Just upload the files to create your own instant exhibition! For example, this exhibition was generated from this Trove list.

More recently, I updated and reorganised the documentation pages. In particular, I updated the links to the pre-harvested datasets which are now all saved in Zenodo:

These datasets have also been added to the Trove historical data collection in Zenodo.

Trove maps

The Trove Maps section also needed a refresh and rebuild. The existing notebook harvested metadata about digitised maps to build an overview of what was available. I updated the harvesting code to capture a bit more information, including spatial coordinates where available. These coordinates aren’t available through the API, and aren’t visible on a map’s web page, but they are embedded within the HTML (this is a little trick I’ve used with other digitised materials to get additional metadata). You can download the updated dataset.

I also added a new notebook that attempts to parse the spatial data strings and convert the coordinates into decimal values that we can display on a map. Some of the coordinates were bounding boxes, while others were just points. If there was a bounding box, I calculated the centre point and saved that as well. I ended up with decimal coordinates for 26,591 digitised maps. You can download the parsed and converted coordinates as a dataset.

From this data I was able generate some interesting visualisations. I used Folium and the FastMarkerCluster plugin to map all of the centre points. I’ve had trouble before displaying lots of markers in Jupyter, but FastMarkerCluster handled it easily. I also saved the Folium map as a HTML page to make it easy for people to explore. Just zoom around and click on markers to display the map’s title and a link to Trove.

Where there are bounding boxes, you can overlay the map images themselves on a modern map. Of course, this isn’t as accurate as georectifying the map, particularly if the map doesn’t fill the whole image, but it’s still pretty fun. There’s a demonstration in the notebook that selects a random map and overlays the image on a modern basemap using IPyleaflet. It includes a widget to adjust the opacity of the map image (something I didn’t seem to be able to include using Folium?).

Trove People & Organisations

I’ve finally added a section for the Trove People & Organisations zone! This has been in the works for a while, but thanks to the Australian Cultural Data Engine I was able to devote some time to it. Trove’s People and Organisations zone aggregates information about individuals and organisations, bringing multiple sources together under a single identifier. Data is available from a series of APIs, which are not well-documented. The notebooks show you how to harvest all the available people and organisations data as EAC-CPF encoded XML files. Once you have the data, you can extract some summary information about sources and occupations, and use this to explore the way that records from different sources have been merged into unique identities. For example you can create a network graph of relationships between sources.

Or use an UpSet chart to show the most common groupings of sources.

There’s also a couple of notebooks with some handy examples of code to convert the XML results from the SRU API to JSON, and to find extra identity links through VIAF. I’ve also shared a pre-harvested version of the complete dataset and the extracted summaries.

Recent presentations – Library of Congress Data Jam, Everyday Heritage, Wikidata, and GLAM Workbench!

Sat, 10 Dec 2022 14:00:33 +1100

October and November brought a flurry of presentations from which I’m still recovering. Here’s a few details and links.

Library of Congress Data Jam

In October, the Computing Cultural Heritage in the Cloud project at the Library of Congress organised a Data Jam. I was invited to spend a couple of weeks playing around with one of their datasets and to report on the results. I ended up trying to find references to countries in a collection of 90,000 OCRd books. Of course I struck a few problems along the way, and didn’t get nearly as much done as I’d hoped, but that was really part of the point – to find the problems and explore the possibilities. You can watch a video of my presentation, or the whole Data Jam.

You can also:

Wikimedia Australia Community Meeting

Thanks to a grant from Wikimedia Australia, I’ve been able to spend some time this year working to align information about government agencies in Wikidata with details from the National Archives of Australia’s RecordSearch database. You can read about the project in this blog post. In November, I gave a report on the project at Wikimedia Australia’s monthly Community Meeting. You can watch the video on YouTube. You can also explore the new Wikidata section of the GLAM Workbench.

Everyday Heritage workshop and symposium

November also brought the first of what will be a series of annual symposia relating to the ARC-funded Everyday Heritage project. On 9 November, Kate Bagnall and I ran a Connecting People and Place workshop at the University of Canberra. We walked participants through some ways of finding and using digital collections such as Trove, and worked with them to create projects based on their own research using StoryMapJS and CollectionBuilder. You can view the slides of my Trove tips & tricks presentation. In preparation for the workshop I also created a new notebook in the GLAM Workbench that converts a Trove List into a Collection Builder exhibition.

The following day the Everyday Heritage Symposium, Connecting digital archives people & place, was held at the National Film and Sound Archive in Canberra. You can view the slides from my presentation Access to the everyday through digital collections. You can also watch a video of the full symposium.

Building DH

On 15 November I took part in a panel discussion on Designing user-friendly Platforms and Toolkits for Digital Humanities as part of the Building DH online conference. You can watch the video of the session on YouTube.

CAPOS 2022

On 16 November I gave the feature presentation at Reviewing, Revising, and Refining Open Social Scholarship: Australasia, and event organised by the Canadian-Australian Partnership for Open Scholarship. My talk, ‘DIY Infrastructure – Building the GLAM Workbench’, described work I’ve been doing over the past year or so to make the GLAM Workbench more sustainable by automating and standardising basic tasks, and integrating it with a range of existing services and tools. You can view the video of my talk, or browse the slides.

Worlds of Wikimedia 2022

Finally, on 17 November, I gave a keynote presentation at the Worlds of Wikimedia Conference. My talk, ‘Portals, platforms, and participation: building online collaboration around GLAM collections’, revisited my Portals to platforms paper from 2014, looking at ways that people can engage and create in the space around GLAM collections. You can view my slides online.

Do you want your Trove newspaper articles in bulk? Meet the new Trove Newspaper Harvester Python package!

Thu, 22 Sep 2022 14:53:49 +1100

The Trove Newspaper Harvester has been around in different forms for more than a decade. It helps you download all the articles in a Trove newspaper search, opening up new possibilities for large-scale analysis. You can use it as a command-line tool by installing a Python package, or through the Trove Newspaper Harvester section of the GLAM Workbench.

I’ve just overhauled development of the Python package. The new trove-newspaper-harvester replaces the old troveharvester repository. The command-line interface remains the same (with a few new options), so it’s really a drop in replacement. Read the full documentation of the new package for more details.

Here’s a summary of the changes:

the package can now be used as a library (that you incorporate into your own code) as well as a standalone command-line tool – this means you can embed the harvester in your own tools or workflows
both the library and the CLI now let you set the names of the directories in which your harvests will be saved – this makes it easier to organise your harvests into groups and give them meaningful names
the harvesting process now saves results into a newline-delimited JSON file (one JSON object per line) – the library has a save_csv() option to convert this to a CSV file, while the CLI automatically converts the results to CSV to maintain compatibility with previous versions
behind the scenes, the package is now developed and maintained using nbdev – this means the code and documentation are all generated from a set of Jupyter notebooks
the Jupyter notebooks include a variety of automatic tests which should make maintenance and development much easier in the future

I’ve also updated the Trove Newspaper Harvester section of the GLAM Workbench to use the new package. The new core library will make it easier to develop more complex harvesting examples – for example, searching for articles from a specific day across a range of years. If you find any problems, or want to suggest improvements, please raise an issue.

From 48 PDFs to one searchable database – opening up the Tasmanian Post Office Directories with the GLAM Workbench

Thu, 15 Sep 2022 23:15:21 +1100

A few weeks ago I created a new search interface to the NSW Post Office Directories from 1886 to 1950. Since then, I’ve used the same process on the Sydney Telephone Directories from 1926 to 1954. Both of these publications had been digitised by the State Library of NSW and made available through Trove. To build the new interfaces I downloaded the text from Trove, indexed it by line, and linked it back to the online page images.

But there are similar directories from other states that are not available through Trove. The Tasmanian Post Office Directory, for example, has been digitised between 1890 and 1948 and made available as 48 individual PDF files from Libraries Tasmania. While it’s great that they’ve been digitised, it’s not really possible to search them without downloading all the PDFs.

As part of the Everyday Heritage project, Kate Bagnall and I are working on mapping Tasmania’s Chinese history – finding new ways of connecting people and places. The Tasmanian Post Office Directories will be a useful source for us, so I thought I’d try converting them into a database as I had with the NSW directories. But how?

There were several stages involved:

Downloading the 48 PDF files
Extracting the text and images from the PDFs
Making the separate images available online so they could be integrated with the search interface
Loading all the text and image links into a SQLite database for online delivery using Datasette

And here’s the result!

Search for people and places in Tasmania from 1890 to 1948!

The complete process is documented in a series of notebooks, shared through the brand new Libraries Tasmania section of the GLAM Workbench. As with the NSW directories, the processing pipeline I developed could be reused with similar publications in PDF form. Any suggestions?

Some technical details

There were some interesting challenges in connecting up all the pieces. Extracting the text and images from the PDFs was remarkably easy using PyMuPDF, but the quality of the text wasn’t great. In particular, I had trouble with columns – values from neighbouring columns would be munged together, upsetting the order of the text. I tried working with the positional information provided by PyMuPDF to improve column detection, but every improvement seemed to raise another issue. I was also worried that too much processing might result in some text being lost completely.

I tried a few experiments re-OCRing the images with Textract ( a paid service from Amazon) and Tesseract. The basic Textract product provides good OCR, but again I needed to work with the positional information to try and reassemble the columns. On the other hand, Tesseract’s automatic layout detection seemed to work pretty well with just the default settings. It wasn’t perfect, but good enough to support search and navigation. So I decided to re-OCR all the images using Tesseract. I’m pretty happy with the result.

The search interfaces for the NSW directories display page images loaded directly from Trove into an OpenSeadragon viewer. The Tasmanian directories have no online images to integrate in this way, so I had to set up some hosting for the images I extracted from the PDFs. I could have just loaded them from an Amazon s3 bucket, but I wanted to use IIIF to deliver the images. Fortunately there’s a great project that uses Amazon’s Lambda service to provide a Serverless IIIF Image API. To prepare the images for IIIF, you convert them to pyramidal TIFFs (a format that contains an image at a number of different resolutions) using VIPS. Then you upload the TIFFs to an s3 bucket and point the Serverless IIIF app at the bucket. There’s more details in this notebook. It’s very easy and seems to deliver images amazingly quickly.

The rest of the processing followed the process I used with the NSW directories – using SQLite-utils and Datasette to package the data and deliver it online via Google Cloudrun.

Postscript: Time and money

I thought I should add a little note about costs (time and money) in case anyone was interested in using this workflow on other publications. I started working on this on Sunday afternoon and had a full working version up about 24 hours later – that includes a fair bit of work that I didn’t end up using, but doesn’t include the time I spent re-OCRing the text a day or so later. This was possible because I was reusing bits of code from other projects, and taking advantage of some awesome open-source software. Now that the processing pipeline is pretty well-defined and documented it should be even faster.

The search interface uses cloud services from Amazon and Google. It’s a bit tricky to calculate the precise costs of these, but here’s a rough estimate.

I uploaded 63.9gb of images to Amazon s3. These should cost about US$1.47 per month to store.

The Serverless IIIF API uses Amazon’s Lambda service. At the moment my usage is within the free tier, so $0 so far.

The Datasette instance uses Google Cloudrun. Costs for this service are based on a combination of usage, storage space, and the configuration of the environment. The size of the database for the Tasmanian directories is about 600mb, so I can get away with 2gb of application memory. (The NSW Post Office directory currently uses 8gb.) These services scale to zero – so basically they shut down if they’re not being used. This saves a lot of money, but means there can be a pause if they need to start up again. I’m running the Tasmanian and NSW directories, as well as the GLAM Name Index search, within the same Google Cloud account, and I’m not quite sure how to itemise the costs. But overall, it’s costing me about US$4.00 a month to run them all. Of course if usage increases, so will the costs!

So I suppose the point is that these sorts of approaches can be quite a practical and cost-effective way of improving access to digitised resources, and don’t need huge investments in time or infrastructure.

If you want to contribute to the running costs of the NSW and Tasmanian directories you can sponsor me on GitHub.

Fresh harvest of OCRd text from Trove's digitised periodicals – 9gb of text to explore and analyse!

Mon, 05 Sep 2022 17:13:16 +1100

I’ve updated the GLAM Workbench’s harvest of OCRd text from Trove’s digitised periodicals. This is a completely fresh harvest, so should include any corrections made in recent months. It includes:

1,430 periodicals
OCRd text from 41,645 issues
About 9gb of text

The easiest way to explore the harvest is probably this human-readable list. The list of periodicals with OCRd text is also available as a CSV. You can find more details in the Trove journals section of the GLAM Workbench, and download the complete corpus from CloudStor.

Finding which periodical issues in Trove have OCRd text you can download is not as easy as it should be. The fullTextInd index doesn’t seem to distinguish between digitised works (with OCR) and born-digital publications (like PDFs) without downloadable text. You can use has:correctabletext to find articles with OCR, but you can’t get a full list of the periodicals the articles come from using the title facet. As this notebook explains, you can search for nla.obj, but this returns both digitised works and publications supplied through edeposit. In previous harvests of OCRd text I processed all of the titles returned by the nla.obj search, finding out whether there was any OCRd text by just requesting it and seeing what came back. But the number of non-digitised works on the list of periodicals in digital form has skyrocketed through the edeposit scheme and this approach is no longer practical. It just means you waste a lot of time asking for things that don’t exist.

For the latest harvest I took a different approach. I only processed periodicals in digital form that weren’t identified as coming through edeposit. These are the publications with a fulltext_url_type value of either ‘digitised’ or ‘other’ in my dataset of digital periodicals. Is it possible that there’s some downloadable text in edeposit works that’s now missing from the harvest? Yep, but I think this is a much more sensible, straightforward, and reproduceable approach.

That’s not the only problem. As I noted when creating the list of periodicals in digital form, there are duplicates in the list, so they have to be removed. You then have to find information about the issues available for each title. This is not provided by the Trove API, but there is an internal API used in the web interface that you can access – see this notebook for details. I also noticed that sometimes where there’s a single issue of a title, it’s presented as if each page is an issue. I think I’ve found a work around for that as well.

All these doubts, inconsistencies and workarounds mean that I’m fairly certain I don’t have everything. But I do think I have most of the OCRd text available from digitised periodicals, and I do have a methodology, documented in this notebook, that at least provides a starting point for further investigation. As I noted in my wishlist for a Trove Researcher Platform, it would be great if more metadata for digitised works, other than newspapers, was made available through the API.

Explore Trove's digitised newspapers by place

Mon, 05 Sep 2022 16:34:05 +1100

I’ve updated my map displaying places where Trove digitised newspapers were published or distributed. You can view all the places on single map – zoom in for more markers, and click on a marker for title details and a link back to Trove.

If you want to find newspapers from a particular area, just click on a location using this map to view the 10 closest titles.

You can view or download the dataset used to construct the map. Place names were extracted from the newspaper titles using the Geoscience Gazetteer.

Making NSW Postal Directories (and other digitised directories) easier to search with the GLAM Workbench and Datasette

Thu, 01 Sep 2022 17:22:09 +1100

As part of my work on the Everyday Heritage project I’m looking at how we can make better use of digitised collections to explore the everyday experiences woven around places such as Parramatta Road in Sydney. For example, the NSW Postal Directories from 1886 to 1908 and 1909 to 1950 have been digitised by the State Library of NSW and made available through Trove. The directories list residences and businesses by name and street location. Using them we can explore changes in the use of Parramatta Road across 60 years of history. But there’s a problem. While you can browse the directories page by page, searching is clunky. Trove’s main search indexes the contents of the directories by ‘article’. Each ‘article’ can be many pages long, so it’s difficult to focus in on the matching text. Clicking through from the search results to the digitised volume lands you in another set of search results, showing all the matches in the volume. However, the internal search index works differently to the main Trove index. In particular it doesn’t seem to understand phrase or boolean searches. If you start off searching for “parramatta road” , Trove tells you there’s 50 matching articles, but if you click through to a volume you’re told there’s no results. If you remove the quotes you get every match for ‘parramatta’ or ‘road’. It’s all pretty confusing.

The idea of ‘articles’ is really not very useful for publications like the Post Office Directories where information is mostly organised by column, row or line. You want to be able to search for a name, and go directly to the line in the directory where that name is mentioned. And now you can! Using Datasette, I’ve created an interface that searches by line across all 54 volumes of the NSW Post Office Directory from 1886 to 1950 (that’s over 30 million lines of text).

Try it now!

Basic features

The full text search supports phrase, boolean, and wildcard searches. Just enter your query in the main search box to get results from all 54 volumes in a flash.
Each search result is a single line of text. Click on the link to view this line in context – it’ll show you 5 lines above and below your match, as well as a zoomable image of the digitised page from Trove.
For more context, you can click on View full page to see all the lines of text extracted from that page. You can then use the Next and Previous buttons to browse page by page.
To view the full digitised volume, just click on the View page in Trove button.

How it works

There were a few stages involved in creating this resource, but mostly I was able to reuse bits of code from the GLAM Workbench’s Trove journals and books sections, and other related projects such as the GLAM Name Index Search. Here’s a summary of the processing steps:

I started with the two top-level entries for the NSW Postal Directories, harvesting details of the 54 volumes under them.
For each of these 54 volumes, I downloaded the OCRd text page by page. Downloading the text by page, rather than volume, was very slow, but I thought it was important to be able to link each line of text back to its original page.
To create links back to pages, I also needed the individual identifiers for each page. A list of page identifiers is embedded as a JSON string within each volume’s HTML, so I extracted this data and matched the page ids to the text.
Using sqlite-utils, I created a SQLite database with a separate table for every volume. Then I processed the text by volume, page, and line – adding each line line of text and its page details as a individual record in the database.
I then ran full text indexing across each line to make it easily searchable.
Using Datasette and its search-all plugin, I loaded up the database and BINGO! More than 30 million lines of text across 54 digitised volumes were instantly searchable.
To make it all public, I used Datasette’s publish function to push the database to Google’s Cloudrun service.

All the code is available in the journals section of the GLAM Workbench.

Future developments

One of the most exciting things to me is that this processing pipeline can be used with any digitised publication on Trove where it would be easier to search by line rather than article. Any suggestions?

Mon, 29 Aug 2022 16:55:30 +1100

Interested in Victorian shipwrecks? Kim Doyle and Mitchell Harrop have added a new notebook to the Heritage Council of Victoria section of the GLAM Workbench exploring shipwrecks in the Victorian Heritage Database: glam-workbench.net/heritage-…

Mon, 29 Aug 2022 14:53:26 +1100

Updates!

troveharvester Python package updated to v0.5.1: github.com/wragge/tr…
Trove Newspaper Harvester section of #GLAMWorkbench updated to v1.1.1 to use latest troveharvester: glam-workbench.net/trove-har…

Thu, 25 Aug 2022 14:29:39 +1100

Minor update to RecordSearch Data Scraper – now captures ‘institution title’ for agencies if it is present. pypi.org/project/r…

Many thanks to the British Library – sponsors of the GLAM Workbench’s web archives section!

Tue, 16 Aug 2022 11:21:59 +1100

You might have noticed some changes to the web archives section of the GLAM Workbench.

I’m very excited to announce that the British Library is now sponsoring the web archives section! Many thanks to the British Library and the UK Web Archive for their support – it really makes a difference.

The web archives section was developed in 2020 with the support of the International Internet Preservation Consortium’s Discretionary Funding Programme, in collaboration with the British Library, the National Library of Australia, and the National Library of New Zealand. It’s intended to help historians, and other researchers, understand what sort of data is available through web archives, how to get it, and what you can do with it. It provides a series of tools and examples that document existing APIs, and explore questions such as how web pages change over time. The notebooks focus on four particular web archives: the UK Web Archive, the Australian Web Archive (National Library of Australia ), the New Zealand Web Archive (National Library of New Zealand), and the Internet Archive. However, the tools and approaches could be easily extended to other web archives (and soon will be!). I introduced the web archives section of the GLAM Workbench in this seminar for the IIPC in August 2020:

According to the Binder launch stats, the web archives section is the most heavily used part of the GLAM Workbench. In December 2020, it won the British Library Labs Research Award. Last year I updated the repository, automating the build of Docker images, and adding integrations with Zenodo, Reclaim Cloud, and Australia’s Nectar research cloud. I’m also thinking about some new notebooks – watch this space!

The GLAM Workbench receives no direct funding from government or research agencies, and so the support of sponsors like the British Library and all my other GitHub sponsors is really important. Thank you! If you think this work is valuable, have a look at the Get involved! page to see how you can contribute. And if your organisation would like to sponsor a section of the GLAM Workbench, let me know!

New GLAM data to search, visualise and explore using the GLAM Workbench!

Mon, 15 Aug 2022 17:43:44 +1100

There’s lots of GLAM data out there if you know where to look! For the past few years I’ve been harvesting a list of datasets published by Australian galleries, libraries, archives, and museums through open government data portals. I’ve just updated the harvest and there’s now 463 datasets containing 1,192 files. There’s a human-readable version of the list that you can browse. If you just want the data you can download it as a CSV. Or if you’d like to search the list there’s a database version hosted on Glitch. The harvesting and processing code is available in this notebook.

The GLAM data from government portals section of the GLAM Workbench provides more information and a summary of results. For example, here’s a list of the number of data files by GLAM institution.

Most of the datasets are in CSV format, and most have a CC-BY licence.

What’s inside?

Obviously it’s great that GLAM organisations are sharing lots of open data, but what’s actually inside all of those CSV files? To help you find out, I created the GLAM CSV Explorer. Click on the blue button to run it in Binder, then just select a dataset from the dropdown list. The CSV Explorer will download the file, and examine the contents of every field to try and determine the type of data it holds – such as text, dates, or numbers. It then summarises the results and builds a series of visualisations to give you an overview of the dataset.

Search for names

Many of the datasets are name indexes to collections of records – GLAM staff or volunteers have transcribed the names of people mentioned in records as an aid to users. For Family History Month last year I aggregated all of the name indexes and made them searchable through a single interface using Datasette. The GLAM Name Index Search has been updated as well – it searches across 10.3 million records in 253 indexes from 10 GLAM organisations. And it’s free!

And a bit of maintenance…

As well as updating the data, I also updated the code repository adding the features that I’m rolling out across the whole of the GLAM Workbench. This includes automated Docker builds saved to Quay.io, integrations with Reclaim Cloud and Zenodo, and some basic quality controls through testing and code format checks.

Zotero now saves links to digitised items in Trove from the NLA catalogue!

Tue, 09 Aug 2022 10:52:53 +1100

I’ve made a small change to the Zotero translator for the National Library of Australia’s catalogue. Now, if there’s a link to a digitised version of the work in Trove, that link will be saved in Zotero’s url field. This makes it quicker and easier to view digitised items – just click on the ‘URL’ label in Zotero to open the link.

It’s also handy if you’re viewing a digitised work in Trove and want to capture the metadata about it. Just click on the ‘View catalogue’ link in the details tab of a Trove item, then use Zotero to save the details from the catalogue.

View embedded JSON metadata for Trove's digitised books and journals

Mon, 01 Aug 2022 19:22:47 +1100

The metadata for digitised books and journals in Trove can seem a bit sparse, but there’s quite a lot of useful metadata embedded within Trove’s web pages that isn’t displayed to users or made available through the Trove API. This notebook in the GLAM Workbench shows you how you can access it. To make it even easier, I’ve added a new endpoint to my Trove Proxy that returns the metadata in JSON format.

Just pass the url of a digitised book or journal as a parameter named url to https://trove-proxy.herokuapp.com/metadata/. For example:

https://trove-proxy.herokuapp.com/metadata/?url=https://nla.gov.au/nla.obj-2906940941

I’ve created a simple bookmarklet to make it simpler to open the proxy. To use it just:

Drag this link to your bookmarks toolbar: Get Trove work metadata
View a digitised book or journal in Trove.
Click on the bookmarklet to view the metadata in JSON format.

To view the JSON data in your browser you might need to install an extension like JSONView.

Where did all those NSW articles go? Trove Newspapers Data Dashboard update!

Fri, 29 Jul 2022 14:20:34 +1100

I was looking at my Trove Newspapers Data Dashboard again last night trying to figure out why the number of newspaper articles from NSW seemed to have dropped by more than 700,000 since my harvesting began. It took me a while to figure out, but it seems that the search index was rebuilt on 31 May, and that caused some major shifts in the distribution of articles by state, as reported by the main result API. So the indexing of the articles changed, not the actual number of articles. Interestingly, the number of articles by state reported by the newspaper API doesn’t show the same fluctuations.

This adds another layer of complexity to understanding how Trove changes over time. To try and document such things, I’ve added a ‘Significant events’ section to the Dashboard. I’ve also included a new ‘Total articles by publication state’ section that compares results from the result and newspaper APIs. This should make it easier to identify such issues in the future.

Stay alert people – remember, search interfaces lie!

Catching up – some recent GLAM Workbench updates!

Thu, 28 Jul 2022 15:59:13 +1100

There’s been lots of small updates to the GLAM Workbench over the last couple of months and I’ve fallen behind in sharing details. So here’s an omnibus list of everything I can remember…

Data

Weekly harvests of basic Trove newspaper data continue, there’s now about three months worth. You can view a summary of the harvested data through the brand new Trove Newspaper Data Dashboard. The Dashboard is generated from a Jupyter notebook and is updated whenever there’s a new data harvest.
There’s also weekly harvests of files digitised by the NAA, now 16 months worth of data.
Updated harvest of Trove public tags (Zenodo) – includes 2,201,090 unique public tags added to 9,370,614 resources in Trove between August 2008 and July 2022.
I’ve started moving other pre-harvested datasets out of the GLAM Workbench code repositories, into their own data repositories. This means better versioning and citability. The first example is the list of Trove newspapers with articles post the 1955 copyright cliff of death – here’s the GH repo, and the Zenodo record.
To bring together datasets that provide historical data about Trove itself, I’ve created a Trove historical data community on Zenodo. Anyone’s welcome to contribute. There’s much more to come.

Tag cloud generated from the latest harvest of Trove Tags

Code

Big thanks to Mitchell Harrop who contributed a new Heritage Council of Victoria section to the GLAM Workbench providing examples using the Victorian Heritage Database API.
The troveharvester Python package has been updated. Mainly to remove annoying Pandas warnings and to make use of the trove-query-parser package.
As a result of the above, the Trove Newspaper & Gazette Harvester section of the GLAM Workbench has been updated. No major changes to notebooks, but I’ve implemented basic testing and linting to improve code quality.
The Trove newspapers section of the GW has been updated. There were a few bug fixes and minor improvements. In particular there was a problem downloading data and HTML files from QueryPic, and some date queries in QueryPic were returning no results.
The tool to download complete, high-res newspaper page images has been updated so that you now no longer need to supply an API key. Also fixed a problem with displaying the images in Voila.
The recordsearch_data_scraper Python package has been updated. This fixes a bug where agency and series searches with only one result weren’t being captured properly.
The RecordSearch section of the GW has been updated. This is incorporates the above update, but I took the opportunity to update all packages, and implement basic testing and linting. The Harvest items from a search in RecordSearch notebook has been simplified and reorganised. There are two new notebooks: Exploring harvested series data, 2022 – generates some basic statistics from the harvest of series data in 2022 and compares the results to the previous year; Summary of records digitised in the previous week – run this notebook to analyse the most recent dataset of recently digitised files, summarising the results by series.
A new Zotero translator for Libraries Tasmania has been developed

Thu, 14 Jul 2022 00:04:24 +1100

Updated dataset! Harvests of Trove list metadata from 2018, 2020, and 2022 are now available on Zenodo: doi.org/10.5281/z… Another addition to the growing collection of historical Trove data. #GLAMWorkbench

Sat, 09 Jul 2022 18:34:26 +1100

Coz I love making work for myself, I’ve started pulling datasets out of #GLAMWorkbench code repos & creating new data repos for them. This way they’ll have their own version histories in Zenodo. Here’s the first: github.com/GLAM-Work…

Tue, 28 Jun 2022 09:48:32 +1100

Ahead of my session at #OzHA2022 tomorrow, I’ve updated the NAA section of the #GLAMWorkbench. Come along to find out how to harvest file details, digitsed images, and PDFs, from a search in RecordSearch! github.com/GLAM-Work…

Sun, 26 Jun 2022 12:48:52 +1100

Noticed that QueryPic was having a problem with some date queries. Should be fixed in the latest release of the Trove Newspapers section of the #GLAMWorkbench: glam-workbench.net/trove-new… #maintenance #researchinfrastructure

Fri, 24 Jun 2022 17:38:04 +1100

The Trove Newspapers section of the #GLAMWorkbench has been updated! Voilá was causing a problem in QueryPic, stopping results from being downloaded. A package update did the trick! Everything now updated & tested. glam-workbench.net/trove-new…

Fri, 24 Jun 2022 15:40:33 +1100

Some more #GLAMWorkbench maintenance – this app to download a high-res page images from Trove newspapers now doesn’t require an API key if you have a url, & some display problems have been fixed. trove-newspaper-apps.herokuapp.com/voila/ren…

Thu, 23 Jun 2022 17:11:35 +1100

The Trove Newspaper and Gazette Harvester section of the #GLAMWorkbench has been updated! No major changes to notebooks, just lots of background maintenance stuff such as updating packages, testing, linting notebooks etc. glam-workbench.net/trove-har…

Wed, 01 Jun 2022 17:43:27 +1100

Ordering some #GLAMWorkbench stickers…

Using Datasette on Nectar

Thu, 26 May 2022 16:24:49 +1100

If you have a dataset that you want to share as a searchable online database then check out Datasette – it’s a fabulous tool that provides an ever-growing range of options for exploring and publishing data. I particularly like how easy Datasette makes it to publish datasets on cloud services like Google’s Cloudrun and Heroku. A couple of weekends ago I migrated the TungWah Newspaper Index to Datasette. It’s now running on Heroku, and I can push updates to it in seconds.

I’m also using Datasette as the platform for sharing data from the Sydney Stock Exchange Project that I’m working on with the ANU Archives. There’s a lot of data – more than 20 million rows – but getting it running on Google Cloudrun was pretty straightforward with Datasette’s publish command. The problem was, however, that Datasette is configured to run on most cloud services in ‘immutable’ mode and we want authenticated users to be able to improve the data. So I needed to explore alternatives.

I’ve been working with Nectar over the past year to develop a GLAM Workbench application that helps researchers do things like harvesting newspaper articles from a Trove search. So I thought I’d have a go at setting up Datasette in a Nectar instance, and it works! Here’s a few notes on what I did…

First of course you need get yourself a resource allocation on Nectar. I’ve also got a persistent volume storage allocation that I’m using for the data.
From the Nectar Dashboard I made sure that I had an SSH keypair configured, and created a security group to allow access via SSH, HTTP and HTTPS. I also set up a new storage volume.
I then created a new Virtual Machine using the Ubuntu 22.04 image, and attaching the keypair, security group, and volume storage. For the stock exchange data I’m currently used the ‘m3.medium’ flavour of virtual machine which provides 8gb of RAM and 4 VCPUs. This might be overkill, but I went with the bigger machine because of the size of the SQLite database (around 2gb). This is similar to what I used on Cloudstor after I ran into problems with the memory limit. I think most projects would run perfectly well using one of the ‘small’ flavours. In any case, it’s easy to resize if you run into problems.
Once the new machine was running I grabbed the IP address. Because I have DNS configured on my Nectar project, I also created a ‘datasette’ subdomain from the DNS dashboard by pointing an ‘A’ (alias) record to the IP address.
Using the IP address I logged into the new machine via SSH.
With all the Nectar config done, it was time to set up Datasette. I mainly just followed the excellent instructions in the Datasette documention for deploying Datasette using systemd. This involved installing datasette via pip, creating a folder for the Datasette data and configuration files, creating a datasette.service file for systemd.
I also used the datasette install command to add a couple of Datasette plugins. One of these is the datasette-github-auth plugin, which needs a couple of secret tokens set. I added these as environment variables in the datasette.service file.
The systemd setup uses Datasette’s configuration directory mode. This means you can put your database, metadata definitions, custom templates and CSS, and any other settings all together in a single directory and Datasette will find and use them. I’d previously passed runtime settings via the command line, so I had to create a settings.json for these.
Then I just uploaded all my Datasette database and configuration files to the folder I created on the virtual machine using rsync and started the Datasette service. It worked!
The next step was to use the persistent volume storage for my Datasette files. The persistent storage exists independently of the virtual machine, so you don’t need to worry about losing data if there’s a change to the instance. I mounted the storage volume as /pvol in the virtual machine as the Nectar documentation describes.
I created a datasette-root folder under pvol, copied the Datasette files to it, and changed the datasette.service file to point to it. This didn’t seem to work and I’m not sure why. So instead I created a symbolic link between /home/ubuntu/datasette-root and /pvol/datasette-root and and set the path in the service file back to /home/ubuntu/datasette-root. This worked! So now the database and configuration files are sitting in the persistent storage volume.
To make the new Datasette instance visible to the outside world, I installed nginx, and configured it as a Datasette proxy using the example in the Datasette documentation.
Finally I configured HTTPS using certbot.

Although the steps above might seem complicated, it was mainly just a matter of copying and pasting commands from the existing documentation. The new Datasette instance is running here, but this is just for testing and will disappear soon. If you’d like to know more about the Stock Exchange Project, check out the ANU Archives section of the GLAM Workbench.

Convert your Trove newspaper searches to an API query with just one click!

Fri, 20 May 2022 16:43:00 +1100

I’m thinking about the Trove Researcher Platform discussions & ways of integrating Trove with other apps and platforms (like the GLAM Workbench).

As a simple demo I modifed my Trove Proxy app to convert a newspaper search url from the Trove web interface into an API query (using the trove-query-parser package). The proxy app then redirects you to the Trove API Console so you can see the results of the API query without needing a key.

To make it easy to use, I created a bookmarklet that encodes your current url and feeds it to the proxy. To use it just:

Drag this link to your bookmarks toolbar: Open Trove API Console.
Run a search in Trove’s newspapers.
Click on the bookmarklet.

This little hack provides a bit of ‘glue’ to help researchers think their about search results as data, and explore other possibilities for download and analysis. #dhhacks

My Trove researcher platform wishlist

Wed, 11 May 2022 14:26:00 +1100

The ARDC is collecting user requirements for the Trove researcher platform for advanced research. This is a chance to start from scratch, and think about the types of data, tools, or interface enhancements that would support innovative research in the humanities and social sciences. The ARDC will be holding two public roundtables, on 13 and 20 May, to gather ideas. I created a list of possible API improvements in my response to last year’s draft plan, and thought it might be useful to expand that a bit, and add in a few other annoyances, possibilities, and long-held dreams.

My focus is again on the data; this is for two reasons. First because public access to consistent, good quality data makes all other things possible. But, of course, it’s never just a matter of OPENING ALL THE DATA. There will be questions about priorities, about formats, about delivery, about normalisation and enrichment. Many of these questions will arise as people try to make use of the data. There needs to be an ongoing conversation between data providers, research tool makers, and research tool users. This is the second reason I think the data is critically important – our focus should be on developing communities and skills, not products. A series of one-off tools for researchers might be useful, but the benefits will wane. Building tools through networks of collaboration and information sharing based on good quality data offers much more. Researchers should be participants in these processes, not consumers.

Anyway, here’s my current wishlist…

APIs and data

Bring the web interface and main public API back into sync, so that researchers can easily transfer queries between the two. The Trove interface update in 2020 reorganised resources into ‘categories’, replacing the original ‘zones’. The API, however, is still organised by zone and knows nothing about these new categories. Why does this matter? The web interface allows researchers to explore the collection and develop research questions. Some of these questions might be answered by downloading data from the API for analysis or visualisation. But, except for the newspapers, there is currently no one-to-one correspondence between searches in the web interface and searches using the API. There’s no way of transferring your questions – you need to start again.
Expand the metadata available for digitised resources other than newspapers. In recent years, the NLA has digitised huge quantities of books, journals, images, manuscripts, and maps. The digitisation process has generated new metadata describing these resources, but most of this is not available through the public API. We can get an idea of what’s missing by comparing the digitised journals to the newspapers. The API includes a newspaper endpoint that provides data on all the newspapers in Trove. You can use it to get a list of available issues for any newspaper. There is no comparable way of retrieving a list of digitised journals, or the issues that have been digitised. The data’s somewhere – there’s an internal API that’s used to generate lists of issues in the browse interface and I’ve scraped this to harvest issue details. But this information should should be in the public API. Manuscripts are described using finding aids, themselves generated from EAD formatted XML files, but none of this important structured data is available from the API, or for download. There’s also other resource metadata, such as parent/child relationships between different levels in the object hierarchy (eg publication > pages). These are embedded in web pages but not exposed in the API. The main point is that when it comes to data-driven research, digitised books, journals, manuscripts, images, and maps are second-class citizens, trailing far behind the newspapers in research possibilities. There needs to be a thorough stocktake of available metadata, and a plan to make this available in machine actionable form.
Standardise the delivery of text, images, and PDFs and provide download links through the API. As noted above, digitised resources are treated differently depending on where they sit in Trove. There are no standard mechanisms for downloading the products of digitisation, such as OCRd text and images. OCRd text is available directly though the API for newspaper and journal articles, but to download text from a book or journal issue you need to hack the download mechanisms from the web interface. Links to these should be included in the API. Similarly, machine access to images requires various hacks and workarounds. There should be a consistent approach that allows researchers to compile image datasets from digitised resources using the API. Ideally IIIF standard APIs should be used for the delivery of images and maps. This would enable the use of the growing ecosystem of IIIF compliant tools for integration, analysis, and annotation.
Provide an option to exclude search results in tags and comments. The Trove advanced search used to give you the option of excluding search results which only matched tags or comments, rather than the content of the resource. Back when I was working at Trove, the IT folks said this feature would be added to a future version of the API, but instead it disappeared from the web interface with the 2020 update! Why is this important? If you’re trying to analyse the occurance of search terms within a collection, such as Trove’s digitised newspapers, you want to be sure that the result reflects the actual content, and not a recent annotation by Trove users.
Finally add the People & Organisations data to the main API. Trove’s People & Organisations section was ahead of the game in providing machine-readable access, but the original API is out-of-date and uses a completely different query language. Some work was done on adding it to the main RESTful API, but it was never finished. With a bit of long-overdue attention, the People & Organisations data could power new ways of using and linking biographical resources.
Improve web archives CDX API. Although the NLA does little to inform researchers of the possibilities, the web archives software it uses (Pywb) includes some baked in options for retrieving machine-readable data. This includes support for the Memento protocol, and the provision of a CDX API that delivers basic metadata about individual web page captures. The current CDX API has some limitations ( documented here ). In particular, there’s no pagination or results, and no support for domain-level queries. Addressing these limitations would make the existing CDX API much more useful.
Provide new data sources for web archives analysis. There needs to be an constructive, ongoing discussion about the types of data that could be extracted and shared from the Australian web archive. For example, a search API, or downloadable datasets of word frequencies. The scale is a challenge, but some pilot studies could help us all understand both the limits and the possibilities.
Provide a Write API for annotations. Integration between components in the HASS RDC would be greatly enhanced if other projects could automatically add structured annotations to existing Trove resources. Indeed, this would create exciting possibilities for embedding Trove resources within systems of scholarly analysis, allowing insights gained through research to be automatically fed back into Trove to enhance discovery and understanding.
Provide historical statistics on Trove resources. It’s important for researchers to understand how Trove itself changes over time. There used to be a page that provided regularly-updated statistics on the number of resources and user annotations, but this was removed by the interface upgrade in 2020. I’ve started harvesting some basic stats relating to Trove newspapers, but access to general statistics should be reinstated.
Reassess key authentication and account limits. DigitalNZ recently changed their policy around API authentication, allowing public access without a key. Authentication requirements hinder exploration and limit opportunities for using the API in teaching and workshops. Similarly, I don’t think the account usage limits have been changed since the API was released, even though the capacity of the systems has increased. It seems like time that both of these were reassessed.

Ok, I’ll admit, that’s a pretty long list, and not everything can be done immediately! I think this would be a good opportunity for the NLA to develop and share an API and Data Roadmap, that is regularly updated, and invites comments and suggestions. This would help researchers plan for future projects, and build a case for further government investment.

Integration

Unbreak Zotero integration. The 2020 interface upgrade broke the existing Zotero integration and there’s no straightforward way of fixing it without changes at the Trove end. Zotero used to be able to capture search results, metadata and images from most of the zones in Trove. Now it can only capture individual newspaper articles. This greatly limits the ability of researchers to assemble and manage their own research collections. More generally, a program to examine and support Zotero integration across the GLAM sector would be a useful way of spending some research infrastructure dollars.
Provide useful page metadata. Zotero is just one example of a tool that can extract structured metadata from web pages. Such metadata supports reuse and integration, without the need for separate API requests. Only Trove’s newspaper articles currently provide embedded metadata. Libraries used to lead the way is promoting the use of standardised, structured, embedded page metadata (Dublin Core anyone?), but now?
Explore annotation frameworks. I’ve mentioned the possibility of a Write API for annotations above, but there are other possibilities for supporting web scale annotations, such as Hypothesis. Again, the current Trove interface makes the use of Hypothesis difficult, and again this sort of integration would be usefully assessed across the whole GLAM sector.

Tools & interfaces

Obviously any discussion of new tools or interfaces needs to start by looking at what’s already available. This is difficult when the NLA won’t even link to existing resources such as the GLAM Workbench. Sharing information about existing tools needs to be the starting point from which to plan investment in the Trove Researcher Platform. From there we can identify gaps and develop processes and collaborations to meet specific research needs. Here’s a list of some Trove-related tools and resources currently available through the GLAM Workbench.

Update (18 May): some extra bonus bugs

I forgot to add these annoying bugs:

The newspaper endpoint returns both newspaper and gazette titles, even though there’s a separate gazette endpoint. This forces you to do silly workarounds like this in the GLAM Workbench.
The list zone has recurring problems. At the moment it’s impossible to harvest a complete set of Trove lists.

Tue, 10 May 2022 22:26:35 +1100

Spending the evening updating the NAA section of the #GLAMWorkbench. Here’s a fresh harvest of the agency functions currently being used in RecordSearch… gist.github.com/wragge/d1…

Working with Trove data – a collection of tools and resources

Mon, 02 May 2022 15:23:00 +1100

The ARDC is organising a couple of public forums to help gather researcher requirements for the Trove component of the HASS RDC. One of the roundtables will look at ‘Existing tools that utilise Trove data and APIs’. Last year I wrote a summary of what the GLAM Workbench can contribute to the development of humanities research infrastructure, particularly in regard to Trove. I thought it might be useful to update that list to include recent additions to the GLAM Workbench, as well as a range of other datasets, software, tools, and interfaces that exist outside of the GLAM Workbench.

Since last year’s post I’ve also been working hard to integrate the GLAM Workbench with other eResearch services such as Nectar and CloudStor, and to document and support the ways that individuals and institutions can contribute code and documentation.

Getting and moving data

There’s lots of fabulous data in Trove and other GLAM collections. In fact, there’s so much data that it can be difficult for researchers to find and collect what’s relevant to their interests. There are many tools in the GLAM Workbench to help researchers assemble their own datasets. For example:

Get newspaper articles in bulk with the Trove Newspaper and Gazette Harvester – This has been around in some form for more than ten years (it pre-dates the Trove API!). Give it the url of a search in Trove’s newspapers and gazettes and the harvester will save all the metadata in a CSV file, and optionally download the complete articles as OCRd text, images, or PDFs. The amount of data you harvest is really only limited by your patience and disk space. I’ve harvested more than a million articles in the past. The GLAM Workbench includes a web app version of the harvester that runs live in the cloud – just paste in your Trove API key and the search url, and click the button.
Harvest information about newspaper issues – When you search Trove’s newspapers, you find articles – these articles are grouped by page, and all the pages from a particular date make up an issue. But how do you find out what issues are available? On what dates were newspapers published? This notebook shows how you can get information about issues from the Trove API.
Get Trove newspaper pages as images – If you need a nice, high-resolution version of a newspaper page you can use this web app. If you want to harvest every front page (or some other particular page) here’s an example that gets all the covers of the Australian Women’s Weekly. A pre-harvested collection of the AWW covers is included as a bonus extra.
Get Trove newspaper articles as images – The Trove web interface makes it difficult to download complete images of articles, but this tool will do the job. There’s a handy web app to grab individual images, but the code from this tool is reused in other places such as the Trove Newspaper Harvester and the Omeka uploader, and could be built-in to your own research workflows.
Harvest the issues of a newspaper as PDFs – This notebook harvests whole issues of newspapers as PDFs – one PDF per issue.
Upload Trove newspaper articles to Omeka – Whether you’re creating on online exhibition or building a research database, Omeka can be very useful. This notebook connects Trove’s newspapers to Omeka for easy upload. Your selected articles can come from a search query, a Trove list, a Zotero library, or just a list of article ids. Metadata records are created in Omeka for each article and newspaper, and an image of each article is attached. My Omeka S Tools software package also includes an example using Trove newspapers.
Get OCRd text from digitised periodicals in Trove – They’re often overshadowed by the newspapers, but there’s now lots of digitised journals, magazines, and parliamentary papers in Trove. You can get article-level data from the API, but not issue data. This notebook enables researchers to get metadata and OCRd text from every available issue of a periodical. To make researchers’ lives even easier, I regularly harvest all the available OCRd text from digitised periodicals in Trove. The latest harvest downloaded 51,928 issues from 1,163 periodicals – that’s about 10gb of text. You can browse the list of periodicals with OCRd text, or search this database. All the OCRd text is stored in a public repository on CloudStor.
Get page images from digitised periodicals in Trove – There’s more than text in digitised periodicals, and you might want to download images of pages for visual analysis. This notebook shows you how to get cover images, but could be easily modified to get another page, or a PDF. I used a modified version of this to create a collection of 3,471 full page editorial cartoons from The Bulletin, 1886 to 1952 – all available to download from CloudStor.
Get OCRd text from digitised books in Trove – Yep, there’s digitised books as well as newspapers and periodicals. You can download OCRd text from an individual book using the Trove web interface, but how do you make a collection of books without all that pointing and clicking? This notebook downloads all the available OCRd text from digitised books in Trove. The latest harvest includes text from 26,762 works. You can explore the results using this database.
Harvest parliamentary press releases from Trove – Trove includes more than 380,000 press releases, speeches, and interview transcripts issued by Australian federal politicians and saved by the Parliamentary Library. This notebook shows you how to harvest both metadata and fulltext from a search of the parliamentary press releases. For example, here’s a collection of politicians talking about ‘refugees’, and another relating to COVID-19.
Harvest details of Radio National programs from Trove – Trove creates records for programs broadcast on ABC Radio National, for the major current affairs programs these records at at segment level. Even though they don’t provide full transcripts, this data provide a rich, fine-grained record of Australia’s recent political, social, and economic history. This notebook shows you how to download the Radio National data. If you just want to dive straight in, there’s also a pre-harvested collection containing more than 400,000 records, with separate downloads for some of the main programs.
Find all the versions of an archived web page in Trove – Many of the tools in the Web Archives section of the GLAM Workbench will work with the Australian Web Archive, which is part of Trove. This notebook shows you how to get data about the number of times a web page has been archived over time.
Harvesting collections of text from archived web pages in Trove – If you want to explore how the content of a web page changes over time, you can use this notebook to capture the text content of every archived version of a web page.
Convert a Trove list into a CSV file – While Trove provides a data download option for lists, it leaves out a lot of useful data. This notebook downloads full details of newspaper articles and other works in a list and saves them as CSV files. Like the Trove Newspaper Harvester, it lets you download OCRd text and images from newspaper articles.
Collecting information about Trove user activity – It’s not just the content of Trove that provides interesting research data, it’s also the way people engage with it. Using the Trove API it’s possible to harvest details of all user created lists and tags. And yes, there’s pre-harvested collections of lists and tags for the impatient.

While I’m focusing here on Trove, there’s also tools to create datasets from the National Archives of Australia, Digital NZ and Papers Past, the National Museum of Australia and more. And there’s a big list of readily downloadable datasets from Australian GLAM organisations.

Visualisation and analysis

Many of the notebooks listed above include examples that demonstrate ways of exploring and analysing your harvested data. There are also a number of companion notebooks that examine some possibilities in more detail, for example:

But there are also many other notebooks that demonstrate methods for analysing Trove’s content, for example:

QueryPic – Another tool that’s been around in different forms for a decade, QueryPic visualises searches in Trove’s newspapers. The latest web app couldn’t be simpler, just paste in your API key and a search url and create charts showing the number of matching articles over time. You can combine queries, change time scales, and download the data and visualisations. Interested to see how other researchers have used it? Here’s a Twitter thread with links to some publications.
Visualise Trove newspaper searches over time – This is like a deconstructed version of QueryPic that walks you through the process of using Trove’s facets to assemble a dataset of results over time. It provide a lot of detail on the sorts of data available, and the questions we can ask of it.
Visualise the total number of newspaper articles in Trove by year and state – This notebook uses a modified version of the code above to analyse the construction and context of Trove’s newspaper corpus itself. What are you actually searching? Meet the WWI effect and the copyright cliff of death! This is a great place to start if you want to get people thinking critically about digital resources are constructed.
Trove newspapers – number of issues per day, 1803–2020 – visualisation of the number of newspaper issues published every day in Trove.
Analyse rates of OCR correction – Some more meta-analysis of the Trove corpus itself, this time focusing on patterns of OCR correction by Trove users.
Identifying non-English language newspapers in Trove – There are a growing number of non-English language newspapers digitised in Trove. However, if you’re only searching using English keywords, you might never know that they’re there. This notebook analyses a sample of articles from every newspaper in Trove to identify non-English content.
Beyond the copyright cliff of death – Most of the newspaper articles on Trove were published before 1955, but there are some from the later period. This notebook helps you find out how many, and which newspapers they were published in.
Map Trove newspaper results by state – This notebook uses the Trove state facet to create a choropleth map that visualises the number of search results per state.
Map Trove newspaper results by place of publication – This notebook uses the Trove title facet to find the number of results per newspaper, then merges the results with a dataset of geolocated newspapers to map where articles were published.
Compare two versions of an archived web page – This notebook demonstrates a number of different ways of comparing versions of archived web pages. Just choose a repository, enter a url, and select two dates to see comparisons based on: page metadata, basic statistics such as file size and number of words, numbers of internal and external links, cosine similarity of text, line by line differences in text or code, and screenshots.
Display changes in the text of an archived web page over time – This web app gathers all the available versions of a web page and then visualises changes in its content between versions – what’s been added, removed, and changed?
Use screenshots to visualise change in a page over time– Create a series of full page screenshots of a web page over time, then assemble them into a time series.

There are also possibilities for using Trove data creatively. For example you can create ‘scissors and paste’ messages from Trove newspaper articles.

Documentation and examples

All the Trove notebooks in the GLAM Workbench help document the possibilities and limits of the Trove API. The examples above can be modified and reworked to suit different research interests. Some notebooks also explore particular aspects of the API, for example:

Trove API Introduction – Some very basic examples of making requests and understanding results.
Today’s news yesterday – Uses the date index and the firstpageseq parameter to find articles from exactly 100 years ago that were published on the front page. It then selects one of the articles at random and downloads and displays an image of the front page.
The use of standard licences and rights statements in Trove image records – Version 2.1 of the Trove API introduced a new rights index that you can use to limit your search results to records that include one of a list of standard licences and rights statements. We can also use this index to build a picture of which rights statements are currently being used, and by who.
Random items from Trove – Changes to the Trove API meant that techniques you could previously use to select resources at random no longer work. This section documents some alternative ways of retrieving random-ish works and newspaper articles from Trove.

And while it’s not officially part of the GLAM Workbench, I also maintain the Trove API Console which provides lots of examples of the API in action.

Videos

I’ve started making videos to help you get started with the GLAM Workbench.

Datasets

A number of pre-harvested datasets are noted above in the ‘Getting and moving data’ section. Here’s a fairly complete list of ready-to-download datasets harvested from Trove.

Newspapers

Books and periodicals

Other

See also Sources of Australian GLAM data in the GLAM Workbench.

Software

The GLAM Workbench makes use of a number of software packages that I’ve created in Python to work with Trove data. These are openly-licensed and available for installation from PyPi.

Trove Harvester – harvest newspaper and gazette articles
Trove Query Parser – convert search parameters from the Trove web interface into a form the API understands
Trove Newspaper Images – tools for downloading images from Trove’s digitised newspapers and gazettes

Other tools and interfaces

Over the years I’ve developed many tools and interfaces using Trove data. Some have been replaced by the GLAM Workbench, but others keep chugging along, for example:

Explore Trove’s Digitised Journals – a tool to help you browse, search, and explore Trove’s digitised journals
Headline Roulette – a simple, customisable game using Trove newspapers
In a word… – currents in Australian affairs, 2003–2013, using data from ABC Radio National
Trove API Console – use this to learn how to construct API queries
Trove Traces – archived version of a 2014 experiment to see who was citing Trove newspapers
@TroveNewsBot – unlike most GLAM Twitter bots, @TroveNewsBot doesn’t just tweet random stuff, you can use it to search Trove from Twitter, see the full operating instructions.
Trove Places – click on a map to find Trove newspapers by place
Trove Zotero Translator – lets you capture metadata, OCRd text, and a PDF from an article in Trove newspapers, installed as part of Zotero.
Keyboard shortcuts for Trove newspapers – this userscript activates your arrow keys to help you navigate newspapers by page and issue.

See also More GLAM tools and interfaces in the GLAM Workbench. #dhhacks

Sat, 30 Apr 2022 16:39:07 +1100

And so it starts… #GLAMWorkbench

Thu, 28 Apr 2022 23:49:31 +1100

Ok, I’ve created a new #GLAMWorkbench meta issue to try and bring together all the things I’m trying to do to improve & automate the code & documentation. This should help me keep track of things… github.com/GLAM-Work… #DayofDH2022

Thu, 28 Apr 2022 22:33:41 +1100

A couple of hours of #DayofDH2022 left – feeling a bit uninspired, so I’m going to do some pruning & reorganising of the #GLAMWorkbench issues list: github.com/GLAM-Work…

Tracking Trove changes over time

Wed, 20 Apr 2022 15:49:46 +1100

I’ve been doing a bit of cleaning up, trying to make some old datasets more easily available. In particular I’ve been pulling together harvests of the number of newspaper articles in Trove by year and state. My first harvests date all the way back to 2011, before there was even a Trove API. Unfortunately, I didn’t run the harvests as often as I should’ve and there are some big gaps. Nonetheless, if you’re interested in how Trove’s newspaper corpus has grown and changed over time, you might find them useful. They’re available in this repository and also in Zenodo.

This chart shows how the number of newspaper articles per year in Trove has changed from 2011 to 2022. Note the rapid growth between 2011 and 2015.

To try and make sure that there’s a more consistent record from now on, I’ve also created a new Git Scraper – a GitHub repository that automatically harvests and saves data at weekly intervals. As well as the number of articles by year and state, it also harvests the number of articles by newspaper and category. As mentioned, these four datasets are updated weekly. If you want to get all the changes over time, you can retrieve earlier versions from the repository’s commit history.

All the datasets are CC-0 licensed and validated with Frictionless.

There’s also a notebook in the GLAM Workbench that explores this sort of data.

The GLAM Workbench wants you!

Wed, 02 Mar 2022 15:52:20 +1100

Over the past few months I’ve been doing a lot of behind-the-scenes work on the GLAM Workbench – automating, standardising, and documenting processes for developing and managing repositories. These sort of things ease the maintenance burden on me and help make the GLAM Workbench sustainable, even as it continues to grow. But these changes are also aimed at making it easier for you to contribute to the GLAM Workbench!

Perhaps you’re part of a GLAM organisation that wants to help researchers explore its collection data – why not create your own section of the GLAM Workbench? It would be a great opportunity for staff to develop their own digital skills and learn about the possibilities of Jupyter notebooks. I’ve developed a repository template and some detailed documentation to get you started. The repository template includes everything you need to create and test notebooks, as well as built-in integration with Binder, Docker, Reclaim Cloud, and Zenodo. And, of course, I’ll be around to help you through the process.

Or perhaps you’re a researcher who wants to share some code you’ve developed that extends or improves an existing GLAM Workbench repository. Yes please! Or maybe you’re a GLAM Workbench user who has something to add to one of the lists of resources; or you’ve noticed a problem with some of the documentation that you’d like to fix. All contributions welcome!

The Get involved! page includes links to all this information, as well as some other possibilities such as becoming a sponsor, or sharing news. And to recognise those who make a contribution to the code or documentation there’s also a brand new contributors page.

I’m looking forward to exploring how we can build the GLAM Workbench together. #dhhacks

Omeka S Tools – new Python package

Thu, 17 Feb 2022 10:31:00 +1100

Over the last couple of years I've been fiddling with bits of Python code to work with the Omeka S REST API. The Omeka S API is powerful, but the documentation is patchy, and doing basic things like uploading images can seem quite confusing. My code was an attempt to simplify common tasks, like creating new items.

In case it's of use to others, I've now shared my code as a Python package. So you can just `pip install omeka-s-tools` to get started. The code helps you:

download lists of resources
search and filter lists of items
create new items
create new items based on a resource template
update and delete resources
add media to items
add map markers to items (assuming the Mapping module is installed)
upload templates exported from one Omeka instance to a new instance

There's quite detailed documentation available, including an example of adding a newspaper article from Trove to Omeka. If you want to see the code in action, there's also a notebook in the Trove newspapers section of the GLAM Workbench that uploads newspaper articles (including images and OCRd text) to Omeka from a variety of sources, including Trove searches, Trove lists, and Zotero libraries.

If you find any problems, or would like additional features, feel free to create an issue in the GitHub repository. #dhhacks

Testing, testing...

Fri, 28 Jan 2022 15:01:15 +1100

I regularly update the Python packages used in the different sections of the GLAM Workbench; though probably not as often as I should. Part of the problem is that once I've updated the packages, I have to run all the notebooks to make sure I haven't inadvertently broken something -- and this takes time. And in those cases where the notebooks need an API key to run, I have to copy and paste the key in at the appropriate spots, then remember to delete them afterwords. They're little niggles, but they add up, particularly as the GLAM Workbench itself expands.

I've been looking around at Jupyter notebook automated testing options for a while. There's nbmake, testbook, and nbval, as well as custom solutions involving things like papermill and nbconvert. After much wavering, I finally decided to give `nbval` a go. The thing that I like about `nbval` is that I can start simple, then increase the complexity of my testing as required. The `--nbval-lax` option just checks to make sure that all the cells in a notebook run without generating exceptions. You can also tag individual cells that you want to exclude from testing. This gives me a testing baseline -- this notebook runs without errors -- it might not do exactly what I think it's doing, but at least it's not exploding in flames. Working from this baseline, I can start tagging individual cells where I want the output of the cell to be checked. This will let me test whether a cell is doing what it's supposed to.

This approach means that I can start testing without making major changes to existing notebooks. The main thing I had to think about is how to handle API keys or other variables which are manually set by users. I decided the easiest approach was to store them in a `.env` file and use dotenv to load them within the notebook. This also makes it easy for users to save their own credentials and use them across multiple notebooks -- no more cutting and pasting of keys! Some notebooks are designed to run as web apps using Voila, so they expect human interaction. In these cases, I added extra cells that only run in the testing environment -- they populate the necessary fields and simulate button clicks to start.

While I was in a QA frame of mind, I also started playing with nbqa -- a framework for all sorts of code formatting, linting, and checking tools. I decided I'd try to standardise the formatting of my notebook code by running isort, black, and flake8. As well ask making the code cleaner and more readable, they pick up things like unused imports or variables. To further automate this process, I configured the `nbqa` checks to run when I try to commit any notebook code changes using `git`. This was made easy by the pre-commit package.

This is all set up and running in the Trove newspapers repository -- you can see the changes here. Now if I update the Python packages or make any other changes to the repository, I can just run `pytest --nbval-lax` to test every notebook at once. And if I make changes to an individual notebook, `nbqa` will automatically give the changes a code quality check before I save them to the repository. I'm planning to roll these changes out across the whole of the GLAM Workbench in coming months.

Developments like these are not very exciting for users, but they're important for the management and sustainability of the GLAM Workbench, and help create a solid foundation for future development and collaboration. Last year I created a GLAM Workbench repository template to help people or organisations thinking about contributing new sections. I can now add these testing and QA steps into the template to further share and standardise the work of developing the GLAM Workbench.

Some big pictures of newspapers in Trove and DigitalNZ

Thu, 09 Dec 2021 10:44:24 +1100

One of the things I really like about Jupyter is the fact that I can share notebooks in a variety of different formats. Tools like QueryPic can run as simple web apps using Voila, static versions of notebooks can be viewed using NBViewer, and live versions can be spun up as required on Binder. It’s also possible to export notebooks at PDFs, slideshows, or just plain-old HTML pages. Just recently I realised I could export notebooks to HTML using the same template I use for Voila. This gives me another way of sharing – static web pages delivered via the main GLAM Workbench site.

Here’s a couple of examples:

Papers Past newspapers in DigitalNZ – showing which Papers Past newspapers are available through DigitalNZ
Trove newspapers – number of issues per day, 1803–2020 – the number of newspaper issues published every day in Trove

Both are HTML pages that embed visualisations created using Altair. The visualisations are rendered using javascript, and even though the notebook isn’t running in a live computing environment, there’s some basic interactivity built-in – for example, you can hover for more details, and click on the DigitalNZ chart to search for articles from a newspaper. More to come! #dhhacks

Exploring GLAM data at ResBaz

Thu, 09 Dec 2021 10:13:48 +1100

The video of my key story presentation at ResBaz Queensland (simulcast via ResBaz Sydney) is now available on Vimeo. In it, I explore some of the possibilities of GLAM data by retracing my own journey through WWI service records, The Real Face of White Australia, #redactionart, and Trove – ending up at the GLAM Workbench, which brings together a lot of my tools and resources in a form that anyone can use. The slides are also available, and there’s an archived version of everything in Zenodo.

This and many other presentations about the GLAM Workbench are listed here. It seems I’ve given at least 11 talks and workshops this year! #dhhacks

GLAM Workbench Nectar Cloud Application updated!

Wed, 01 Dec 2021 11:20:53 +1100

The newly-updated DigitalNZ and Te Papa sections of the GLAM Workbench have been added to the list of available repositories in the Nectar Research Cloud’s GLAM Workbench Application. This means you can create your very own version of these repositories running in the Nectar Cloud, simply by choosing them from the app’s dropdown list. See the Using Nectar help page for more information.

I’ve also taken the opportunity to make use of the new container registry service developed by the ARDC as part of the ARCOS project. The app now pulls the GLAM Workbench Docker images from Quay.io via the container registry’s cache. This means that copies of the images are cached locally, speeding things up and saving on data transfers. Yay for integration!

Thanks again to Andy and the Nectar Cloud staff for their help! #dhhacks

DigitalNZ & Te Papa sections of the GLAMWorkbench updated!

Wed, 01 Dec 2021 10:41:00 +1100

In preparation for my talk at ResBaz Aotearoa, I updated the DigitalNZ and Te Papa sections of the GLAM Workbench. Most of the changes are related to management, maintenance, and integration of the repositories. Things like:

Setting up GitHub actions to automatically generate Docker images when the repositories change, and to upload the images to the Quay.io container registry
Automatic generation of an index.ipynb file based on README.md to act as a front page within Jupyter Lab
Addition of a reclaim-manifest.jps file to allow for one-click installation of the repository in Reclaim Cloud
Additional documentation in README.md with instructions on how to run the repository via Binder, Reclaim Cloud, Nectar Research Cloud, and Docker Desktop.
Addition of a .zenodo.json metadata file so that new releases are preserved in Zenodo
Switch to using pip-tools for generating requirements.txt files, and the include unpinned requirements in requirements.in
Update of all Python packages

From the user’s point of view, the main benefit of these changes is the ability to run the repositories in a variety of different environments depending on your needs and skills. The Docker images, generated using repo2Docker, are used by Binder, Reclaim Cloud, Nectar, and Docker Desktop. Same image, multiple environments! See ‘Run these notebooks’ in the DigitalNZ and Te Papa sections of the GLAM Workbench for more information.

Of course, I’ve also re-run all of the notebooks to make sure everything works and to update any statistics, visualisations, and datasets. As a bonus, there’s a couple of new notebooks in the DigitalNZ repository:

QueryPic DigitalNZ – a web app to visualise searches in Papers Past over time (I’ll post some further info about this shortly)
Papers Past newspapers in DigitalNZ – displays details of the Papers Past newspapers available through DigitalNZ (you can view the results as an HTML page)

#dhhacks

A template for GLAM Workbench development

Thu, 11 Nov 2021 17:03:43 +1100

I’m hoping that the GLAM Workbench will encourage GLAM organisations and GLAM data nerds (like me) to create their own Jupyter notebooks. If they do, they can put a link to them in the list of GLAM Jupyter resources. But what if they want to add the notebooks to the GLAM Workbench itself?

To make this easier, I’ve been working on a template repository for the GLAM Workbench. It generates a new skeleton repository with all the files you need to develop and manage your own section of the GLAM Workbench. It uses GitHub’s built in templating feature, together with Cookiecutter , and this GitHub Action by Stefan Buck. Stefan has also written a very helpful blog post.

The new repository is configured to do various things automatically, such as generate and save Docker images, and integrate with Reclaim Cloud and Zenodo. Lurking inside the dev folder of each new repository, you’ll find some basic details on how to set up and manage your development environment.

This is just the first step. There’s more documentation to come, but you’re very welcome to try it out. And, of course, if you are interested in contributing to the development of the GLAM Workbench, let me know and I’ll help get you set up!

Coming up! GLAM Workbench at ResBaz(s)

Thu, 04 Nov 2021 15:28:30 +1100

Want a bit of added GLAM with your digital research skills? You’re in luck, as I’ll be speaking at not one, but three ResBaz events in November. If you haven’t heard of it before, ResBaz (Research Bazaar) is ‘a worldwide festival promoting the digital literacy at the centre of modern research’.

On Wednesday, 24 November I’ll be giving a key story presentation (like a keynote, but with more story!) entitled Exploring GLAM data for ResBaz Queensland. This presentation will also be simulcast through ResBaz Sydney.
On Thursday, 25 November I’ll be giving a presentation on the GLAM Workbench for ResBaz Aotearoa – focusing in particular on NZ GLAM data.

The programs of all three ResBaz events are chock full of excellent opportunities to develop your digital skills, learn new research methods, and explore digital tools. If you’re an HDR student you should check out what’s on offer.

New video – using the Trove Newspaper & Gazette Harvester

Mon, 01 Nov 2021 10:32:00 +1100

The latest help video for the GLAM Workbench walks through the web app version of the Trove Newspaper & Gazette Harvester. Just paste in your search url and Trove API key and you can harvest thousands of digitised newspaper articles in minutes!

Harvest newspaper issues as PDFs

Mon, 01 Nov 2021 09:53:00 +1100

An inquiry on Twitter prompted me to put together a notebook that you can use to download all available issues of a newspaper as PDFs. It was really just a matter of copying code from other tools and making a few modifications. The first step harvests a list of available issues for a particular newspaper from Trove. You can then download the PDFs of those issues, supplying an optional date range. Beware – this could consume a lot of disk space!

The PDF file names have the following structure:

[newspaper identifier]-[issue date as YYYYMMDD]-[issue identifier].pdf

For example:

903-19320528-1791051.pdf

903 – the Glen Innes Examiner
19320528 – 28 May 1932
1791051 – to view in Trove just add this to http://nla.gov.au/nla.news-issue, eg http://nla.gov.au/nla.news-issue1791051

I also took the opportunity to create a new Harvesting data heading in the Trove newspapers section of the GLAM Workbench. #dhhacks

GLAM Workbench now in the Nectar Research Cloud!

Thu, 21 Oct 2021 10:18:11 +1100

The GLAM Workbench isn’t dependent on one big piece of technological infrastructure. It’s basically a collection of Jupyter notebooks, and those notebooks can be used within a variety of different environments. This helps make the GLAM Workbench more sustainable – new components can be swapped in and out as required. It also makes it possible to create different pathways for users, depending on their digital skills, institutional support, and research needs. For example, links to Binder make it easy for users to explore the possibilities of the GLAM Workbench and accomplish quick tasks. But Binder has limits. Where do you go when your research project scales up?

Earlier this year I added one-click installation of GLAM Workbench repositories in Reclaim Cloud. Today I’m very pleased to announce that selected GLAM Workbench repositories can be installed as applications within the Nectar Research Cloud. Using nationally-funded digital infrastructure, researchers in Australian universities can now create their own workbenches in minutes. So whether you’re harvesting truckloads of data from Trove or analysing web archives at scale, you can move beyond Binder and set up an environment dedicated to your research project. Cool huh?

Currently four repositories can be installed on Nectar in this way:

Trove newspapers
Trove newspaper and gazette harvester
Recordsearch (National Archives of Australia)
Web archives

But more will be added in the future as I update repositories to generate the necessary Docker images. Nectar installation information is now included in each of these four repositories, and I’ve added a Using the Nectar Cloud section to the help documentation that includes a detailed walkthrough of the installation process. If you strike any problems either raise an issue on GitHub, or ask a question at OzGLAM Chat.

Huge thanks to Andy, Jacob, and Jo at the Australian Research Data Commons (ARDC) who responded enthusiastically to my tweeted query, and packaged the repositories up into an easy-to-install, reusable application. After all the work I’ve put into the GLAM Workbench, it’s really exciting to see it embedded within Australia’s digital research infrastructure. #dhhacks

More GLAM Name Index updates from Queensland State Archives and SLWA

Mon, 18 Oct 2021 10:40:00 +1100

A new version of the GLAM Name Index Search is available. An additional 49 indexes have been added, bringing the total to 246. You can now search for names in more than 10.2 million records from 9 organisations.

The new indexes come from Queensland State Archives and the State Library of WA. QSA announced on Friday that they’d added two new indexes to their site. When I went to harvest them, I realised there was another 25 indexes that I hadn’t previously picked up. It seems that some QSA datasets are tagged as ‘Queensland State Archives’ in the data.qld.gov.au portal, but others are tagged as ‘queensland state archives’ – and the tag search is case sensitive! I now search for both the upper and lower case tags.

There’s also a number of additions from the State Library of WA. These datasets were already in my harvest, but because of some oddities in their formatting, I hadn’t included them in the Index Search. Looking at them again, I realised they were right to go, so I’ve added them in.

Here’s the list of additions:

Queensland State Archives

Australian South Sea Islanders 1867 to 1908 - A-K
Australian South Sea Islanders 1867 to 1908 L-Z
Beaudesert Shire Burials - Logan Village 1878-2000 - Beaudesert Shire and Logan Village Burials 1878-2000
Immigrants, Bowen Immigration Depot 1885-1892
Brisbane Gaol Hospital Admission registers 1889-1911 - Index to Brisbane Gaol Hospital Admission Registers 1889-1911
Index to Correspondence of Queensland Colonial Secretary 1859-1861 - Index to Colonial Secretary s Correspondence Bundles 1859 - 1861.csv
Dunwich Benevolent Asylum records - Index to Dunwich Benevolent Asylum 1859-1948
Dunwich Benevolent Asylum records - Index to Dunwich Benevolent Asylum 1885-1907
Immigrants and Crew 1860-1865 (COL/A) - Index to Immigrants and Crew 1860 - 1964
Index to Immigration 1909-1932
Outdoor Relief 1900-1904 - Index to Outdoor Relief 1892-1920
Pensions 1908-1919 - Index to Pensions 1908-1909
Cases & treatment Moreton Bay Hospital 1830-1862 - Index to Register of Cases and treatment at Moreton Bay Hospital 1830-1862
Index to Registers of Agricultural Lessees 1885-1908
Index to Registers of Immigrants, Rockhampton 1882-1915
Pneumonic influenza patients, Wallangarra Quarantine Compound - Index to Wallangarra Flu Camp 1918-1919
Land selections 1885-1981
Lazaret patient registers - Lazaret Patient Registers
Leases, Selections and Pastoral Runs and other related records 1850-2014
Perpetual Lease Selections of soldier settlements 1917 - 1929 - Perpetual Lease Selections of soldier settlements 1917-1929
Photographic records of prisoners 1875-1913 - Photographic Records of Prisoners 1875-1913
Redeemed land orders 1860-1906 - Redeemed land orders 1860-1907
Register of the Engagement of Immigrants at the Immigration Depot - Bowen 1873-1912
Registers of Applications by Selectors 1868-1885
Registers of Immigrants Promissory Notes (Maryborough)
Education Office Gazette Scholarships 1900 - 1940 - Scholarships in the Education Office Gazette 1900 - 1940
Teachers in the Education Office Gazettes 1899-1925

State Library of Western Australia

WABI Subset: Eastern Goldfields - Eastern Goldfields
Western Australian Biographical Index (WABI) - Index entries beginning with A
Western Australian Biographical Index (WABI) - Index entries beginning with B
Western Australian Biographical Index (WABI) - Index entries beginning with C
Western Australian Biographical Index (WABI) - Index entries beginning with D and E
Western Australian Biographical Index (WABI) - Index entries beginning with F
Western Australian Biographical Index (WABI) - Index entries beginning with G
Western Australian Biographical Index (WABI) - Index entries beginning with H
Western Australian Biographical Index (WABI) - Index entries beginning with I and J
Western Australian Biographical Index (WABI) - Index entries beginning with K
Western Australian Biographical Index (WABI) - Index entries beginning with L
Western Australian Biographical Index (WABI) - Index entries beginning with M
Western Australian Biographical Index (WABI) - Index entries beginning with N
Western Australian Biographical Index (WABI) - Index entries beginning with O
Western Australian Biographical Index (WABI) - Index entries beginning with P and Q
Western Australian Biographical Index (WABI) - Index entries beginning with R
Western Australian Biographical Index (WABI) - Index entries beginning with S
Western Australian Biographical Index (WABI) - Index entries beginning with T
Western Australian Biographical Index (WABI) - Index entries beginning with U-Z
Digital Photographic Collection - Pictorial collection_csv
WABI subset: Police - WABI police subset
WABI subset: York - York and districts subset

Bonus update

After a bit more work last night I added in a dataset from the State Library of Victoria:

Melbourne and metropolitan hotels, pubs and publicans

That’s an extra 21,000 records, and takes the total number of indexes to 247 from 10 different GLAM organisations!

#dhhacks

Getting data about newspaper issues in Trove

Fri, 15 Oct 2021 10:44:00 +1100

When you search Trove’s newspapers, you find articles – these articles are grouped by page, and all the pages from a particular date make up an issue. But how do you find out what issues are available? How do you get a list of dates when newspapers were published? This notebook in the GLAM Workbench shows how you can get information about issues from the Trove API.

Using the notebook, I’ve created a couple of datasets ready for download and use.

Total number of issues per year for every newspaper in Trove

Harvested 10 October 2021

CSV formatted dataset containing the number of newspaper issues available on Trove, totalled by title and year – comprises 27,604 rows with the fields:

title – newspaper title
title_id – newspaper id
state – place of publication
year – year published
issues – number of issues

Download from Cloudstor: newspaper_issues_totals_by_year_20211010.csv (2.1mb)

Complete list of issues for every newspaper in Trove

Harvested 10 October 2021

CSV formatted dataset containing a complete list of newspaper issues available on Trove – comprises 2,654,020 rows with the fields:

title – newspaper title
title_id – newspaper id
state – place of publication
issue_id – issue identifier
issue_date – date of publication (YYYY-MM-DD)

To keep the file size down, I haven’t included an issue_url in this dataset, but these are easily generated from the issue_id. Just add the issue_id to the end of http://nla.gov.au/nla.news-issue. For example: http://nla.gov.au/nla.news-issue495426. Note that when you follow an issue url, you actually get redirected to the url of the first page in the issue.

Download from Cloudstor: newspaper_issues_20211010.csv (222mb)

For more information see the Trove newspapers section of the GLAM Workbench.

GLAM Workbench at eResearch Australasia 2021

Fri, 15 Oct 2021 09:50:58 +1100

Way back in 2013, I went to the eResearch Australasia conference as the manager of Trove to talk about new research possibilities using the Trove API. Eight years years later I was back, still spruiking the possibilities of Trove data. This time, however, I was discussing Trove in the broader context of GLAM data – all the exciting possibilities that have emerged as galleries, libraries, archives and museums make more of their collections available in machine-readable form. The big question is, of course, how do researchers, particularly those in the humanities, make use of that data? The GLAM Workbench is my attempt to address that question – to provide humanities researchers with both the tools and information they need, and an understanding of the possibilities that might emerge if they invest a bit of time in working with GLAM data. My eResearch Australasia 2021 presentation provides a quick introduction to the GLAM Workbench, here’s the video, and the slides.

A GLAM Workbench for humanities researchers from Tim Sherratt on Vimeo.

The presentation was pre-recorded, but I managed to sneak in an update via chat for those who attended the session. More news on this next week… 🥳

New Python package to download Trove newspaper images

Tue, 05 Oct 2021 12:03:00 +1100

There’s no reliable way of downloading an image of a Trove newspaper article from the web interface. The image download option produces an HTML page with embedded images, and the article is often sliced into pieces to fit the page.

This Python package includes tools to download articles as complete JPEG images. If an article is printed across multiple newspaper pages, multiple images will be downloaded – one for each page. It’s intended for integration into other tools and processing workflows, or for people who like working on the command line.

You can use it as a library:

from trove_newspaper_images.articles import download_images

images = download_images('107024751')

Or from the command line:

trove_newspaper_images.download 107024751 --output_dir images

If you just want to quickly download an article as an image without installing anything, you can use this web app in the GLAM Workbench. To download images of all articles returned by a search in Trove, you can also use the Trove Newspaper and Gazette Harvester.

See the documentation for more information. #dhhacks

More records for the GLAM Name Index Search

Wed, 29 Sep 2021 12:17:35 +1100

Two more datasets have been added to the GLAM Name Index Search! From the History Trust of South Australia and Collab, I’ve added:

Passengers in History – that’s 371,894 records of people arriving in South Australia from 1836 to 1961
Women’s Suffrage Petition 1894 (South Australia) – another 10,638 names

In total there’s 9.67 million name records to search across 197 datasets provided by 9 GLAM organisations!

More QueryPic in action

Wed, 29 Sep 2021 11:30:26 +1100

Recently I created a list of publications that made use of QueryPic, my tool to visualise searches in Trove’s digitised newspapers. Here’s another example of the GLAM Workbench and QueryPic in action, in Professor Julian Meyrick’s recent keynote lecture, ‘Looking Forward to the 1950s: A Hauntological Method for Investigating Australian Theatre History’.

Some thoughts on the ‘Trove Researcher Platform for Advanced Research’ draft plan

Fri, 10 Sep 2021 11:12:00 +1100

Late last year the Federal Government announced it was making an $8.9 million investment in HASS and Indigenous research infrastructure. This program is being managed by the ARDC and will lead to the development of a HASS Research Data Commons. According to the ARDC, a research data commons:

brings together people, skills, data, and related resources such as storage, compute, software, and models to enable researchers to conduct world class data-intensive research

Sounds awesome!

Based on scoping studies commissioned by the Department of Education, Skills, and Employment (which have not yet been made public), four activities were selected for initial funding under this program. Draft project plans for these four activities have now been released for public comment.

One of these activities aims to develop a ’Trove researcher platform for advanced research’:

Augmenting existing National Library of Australia resources, this platform will enable a focus on the delivery of researcher portals accessible through Trove, Australia’s unique public heritage site. The platform will create tools for visualisation, entity recognition, transcription and geocoding across Trove content and other corpora.

You can download the draft project plan for the Trove platform. Funding for this activity will be capped at $2,301,185 across 2021-23. In this post I’ll try to pull together some of my own thoughts on this plan.

I suppose I’d better start with a disclaimer – I’m not a neutral observer in this. I started scraping data from Trove newspapers way back in 2010, building the first versions of tools like QueryPic and the Trove Newspaper Harvester. While I was manager of Trove, from 2013 to 2016, I argued for recognition of Trove as a key part of Australia’s humanities research infrastructure, and highlighted possible research uses of Trove data available through the API. Since then I’ve worked to bring a range of digital tools, examples, tutorials, and hacks together for researchers in the GLAM Workbench – a large number of these work with data from Trove.

I strongly believe that Trove should receive ongoing funding through NCRIS as a piece of national research infrastructure. Unfortunately though, the draft project plan does not make a strong case for investment – it’s vague, unimaginative, and makes little attempt to integrate with existing tools and services. I think it scores poorly against the ARDC’s evaluation criteria, and doesn’t seem to offer good value for money. As someone who has championed the use of Trove data for research across the last decade, I’m very disappointed.

What’s planned?

So what is being proposed? There seems to be three main components:

Authenticated ‘project’ spaces for researchers where datasets relating to a particular research topic can be stored
The ability to create custom datasets from a search in Trove
Tools to visualise stored datasets.

There’s no doubt that these are all useful functions for researchers, but many problems arise when we look at how they’re going to be implemented.

1. Authenticated project spaces

The draft plan indicates that authentication of users through the Australian Access Federation is preferred. Why? Trove already has a system for the creation of user accounts. Using AAF would limit use of the new platform to those attached to universities or research agencies. I don’t understand what the use of AAF adds to the project, except perhaps to provide an example of integration with existing infrastructure services.

The plan notes that project spaces could be ‘public’ or ‘private’. Presumably a ‘public’ space would give access to stored datasets, but what sort of access controls would be available in relation to individual datasets? It’s also noted (Deliverable 7) that researchers would have ‘an option to “publish” their research findings for public consumption‘. Does this mean datasets and visualisations would be assigned a DOI (or other persistent identifier) and preserved indefinitely? How might these spaces integrate with existing data repositories?

2. Create custom datasets

The lack of detail in the plan makes it difficult to assess what’s being proposed here. But it seems that users would be able to construct a search using the Trove web interface (or a new search interface?) and save the results as a dataset.

What data would be searched? It’s not clear, but in reference to the visualisations it’s stated that data would come from ’Trove’s existing full text collections (newspapers and gazettes, magazines and newsletters, books)’. So no web archives, and no metadata from any of Trove’s aggregated collections (even without full text, collection metadata can create interesting research possibilities, see for example the Radio National records in the GLAM Workbench).

What will be included in each dataset? There’s few details, but at a minimum you’d expect something like a CSV containing the metadata of all the matching records, and files containing the full text content of the items. These could potentially be very large. There’s no indication about how storage and processing demands would be managed, but presumably there would be some per user, or per project, limits.

Deliverable 8, ‘Data and visual download’, states that:

All query results must be available as downloadable files, this would include CSV, JSON and XML for the query results list.

But there’s no mention of the full text content at all. Will it be included in downloadable datasets?

As well as the record metadata and full text, you’d want there to be some metadata captured about the dataset itself – the search query used, when it was captured, the number of records, etc. To support integration and reuse, it would be good to align this with something like RO Crate.

How will searches be constructed? It’s not clear if this will be integrated with the existing search interface, or be something completely separate; however, the plan does note that ‘limitations are put onto the dataset like keyword search terms and filters corresponding to the filters currently available in the interface’. So it seems that the new platform will be using the existing search indexes. It’s obviously important for the relationship between existing search functions and the new dataset creation tool to be explicit and transparent so that researchers understand what they’re getting.

It’s also worth noting that changes to the search interface last year removed some useful options from the advanced search form. In particular, you can no longer exclude matches in tags or comments. If you’re a researcher looking for the occurrence of a particular word, you generally don’t want to include records where that word only appears in a user added tag (I have a story about ‘Word War I’ that illustrates this!).

This raises a broader issue. There doesn’t seem to be any mention in the project plan of work to improve the metadata and indexing in response to research needs. Even just identifying digitised books in the current web interface can be a bit of a challenge, and digitised books and periodicals can be grouped into work records with other versions. We need to recognise that the needs of discovery sometimes compromise specific research uses.

I’m trying to be constructive in my responses here, but at this point I just have to scream – WHAT ABOUT THE TROVE NEWSPAPER HARVESTER? A tool has existed for ten years that lets users create a dataset containing metadata and full text from a search in Trove’s newspapers and gazettes. I’ve spent a lot of time over recent years adding features and making it easier to use. Now you can download not only full text, but also PDFs and images of articles. The latest web app version in the GLAM Workbench runs in the cloud. Just one click to start it up, then all you need to do is paste in your Trove API key and the url of your search. It can’t get much easier.

The GLAM Workbench also includes tools to create datasets and download OCRd text from Trove’s books and digitised journals. These are still in notebook form, so are not as easy to use, but I have created pre-harvested datasets of all books and periodicals with OCRd text, and stored them on CloudStor. What’s missing at the moment is something to harvest a collection of journal articles, but this would not be difficult. As an added bonus, the GLAM Workbench has tools to create full text datasets from the Australian Web Archive.

So what is this project really adding? And why is there no attempt to leverage existing tools and resources?

3. Visualise datasets

Again, there’s a fair bit of hand waving in the plan, but it seems that users will be able to select a stored dataset and then choose a form of visualisation. The plan says that:

An initial pilot would allow users to create line graphs that plot the frequency of a search term over time and maps that display results based on state-level geolocation.

Up to three additional visualisations would be created later based on research feedback. It’s not clear which researchers will be consulted and when their feedback will be sought.

The value of these sorts of visualisations is obviously dependent on the quality and consistency of the metadata. There’s nothing built into this plan that would, for example, allow a researcher to clean or normalise any of the saved data. You have to take what you’re given. The newspaper metadata is generally consistent, but books and periodicals less so.

It’s also important to clarify what’s meant by ‘the frequency of a search term over time’. Does this mean the number of records matching a search term, or the number of times that the search term actually appears in the full text of all matched records? If the latter, then this would be a major enrichment of the available data. Though if this data was available it should be pushed through the API and/or made available as a downloadable dataset for integration with other platforms (perhaps along the lines of the Hathi Trust’s Extracted Features Dataset). I suspect, however, that what is actually meant is the number of matching search results.

Again, the value of any geospatial visualisation depends on what is actually being visualised! The state facet in newspapers indicates place of publication, it’s not clear what the place facet in other categories represents. For this sort of visualisation to be useful in a research context, there would need to be some explanation of how these values were created, and any gaps or uncertainties.

Time for another scream of frustration — WHAT ABOUT QUERYPIC? Another long-standing tool which has already been cited a number of times in research literature. QueryPic visualises searches in Trove’s newspapers and gazettes over time. You can adjust time scales and intervals, and download the results as images, a CSV file, and an HTML page. The project plan makes a point of claiming that its tools would not require any coding, but neither does QueryPic. Just plug in an API key and a search URL. I even made some videos about it! The GLAM Workbench also includes a number of examples of how you can visualise places of publication of newspaper articles.

But it’s not just the GLAM Workbench. The Linguistics Data Commons of Australia, another activity to be funded as part of the HASS Research Data Commons, will include tools for text analysis and visualisation. The Time Layered Cultural Map is developing tools for geospatial visualisation of Australian collections. Surely the focus should be on connecting and reusing what’s available. Again I’m wondering what this project is really adding.

Portals and platforms

The original language describing the funded activity is interesting — it is intended to ‘focus on the delivery of researcher portals accessible through Trove’.

Portals (plural) accessible through (not in) Trove.

The NLA could meet a fair proportion of its stated objectives right now, simply by including links to QueryPic and the Trove Newspaper and Gazette Harvester. Done! There’s a million dollars saved.

More seriously, there’s no reason why the outcome of this activity should be a new interface attached to Trove and managed by the NLA. Indeed, such an approach works against integration, reuse, and data sharing. I believe the basic assumptions of the draft plan are seriously flawed. We need to separate out the strands of what’s meant by a ‘platform for advanced research’, and think more creatively and collaboratively about how we could achieve something useful, flexible, and sustainable.

Where’s the API?

I think the primary role of the NLA in the development of this research platform should be as the data provider. There are numerous ways in which Trove’s data might be improved and enriched in support of new research uses. These improvements could then be pushed through the API to integrate with a range of tools and resources. Which raises the question — where is the API in this plan?

The only mention of the API comes as an option for a user with ‘high technical expertise’ to extend the analysis provided by the built-in visualisations. This is all backwards. The API is the key pipeline for data-sharing and integration and should be at the heart of this plan.

This program offers an opportunity to make some much-needed improvements to the API. Here’s a few possibilities:

Bring the web interface and API back into sync so that researchers can easily transfer queries between the two (Trove’s interface update introduced new categories, while the API still groups resources by the original zones).
Provide public API access to additional data about digitised items. For example, you can get lists of newspaper titles and issues from the API, but there’s no comparable method to get titles and issues for digitised periodicals. The data’s there – it’s used to generate lists of issues in the browse interface – but it’s not in the API. There’s also other resource metadata, such as parent/child relationships, which are embedded in web pages but not exposed in the API.
Standardise the delivery of OCRd text for different resource types.
Finally add the People & Organisations data to the main RESTful API.
Fix the limitations of the web archives CDX API (documented here).
Add a search API for the web archives.
And what about a Write API? Integration between components in the HASS RDC would be greatly enhanced if other projects could automatically add structured annotations to existing Trove resources.

I think the HASS RDC would benefit greatly by thinking much more about the role of the Trove API in establishing reusable data flows, and connecting up components.

Pathways

Anyone who’s been to one of my GLAM Workbench talks will know that I talk a lot about ‘pathways’. My concern is not just to provide useful tools and examples, but to try and connect them in ways that encourage researchers to develop their skills and confidence. So a researcher with limited digital skills can spin up QueryPic and start making visualisations without any specialised knowledge. But if they want to explore the data and assumptions behind QueryPic, they can view a notebook that walks them through the process of getting data from facets and assembling a time series. If they find something interesting in QueryPic, they can go to the Newspaper Harvester and assemble a dataset that helps them zoom into a particular period. There are places to go.

Similarly, users can start making use of the GLAM Workbench in the cloud using Binder – one click and it’s running. But as their research develops they might find Binder a bit limiting, so there are options to spin up the GLAM Workbench using Reclaim Cloud or Docker. As a researcher’s skills, needs, and questions change, so does their use of the GLAM Workbench. At least that’s the plan – I’m very aware that there’s much, much more to do to build and document these pathways.

The developments described in the draft plan are focused on providing simple tools for non-technical users. That’s fair enough, but you have to give those users somewhere to go, some path beyond, or else it just becomes another dead end. Users can download their data or visualisation, but then what?

Of course you don’t point a non-coder to API documentation and say ‘there you go’. But coders can use the API to build and share a range of tools that introduce people to the possibilities of data, and scaffold their learning. Why should there be just one interface? It’s not too difficult to imagine a range of introductory visualisation tools aimed at different humanities disciplines. Instead of focusing inward on a single Trove Viz Lite tool, why not look outwards at ways of embedding Trove data within a range of research training contexts?

Integration

A number of the HASS RDC Evaluation Criteria focus on issues of integration, collaboration, and reuse of existing resources. For example:

Project plans should display robust proposal planning including the maximisation of the use or re-use of existing research infrastructure, platforms, tools, services, data storage and compute.

Project plans should display integrated infrastructure layers with other HASS RDC activities, in particular by linking together elements such as data storage, tools, authentication, licensing, networks, cloud and high-performance computing, and access to data resources for reuse.
Project plans must be robust and contribute to the HASS RDC as a coherent whole that capitalises on existing data collections, adheres to the F.A.I.R. principles, develops collaborative tools, utilises shared underlying infrastructure and has appropriate governance planning.

There’s little evidence of this sort of thinking in the draft project plan. I’ve mentioned a few obvious opportunities for integration above, but there are many more. Overall, I think the proposed ‘platform for advanced research’ needs to be designed as a series of interconnected components, and not be seen as the product of a single institution.

We could imagine, for example, a system where the NLA focused on the delivery of research-ready data via the Trove API. A layer of data filtering, cleaning, and packaging tools could be built on top of the API to help users assemble actionable datasets. The packaging processes could use standards such as RO-Crate to prepare datasets for ingest into data repositories. Existing storage services, such as CloudStor, could be used for saving and sharing working datasets. Another layer of visualisation and analysis tools could either process these datasets, or integrate directly with the API. These tools could be spread across different projects including LDaCA, TLCMap, and the GLAM Workbench — using standards such as Jupyter to encourage sharing and reuse of individual components, and running on a variety of cloud-hosted platforms. Instead of just adding another component to Trove, we’d be building a collaborative network of tool builders and data wranglers — developing capacities across the research sector, and spreading the burden of maintenance.

Sustainability

The draft project plan includes some pretty worrying comments about long-term support for the new platform. Work Package 5 notes:

The developed product will require support post release which can be guaranteed for a period not exceeding the contracted period for this project

And:

ARDC will be responsible for providing ongoing financial support for this phase. It has not been included in the proposal.

So once the project is over, the NLA will not support the new platform unless the ARDC provides ongoing funding. What researcher would want to ‘publish’ their data on a platform that could disappear at any time? We all know that sustainability is hard, but you would think that the NLA could at least offer to work collaboratively with the research sector to develop a plan for sustainability, instead of just asking for more money. Why would anyone invest so much for so little?

Leadership and community

The development of collaborations and communities also figure prominently in the HASS RDC Evaluation Criteria. For example:

Project plans should clearly demonstrate that they enable collaboration and build communities across geographically dispersed research groups through facilitated sharing of high-quality data, particularly for computational analysis; the development of new platforms for collaboration and sharing; and, the encouragement of innovative methodologies through the use of analytic tools.

Project plans must include a demonstrated commitment to ongoing community development to ensure the sustainability of the development is vital. The deliverables will act as ongoing national research infrastructure. They must be broadly usable by more than just the project partners and serve as input to a wide range of research.
Project plans, and project leads in particular, should demonstrate the research leadership that will foster and encourage the uptake and use of the HASS RDC.

Once again the draft project plan falls short. There are no project partners listed. Instead the plan refers broadly to all of Trove’s content partners, none of whom have direct involvement in this project. Indeed, as noted above, data aggregated from project parters is excluded from the new platform.

There are no new governance arrangements proposed for this project. Instead the plan refers to the Trove Strategic Advisory Committee which includes representatives from partner organisations. But there are no researcher representatives on this committee.

The only consultation with the research sector undertaken in the ‘Consultation Phase’ of the project is that undertaken by the ARDC itself. Does that mean this current process whereby the ARDC is soliciting feedback on the project plans? Whoa, meta…

The plan notes that during the testing phase described in Work Package 3, ‘HASS community members would gain access to a beta version of the product for comment’. However, later it is stated that access would be provided to ’a subset of researchers’, and that only system bugs and ‘high priority improvements’ would be acted upon.

Generally speaking, it seems that the NLA is seeking as little consultation as possible. It’s not exploring options for collaboration. It’s not engaging with the research community about these developments. That doesn’t seem like an effective way to build communities. Nor does it demonstrate leadership.

Summing up

This project plan can’t be accepted in its current form. We’ve had failures and disappointments in the development of HASS research infrastructure in the past. The HASS RDC program gives us a chance to start afresh, and the focus on integration, data-sharing, and reuse give hope that we can build something that will continue to grow and develop, and not wither through lack of engagement and support. So should the NLA be getting $2 million to add a new component to Trove that is not integrated with other HASS RDC projects, and substantially duplicates tools available elsewhere? No, I don’t think so. They need to go back to the drawing board, undertake some real consultation, and build collaborations, not products.

Some research projects that have used QueryPic

Mon, 30 Aug 2021 12:48:11 +1100

A Twitter thread about some of the research uses of QueryPic…

QueryPic, my tool for visualising searches in @TroveAustralia’s digitised newspapers, has been around in different forms for more than 10 years. The latest version is part of the #GLAMWorkbench: https://t.co/qnY5tVDwgY #researchinfrastructure pic.twitter.com/QyHWJwGV3u
— Tim Sherratt (@wragge) August 29, 2021

I thought I’d highlight some of the research publications that have made use of QueryPic over the years, so, in no particular order...
— Tim Sherratt (@wragge) August 29, 2021

There’s @Airminded’s article in @HistAustJournal – Brett Holman (2013) 'Dreaming War: Airmindedness and the Australian Mystery Aeroplane Scare of 1918', History Australia, 10:2, 180-201, DOI: https://t.co/2wgiLueHGL
— Tim Sherratt (@wragge) August 29, 2021

A book! Simon Sleight, Young People and the Shaping of Public Space in Melbourne, 1870–1914, Ashgate Publishing, Ltd., 2013. https://t.co/CPgGMrYYYq pic.twitter.com/XryAF0hJ5K
— Tim Sherratt (@wragge) August 29, 2021

Yorick Smaal (2013) Keeping it in the family: prosecuting incest in colonial Queensland, Journal of Australian Studies, 37:3, 316-332, DOI: https://t.co/n5tQlER9Vo pic.twitter.com/tKzpAosu1i
— Tim Sherratt (@wragge) August 29, 2021

In @AHSjournal there’s – Murray G. Phillips & Gary Osmond (2015) Australia's Women Surfers: History, Methodology and the Digital Humanities, Australian Historical Studies, 46:2, 285-303, DOI: https://t.co/Gxs1Ru6Ojt
— Tim Sherratt (@wragge) August 29, 2021

Gary Osmond (2015) ‘Pink Tea and Sissy Boys’: Digitized Fragments of Male Homosexuality, Non-Heteronormativity and Homophobia in the Australian Sporting Press, 1845–1954, The International Journal of the History of Sport, 32:13, 1578-1592, DOI: https://t.co/C6FndD7C4E
— Tim Sherratt (@wragge) August 29, 2021

Murray G. Phillips, Gary Osmond & Stephen Townsend (2015) A Bird’s-Eye View of the Past: Digital History, Distant Reading and Sport History, The International Journal of the History of Sport, 32:15, 1725-1740, DOI: https://t.co/4rB2hkmmDM
— Tim Sherratt (@wragge) August 29, 2021

Sarah Ailwood and Maree Sainsbury, ‘Copyright Law, Readers and Authors in Colonial Australia’, Journal of the Association for the Study of Australian Literature, vol. 14, no. 3, 2014. https://t.co/XWqx8XJGLQ
— Tim Sherratt (@wragge) August 29, 2021

Sarah Ailwood and Maree Sainsbury, ‘The Imperial Effect: Literary Copyright Law in Colonial Australia’, Law, Culture and the Humanities, vol. 12, no. 3, 1 October 2016, pp. 716–740. https://t.co/s6HrBZmQ6N
— Tim Sherratt (@wragge) August 29, 2021

A book chapter by @JVLamond – Lamond, J, 2016, 'Zones of Connection: Common Reading in a Regional Australian Library', in Print Culture Histories Beyond the Metropolis, University of Toronto Press, Toronto, pp. 355-374. https://t.co/o3oAmreYne
— Tim Sherratt (@wragge) August 29, 2021

Not just history – Scifleet, P., Henninger, M. & Albright, K.H. (2013). When social media are your source. Information Research, 18(3) paper C41. https://t.co/qOYbZ3TMTf pic.twitter.com/GDP2TmeUzp
— Tim Sherratt (@wragge) August 29, 2021

There’s also a number of references to QueryPic as a tool in the DH & library literature, that I won’t list.

There’s probably more – citation of tools like QueryPic can be a bit hit and miss.
— Tim Sherratt (@wragge) August 29, 2021

The latest version of QueryPic is designed to be both easy-to-use and flexible – click a link to start it up, paste in your @TroveAustralia API key, and a search url from Trove… and bingo!

For a quick intro, see this video: https://t.co/Hh1oDIOh9a
— Tim Sherratt (@wragge) August 29, 2021

But even though it’s easy to get started, QueryPic can do interesting things like compare queries. You can also adjust facets, date ranges, and time scales.

This video shows you how to create more complex queries: https://t.co/0CoJhO7vaJ
— Tim Sherratt (@wragge) August 29, 2021

As I often say, not all #researchinfrastructure has to be big. A simple tool like this can help researchers see their topics in new ways.

And from this starting point, there’s all sorts of pathways to follow in the #GLAMWorkbench https://t.co/AC2tipN8eY
— Tim Sherratt (@wragge) August 29, 2021

Government publications in Trove

Mon, 30 Aug 2021 12:21:46 +1100

Over the last few weeks I’ve been updating my harvests of OCRd text from digitised books and periodicals in Trove. As part of the harvesting process, I’ve created lists of both that are available in digital form – this includes digitised works, as well as those that are born-digital (such as PDFs or epubs). I’ve published the full lists of books and periodicals as searchable databases to make them easy to explore.

One thing that you might notice is that works with the format ‘Government publication’ pop up in both lists – sometimes it’s not clear whether something is a ‘book’ or ‘periodical’. To make it easier to find these items, no matter what their format, I’ve combined data from my two harvests and created a searchable dataset of government publications. It includes links to download OCRd text from CloudStor if available.

All three databases make use of Datasette, which I’ve also used for the GLAM Name Index Search. One of the cool things about Datasette is that it provides it’s own API, so if you find some interesting in any of these databases, you can easily download the machine-readable data for further analysis. #dhhacks

GLAM Workbench – a platform for digital HASS research

Thu, 26 Aug 2021 17:31:00 +1100

We’re in the midst of planning for the HASS Research Data Commons, which will deliver some much-needed investment in digital research infrastructure for the humanities and social sciences. Amongst the funded programs are tools for text analysis as part of the Linguistics Data Commons, and a platform for more advanced research using Trove. I’m hoping that this will be an opportunity to take stock of existing tools and resources, and build flexible pathways for researchers that enable them to collect, move, analyse, preserve, and share data across different platforms and services.

To this end, I thought it might be useful to try and summarise what the GLAM Workbench offers, particularly for Trove researchers. The GLAM Workbench doesn’t really have an institutional home, and is mostly unfunded – it’s my passion project. That means that it’s easy to overlook, particularly when the big grants are being doled out. But I think it has a lot to offer and I’m looking forward to exploring ways it can connect with these new initiatives.

Getting and moving data

Get newspaper articles in bulk with the Trove Newspaper and Gazette Harvester – This has been around in some form for more than ten years (it pre-dates the Trove API!). Give it the url of a search in Trove’s newspapers and gazettes and the harvester will save all the metadata in a CSV file, and optionally download the complete articles as OCRd text, images, or PDFs. The amount of data you harvest is really only limited by your patience and disk space. I’ve harvested more than a million articles in the past. The GLAM Workbench includes a web app version of the harvester that runs live in the cloud – just paste in your Trove API key and the search url, and click the button.
Get Trove newspaper pages as images – If you need a nice, high-resolution version of a newspaper page you can use this web app. If you want to harvest every front page (or some other particular page) here’s an example that gets all the covers of the Australian Women’s Weekly. A pre-harvested collection of the AWW covers is included as a bonus extra.
Get Trove newspaper articles as images – The Trove web interface makes it difficult to download complete images of articles, but this tool will do the job. There’s a handy web app to grab individual images, but the code from this tool is reused in other places such as the Trove Newspaper Harvester and the Omeka uploader, and could be built-in to your own research workflows.
Upload Trove newspaper articles to Omeka – Whether you’re creating on online exhibition or building a research database, Omeka can be very useful. This notebook connects Trove’s newspapers to Omeka for easy upload. Your selected articles can come from a search query, a Trove list, a Zotero library, or just a list of article ids. Metadata records are created in Omeka for each article and newspaper, and an image of each article is attached.
Get OCRd text from digitised periodicals in Trove – They’re often overshadowed by the newspapers, but there’s now lots of digitised journals, magazines, and parliamentary papers in Trove. You can get article-level data from the API, but not issue data. This notebook enables researchers to get metadata and OCRd text from every available issue of a periodical. To make researchers’ lives even easier, I regularly harvest all the available OCRd text from digitised periodicals in Trove. The latest harvest downloaded 51,928 issues from 1,163 periodicals – that’s about 10gb of text. You can browse the list of periodicals with OCRd text, or search this database. All the OCRd text is stored in a public repository on CloudStor.
Get page images from digitised periodicals in Trove – There’s more than text in digitised periodicals, and you might want to download images of pages for visual analysis. This notebook shows you how to get cover images, but could be easily modified to get another page, or a PDF. I used a modified version of this to create a collection of 3,471 full page editorial cartoons from The Bulletin, 1886 to 1952 – all available to download from CloudStor.
Get OCRd text from digitised books in Trove – Yep, there’s digitised books as well as newspapers and periodicals. You can download OCRd text from an individual book using the Trove web interface, but how do you make a collection of books without all that pointing and clicking? This notebook downloads all the available OCRd text from digitised books in Trove. The latest harvest includes text from 26,762 works. You can explore the results using this database.
Harvest parliamentary press releases from Trove – Trove includes more than 380,000 press releases, speeches, and interview transcripts issued by Australian federal politicians and saved by the Parliamentary Library. This notebook shows you how to harvest both metadata and fulltext from a search of the parliamentary press releases. For example, here’s a collection of politicians talking about ‘refugees’, and another relating to COVID-19.
Harvest details of Radio National programs from Trove – Trove creates records for programs broadcast on ABC Radio National, for the major current affairs programs these records at at segment level. Even though they don’t provide full transcripts, this data provide a rich, fine-grained record of Australia’s recent political, social, and economic history. This notebook shows you how to download the Radio National data. If you just want to dive straight in, there’s also a pre-harvested collection containing more than 400,000 records, with separate downloads for some of the main programs.
Find all the versions of an archived web page in Trove – Many of the tools in the Web Archives section of the GLAM Workbench will work with the Australian Web Archive, which is part of Trove. This notebook shows you how to get data about the number of times a web page has been archived over time.
Harvesting collections of text from archived web pages in Trove – If you want to explore how the content of a web page changes over time, you can use this notebook to capture the text content of every archived version of a web page.
Convert a Trove list into a CSV file – While Trove provides a data download option for lists, it leaves out a lot of useful data. This notebook downloads full details of newspaper articles and other works in a list and saves them as CSV files. Like the Trove Newspaper Harvester, it lets you download OCRd text and images from newspaper articles.
Collecting information about Trove user activity – It’s not just the content of Trove that provides interesting research data, it’s also the way people engage with it. Using the Trove API it’s possible to harvest details of all user created lists and tags. And yes, there’s pre-harvested collections of lists and tags for the impatient.

Visualisation and analysis

But there are also many other notebooks that demonstrate methods for analysing Trove’s content, for example:

QueryPic – Another tool that’s been around in different forms for a decade, QueryPic visualises searches in Trove’s newspapers. The latest web app couldn’t be simpler, just paste in your API key and a search url and create charts showing the number of matching articles over time. You can combine queries, change time scales, and download the data and visualisations.
Visualise Trove newspaper searches over time – This is like a deconstructed version of QueryPic that walks you through the process of using Trove’s facets to assemble a dataset of results over time. It provide a lot of detail on the sorts of data available, and the questions we can ask of it.
Visualise the total number of newspaper articles in Trove by year and state – This notebook uses a modified version of the code above to analyse the construction and context of Trove’s newspaper corpus itself. What are you actually searching? Meet the WWI effect and the copyright cliff of death! This is a great place to start if you want to get people thinking critically about digital resources are constructed.
Analyse rates of OCR correction – Some more meta-analysis of the Trove corpus itself, this time focusing on patterns of OCR correction by Trove users.
Identifying non-English language newspapers in Trove – There are a growing number of non-English language newspapers digitised in Trove. However, if you’re only searching using English keywords, you might never know that they’re there. This notebook analyses a sample of articles from every newspaper in Trove to identify non-English content.
Beyond the copyright cliff of death – Most of the newspaper articles on Trove were published before 1955, but there are some from the later period. This notebook helps you find out how many, and which newspapers they were published in.
Map Trove newspaper results by state – This notebook uses the Trove state facet to create a choropleth map that visualises the number of search results per state.
Map Trove newspaper results by place of publication – This notebook uses the Trove title facet to find the number of results per newspaper, then merges the results with a dataset of geolocated newspapers to map where articles were published.
Compare two versions of an archived web page – This notebook demonstrates a number of different ways of comparing versions of archived web pages. Just choose a repository, enter a url, and select two dates to see comparisons based on: page metadata, basic statistics such as file size and number of words, numbers of internal and external links, cosine similarity of text, line by line differences in text or code, and screenshots.
Display changes in the text of an archived web page over time – This web app gathers all the available versions of a web page and then visualises changes in its content between versions – what’s been added, removed, and changed?
Use screenshots to visualise change in a page over time– Create a series of full page screenshots of a web page over time, then assemble them into a time series.

There are also possibilities for using Trove data creatively. For example you can create ‘scissors and paste’ messages from Trove newspaper articles.

Documentation and examples

Trove API Introduction – Some very basic examples of making requests and understanding results.
Today’s news yesterday – Uses the date index and the firstpageseq parameter to find articles from exactly 100 years ago that were published on the front page. It then selects one of the articles at random and downloads and displays an image of the front page.
The use of standard licences and rights statements in Trove image records – Version 2.1 of the Trove API introduced a new rights index that you can use to limit your search results to records that include one of a list of standard licences and rights statements. We can also use this index to build a picture of which rights statements are currently being used, and by who.
Random items from Trove – Changes to the Trove API meant that techniques you could previously use to select resources at random no longer work. This section documents some alternative ways of retrieving random-ish works and newspaper articles from Trove.

And while it’s not officially part of the GLAM Workbench, I also maintain the Trove API Console which provides lots of examples of the API in action.

Pathways

In developing the GLAM Workbench I’m very aware that people will arrive with different levels of digital skill, confidence, and experience. That’s why I’ve been putting a lot of thought and effort into ways of providing a range of entry points.

Someone who might not identify as a ‘digital’ researcher can, with a single click, start up QueryPic and start exploring changes over time in Trove’s newspapers. This is possible because the GLAM Workbench is configured to make use of Binder, a service that spins up customised computing environments as needed.

Another researcher might start running the Trove Newspaper Harvester using Binder, but find that they want to run bigger and longer harvests. In that case, the GLAM Workbench offers a one-click installation of the Trove Newspaper Harvester on Reclaim Cloud. Unlike Binder, Reclaim Cloud environments are persistent, so you can run the harvester for as long as you want without the worry of interruptions.

Yet another researcher might want to understand how the Trove API works and the sorts of data that it makes available. By exploring the various notebooks they’ll find useful snippets of code they can try out in their own projects.

The GLAM Workbench connects outwards to make use of a range of other services – the notebooks run in Binder, Reclaim Cloud, and Docker; the code is all openly licensed and publicly available through GitHub and Zenodo; data is hosted on GitHub, CloudStor, and Zenodo; datasets can be explored using Datasette running on Glitch or Google CloudRun. I’m hoping that the new investments in HASS research infrastructure will embed a similar philosophy, connecting up existing services rather than starting from scratch.

The future

This is just an outline on what the GLAM Workbench currently offers researchers wanting to make use of the data available from Trove. It’s all there now, publicly accessible, openly licensed, and ready to use – take it, use it, change it, share it. But there’s much more I’d like to do, both in regard to Trove and to encourage use of GLAM data more generally. I’m also interested in your ideas for new tools, examples, or data sources – what would help your research? You can add a suggestion in GitHub, or post a comment in the GLAM Workbench channel of OzGLAM Help.

See the Getting Started section of the GLAM Workbench for more hints and examples. And keep an eye on the news feed for the latest additions and updates.

A Family History Month experiment – search millions of name records from GLAM organisations

Mon, 23 Aug 2021 11:05:00 +1100

There’s a lot of rich historical data contained within the indexes that Australian GLAM organisations provide to help people navigate their records. These indexes, often created by volunteers, allow access by key fields such as name, date or location. They aid discovery, but also allow new forms of analysis and visualisation. Kate Bagnall and I wrote about some of the possibilities, and the difficulties, in this recently published article.

Many of these indexes can be downloaded from government data portals. The GLAM Workbench demonstrates how these can be harvested, and provides a list of available datasets to browse. But what’s inside them? The GLAM CSV Explorer visualises the contents of the indexes to give you a sneak peek and encourage you to dig deeper.

There’s even more indexes available from the NSW State Archives. Most of these aren’t accessible thought the NSW government data portal yet, but I managed to scrape them from the website a couple of years ago and made them available as CSVs for easy download.

It’s Family History Month at the moment, and the other night I thought of an interesting little experiment using the indexes. I’ve been playing round with Datasette lately. It’s a fabulous tool for exploring tabular data, like CSVs. I also noticed that Datasette’s creator Simon Willison had added a search-all plugin that enabled you to run a full text search across multiple databases and tables. Hmmm, I wondered, would it be possible to use Datasette to provide a way of searching for names across all those GLAM indexes?

After a few nights work, I found the answer was yes.

Try out my new aggregated search interface here!.

(The cloud service it uses runs on demand, so if it has gone to sleep, it might take a little while to wake up again – just be patient for a few seconds.)

Currently, the GLAM Name Search interface lets you search for names across 195 indexes from eight GLAM organisations. All together, there’s a total of more than 9.2 million rows of data to explore!

It’s simple to use – just enter a name in the search box and Datasette will search each index in turn, displaying the first five matching results. You can click through to view all results from a specific index. Not surprisingly, the aggregated name search only searches columns containing names. However, once you click through to an individual table, you can apply additional filters or facets.

To create the aggregated search interface I worked through the list of CSVs I’d harvested from government data portals to identify those that contained names of people, and discard those that contained administrative, rather than historical data. I also made a note of the columns that contained the names so I could index their contents once they’d been added to the database. Usually these were fields such as Surname or Given names, but sometimes names were in the record title or notes.

Datasette uses SQLite databases to store its data. I decided to create one database for each GLAM organisation. I wrote some code to work through my list of datasets, saving them into an SQLite database, indexing the name columns, and writing information about the dataset to a metadata.json file. This file is used by Datasette to display information such as the title, source, licence, and last modified date of each of the indexes.

Once that was done, I could fire up Datasette and feed it all the SQLite databases. Amazingly it all worked – searching across all the indexes was remarkably quick! To make it publicly available I used the Datasette publish to push everything to Google CloudRun (about 1.4 gb of data). The first time I used CloudRun it took some time to get the authentication and other settings working properly. This time was much smoother. Before long it was live!

Once I knew it all worked, I decided to add in another 59 indexes from the NSW State Archives. I also plugged in a few extra indexes from the Public Record Office of Victoria. These datasets are stored as ZIP files in the Victorian government data portal, so it took a little bit of extra manual processing to get everything sorted. But finally I had all 195 indexes loaded.

What now? That depends on whether people find this experiment useful. I have a few ideas for improvements. But if people do use it, then the costs will go up. I’m going to have to monitor this over the next couple of months to see if I can afford to keep it going. If you want to help with the running costs, you might like to sign up as a GitHub sponsor.

And please let me know if you think it’s worth developing! #dhhacks

Explore Trove’s digitised books

Mon, 16 Aug 2021 16:40:15 +1100

The Trove books section of the GLAM Workbench has been updated! There’s freshly-harvested data, as well as updated Python packages, integration with Reclaim Cloud, and automated Docker builds.

Included is a notebook to harvest details of all books available from Trove in digital form. This includes both digitised books, that have been scanned and OCRd, as well as born digital publications, such as PDFs and epubs. The definition of ‘books’ is pretty loose – I’ve harvested details of anything that has been assigned the format ‘Book’ in Trove, but this includes ephemera, such as posters, pamphlets, and advertising.

In the latest harvest, I ended up with details of 42,174 ‘books’. This includes some duplicates, because multiple metadata entries can point to the same digital object. I thought it was best to preserve the duplicates, rather than discard the metadata.

Once I’d harvested the details of the books, I tried to see if there was any OCRd text available for download. If there was, I saved it to a public folder on CloudStor. In total, I was able to download 26,762 files of OCRd text.

The easiest way to explore the books is using this searchable database. It’s created using Datasette and is running on Glitch. Full text search is available on the ‘title’ and ‘contributors’ fields, and you can filter on things like date, copyright status, number of pages, and whether OCRd text is available for download. If there is OCRd text, a direct link to the file on CloudStor is included. You can use the database to filter the titles, creating your own dataset that you can download in CSV or JSON format.

If you just want the full list of books as a CSV file, you can download it here. And if you want all the OCRd text, you can go straight to the public folder on CloudStor – there’s about 3.6gb of text files to explore! #dhhacks

A miscellany of ephemera, oddities, & estrays

Fri, 13 Aug 2021 12:02:00 +1100

I’m just in the midst of updating my harvest of OCRd text from Trove’s digitised books (more about that soon!). But amongst the items catalogued as ‘books’ are a wide assortment of ephemera, posters, advertisements, and other oddities. There’s no consistent way of identifying these items through the search interface, but because I’ve found the number of pages in each ‘book’ as part of the harvesting process, I can limit results to items with just a single digitised page – there’s more than 1,500! To make it easy to explore this collection of odds and ends, I’ve downloaded all the single page images and compiled them into one big PDF with links back to their entries in Trove. Enjoy your browsing!

This is another example of the ways in which we can extend and enrich existing collection interfaces using simple technologies like PDFs and CSVs. We can create slices across existing categories to expose interesting features, and provide new entry points for researchers. Some other examples in the GLAM Workbench are the collection of editorial cartoons from The Bulletin, the list of Trove newspapers with non-English content, the harvest of ABC Radio National programs, and the recent collection of politicians talking about COVID. Let me know if you have any ideas for additional slices! #dhhacks

Everyday heritage and the GLAM Workbench

Mon, 09 Aug 2021 12:26:38 +1100

Some good news on the funding front with the success of the Everyday Heritage project in the latest round of ARC Linkage grants. The project aims to look beyond the formal discourses of ‘national’ heritage to develop a more diverse range of heritage narratives. Working at the intersection of place, digital collections, and material culture, team members will develop a series of ‘heritage biographies’, that document everyday experience, and provide new models for the heritage sector.

Digital methods will play a major role in the project. I’ll be leading the ‘Heritage Hacks’ work package that will support the creation of the heritage biographies and develop a range of new tools and tutorials for use in heritage management contexts. All the tools, methods, and data generated through the project will be documented using Jupyter notebooks and published through the GLAM Workbench. Watch this space!

The project is led by Tracy Ireland (University of Canberra), with Jane Lydon (UWA), Kate Bagnall (UTAS), and me as chief investigators. Our industry partner is GML Heritage.

Recent GLAM Workbench presentations

Fri, 06 Aug 2021 18:42:17 +1100

So far this year I’ve given eight workshops or presentations relating to the GLAM Workbench, with probably a few more yet to come. Here’s the latest:

Introducing the GLAM Workbench, presentation for the Griffith University Centre for Social and Cultural Research, Digital Humanities Seminar Series, 6 August 2021
Exploring the GLAM Workbench (slides), presentation for the UTS Digital Histories Seminar Series, 8 July 2021
The GLAM Workbench: A Labs approach?, presentation for the panel ‘Research use of web archives: A labs approach’, at the IIPC Web Archiving Conference, 15 June 2021
Hands-on introduction to the GLAM Workbench, workshop for the Professional Historians Association of Victoria and Tasmania, 27 May 2021
Exploring collections through the GLAM Workbench, keynote presentation for the XVIII Congrés d’Arxvística i Gestió de Documents de Catalunya, 11 May 2021
Quick hacks and DIY data: Innovations for the discerning historian, presentation for AHA ECR digital skills seminar, 5 March 2021

You can view all the GLAM Workbench presentations here.

Updated! Lots and lots of text freshly harvested from Trove periodicals

Fri, 06 Aug 2021 10:49:54 +1100

For a few years now I’ve been harvesting downloadable text from digitised periodicals in Trove and making it easily available for exploration and research. I’ve just completed the latest harvest – here’s the summary:

1,163 digitised periodicals had text available for download
Text was downloaded from 51,928 individual issues
Adding up to a total of around 12gb of text

If you want to dive straight in, here’s a list of all the harvested periodicals, with links to download a summary of available issues, as well as all the harvested text (there’s one file per issue). You’ll notice that the list includes a large number of parliamentary papers and government reports as well as published journals.

All of the harvested text is available from a public folder on CloudStor.

The harvesting process involves a few different steps:

First I generate a list of periodicals available in digital form from Trove. This includes digitised titles, as well as born-digital titles submitted through e-Legal Deposit. This produced a CSV file containing the details of 7,270 titles. See this notebook for details.
Then I work through this list of titles to find out how many issues of each title are available through Trove. This information isn’t accessible through the API, so I have to do some screen scraping.
Next I work through the list of issues and try to download the text contents. Most of the born-digital titles don’t have downloadable text.
Once I’ve downloaded all the text I can from a title, I create a CSV file for it that lists the available issues and notes whether text is available for each. This file is stored with the text on CloudStor.
Once I’ve checked all the titles, I generate another CSV file that lists the details of all the periodicals that have downloadable text.
The code to harvest and document the downloaded text is available in this notebook. #dhhacks

New dataset – Politicians talking about COVID

Mon, 02 Aug 2021 11:23:00 +1100

The Trove Journals section of the GLAM Workbench includes a notebook that helps you download press releases, speeches, and interview transcripts by Australian federal politicians. These documents are compiled and published by the Parliamentary Library, and the details are regularly harvested into Trove.

Using this notebook, I’ve created a collection of documents that include the words ‘COVID’ or ‘Coronavirus’. It includes all the metadata from Trove, as well as the full text of each document downloaded from the Parliamentary Library. There’s 3,995 documents in total, covering the period up until early April 2021. You can download them all as a zip file (12 mb).

While I was compiling this dataset, I also made a few improvements to the notebook. You can now filter the results to weed out false positives, and identify duplicates. #dhhacks

8 million Trove tags to explore!

Wed, 14 Jul 2021 18:06:00 +1100

I’ve always been interested in the way people add value to resources in Trove. OCR correction tends to get all the attention, but Trove users have also been busy organising resources using tags, lists, and comments. I used to refer to tagging quite often in presentations, pointing to the different ways they were used. For example, ‘TBD’ is a workflow marker, used by text correctors to label articles that are ‘To Be Done’. My favourite was ‘LRRSA’, one of the most heavily-used tags across the whole of Trove. What does it mean? It stands for the Light Rail Research Society of Australia, and the tag is used by members to mark items of shared interest. It’s a great example of how something as simple as plain text tags can be used to support collaboration and build communities.

Until its update last year, Trove used to provide some basic stats about user activity. There was also a tag cloud that let you explore the most commonly-used tags. It’s now much harder to access this sort of information. However, you can extract some basic information about tags from the Trove API. First of all, you can filter a search using ‘has:tags’ to limit the results to items that have tags attached to them. Then to find out what the tags actually are, you can add the include=tags parameter. This embeds the tags within the item record, so you can work through a set of results, extracting all the tags as you go. To save you the trouble, I’ve done this for the whole of Trove, and ended up with a dataset containing more than 8 million tags!

The dataset is saved as a 500mb CSV file, and contains the following fields:

tag – lower-cased version of the tag
date – date the tag was added
zone – the API zone that contains the tagged resource
resource_id – the identifier of he tagged resource

There’s a few things to note about the data:

Works (such as books) in Trove can have tags attached at either work or version level. This dataset aggregates all tags at the work level, removing any duplicates.
A single resource in Trove can appear in multiple zones – for example, a book that includes maps and illustrations might appear in the ‘book’, ‘picture’, and ‘map’ zones. This means that some of the tags will essentially be duplicates – harvested from different zones, but relating to the same resource. Depending on your interests, you might want to remove these duplicates.
While most of the tags were added by Trove users, more than 500,000 tags were added by Trove itself in November 2009. I think these tags were automatically generated from related Wikipedia pages. Depending on your interests, you might want to exclude these by limiting the date range or zones.
User content added to Trove, including tags, is available for reuse under a CC-BY-NC licence.

You can download the complete dataset from Zenodo, or from CloudStor. For more information on how I harvested the data, and some of its limits and complexities, see the notebooks in the new ‘Tags’ section in the GLAM Workbench. There’s also some examples of analysing and visualising the tags. As an extra bonus, there’s a more compact 50mb CSV dataset which lists each unique tag and the number of times it has been used.

Of course, it’s worth remembering that this sort of dataset is out of date before the harvest is even finished. More tags are being added all the time! But hopefully this data will help us better understand the way people work to organise and enrich complex resources like Trove. #dhhacks

Integrating GLAM Workbench news and discussion

Thu, 01 Jul 2021 16:18:41 +1100

I’ve spent a lot of time this year working on ways of improving the GLAM Workbench’s documentation and its integration with other services. Last year I created OzGLAM Help to provide a space where users of GLAM collections could ask questions and share discoveries – including a dedicated GLAM Workbench channel. Earlier this year, I tweaked my Micro.blog powered updates to include a dedicated GLAM Workbench news feed. Now I’ve brought the two together! What does this mean?

Any GLAM Workbench news that I post to my updates feed is now automatically added to OzGLAM Help
Links are automatically added to items in the news feed that let you add comments or questions in OzGLAM Help

So now there’s two-way communication between the services providing more ways for people to discover and discuss how the GLAM Workbench can help them.

GLAM Workbench now on YouTube!

Thu, 01 Jul 2021 15:51:00 +1100

I’ve started creating short videos to introduce or explain various components of the GLAM Workbench. The first video shows how you can visualise searches in Trove’s digitised newspapers using the latest version of QueryPic. It’s a useful introduction to the way access to collection data enables us to ask different types of questions of historical sources.

As with all GLAM Workbench resources, the video is openly-licensed – so feel free to stop it into your own course materials or workshops. It could, for example, provide an interesting little digital methods task in an Australian history unit.

I’ll be creating a second QueryPic video shortly, demonstrating how you can work with complex queries and differing timescales. Let me know if you find it useful, or if you have any ideas for future topics. #dhhacks

GLAM Workbench office hours

Mon, 28 Jun 2021 15:04:56 +1100

To help you make use of the GLAM Workbench, I’ve set up an ‘office hours’ time slot every Friday when people can book in for 30 minute chats via Zoom. Want to talk about how you might use the GLAM Workbench in your latest research project? Are you having trouble getting started with GLAM data? Or perhaps you have some ideas for future notebooks you’d like to share? Just click on the ‘Book a chat’ link in the GLAM Workbench, or head straight to the scheduling page to set up a time!

This is yet another experiment to see how I can support the use of GLAM data and the development of digital skills with the GLAM Workbench. Let me know if you think it’s worthwhile. #dhhacks

QueryPic: The Next Generation

Mon, 21 Jun 2021 11:50:19 +1100

QueryPic is a tool to visualise searches in Trove’s digitised newspapers. I created the first version way back in 2011, and since then it’s taken a number of different forms. The latest version introduces some new features:

Automatic query creation – construct your search in the Trove web interface, then just copy and paste the url into QueryPic. This means you can take advantage of Trove’s advanced search and facets to build complex queries.
Multiple time scales – previous versions only aggregated search results by year, but now you can also aggregate by month, or by day. QueryPic will automatically choose a time unit based on the date range of your query, but if you’re not happy with the result you can change it!
Links back to Trove – click on any of the points on the chart to search Trove within that time period. This enables you to zoom in and out of your results, from the high-level visualisation, to individual articles.

This version of QueryPic is built within a Jupyter notebook, and designed to run using Voila (which hides all the code and makes the notebook look like a web app). See the Trove Newspapers section of the GLAM Workbench for more information. If you’d like to give it a try, just click the button below to run it live using Binder.

Hope you find it useful! #dhhacks

Everyone gets a Lab!

Mon, 21 Jun 2021 11:00:30 +1100

I recently took part in a panel at the IIPC Web Archiving Conference discussing ‘Research use of web archives: a Labs approach’. My fellow panellists described some amazing stuff going on in European cultural heritage organisations to support researchers who want to make use of web archives. My ‘lab’ doesn’t have a physical presence, or an institutional home, but it does provide a starting point for researchers, and with the latest Reclaim Cloud and Docker integrations, everyone can have their own web archives lab! Here’s my 8 minute video. The slides are available here.

Minor change to Reclaim Cloud config

Mon, 14 Jun 2021 15:44:00 +1100

When the 1-click installer for Reclaim Cloud works its magic and turns GLAM Workbench repositories into your own, personal digital labs, it creates a new work directory mounted inside of your main Jupyter directory. This new directory is independent of the Docker image used to run Jupyter, so it’s a handy place to copy things if you ever want to update the Docker image. However, I just realised that there was a permissions problem with the work directory which meant you couldn’t write files to it from within Jupyter.

To fix the problem, I’ve added an extra line to the reclaim-manifest.jps config file to make the Jupyter user the owner of the work directory:

	- cmd[cp]: chown -R jovyan:jovyan /home/jovyan/work

This takes care of any new installations. If you have an existing installation, you can either just create a completely new environment using the updated config, or you can manually change the permissions:

Hover over the name of your environment in the control panel to display the option buttons.
Click on the Settings button. A new box will open at the bottom of the control panel with all the settings options.
Click on ‘SSH Access’ in the left hand menu of the settings box.
Click on the ‘SSH Connection’ tab.
Under ‘Web SSH’ click on the Connect button and select the default node.
A terminal session will open. At the command line enter the following:

	chown -R jovyan:jovyan /home/jovyan/work

Done! See the Using Reclaim Cloud section of the GLAM Workbench for more information.

Trove Query Parser

Mon, 14 Jun 2021 13:46:01 +1100

Here’s a new little Python package that you might find useful. It simply takes a search url from Trove’s Newspapers & Gazettes category and converts it into a set of parameters that you can use to request data from the Trove API. While some parameters are used both in the web interface and the API, there are a lot of variations – this package means you don’t have to keep track of all the differences!

It’s very simple to use.

The code for the parser has been basically lifted from the Trove Newspaper Harvester. I wanted to separate it out so that I could use it at various spots in the GLAM Workbench and in other projects.

This package, the documentation, and the tests were all created using nbdev, which is really quite a fun way to develop Python packages. #dhhacks

Some GLAM Workbench stats

Sun, 13 Jun 2021 19:01:23 +1100

I deliberately don’t keep any stats about GLAM Workbench visits, because I think they’re pretty meaningless. On the other hand, I’m always interested to see how often GLAM Workbench repositories are launched on Binder. Rather than just random clicks, these numbers represent the number of times users started new computing sessions using the GLAM Workbench. I just compiled these stats for the past year, and I was very pleased to see that the Web Archives section has been launched over 1,000 times in the past twelve months! The Trove Newspapers and Trove Newspaper Harvester repositories are also well used – on average these are both being launched more than once a day.

The GLAM Workbench is never going to attract massive numbers of users – it’s all about being there when a researcher needs help to use GLAM collections. One or two launches per day means one or two researchers from somewhere around the world are able to explore new datasets, or ask new questions. I think that’s pretty important.

More Reclaim Cloud integrations!

Sun, 13 Jun 2021 18:28:12 +1100

Five of the GLAM Workbench repositories now have automatically built Docker images and 1-click integration with Reclaim Cloud – ANU Archives, Trove Newspapers, Trove Newspaper Harvester, NAA RecordSearch, & Web Archives.

This means you can launch your very own version of these GLAM Workbench repositories in the cloud, where all your downloads and experiments will be saved! Find out more on the Using Reclaim Cloud page.

Get your GLAM datasets here!

Sun, 13 Jun 2021 18:12:37 +1100

I’ve updated my harvest of Australian GLAM datasets from state/national government open data portals. There’s now 387 datasets, containing 1049 files (including 684 CSVs). There’s a list if you want to browse, and a CSV file if you want to download all the metadata. For more more information see the data portals section of the GLAM Workbench.

If you’re interested in finding out what’s inside all those 684 CVS files, take the GLAM CSV Explorer for a spin! It’s also been given a refresh, with new data and a new interface. #dhhacks

NAA RecordSearch section of the GLAM Workbench updated!

Mon, 24 May 2021 11:50:00 +1100

If you work with the collections of the National Archives of Australia, you might find the RecordSearch section of the GLAM Workbench helpful. I’ve just updated the repository to add new options for running the notebooks, including 1-click installation on Reclaim Cloud. There’s also a few new notebooks.

New notebooks and datasets

Harvest details of all series in RecordSearch – get details of all series registered in RecordSearch, also generates a summary dataset with the total number of items digitised, described and in each access category
Exploring harvested series data – generates some basic statistics from the harvest of series data
Summary data about all series in RecordSearch (15mb CSV) – contains basic descriptive information about all the series currently registered on RecordSearch (May 2021) as well as the total number of items described, digitised, and in each access category

Updated

I’ve started (but not completed) updating all the notebooks in this repository to use my new RecordSearch Data Scraper. The new scraper is simpler and more efficient, and enables me to get rid of a lot of boilerplate code. Updated notebooks include:

Harvest items from a search in RecordSearch – save the results of an item search in RecordSearch as a downloadable dataset, you can also save images and PDFs from digitised files (PDF saving is new!)
Harvest files with the access status of ‘closed’ – find out what we’re not allowed to see by harvesting details of ‘closed’ files

Other updates include:

Python packages updated
Integration with Reclaim Cloud allowing 1-click installation of the whole repository and environment
Automatic creation of Docker images when the repository is updated
Updated README and repository index with list of all notebooks
Notebooks intended to run as apps now use Voila rather than Appmode for better integration with Jupyter Lab
requirements-unpinned.txt added to repository for people who want to develop the notebooks in their own clean environment

Hope you find these changes useful! #dhhacks

Web archives section of GLAM Workbench updated!

Mon, 17 May 2021 13:22:00 +1100

My program of rolling out new features and integrations across the GLAM Workbench continues. The latest section to be updated is the Web Archives section!

There are no new notebooks with this update, but some important changes under the hood. If you haven’t used it before, the Web Archives section contains 16 notebooks providing documentation, tools, apps, and examples to help you make use of web archives in your research. The notebooks are grouped by the following topics: Types of data, Harvesting data and creating datasets, and Exploring change over time.

I’ve updated all the Python packages used in this repository and changed the app-ified notebooks to run using Voila (which is better integrated with Jupyter Lab than Appmode). But most importantly, you can now install the repository into your own persistent environment using Reclaim Cloud or Docker.

As Christie Moffatt noted recently harvesting data from web archives can take a long time, and you might hit the limits of the free Binder service. These new integrations mean you don’t have to worry about your notebooks timing out. Just click on the Launch on Reclaim Cloud button and you can have your own fully-provisioned, persistent environment up and running in minutes!

This is possible because every change to the Web Archives repository now triggers the build of a new Docker image with all the software that you need pre-installed. You can also run this Docker image on your own computer, or using another cloud service.

The Web Archives section now includes documentation on running the notebooks using Binder, Reclaim, Cloud or Docker. #dhhacks

Using web archives to find out when newspapers were added to Trove

Wed, 12 May 2021 12:36:00 +1100

There’s no doubt that Trove’s digitised newspapers have had a significant impact on the practice of history in Australia. But analysing that impact is difficult when Trove itself is always changing – more newspapers and articles are being added all the time.

In an attempt to chart the development of Trove, I’ve created a dataset that shows (approximately) when particular newspaper titles were first added. This gives a rough snapshot of what Trove contained at any point in the last 12 years.

I say approximately because the only public source of this information are web archives like the Internet Archive’s Wayback Machine and Trove itself. By downloading captures of Trove’s browse page, I was able to extract a list of newspaper titles available when that capture was made. Depending on the frequency of captures, the titles may have been first made available some time earlier.

The method I used to create the dataset is documented in the Trove Newspapers section of the GLAM Workbench. I used the Internet Archive as my source rather than Trove just because there were more captures available. Most of the code I could conveniently copy from the Web Archives section of the GLAM Workbench, in particular the Find all the archived versions of a particular web page notebook.

The result was actually two datasets:

trove_newspaper_titles_2009_2021.csv – complete dataset of captures and titles
trove_newspaper_titles_first_appearance_2009_2021.csv – filtered dataset, showing only the first appearance of each title / place / date range combination

There’s also an alphabetical list of newspaper titles for easy browsing. The list shows the date of the capture in which the title was first recorded, as well as any changes to its date range. #dhhacks

GLAM Jupyter Resources

Wed, 12 May 2021 11:58:29 +1100

To make it easier for people to suggest additions, I’ve created a GitHub repository for my list of GLAM Jupyter examples and resources. Contributions are welcome!

This list is automatically pulled into the GLAM Workbench’s help documentation. #dhhacks

Running notebooks – a sign of things to come in the GLAM Workbench

Wed, 12 May 2021 11:51:11 +1100

I recently made some changes in the GLAM Workbench’s Help documentation, adding a new Running notebooks section. This section provides detailed information of running and managing GLAM Workbench repositories using Reclaim Cloud and Docker.

I’m still rolling out this functionality across all the repositories, but it’s going to take a while. When I’m finished you’ll be able to create your own persistent environment on Reclaim Cloud from any repository with just the click of a button. See the Trove Newspapers section to try this out now! #dhhacks

Sponsor my work on GitHub!

Wed, 12 May 2021 11:25:00 +1100

As I foreshadowed some weeks ago, I’ve shut down my Patreon page. Thanks to everyone who has supported me there over the last few years!

I’ve now shifted across to GitHub Sponsors, which is focused on supporting open source projects. This seems like a much better fit for the things that I do, which are all free and open by default.

So if you think things like the GLAM Workbench, Historic Hansard, OzGLAM Help, and The Real Face of White Australia are worth supporting, you can sign up using my GitHub Sponsors page. Sponsorship tiers start at just $1 a month. Financially, your contributions help pay some of my cloud hosting bills and keep everything online. But just as important is the encouragement and motivation I get from knowing that there are people out there who think this work is important and useful.

To recognise my GitHub sponsors, I’ve also created a new Supporters page in the GLAM Workbench.

Thanks!

Updates to the Trove Newspapers section of GLAM Workbench

Wed, 12 May 2021 10:52:00 +1100

I’ve updated, refreshed, and reorganised the Trove newspapers section of the GLAM Workbench. There’s currently 22 Jupyter notebooks organised under the following headings:

Trove newspapers in context – Notebooks in this section look at the Trove newspaper corpus as a whole, to try and understand what’s there, and what’s not.
Visualising searches – Notebooks in this section demonstrate some ways of visualising searches in Trove newspapers – seeing everything rather than just a list of search results.
Useful tools – Notebooks in this section provide useful tools that extend or enhance the Trove web interface and API.
Tips and tricks – Notebooks in this section provide some useful hints to use with the Trove API.
Get creative – Notebooks in this section look at ways you can use data from Trove newspapers in creative ways.

There’s also a number of pre-harvested datasets.

Recently refreshed analyses, visualisations, and datasets include:

Number of Trove newspaper articles by year and state (notebook)
Analysing OCR correction in Trove’s newspapers (notebook)
List of Trove newspapers in languages other than English (markdown formatted list)
Newspapers with content from beyond the 1954 copyright ‘cliff of death’ (CSV file)

As part of the update, notebooks that are intended to run as apps (with all the code hidden) have been updated to use Voila. But perhaps the thing I’m most excited about are the new options for running the notebooks. As well as being able to launch the notebooks on Binder, you can now create your very own, persistent environment on Reclaim Cloud with just a click of a button.

There’s also an automatically-built Docker image of this repository, containing everything you need to run the notebooks on your own computer. Check out the new Run these notebooks section for details. I’m gradually rolling this out across all the repositories in the GLAM Workbench. #dhhacks

Introducing the new, improved RecordSearch Data Scraper!

Tue, 27 Apr 2021 10:55:00 +1100

It was way back in 2009 that I created my first scraper for getting machine-readable data out of the National Archives of Australia’s online database, RecordSearch. Since then I’ve used versions of this scraper in a number of different projects such as The Real Face of White Australia, Closed Access, and Redacted (including the recent update). The scraper is also embedded in many of the notebooks that I’ve created for the RecordSearch section of the GLAM Workbench.

However, the scraper was showing its age. The main problem was that one of its dependencies, Robobrowser, is no longer maintained. This made it difficult to update. I’d put off a major rewrite, thinking that RecordSearch itself might be getting a much-needed overhaul, but I could wait no longer. Introducing the brand new RecordSearch Data Scraper.

Just like the old version, the new scraper delivers machine-readable data relating to Items, Series and Agencies – both from individual records, and search results. It also adds a little extra to the basic metadata, for example, if an Item is digitised, the data includes the number of pages in the file. Series records can include the number of digitised files, and the breakdown of files by access category.

The new scraper adds in some additional search parameters for Series and Agencies. It also makes use of a simple caching system to improve speed and efficiency. RecordSearch makes use of an odd assortment of sessions, redirects, and hidden forms, which make scraping a challenge. Hopefully I’ve nailed down the idiosyncrasies, but I expect to catching bugs for a while.

I created the new scraper in Jupyter using NBDev. NBDev helps you to keep your code, examples, tests, and documentation all together in Jupyter notebooks. When you’re ready, it converts the code from the notebooks into distributable Python libraries, runs all your tests, and builds a documentation site. It’s very cool.

Having updated the scraper, I now need to update the notebooks in the GLAM Workbench – more on that soon. The maintenance never ends! #dhhacks

Recently digitised files in the National Archives of Australia

Mon, 29 Mar 2021 10:00:00 +1100

I’m interested in understanding what gets digitised and when by our cultural institutions, but accessible data is scarce. The National Archives of Australia lists ‘newly scanned' records in RecordSearch, so I thought I’d see if I could convert that list into a machine-readable form for analysis. I’ve had a lot of experience trying to get data out of RecordSearch, but even so it took me a while to figure out how the ‘newly scanned’ page worked. Eventually I was able to extract all the file metadata from the list and save it to a CSV file. The details are in this notebook in the GLAM Workbench.

I used the code to create a dataset of all the files digitised in the past month. The ‘newly scanned' list only displays a month’s worth of additions, so that’s as much as I could get in one hit. In the past month, 24,039 files were digitised. 22,500 of these (about 93%) come from just four series of military records. This is no surprise, as the NAA is currently undertaking a major project to digitise WW2 service records. What is perhaps more interesting is the long tail of series from which a small number of files were digitised. 357 of the 375 series represented in the dataset (about 95%) appear 20 or fewer times. 210 series have had only one file digitised in the last month. I’m assuming that this diversity represents research interests, refracted through the digitisation on demand service. But this really needs more data, and more analysis.

As I mentioned, only one month’s data is available from RecordSearch at any time. To try and capture a longer record of the digitisation process, I’ve set up an automated ‘git scraper’ that runs every Sunday and captures metadata of all the files digitised in the preceding week. The weekly datasets are saved as CSV files in a public GitHub repository. Over time, this should become a useful dataset for exploring long-term patterns in digitisation. #dhhacks

Moving on from Patreon...

Fri, 26 Mar 2021 13:37:00 +1100

Over the last few years, I’ve been very grateful for the support of my Patreon subscribers. Financially, their contributions have helped me cover a substantial proportion of the cloud hosting costs associated with projects like Historic Hansard and The Real Face of White Australia. But, more importantly, just knowing that they thought my work was of value has helped keep me going, and inspired me to develop a range of new resources.

However, while I’ve been grateful for the platform provided by Patreon, I’ve increasingly felt that it’s not a good fit for the sort of work I do. Patreon is geared towards providing special content to supporters, but, as you know, all my work is open. And that’s really important to me.

Recently GitHub opened up its own sponsorship program for the development of open source software. This program seems to align more closely with what I do. I already share and manage my code through GitHub, so integrating sponsorship seems to make a lot of sense. It’s worth noting too, that, unlike Patreon, GitHub charges no fees and takes no cut of your contributions. As a result I’ve decided to close my Patreon account by the end of April, and create a GitHub sponsors page.

What does this mean for you?

If you’re a Patreon subscriber and you’d like to keep supporting me, you should cancel your Patreon contribution, then head over to my brand new GitHub sponsors page and sign up! Thanks for your continued support!

If you’d prefer to let your contributions lapse, just do nothing. Your payments will stop when I close the account at the end of April. I understand that circumstances change – thank you so much for your support over the years, and I hope you will continue to make use of the things I create.

If you make use of any of my tools or resources and would like to support their continued development, please think about becoming a sponsor. For a sample of the sorts of things I’ve been working on lately, see my updates feed.

The future!

I’m very excited about the possibilities ahead. The GLAM Workbench has received a lot of attention around the world (including a Research Award from the British Library Labs), and I’m planning some major developments over coming months. And, of course, I won’t forget all my other resources – I spent a lot of time in 2020 migrating databases and platforms to keep everything chugging along.

On my GitHub sponsors page, I’ve set an initial target of 50 sponsors. That might be ambitious, but as I said above, it’s not just about money. Being able to point to a group of people who use and value this work will help me argue for new ways of enabling digital research in the humanities. So please help me spread the word – let’s make things together!

What can you do with the GLAM Workbench?

Thu, 25 Mar 2021 11:43:47 +1100

You might have noticed some changes to the GLAM Workbench home page recently. One of the difficulties has always been trying to explain what the GLAM Workbench actually is, so I thought it might be useful to put more examples up front. The home page now lists about 25 notebooks under the headings:

Hopefully they give a decent representation of the sorts of things you can do using the GLAM Workbench. I’ve also included a little rotating slideshow built using Slides.com.

Other recent additions include a new Grants and Awards page. #dhhacks

Reclaim Cloud integration coming soon to the GLAM Workbench

Thu, 25 Mar 2021 11:18:00 +1100

I’ve been doing a bit of work behind the scenes lately to prepare for a major update to the GLAM Workbench. My plan is to provide one click installation of any of the GLAM Workbench repositories on the Reclaim Cloud platform. This will provide a useful step up from Binder for any researcher who wants to do large-scale or sustained work using the GLAM Workbench. Reclaim Cloud is a paid service, but they do a great job supporting digital scholarship in the humanities, and it’s fairly easy to minimise your costs by shutting down environments when they’re not in use.

I’ve still got a lot of work to do to roll this out across the GLAM Workbench’s 40 repositories, but if you’d like a preview head to the Trove Newspaper and Gazette Harvester repository on GitHub. Get yourself a Reclaim Cloud account and click on the Launch on Reclaim Cloud button. It’s that easy!

There’s some technical notes in the Reclaim Hosting forum, and a post by Reclaim Hosting guru Jim Groom describing his own experience spinning up the GLAM Workbench.

Watch this space for more news! #dhhacks

Some recent GLAM Workbench presentations

Thu, 25 Mar 2021 10:40:00 +1100

I’ve given a couple of talks lately on the GLAM Workbench and some of my other work relating to the construction of online access to GLAM collections. Videos and slides are available for both:

From collections as data to collections as infrastructure: Building the GLAM Workbench, seminar for the Centre for Creative and Cultural Research, University of Canberra, 22 February 2021 – video (40 minutes) and slides

Building the GLAM Workbench (and various other projects such as The Real Face of White Australia, Closed Access, and redacted), guest lecture for the Cultural Data Sculpting course, EPFL, Switzerland, 18 March 2021 – video (1hr 40mins) and slides

I’ve also updated the presentations page in the GLAM Workbench. #dhhacks

Some GLAM Workbench datasets to explore for Open Data Day

Mon, 08 Mar 2021 14:54:00 +1100

It was Open Data Day on Saturday 6 March – here’s some of the ready-to-go datasets you can find in the GLAM Workbench – there’s something for historians, humanities researchers, teachers & more!

First here’s a list of Australian GLAM (that’s galleries, libraries, archives & museums) data sources. It includes APIs, portals, and downloadable datasets. Suggested additions welcome!
There’s also a list of Australian GLAM datasets that are available through government open data portals. There’s hundreds of them, but they’re not always easy to find. Convicts, immigration, hospitals, WWI – includes lots of useful biographical data.
If you’re not sure where to start with a list of 600 CSV files, have a look at the GLAM CSV Explorer! Select a file and this Jupyter-powered app will build a series of visualisations based on the contents of each column.
While they’re not yet in an open data portal, NSW State Archives has a rich collection of indexes transcribed by volunteers. I’ve scraped 64 indexes, with over 1.4 million rows of data and put them in a repository for easy download. There’s even a version of the CSV Explorer, just for the NSW State Archives indexes.
Here’s a CSV file containing details of every issue of the Australian Women’s Weekly in Trove.
A collection of front covers from the Australian Women’s Weekly from 1933 to 1982! That’s 2,566 images you can download from Cloudstor or browse in a series of convenient PDFs.

Here’s a list of non-English language newspapers in Trove.
And another list of newspapers in Trove with articles available from beyond the 1954 copyright cliff of death.
While we’re on newspapers, here’s a spreadsheet that identifies places of publication or circulation of Trove newspapers, and provides geocordinates
What about some text? Here’s 24,620 files of OCRd text from digitised books and ephemera in Trove. There’s also a CSV-formatted list with the basic details of each book.
More text! Here’s OCRd text from 26,234 issues of 397 digitised journals in Trove.
Something different – a collection of 12,619 press releases & speeches by Australian politicians that include any of the terms ‘immigrant’, ‘asylum seeker’, ‘boat people’, ‘illegal arrivals’, or ‘boat arrivals’. From the Parliamentary Library via Trove.
Some more images – a collection of 3,471 full-page editorial cartoons from The Bulletin, 1886 to 1952 (with a warning for racist content). Available both as individual images and compiled into PDFs.
From the ABC via Trove, there’s 400,000 records from Radio National programs broadcast since the late 1990s. That includes every segment broadcast on AM, PM, RN Breakfast etc.
This might be handy – from some work I’m doing with ANU Archives, here’s a CSV file containing details of holidays in NSW from 1901 to 1950.
The Department of Prime Minister and Cabinet provides XML versions of more than 20,000 speeches & interviews from recent PMs for download. I’ve saved them to a repository and compiled some indexes.
And finally – Commonwealth Hansard from the Parliamentary Library – lots of well-structured XML files! I’ve created a repo with one file for each sitting day from 1901 to 1980 & 1998 to 2005 (hopefully the gap will be filled soon). There’s also a CSV index to sitting days.

And if that’s not enough data, the GLAM Workbench provides tools to help you create your own datasets from Trove, the National Archives of Australia, the National Museum of Australia, Archives NZ, DigitalNZ, & more! #dhhacks

Thu, 11 Feb 2021 10:24:00 +1100

The NAA recently changed field labels in RecordSearch, so that ‘Barcode' is now ‘Item ID’. This required an update to my recordsearch_tools screen scraper. I also had to make a few changes in the RecordSearch section of the GLAM Workbench. #dhhacks

New! DigitalNZ API Query Builder added to GLAM Workbench

Wed, 03 Feb 2021 10:08:17 +1100

I’ve added an API Query Builder to the DigitalNZ section of the GLAM Workbench. You can use it to learn about the different parameters available from the search API, and experiment with different queries. Just get your API key from DigitalNZ, then try entering keywords and selecting options. Once you understand how the API works, you can start thinking about how you can make use of it in your own projects.

👉🏻 Try it out live on Binder!

Under the hood the API Query Builder is a Jupyter notebook (of course), but it uses ipyvuetify to create good-looking, responsive, form widgets. It’s intended to be run using Voilà, which turns notebooks into interactive apps and dashboards. You can now run any Jupyter notebook using Voilà on Binder, just by changing the url.

If this app seems useful (let me know!) I might put a version on Heroku so the start up time is reduced. I’m also thinking of using this sort of pattern to create apps for other APIs in the GLAM Workbench. #dhhacks

OpenGLAM fireworks! Finding open collections in DigitalNZ

Thu, 28 Jan 2021 11:29:00 +1100

Lately I’ve been updating and expanding the notebooks in the DigitalNZ section of the GLAM Workbench. In particular, I’ve been looking at the usage facet to understand how much of the aggregated content is ‘open’. What do I mean by ‘open’? The Open Knowledge Foundation definition states that ‘open data and content can be freely used, modified, and shared by anyone for any purpose’. Obviously things that are in the public domain, such as out-of-copyright resources, are open. But so are resources with an open licence such as CC-BY or CC-BY-SA. The Creative Commons ‘Non commercial' and ‘No derivatives’ licences are not open because they put limits on how you can use resources.

How does this definition map to DigitalNZ? The usage facet includes five values:

Share
Modify
Use commercially
Unknown

These values have been assigned by DigitalNZ based on the 35,000 different rights statements and 30 different copyright statements that are included in DigitalNZ metadata records. I find I have to turn the usage values inside out to really understand them. A resource that only allows you to ‘Share’, excludes the ‘Modify’ and ‘Use commercially’ permissions and so is sort of equivalent to a CC-BY-ND-NC licence. The only open value, according to the definition above, is ‘Use commercially’, which is like CC-BY. I’m assuming that ‘Use commercially’ has been assigned to resources that either out of copyright (or with no known copyright restrictions) or are openly licensed.

It’s also worth noting that the ‘usage’ values are not mutually-exclusive. A record with a ‘usage’ value of ‘Use commercially’, will also be assigned ‘Share’ and ‘Modify' values. This is because ‘Use commercially’ includes the ‘Share’ and ‘Modify’ permissions. This seems a bit counter-intuitive, but makes sense if you think about doing a search for everything you’re allowed to share.

A rough calculation based on the usage facet indicates that 71.76% of the resources aggregated by DigitalNZ are open. That seems pretty good, though a lot of those are probably out-of-copyright newspaper articles from Papers Past. For a more fine-grained analysis, I decided to look at the ‘usage’ data for each combination of ‘content_partner’ and ‘primary_collection’. How open is each individual collection in DigitalNZ?

For added excitement, and to stretch my knowledge of what Altair can do, I decided to visualise the results as display of colourful fireworks. The higher the explosion, the more open the collection! I’m pretty pleased with the result.

I’ve saved a HTML version of the chart so you can mouseover the explosions for more details. All the code is included in this notebook, along with a CSV file containing all the harvested facet data. #dhhacks

New dataset and notebooks – twenty years of ABC Radio National

Mon, 18 Jan 2021 10:27:00 +1100

There’s a new GLAM Workbench section for working with data from Trove’s Music & Sound zone!

Inside you’ll find out how to harvest all the metadata from ABC Radio National program records – that’s 400,000+ records, from 160 Radio National programs, over more than 20 years.

It’s metadata only, so not full transcripts or audio, though there are links back to the ABC site where you might find transcripts. Most records should at least have a title, a date, the name of the program it was broadcast on, a list of contributors, and perhaps a brief abstract/summary. It’s also worth noting that many of these records, particularly those from the main current affairs programs, represent individual stories or segments – so they provide a detailed record of the major news stories for the last couple of decades!

The harvesting notebook shows you how to get the data from the Trove API. There are a number of duplicate records, and some inconsistencies in the way the data is formatted, so the harvesting code tries to clean things up a bit. You can of course adjust this to meet your own needs.

If you don’t want to do the harvesting yourself, there’s pre-harvested datasets that you can download immediately from Cloudstor and start exploring. The complete harvest of all 400,000+ records is available both in JSONL (newline separated JSON) and CSV formats. There’s also a series of separate datasets for the most frequently occurring programs: RN Breakfast, RN Drive, AM, PM, The World Today, Late Night Live, Life Matters, and the Science Show.

There’s also a notebook that demonstrates a few possible ways you might start to play with the data – looking at the range of programs, the distribution of records over time, the people involved in each story, and words in the titles of each segment.

This is a very rich source of data for examining Australia’s political and social history over the last twenty years. Dive in and see what you can find! #dhhacks

GLAM Workbench wins British Library Labs Research Award!

Wed, 16 Dec 2020 11:46:00 +1100

Asking questions with web archives – introductory notebooks for historians has won the British Library Labs Research Award for 2020. The awards recognise ‘exceptional projects that have used the Library’s digital collections and data’.

This project gave me a chance to work with web archives collections and staff from the British Library, the National Library of Australia, and the National Library of New Zealand, and was supported by the International Internet Preservation Consortium’s Discretionary Funding Program.

We developed a range of tools, examples, and documentation to help researchers use and explore the vast historical resources available through web archives. A new web archives section was added to the GLAM Workbench, and 16 Jupyter notebooks, combining text, images, and live code, were created.

Here’s a 30 second summary of the project!

The judges noted:

“The panel were impressed with the level of documentation and thought that went into how to work computationally through Jupyter notebooks with web archives which are challenging to work with because of their size. These tools were some of the first of their kind.

“The Labs Advisory Board wanted to acknowledge and reward the incredible work of Tim Sherratt in particular. Tim you have been a pioneer as a one-person lab over many years and these 16 notebooks are a fine addition to your already extensive suite in your GLAM Workbench. Your work has inspired so many in GLAM, the humanities community, and BL Labs to develop their own notebooks. To our audience, we strongly recommend that you look at the GLAM Workbench if you’re interested in doing computational experiments with many institutions’ data sources.

Thanks to Andy, Olga, Alex, and Ben for your advice and support. And thanks to the British Library Labs for the award! #dhhacks

The GLAM Workbench as research infrastructure (some basic stats)

Tue, 15 Dec 2020 10:43:00 +1100

Repositories in the GLAM Workbench have been launched on Binder 3,529 times since the start of this year (according to data from the Binder Events log). That’s repository launches, not notebooks. Having launched a repository, users might use multiple notebooks. And of course these stats don’t include people using the notebooks in contexts other than Binder – on their own machines, servers, or services like AARNet’s SWAN. Or just viewing the notebooks in GitHub and copying code into their own projects.

I’m suspicious of web stats, but the Binder data indicates that people have actually done more than ‘visit’ – they’ve spun up a Binder session ready to do some exploration.

Every Jupyter notebook in the GLAM Workbench has a link that opens the notebook in Binder. If you click on the link, Binder reads configuration details from the repository and loads a customised computing environment. All in your browser! That means you can start using the GLAM Workbench without installing any software. Just click on the Binder link and start exploring!

There are about 40 different repositories in the GLAM Workbench, helping you work with data from Trove, DigitalNZ, NAA, SLNSW, NSW Archives, NMA, ArchivesNZ, ANU Archives & more! The image below shows them ranked by number of Binder launches this year.

The web archives section was added this year in collaboration with the IIPC, the UK Web Archive, the Australian Web Archive, and the NZ Web Archive. Its annual number of launches is inflated a bit by the development process. But there’s been 426 launches since it went public in June.

I’m really pleased to see the Trove newspaper harvester up near the top. At least once a day (on average) someone’s been firing up the repository to grab Trove newspaper articles in bulk.

Overall, that’s about 11 GLAM Workbench repository launches a day on Binder. It might not seem like much, but that’s 11 research opportunities that didn’t exist before, 11 GLAM collections opened to exploration, 11 researchers building their digital skills…

As humanities researchers continue to learn of the possibilities of GLAM data and develop their digital skills the numbers will grow. It’s a start. And a reminder that not all research infrastructure needs to be built in Go8 unis, by large teams, with $millions. We can all contribute by sharing our tools and methods. #dhhacks

Fri, 27 Nov 2020 16:02:00 +1100

Earlier this year I gave a seminar for the International Internet Preservation Consortium (IIPC) introducing the web archives section of the GLAM Workbench. The seminar is now available online: youtu.be/rVidh_wex…

Here are the slides if you want to follow along. #dhhacks

Harvest text from the Australian Women's Weekly!

Wed, 25 Nov 2020 15:52:00 +1100

The Trove Newspaper & Gazette Harvester has been updated to version 0.4.0. The major change is that if the OCRd text for an article isn’t available through the API, it will be automatically downloaded via the web interface. What does this mean in practice? Well previously you couldn’t harvest OCRd text from the Australian Women’s Weekly because it’s not included in API results, but now you can!

You don’t need to do anything differently. If there are AWW articles in your search, and you ask for all the OCRd text using the --text option, the AWW text files will automagically appear in your harvest.

Under the hood, I’ve started using html2text to remove tags from the OCRd text. I think this should produce more consistent results. As previously, line breaks are removed by default from the OCRd text files. However, I’ve now added a --include_linebreaks option if you’d like to keep them. This generally produces text that is more human-readable, but note that the line breaks produced by OCR aren’t always accurate.

Head to the GLAM Workbench to try it out, or download the code from PyPi. #dhhacks

Beyond the copyright cliff of death

Fri, 13 Nov 2020 09:56:00 +1100

If you’ve done any searching in Trove’s digitised newspapers, you’ve probably noticed that there aren’t many results after 1954. This is basically because of copyright restrictions (though given the complexities of Australia’s copyright system, you can’t be sure that everything published before 1955 is out of copyright). We can visualise the impact of this by looking at the number of newspaper articles in Trove by year.

You can see why I started referring to it as the copyright cliff of death.

But you can also see a little trickle of articles continuing post-1954. The number of newspapers from beyond the copyright cliff of death continues to increase as agreements are made with publishers to put them online. I just checked and there’s now 83 newspapers that have at least some post-1954 articles available. Here’s the top 10 (by number of articles).

If you’d like to browse the full list of post-1954 newspapers, here’s the data as a CSV (spreadsheet) file.

If you’d like to see how I generated this list, have a look at this notebook in the Trove Newspapers section of the GLAM Workbench.

If you’d like to know how I created the chart above, have a look at Visualise the total number of newspaper articles in Trove by year and state. #dhhacks

Questions? Ask away at OzGLAM Help. #dhhacks

Mon, 26 Oct 2020 17:15:00 +1100

I’ve added a new section to the GLAM Workbench for the ANU Archives. The first set of notebooks relates to the Sydney Stock exchange stock and share lists. As the content note describes:

These are large format bound volumes of the official lists that were posted up for the public to see - 3 times a day - forenoon, noon and afternoon - at the close of the trading session in the call room at the Sydney Stock Exchange. The closing prices of stocks and shares were entered in by hand on pre-printed sheets.

The volumes have been digitised, resulting in a collection of 70,000+ high resolution images. You can browse the details of each volume using this notebook.

I’ve been exploring ways of getting useful, machine-readable data out of the images. There’s more information about the processes involved in this repository. I’ve also been working on improving the metadata and have managed to assign a date and session (Morning, Noon, or Afternoon) to each page. We these, we can start to explore the content!

One of the notebooks creates a calendar-like view of the whole collection, showing the number of pages surviving from each trading day. This makes it easy to find the gaps and changes in process. #dhhacks

Mon, 26 Oct 2020 16:52:00 +1100

I’ve added more years to my repository of Commonwealth Hansard! The repository now includes XML-formatted text files for both houses from 1901 to 1980, and 1998 to 2005. I’ve done some more checking and confirmed that the XML files for 1981 to 1997 aren’t currently available through ParlInfo, however, the Parliamentary Library are looking into it. I’ve also created a CSV-formatted list of sitting days from 1901 to 2005 (based on ParlInfo search results). Details of the harvesting process are available in the GLAM Workbench. #dhhacks

Fri, 14 Aug 2020 18:29:50 +1100

Another #GLAMWorkbench update! Snip words out of @TroveAustralia newspaper pages and create big composite images. OCR art! glam-workbench.github.io/trove-new… #dhhacks

Thu, 30 Jul 2020 12:37:16 +1100

Ok, so do you want to make your own ‘scissors & paste’ messages using words from @TroveAustralia newspaper articles? Go to the notebook in #GLAMWorkbench & click on ‘Run live on Binder in Appmode’. #dhhacks

Wed, 29 Jul 2020 14:17:56 +1100

Another #GLAMWorkbench update! The Trove Harvester will now download both newspaper and gazette articles in bulk. You can optionally include full text, and save copies of the articles as images and PDFs. #dhhacks glam-workbench.github.io/trove-har…

Tue, 28 Jul 2020 10:27:51 +1100

Interested in using web archives in your research? Join us on 5/6 August for a free @netpreserve webinar introducing the tools and examples available in the new #webarchives section of the #GLAMWorkbench. There are two timeslots to cover multiple timezones: www.eventbrite.com/e/iipc-rs… and www.eventbrite.com/e/iipc-rs…

Mon, 27 Jul 2020 18:46:40 +1100

Introducing a brand new section of the #GLAMWorkbench, exploring the @MuseumsVictoria collection API. Harvest species records, display random images, and download ALL THE ANTECHINUSES! glam-workbench.github.io/museumsvi… #dhhacks

Mon, 27 Jul 2020 16:32:34 +1100

New additions to the @TroveAustralia books section of the #GLAMWorkbench – word frequency examples with OCRd text from digitised books, and a random recipe generator powered by a 19th C cook book! glam-workbench.github.io/trove-boo… #dhhacks

Mon, 27 Jul 2020 11:52:18 +1100

With the recent changes to @TroveAustralia, the Australian Women’s Weekly cover browser was retired. As a low-tech alternative, I’ve harvested all the cover images from the Women’s Weekly and saved them into PDFs for easy browsing, one for each decade. There are 2,566 images from 1933 to 1982.

Just click on the link below each image to explore the complete issue on Trove. You can also download the full collection of images from Cloudstor. There’s a CSV file containing all the issue metadata.

The notebook used to harvest the images is in the Trove newspapers section of the GLAM Workbench. You could easily adapt the notebook to harvest the front pages of any newspaper. #dhhacks

Fri, 17 Jul 2020 23:11:35 +1100

The Trove books section of the #GLAMWorkbench has been updated. There’s a fresh harvest of OCRd text & the notebooks have been changed to work with the new @TroveAustralia interface. Download & explore 24,620 files (3gb) of OCRd text! #dhhacks

Fri, 17 Jul 2020 17:07:44 +1100

Revisiting my Historic Hansard XML repository & realising how easy it is to load files as needed via the GitHub API & explore with Pandas & Jupyter. This #GLAMWorkbench notebook helps you explore a particular year/house. #dhhacks

Tue, 14 Jul 2020 14:31:47 +1100

The Trove Journals section of the #GLAMWorkbench has been updated to work with the new @TroveAustralia interface! I’ve also re-harvested ALL the OCRd text from digitised journals — 6gb of text from 397 journals now downloadable in bulk from CloudStor. #dhhacks

Sun, 12 Jul 2020 14:18:00 +1100

New in #GLAMWorkbench! After you’ve used the @TroveAustralia Newspaper Harvester to download lots & lots of articles, try exploring the results in Datasette. This notebook sets everything up, you can even add full text search & images! #dhhacks

Mon, 29 Jun 2020 10:48:53 +1100

Download newspaper articles in bulk! The Trove Newspaper Harvester has been updated to work with the new @TroveAustralia interface. I’ve also added the ability to save articles as .jpg images! The easiest way to get started is via the #GLAMWorkbench. #dhhacks

New GLAM Workbench section on web archives!

Wed, 27 May 2020 14:11:25 +1100

We tend to think of a web archive as a site we go to when links are broken – a useful fallback, rather than a source of new research data. But web archives don’t just store old web pages, they capture multiple versions of web resources over time. Using web archives we can observe change – we can ask historical questions. But web archives store huge amounts of data, and access is often limited for legal reasons. Just knowing what data is available and how to get to it can be difficult. Where do you start?

The GLAM Workbench’s new web archives section can help! Here you’ll find a collection of Jupyter notebooks that document web archive data sources and standards, and walk through methods of harvesting, analysing, and visualising that data. It’s a mix of examples, explorations, apps and tools. The notebooks use existing APIs to get data in manageable chunks, but many of the examples demonstrated can also be scaled up to build substantial datasets for research – you just have to be patient!

Have you ever wanted to find when a particular fragment of text first appeared in a web page? Or compare full-page screenshots of archived sites? Perhaps you want to explore how the text content of a page has changed over time, or create a side-by-side comparison of web archive captures. There are notebooks to help you with all of these.

To dig deeper you might want to assemble a dataset of text extracted from archived web pages, construct your own database of archived Powerpoint files, or explore patterns within a whole domain. The notebooks provide a range of approaches that can be extended or modified according to your research questions.

The development of these notebooks was supported by the International Internet Preservation Consortium’s Discretionary Funding Programme 2019-2020, with the participation of the British Library, the National Library of Australia, and the National Library of New Zealand. #dhhacks

Fri, 08 May 2020 12:18:15 +1100

Thanks to @NetPreserve, I’ve been spending time lately working on a set of web archive exploration notebooks for the #GLAMWorkbench. Here’s an example to create/compare screenshots of captures. #dhhacks

Fri, 03 Apr 2020 00:19:24 +1100

The GLAM CSV Explorer has had a few updates — you can now filter by organisation, and upload your own CSV files! #GLAMWorkbench Try it live on Binder.

Buildings might be closed, but the data is open – explore hundreds of datasets from Australian GLAM organisations!

Tue, 31 Mar 2020 13:48:07 +1100

For a couple of years I’ve been harvesting datasets created or published by Australian GLAM organisations through government data portals. I’ve just completed the latest harvest, and there’s now 369 datasets, containing 983 files, from 23 GLAM organisations. 628 of these files are in CSV (spreadsheet) format.

There’s a number of ways that you can explore the harvested data. You can browse a big list of datasets, or download a CSV containing all the harvested data or just those formatted as CSVs.

With this harvest I’ve added a new way of searching and filtering the harvested data using Datasette running on Glitch. This interface lets you narrow your queries by field or facet, and search text fields for keywords.

But what’s actually in all those CSV files? If you’d like to start exploring the content of the datasets, then give my GLAM CSV Explorer a go! The CSV Explorer looks at each column in the dataset and tries to identify the type of data inside. It then attempts to tell you something useful about it.

For all the details, links, and harvesting code, see the #GLAMWorkbench. #dhhacks

Wed, 11 Mar 2020 20:05:04 +1100

My harvest of OCRd text from @TroveAustralia digitised books, ephemera, and parliamentary papers has been updated! There’s now 19,795 text files (about 3gb) to explore! Harvesting details and links to browse/download files from Cloudstor are in the #GLAMWorkbench. #dhhacks

Tue, 03 Mar 2020 11:09:17 +1100

I’ve added some more documentation to the Trove Newspaper Harvester page in the #GLAMWorkbench. Get your @TroveAustralia newspaper articles in bulk! #dhhacks #collectionsasdata

Thu, 27 Feb 2020 12:47:18 +1100

New section added to the #GLAMWorkbench with examples from @Library_Vic! #slvdata #dhhacks #collectionsasdata

Thu, 27 Feb 2020 12:18:57 +1100

More fun with @iiif_io and images from @library_vic – resize, rotate, crop and more! Try it out with this new notebook in the #GLAMWorkbench. #slvdata #dhhacks

Wed, 26 Feb 2020 22:39:10 +1100

New #GLAMWorkbench notebook! Download images from @Library_Vic using IIIF and Handle… #dhhacks

Fri, 21 Feb 2020 10:10:00 +1100

Want to save @TroveAustralia newspaper articles as images (that aren’t sliced up in annoying ways)? There’s an app for that in the #GLAMWorkbench. #dhhacks

Mon, 17 Feb 2020 21:08:39 +1100

New ‘Trove images' section added to the #GLAMWorkbench! Here you’ll find my latest Jupyter notebook harvesting data about the use of standard licences & rights statements in Trove’s picture zone. #dhhacks

Fri, 14 Feb 2020 09:09:06 +1100

Voting in the 2019 @dhawards is now open! Go and check out all the cool #DigitalHumanities projects from around the world. And while you’re there, you might like to vote for my #GLAMWorkbench in the ‘Tools’ category!

Wed, 20 Nov 2019 22:24:46 +1100

New #GLAMWorkbench section with examples of how to get random-ish works and newspaper articles from @TroveAustralia. #dhhacks

Thu, 05 Sep 2019 00:03:04 +1100

The @naagovau RecordSearch section of the #GLAMWorkbench has been updated with more notebooks to help you get Australian archives data in a usable form. glam-workbench.github.io/recordsea… Useful for #twitterstorians, #ozhist, & #govhack!

Sun, 25 Aug 2019 14:33:18 +1100

I’ve updated my harvest of OCRd text from digitised journals in @TroveAustralia. The complete dataset now includes 33,035 issues from 720 titles – about 8gb of text to explore. Details in the #GLAMWorkbench: glam-workbench.github.io/trove-jou… #dhhacks

Fri, 09 Aug 2019 23:55:23 +1100

There’s a new section of the GLAM Workbench devoted to the National Museum of Australia collection API! Harvest @nma data, then explore it by time and place. #dhhacks

Wed, 24 Jul 2019 18:09:56 +1100

Updates to the Trove newspapers section of GLAM Workbench – adding links to app-ified versions of some notebooks, & direct links to @mybinderteam for everything. If you work with @TroveAustralia newspapers you might find it useful.

Download & explore 1,499,259 rows of open data from NSW State Archives Online Indexes

Wed, 24 Jul 2019 14:34:00 +1100

NSW State Archives publishes a number of detailed indexes containing data manually extracted from their records. These provide additional entry points to the records, such as a person’s name, or a place. But they also provide useful data for analysis. However, to explore the index data we need to get it out of the web interface and into a form that can be easily downloaded and manipulated.

I’ve created a series of Jupyter notebooks to harvest the all the indexes and save the data in a series of CSV-formatted files. I’ve also updated my repository containing all the harvested CSV files. It’s available from the new NSW State Archives section of my GLAM Workbench. There are currently 64 different index datasets, containing 1,499,259 rows of data.

And to help you get a sense of what’s actually in all those CSV files, I’ve created an interactive Index Explorer. Just select an index from the list, and the Index Explorer will generate a series of tables and visualisations that provide an overview of the data. Try running it live on Binder.

Thanks to the State Archives staff and volunteers for preparing all this most excellent data. #dhhacks

Thu, 11 Jul 2019 22:00:00 +1100

New in GLAM Workbench! Notebooks to harvest, index, analyse, and aggregate transcripts of speeches & interviews by Australian prime ministers. Plus links to harvested data and aggregated files. #dhhacks

Thu, 11 Jul 2019 12:26:29 +1100

Reorganising things a little at GLAM Workbench. @statelibrarynsw gets its own section. Hansard and @datagovau GLAM datasets now under ‘Australian government’. Making some space for further additions…

Mon, 17 Jun 2019 18:17:43 +1100

Kicked off a new GLAM Workbench repository dedicated to @SLSA with a quick notebook hack to get higher res versions of digitised photos. #dhhacks

Fri, 07 Jun 2019 14:57:28 +1100

Recent additions to the Trove Newspapers section of the GLAM Workbench: getting images from @TroveAustralia newspaper articles, and uploading article to @Omeka-S: glam-workbench.github.io/trove-new…

Sun, 26 May 2019 17:39:22 +1100

More GLAM Workbench updates! More full text of Australian books! I’ve added the notebook & data from my harvest of @TroveAustralia books in the @InternetArchive. There’s metadata and text of 1,153 books to explore. #dhhacks

Sun, 19 May 2019 23:22:47 +1100

Some overdue updates to the GLAM Workbench. First here’s details, data, and code from a harvest of GLAM datasets on @datagovau. Includes details of more than 400 CSV datasets. #dhhacks

Thu, 09 May 2019 20:14:00 +1100

Over the last week I’ve been downloading editorial cartoons published in The Bulletin from @TroveAustralia. There’s 3,471 cartoons – at least one from every issue published between 4 Sep 1886 and 17 Sep 1952. And you can browse them all…

To make it easier to explore the images, I’ve compiled them into a series of PDFs – one PDF for each decade. The PDFs include lower resolution versions of the images together with their publication details and a link to Trove. They’re all available from DropBox:

1886 to 1889 (45mb PDF)
1890 to 1899 (139mb PDF)
1900 to 1909 (147mb PDF)
1910 to 1919 (153mb PDF)
1920 to 1929 (159mb PDF)
1930 to 1939 (151mb PDF)
1940 to 1949 (146mb PDF)
1950 to 1952 (42mb PDF)

The complete collection of high resolution images (about 60gb in total) can be downloaded from CloudStor. The names of each image file provide useful contextual metadata. For example, the file name 19330412-2774-nla.obj-606969767-7.jpg tells you:

19330412 – the cartoon was published on 12 April 1933
2774 – it was published in issue number 2774
nla.obj-606969767 – the Trove identifier for the issue, can be used to make a url eg [nla.gov.au/nla.obj-6...](https://nla.gov.au/nla.obj-606969767)
7 – on page 7

There’s some details of the method that I used to find the cartoons in this notebook. I’ve also documented everything in the Trove Journals section of my GLAM Workbench.

Be warned – the language, images, and ideas presented in The Bulletin were often racist, anti-Semitic, and sexist. You won’t have to look far within this collection to find something offensive. This was, after all, the journal whose slogan for many years was ‘Australia for the white man’. This is our history… #dhhacks

Sat, 27 Apr 2019 12:44:19 +1100

And now my GLAM Workbench has a ‘Trove Maps’ section to document examples and explorations using data from @TroveAustralia’s ‘map’ zone: glam-workbench.github.io/trove-map… Includes a list of 20,158 maps with high-res downloads. #dhhacks

Tue, 23 Apr 2019 14:49:00 +1100

I’ve been busy lately harvesting LOTS of full text data from @TroveAustralia’s digitised journals – so many opportunities for research! You should be able to get to all the code & data from the new Trove journals section of my GLAM Workbench. #dhhacks

Mon, 22 Apr 2019 23:07:01 +1100

I’ve added a section for the @TroveAustralia ‘book’ zone to the GLAM Workbench.

Mon, 22 Apr 2019 12:11:06 +1100

All 9,738 OCRd text files harvested from books, pamphlets and leaflets in @TroveAustralia’s ‘book' zone have been uploaded to @aarnet’s CloudStor for easy browsing/download. There’s also a 400mb zip file if you want the whole lot.

The harvesting method and code is available in this notebook. All this and more will be documented soon in my GLAM Workbench. #dhhacks

Sun, 31 Mar 2019 21:23:43 +1100

Train from Canberra to Melbourne booked for #VALATechCamp. I’ll be hanging around both days, so let me know if you’d like to chat about the GLAM Workbench, Jupyter, Trove data, or any of the other things I fiddle with…

Sun, 24 Feb 2019 20:15:56 +1100

I’ve updated the notebook for harvesting records from @archivesnz’s Archway database in my GLAM Workbench. I just used it to harvest more than 8,000 records from series 8333 relating to naturalisation. #dhhacks

Thu, 21 Feb 2019 11:46:17 +1100

New section added to my GLAM Workbench for the Queensland State Archives (@qsarchives). Includes a notebook to add series information into their Naturalisations 1851-1904 index. #dhhacks

Sun, 17 Feb 2019 14:33:44 +1100

Suggestions of new topics and collections for my GLAM workbench are welcome!

Sun, 17 Feb 2019 12:40:14 +1100

I’ve added a section for Library and Archives Canada to my GLAM workbench. The first notebook extracts records of people from a specific country from their naturalisations database and saves the results as a CSV file. #dhhacks

Fri, 15 Feb 2019 23:59:24 +1100

Current status — extracting data from Library and Archives Canada’s 1915-1946 naturalisation database. Coming soon to my GLAM Workbench…

Fri, 01 Feb 2019 22:26:26 +1100

I’ve added a ‘save chart’ option to the QueryPic app in my GLAM Workbench. Visualise your searches in @TroveAustralia newspapers, then save the results as HTML for easy download. #dhhacks

Wed, 23 Jan 2019 23:17:20 +1100

One more and I’m done for the night… New GLAM Workbench page for the ‘Trove API introduction’ notebooks.

Wed, 23 Jan 2019 22:22:10 +1100

I’ve finished putting details of all the current GLAM Workbench repositories into the new documentation site. Still a few notebooks to migrate from the original workbench, but getting there! There’s about 50 Jupyter notebooks so far. #dhhacks

Wed, 23 Jan 2019 17:07:30 +1100

Added a ‘data’ section to the GLAM Workbench docs, with info on harvests from government data portals, as well as series from @naagovau relating to ASIO and the White Australia Policy.

Wed, 23 Jan 2019 11:23:08 +1100

And now a GLAM Workbench page for @Te_Papa…

Wed, 23 Jan 2019 10:49:01 +1100

Added a page for @ArchivesNZ’s Archway to the GLAM Workbench docs…

Tue, 22 Jan 2019 17:30:49 +1100

So here’s some fun things to do with @TroveAustralia newspapers… (via GLAM Workbench)

Tue, 22 Jan 2019 15:06:50 +1100

Ok, more documentation for you — page for the @DigitalNZ API in GLAM Workbench updated!

Tue, 22 Jan 2019 13:11:33 +1100

Slowly working my way through the documentation for my GLAM Workbench. Still lots to do, but I think the page for @naagovau’s RecordSearch is now up-to-date.

Tue, 22 Jan 2019 12:38:15 +1100

If there are APIs or other data sources you’d like me to add to my GLAM Workbench, feel free to create an issue. You could also describe what sorts of tools or examples using that data source would be useful.

Wed, 16 Jan 2019 17:41:45 +1100

New notebook added to the #GLAMWorkbench RecordSearch repository — get the basic details of agencies associated with all government functions used in @naagovau’s RecordSearch and save to a single JSON data file. View code and data. #dhhacks

glamworkbench on Tim Sherratt

Generosity in practice – a chat with Paula Bray at the State Library of Victoria

Zotero translator for Libraries Tasmania updated!

my place – exploring SLV collections through a street address

The collections

OpenStreetMap

Sands & MacDougall’s directories

Committee for Urban Action photographs

Georeferenced maps

Parish maps

Newspapers

Photographs

The interface

my house

my street

my suburb/town

nearby

The data

Next steps?

Geolocating photos from the SLV's Committee for Urban Action collection

The process

Finding intersections

Finding road segments

Problems finding intersections

Assembling the data

Constructing the interface

More information and links

Goodbye 2025! A brief summary of the highlights and lowlights…

Highlights

Lowlights

Next year

How you can help

Exploring Victorian newspapers

Assembling the data

Building the apps

Why bother?

Counting down... (to the end of my SLV residency)

Some Sands & Mac tweaks thanks to ALTO and IIIF

A new way of searching Sands & Mac

Technical information follows…

Downloading the PDFs

Downloading the ALTO files

Assembling and publishing the database

More information

Turning the SLV's maps into data with Allmaps and some GLAM plumbing

Me at 63...

What was your first history related job? What path have you taken since then?

What kind of work have you done? What are you working on now?

Research or writing? (What do you enjoy more and why?)

What are the best and hardest things about the kind of work you do?

Creating bounding boxes for parish maps in the SLV collection

The metadata

The method

Creating bounding boxes

Limitations

Visualising the results

Exploring SLV urls

Permalinks

Structured data

IIIF and images

Handles

From old to new

The process of GLAM hacking

Creative Technologist-in-Residence at the State Library of Victoria!

WikiFest at the State Library of Victoria

Bonus userscript

GLAM hacking with userscripts

Some old userscripts

And a new userscript to improve Trove lists

Using userscripts

The rebirth of Wragge Labs (and moving my Heroku apps)

A new (old) home

Moving house

Redirecting Heroku

More experiments to come?

The future of the past... in the present

Mining for meanings

I. Beyond discovery

II. Finding formulas

III. The future of the past

GLAM Workbench – preprint for 'Building User-Friendly Toolkits and Platforms for Digital Humanities'