Tim Sherratt

Generosity in practice – a chat with Paula Bray at the State Library of Victoria

Mon, 16 Mar 2026 15:14:17 +1100

While I was in Melbourne during my time as Creative Technologist-in-Residence at the State Library of Victorian LAB, I had a conversation with Paula Bray for the LAB’s podcast series. Paula is the SLV’s Chief Digital Officer, and has long championed the importance of digital innovation in the GLAM sector. It was fun to chat about stuff that I’ve been doing for the last 30 years, any why openness and generosity is important in working with GLAM collections. You can listen to our conversation on the LAB site.

Zotero translator for Libraries Tasmania updated!

Tue, 10 Mar 2026 10:31:36 +1100

The Zotero translator for Libraries Tasmania has been updated, fixing a problem with attaching images of digitised resources. The fix is in the main Zotero repository now, so it should find its way to your computer automatically.

I created the first version of the Libraries Tasmania translator back in 2022 – this post describes what it does. It works across all three sections of the catalogue, including the archives, and names index. The translator captures metadata, PDFs, and images from records, including things like digitised pages from convict records. This makes it easy for researchers to assemble their own datasets of Tasmanian records in Zotero, where they can add notes and annotations, or share with colleagues.

Capture images and metadata from the Libraries Tasmania catalogue using Zotero

The update was necessary because Libraries Tasmania changed the way some digitised resources were displayed and downloaded. Keeping Zotero translators working across system updates can take a bit of work! I also took the opportunity to update the code to meet current Zotero guidelines and clean up a few lingering problems. If you notice any oddities, please let me know.

There are now at least 8 custom translators to help you work with Australian GLAM collections.

Exploring georeferenced maps from the SLV collection

Thu, 12 Feb 2026 23:05:06 +1100

I’m in the process of tying up all the documentation relating to my time as Creative Technologist-in-Residence at the State Library of Victoria LAB. But as I was looking through the list of outputs, I realised I’d never written anything about the interface I created to explore georeferenced maps from the SLV collection.

I also remembered that there were a few improvements I wanted to make to the interface. So instead of spending a few hours writing up a blog post, I’ve spent several days completely overhauling the Georeferenced Maps Explorer. I’m pretty happy with how it’s working now. Have a play!

Wilson's Prom made up of a patchwork of georeferenced maps and aerial photographs using the Georefrenced Maps Explorer. Try it now!

To get started, just click on the basemap. Details of all georeferenced maps within 50km of your selected point will be displayed in the right-hand column. As you move your mouse over the list of results, the boundaries of the georeferenced maps will be displayed on the basemap. This gives you a preview of their location and size. Click on one of the results to display the georeferenced map as a layer on top of the modern basemap.

Hover over a result to see the map boundaries

You can add as many maps as you like. If your selected maps overlap, you can change the order in which they’re shown. Click on the layers icon in the top left of the basemap. You’ll see a list of the maps that are currently displayed. Use the arrow buttons to move a map backwards or forwards. You can also use the sliders to adjust the opacity of each map. This can make it easier to examine the relationship between maps. For example, you might want to compare the features of a historic map with those of the underlying basemap.

Stitch together multiple maps like this series of seven photomaps, and change the opacity to see the features underneath

The Explorer’s url updates with every selection you make, so you can bookmark or share a url to return to the same position and collection of maps. For example, this link will take you to the collection of maps of Wilson’s Prom shown above.

The background

If you missed the start of this journey back in November last year, you might be wondering what the georeferenced maps are and where they come from. During my SLV LAB residency, I found a way of hooking the SLV’s digitised maps up to a tool called Allmaps that helps you identify points that connect historic maps to our modern coordinate system. When enough points have been identified, the historic maps can be positioned on a modern basemap. This is known as georeferencing, georectifying, or ‘map warping’, as the results can often appear skewed or warped.

Once I had connected things up, I invited the world (or at least the tiny part of it that follows me on social media) to help turn the SLV’s maps into data. And they did! As of today, 1,447 of the SLV’s digitised maps have been georeferenced. This dashboard displays current georeferencing progress.

The total number of SLV maps georeferenced over time. It's still going up!

There’s still plenty more to do. If you’d like to help, the full instructions are available here. Georeferencing is pretty fun, so why not have a go?

You can explore the current collection of georeferenced maps in a few different ways. There’s a dataset you can download or search that gets updated every two hours. This data is loaded into a spatial database that’s used by the Georeferenced Maps Explorer. As part of my recent improvements, I’ve automated this process as well, so the database should be updated with the latest additions every 24 hours.

You can also search for georeferenced maps using the my place app. You just enter an address and my place pulls together data from a variety of sources – mixing the georeferenced maps up with parish maps, newspapers, photos, and entries from the Sands & MacDougall’s directories.

Georeferenced maps in my place results

The interface

The Georeferenced Maps Explorer uses MapLibre and the Allmaps MapLibre plugin to display the georeferenced maps. You might notice that it looks pretty similar to the Newspapers Explorer and the CUA Browser, both of which use MapLibre, as well as Bulma for CSS. I’ve been trying to settle on a fairly standard set of tools that I can use to create and maintain these sorts of interfaces without too much fuss. Basically I just cut and paste a lot of stuff, then modify as needed.

When you click on the basemap in the Explorer, the coordinates are sent off to the spatial database to retrieve details of georeferenced maps within 50km. The spatial database runs in Datasette, which has a built-in JSON API that I use with a set of predefined ‘canned’ queries to pull back the data I need. The results are displayed in the right-hand column, along with square thumbnails generated by the SLV’s IIIF service.

The metadata includes distance and area measures. These are used to find and sort the results. There are two distance measures, one from your selected point to the closest boundary of a map, and the other to the centre of a map. If the point is contained within a map’s boundaries, then the ‘bounds’ distance is zero. The search query finds maps whose closest boundaries are within 50km. Originally I sorted the results by this distance and the area of the maps. But this meant that large scale maps that included the selected point (such as maps of the whole of Victoria) appeared above nearby local maps. To make it easier to find maps within an area, I added the ‘centre’ distance and now sort the results using that. This allows nearby maps that don’t include the current point to bubble up towards the top of the search results, above many of the large scale maps. It’s far from perfect, but I think it strikes an ok balance.

The data also includes the boundaries of each map as GeoJSON. I use this to generate a MapLibre layer that contains all the boundaries as polygons. The boundaries are hidden until you hover over the corresponding search result, then the opacity of the boundary is flipped to 1 and it magically appears.

When you click on a search result, a request is fired off to Allmaps for the full georeferencing data. The Allmaps plugin uses this to retrieve the map image from the SLV’s IIIF service and display the warped map in MapLibre.

I looked around for quite a while to find a good way of changing the opacity and order of the warped maps in MapLibre. I eventually found the Map Libre GL Layer Manager which did a lot of what I wanted. I forked the repository and modified the code to get the opacity slider to work with warped map layers. Warped map layers already have a setOpacity method, it was just a matter of checking for ‘custom’ layers, then finding where the warped map was in the layer object.

if (type == "custom") {
        layer.implementation.setOpacity(opacity);

I also made a few cosmetic changes, such as renaming the tooltips on the reorder buttons from ‘move up’ and ‘move down’ to ‘send back’ and ‘bring forward’ – up and down just confused me.

I tried for a long time to find some way of adding tooltips or popups to the warped maps that would show their details when you moved the mouse over them. I found that if you were displaying multiple maps that looked similar, such as the photomaps above, it was difficult to know which map was which. After a chat with the Allmaps developers in their IIIF Slack channel, I realised that this approach wouldn’t work as the warped map layers don’t currently listen to mouse events. Instead I decided to add hover events to the list of results, rather than the maps, and use them to display the map boundaries as described above. This way I get the connection between the map and metadata that I wanted, as well as a useful way of previewing results.

I think I’ve probably stopped fiddling with the interface for now. I hope you find it useful!

The future?

There’s more that I’d like to do with the georeferenced maps. In particular, I’ve been thinking about an interface with a slider that showed the changing patchwork of maps over time…

Related resources:

the code for the Georeferenced Newspapers Explorer and all the other apps and sites I created during my residency is in this GitHub repository
the code to harvest the georeferenced data from Allmaps and build the dashboard is in this GitHub repository
there’s also the full list of all the apps, code, posts, and talks created during my residency

my place – exploring SLV collections through a street address

Mon, 02 Feb 2026 22:43:20 +1100

‘What can I find out about my house?' My work as Creative Technologist-in-Residence at the SLV LAB was inspired by questions like this that librarians at the SLV hear every day. I wanted to explore how the Library’s place-based collections could be used to provide new entry points for discovery and navigation – entry points based not on words, but locations.

At the end of my residency, I pulled all the different collections I’d been working with into a single interface – my place. It’s not polished or complete, but I think it’s a useful starting point to think about the possibilities. You just type in an address, street name, or place name and my place shows you maps, photos, newspapers, and even extracts from the Sands & MacDougall directories. Try it now!

Try my place! Just enter an address in the search box.

Search results in my place are bookmarkable. So save and share your discoveries!

The collections

my place draws its data from a number of different place-based collections that I’ve been working on during my residency.

OpenStreetMap

When you enter an address in the search box, my place looks it up in OpenStreetMap to get its geospatial coordinates. It then places a marker and re-centres the map at the top of the app.

Map centred on 149 Brunswick Street, Fitzroy

OpenStreetMap is also used to retrieve additional information about the suburb, including its boundaries.

Sands & MacDougall’s directories

my place queries the full-text searchable version of Sands & Mac for addresses. Results will vary based on the OCR quality and the nature of query, but it can give you a potted history of who has lived in your house. The search results are displayed in chronological order, and include an image snippet showing the actual printed entry as well as the text content and metadata.

Occupants of 149 Brunswick Street, Fitzroy from 1875 to 1925

Committee for Urban Action photographs

If you enter a full street address, my place will search the CUA collection for photos associated with the segment of road that includes the current address. It then displays the individual images from any matching photosets.

Photographs from CUA of the currently selected road

Otherwise my space will look for CUA photos that are near the current location, and display a randomly-selected image from each photoset.

Georeferenced maps

my place searches through digitised maps from the SLV collection that have been georeferenced by the public. It finds maps that either intersect with the currently selected location, or are nearby.

If you enter a full street address, the first 6 georeferenced maps will be positioned on a modern basemap with a marker indicating the currently selected point. This means you can see your address on a historical map. The number of georeferenced maps that can be displayed in this way is determined by the browser – so I’ve limited it to 6 to be safe.

Georeferenced maps positioned on a modern basemap, showing the location of the currently selected address

Parish maps

my place searches through parish maps in the SLV collection that have geospatial coordinates or approximate bounding boxes. It finds maps that either intersect with the currently selected location, or are nearby.

Newspapers

my place searches through my dataset of newspapers in the SLV collection that have a place of publication documented in the ‘Place newspaper published’ metadata field. It finds newspapers that are either associated with the current suburb/town, or a nearby suburb/town. This includes digitised and non-digitised titles. Digitised titles include a link to Trove.

Newspapers from the SLV collection published in Fitzroy

Photographs

I thought it would be cool to include a few photographs of the current suburb or town. To do this, I downloaded a list of place names from VicNames, then used the place names to search the SLV catalogue for photographs with relevant subject headings. A random selection of the harvested images is displayed in my place.

A few images of Fitzroy, displayed alongside a map of Fitzroy's current boundaries using data from OpenStreetMap

The interface

The interface is pretty simple. You type an address in the box and hit enter. If the geocoding process finds multiple matches, it’ll give you a list to choose from. Once the location is found, a marker is added and the main map re-centres. Then related resources are displayed below the map.

As you scroll down through the results you gradually zoom out from your initial starting point. This is reflected in the four bands or layers used to group resources: ‘my house’, ‘my street’, ‘my suburb’, and ‘nearby’. Each band contains a mix of resources from different collections.

When I started working on my place, I was thinking about a project from around 2010 called The History Wall. Like my place, The History Wall pulled many different types of resources together into a rich exploratory interface. As you scrolled through The History Wall you moved through time, with randomly selected items appearing from a range of sources including Trove newspapers, the ADB, and museum collections.

A version of The History Wall created for the National Museum of Australia's 'Irish in Australia' exhibition

I originally thought I’d inject some of the same randomness into my place, but I was worried it might just get too confusing. I thought it was important to keep the relationship between the starting point and the resources in focus even as you zoomed out. So my visual metaphor shifted to something more like a blast radius map, or a stratigraphic diagram, that displayed distinct groups and layers as you moved beyond the baseline. My limited CSS skills couldn’t make the vision in my head a reality, but there are lots of headings and colours instead to highlight the transitions!

The actual mix of groups and layers displayed depends on the nature of your query. If you’ve entered a complete street address, and there are results for that address in Sands & Mac, then you’ll see ‘my house’, ‘my suburb’, and ‘nearby’. If you’ve only entered a suburb or town, or your street address can’t be found, you’ll see two layers starting with ‘my suburb’.

Here’s an overview of what you might expect to see.

my house

Sands & MacDougall extracts (text search on full address)
georeferenced maps (search for maps that contain the base point)
parish maps (search for maps that contain the base point)

my street

CUA photos (search for matching street identifiers)

if there’s no ‘my house’ layer:

Sands & MacDougall extracts (text search on street name and suburb)
georeferenced maps (search for intersections between maps and street)
parish maps (search for intersections between maps and street)

my suburb/town

suburb boundaries from OSM
images (search for suburb name in metadata)
newspapers (search for suburb name in metadata)

if there’s no ‘my house’ or ‘my street’ layer:

georeferenced maps (search for intersections between maps and suburb boundaries)
parish maps (search for intersections between maps and suburb boundaries)

nearby

CUA photos (search for photosets within 5km of the base point, filtered to remove current street)
georeferenced maps (search for maps within 10km of base point, ordered by distance, max of 24 displayed)
parish maps (search for maps within 10km of base point, ordered by distance, max of 24 displayed)
newspapers (search for newspapers within 100km of base point, ordered by distance, max of 24 displayed)

The data

Most of the data used in my place is stored in two SQLite databases – one for Sands & Mac, and the other for CUA, georeferenced maps, parish maps, and newspapers. The metadata for the collection images is stored in a JSON file that is directly loaded by the interface.

I’ve published the SQLite databases online using Datasette and Spatialite. Spatialite makes it possible to find geospatial features that intersect, or are near, a given point. For example, you could find maps that include a specific set of coordinates.

Datasette has the ability to create ‘canned queries’ that feed url parameters into pre-defined SQL queries. This coupled with Datasette’s built-in JSON API makes it possible to construct query urls in my place and use them to retrieve JSON results data from my databases.

When you enter an address in my place, multiple queries are fired off to find intersecting or nearby resources. For example, this url finds georeferenced maps within 10km of a point at the centre of Fitzroy: slv-places-481615284700.australia-southeast1.run.app/georefere…&distance=10000&_shape=array.

In the case of Sands & Mac, my place uses a canned query that runs a full-text search across the OCRd content of a volume. Suburb names are often abbreviated in Sands & Mac, so the app first runs a query to find possible abbreviations, then adds them into the main query to inject a bit of fuzziness. This is repeated for all 24 digitised volumes.

Once the metadata is retrieved from the databases, images are loaded from the SLV’s IIIF service.

Next steps?

I’m not sure how much more work I’ll do on my place, but there are a few things I’d like to try. In particular, I’d like to help the user understand more about what data is being presented, or not presented, and why. Not all digitised maps have been georeferenced, not all parish maps have coordinates, street numbers have changed, and the OCR in Sands & Mac varies in quality. my place can only present a sample – a gesture towards the wealth of material available from the SLV. I feel that message needs to be made more explicit. Though I’m not sure how without overloading the interface.

There are additional data sources I’d like to play around with. my place already includes some code to query Wikidata for more information about a suburb. But I haven’t had a chance to do anything with it. I’d like to be able to provide additional contextual information from outside the SLV, such as electoral boundaries, populations, even election results. It would also be fun to display sightings of plants and animals from the Atlas of Living Australia.

What can I find out about my house? It would be great if my place could answer that question by taking the user on an open ended journey through our cultural, historical, and environmental landscape.

Geolocating photos from the SLV's Committee for Urban Action collection

Thu, 29 Jan 2026 18:08:06 +1100

Concerned about the loss of built heritage in the 1970s, the Committee for Urban Action photographed streetscapes across urban and regional Victoria. They compiled a remarkable collection of photographs that is now being digitised by the State Library of Victoria. More than 20,000 images are already available online!

The CUA worked systematically, capturing photos street by street, and recording the locations of each set of photographs. This information is used to prepare the title attached to each photo as it’s uploaded to the SLV catalogue. In general, titles include the name of the road where the photo was taken, the name of the suburb or town, and the names of two intersecting roads that define the boundaries of the current road segment. They can also tell you which side of the road the photo was taken on. For example, the title Gore Street, Fitzroy, from Gertrude Street to Webb Street - east side tells us the photo was taken on the east side of Gore Street, Fitzroy between the intersections with Gertrude Street and Webb Street.

Photos from Gore Street, Fitzroy displayed in the SLV image viewer

It’s great to have this sort of structured information linking photos to specific locations, but to navigate through the collection in space we need more. We need to link each photo to a set of geospatial coordinates by mapping each road segment. That was the challenge I took on as part of my residency in the SLV LAB.

When I started working on the collection I wasn’t really sure what was possible. I had to learn a lot, and ended up revising my processes multiple times as I got deeper into the data. But my aim was always to create some sort of map-based interface, that would allow users to click on a street and see any associated CUA photos. It’s still a bit buggy and incomplete, but here it is – explore the CUA collection street by street!

Gore Street, Fitzroy in the new CUA Browser

The process

My basic plan was to find the intersections using OpenStreetMap, then extract geospatial information about the segment of road between the two intersections. This involved much trial and error, but eventually I ended up with a process that:

parsed each item title to try and extract the names of the main road, the suburb, and the two intersecting roads
queried Nominatim for the suburb bounding box
for each intersecting road, queried OSM to find a node at, or around, its intersection with the main road, within the suburb bounding box
created a new bounding box from the coordinates of the two intersections
queried OSM for the main road within this bounding box
extracted the coordinates of the main road segment, removing any points outside of the bounding box

There’s more details below and in these notebooks: cua_finding_intersections.ipynb and cua_data_processing.ipynb.

Finding intersections

As described, the title of each photograph generally includes 4 pieces of information: the road, suburb, intersecting roads, and side. My plan was to find the intersections first to get the limits of the road segment. This is possible thanks to the awesome OpenStreetMap and its Overpass API. It took me a while to get my head around the Overpass query language, but there are lots of useful examples online. The query to find the intersection between Gore Street and Gertrude Street in Fitzroy looks like this:

[bbox:-37.8089071,144.9732006,-37.7929130,144.9851430];
way['highway'][name="Gore Street"];
node(w)->.n1;
way['highway'][name="Gertrude Street"];
node(w)->.n2;
node.n1.n2; 
out body;

You can try it out using Overpass Turbo’s web interface.

In OpenStreetMap, linear features, such as roads or rivers, are represented as ways. Each way is made up of a series of nodes or points with geospatial coordinates. Every way and node has its own unique identifier. Tags can be added to features to describe what type of things they are.

The query above looks for ways named ‘Gore Street’ and ‘Gertrude Street’ that are tagged as highway (a highway in OpenStreetMap is any road-like feature including things bike paths and foot trails).

way['highway'][name="Gore Street"];

It then extracts the nodes that make up each way and looks to see if there are any nodes in common between the two ways. A node shared between two ways indicates an intersection.

node(w)->.n2;
node.n1.n2;

The query is limited using a bounding box that encloses the suburb of Fitzroy. This avoids false positives and keeps down the query load.

[bbox:-37.8089071,144.9732006,-37.7929130,144.9851430];

The JSON result of this query gives as the latitude and longitude of the node at the intersection of the two roads.

{
  "version": 0.6,
  "generator": "Overpass API 0.7.62.10 2d4cfc48",
  "osm3s": {
    "timestamp_osm_base": "2026-01-27T03:11:45Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "node",
  "id": 224750459,
  "lat": -37.8062302,
  "lon": 144.9817848
}

  ]
}

After a bit of testing, I found this worked pretty well, except for roundabouts… In OpenStreetMap, roads don’t actually cross roundabouts – they end on one side, then begin anew on the other side. In cases like this, looking for shared nodes doesn’t work. Instead you have to look to see if the two roads have nodes that are less than a given distance apart. The query is similar to the one above, but uses around when comparing the nodes. In this case I’m looking for nodes that are within 20 metres of each other.

node(w.w2)(around.w1:20);

Finding road segments

Once I had the coordinates of the two intersections, I could look for the segment of road between between them. To do this I created a bounding box using the coordinates of the intersections, and then searched for ways by name within that defined area.

It’s important to note that there’s no one-to-one correspondence between roads and OSM ways. A single road might be represented in OSM as a series of separate, but connected, ways. For example, at a roundabout, or where a road divides, new ways might have been created to document the change. This means that when we query OSM for details of a road we often get back information about multiple ways. Some of these might be things like bike paths which we can filter using tags, but often they’ll be sections of the road that we want. For example, this query for Gore Street, within the bounds of its intersections with Gertrude Street and Webb Street, returns details of two ways.

way["highway"~"^(trunk|primary|secondary|tertiary|unclassified|residential|service|track|pedestrian|living_street)$"][name="Gore Street"](-37.8062302,144.98128480000003,-37.8040076,144.9826827);
out body;
>;
out body;

You can view the result in Overpass Turbo.

However, that doesn’t mean that the full extent of both ways is contained within the bounding box, just that some of the nodes of both ways are inside. Because of this, I filtered the results from all the ways and only kept nodes whose coordinates were within the desired region.

Problems finding intersections

The method described above works pretty well, and once I understood enough about the Overpass API to get out actual paths that I could display on a map, I fed all of the CUA photos through a script and got useful data for more than 80% of them.

One of my early tests.

Then I spent a lot of time trying to understand where the remainder were failing.

Some of them failed because the titles were missing information, or were formatted in a way I didn’t expect. For example, instead of a second intersecting road, some titles just said ‘to end’. This makes perfect sense to a human looking at a map, but it’s difficult to handle programmatically.

Some photos either recorded the wrong suburb, or the boundaries of the suburb had moved since the photos were taken. For example, many of the photos described as being from Eaglehawk are now in California Gully.

Similarly, some road names were wrong either because of documentation errors, or because the names have changed over time. There are also some variations in the way OSM records road names – in particular, I found that roads with hyphenated names sometimes had spaces around the hyphen and sometimes didn’t. There were also a couple of cases where names weren’t attached to the corresponding road segment in OSM, but I was able to edit these in OSM directly.

Other roads had multiple names, or change names along their path. I mean, what’s going on with Brunswick Street and St Georges Road in Fitzroy? Country towns seemed most prone to this – a highway might become ‘Main Road’ within the town boundaries, or the order of hyphenated places in road names might change. I found one road in Clunes that had four different names within the space of a few hundred metres.

One road, four names!

Finally, the routes of some roads had changed – intersections no longer intersected, roads were closed, or new parks had popped up to split a road in two.

My processing script logged the titles I couldn’t locate and I worked through the list manually, trying to identify what each problem was. I suppose there’s two ways I could’ve handled these problems – building more fuzziness into the process to check for things like alternative names, or by compiling a list of ‘corrected’ titles. I started off using the first approach, but as I worked through more and more anomalies, the checking logic became very complicated and inefficient. Just think about the knots you can tie yourself in trying to handle a title where the suburb is wrong and the main road changes names in between intersections.

I refactored the code multiple times, but it’s still pretty messy. In the end I created a list of ‘corrected’ titles as well, so it was a bit of a hybrid approach. I suspect I could have saved myself a lot of pain if I’d reversed the process – compiling ‘corrected’ titles first, then adapting the logic as patterns emerged.

There are still some photos I haven’t located. In some cases I just don’t have enough information. In others I need to manually record coordinates or way ids to feed into the process, and I haven’t worked out the best way to do this yet. You can see the titles that I’ve haven’t geolocated yet in the files: cua-not-found.txt and cua-not-parsed.txt.

In total, 18,603 out of 20,644 photos have been geolocated. That’s over 90%!

Assembling the data

I processed the data in a couple of phases to get it in the shape I wanted.

The first step was to group all the photos by title, so I could link each group to its location. But remember that titles often record which side of the road a photo was taken on. To bring all sides of a road segment together into a single group, I created a key from a normalised/slugified version of the title with the side value removed. I used this key to save information about each side within the same group.

I ended up with a dataset with this sort of structure (a truncated example):

"iffla-street-south-melbourne-coventry-street-normanby-street":
    {
        "title": "Iffla Street, South Melbourne, from Coventry Street to Normanby Street",
        "sides":
        {
            "east side":
            {
                "title": "Iffla Street, South Melbourne, from Coventry Street to Normanby Street - east side.",
                "images":
                [
                    {
                        "ie_id": "IE20321667",
                        "alma_id": 9939649155207636
                    }
                    more photos...
                ]
            },
            "west side":
            {
                "title": "Iffla Street, South Melbourne, from Normanby Street to Coventry Street - west side.",
                "images":
                [
                    {
                        "ie_id": "IE20320072",
                        "alma_id": 9939655629407636
                    },
					more photos...
                ]
            }
        },
        "ways":
        {
            "27631235":
            [
                [
                    144.9503379,
                    -37.835322
                ],
                more points...
            ]
        }
    },

You can see how the sides and matching ways have been brought together under the key value.

This structure was useful for grouping and processing the data, but to create a map interface I needed to bring the geospatial information to the surface. The first version of the interface used one big GeoJSON file in which the features were MultiLineStrings created from the paths of each road segment. The photo data was saved in the properties of each GeoJSON feature.

It sort of worked. The roads with photos were highlighted, and clicking on the roads displayed the photos. It was only when I changed the opacity of the lines that I realised that, in many cases, different road segments were being piled on top of each other. When the lines were opaque these piles were invisible, but add a bit of transparency and you could see that some lines were darker than others. Clicking on the lines only displayed the top layer, so some groups of photos were effectively invisible.

Version one of the interface showing how the colour of the highlighted roads varied once I decreased the opacity.

Why did this happen? I’d wrongly assumed that each segment of road would only have one group of photos associated with it. But it’s not hard to find cases where this is not true. Consider Moor Street, Fitzroy, between Nicholson Street and Brunswick Street. On the north side, there is a single group of photos that document the buildings between Nicholson Street and Brunswick Street. However, on the south side there’s two groups of photos. One covers the section between Nicholson Street and Fitzroy Street, the other covers Fitzroy Street to Brunswick Street. One section of road, three groups of photos…

Moor Street, Fitzroy, between Nicholson Street and Brunswick Street, in the new CUA Browser, showing the three photosets associated with the one section of road.

To make these layered groups more easily accessible through the interface I had to change the way the data was organised – separating the GeoJSON from the photosets so that multiple photosets could be associated with a single geospatial feature. I decided to create a GeoJSON feature for every OSM way in the dataset. However, I needed to prune the way’s coordinates to only include those that were part of the CUA road segments. To do this, I saved all the way data when I found the road segments. Then in the second processing phase, I grouped the way coordinates associated with the road segments and compared this list to the full way path. Any coordinate in the way path that wasn’t in the road segments was removed. It seems unnecessarily complex, but I wanted to make sure that only the parts of roads associated with photos were highlighted in the interface.

The result was two data files. The first, cua-ways.geojson, contains the pruned way paths and their OSM identifiers. The second, cua-photos.json, contains information about each photo set, including the sides, photos, paths, and associated way identifiers. The datasets are linked by the way identifiers.

Constructing the interface

My plan for the interface was pretty simple. There’d be a map on which all the road segments associated with CUA photos were highlighted. Clicking on a highlighted section would show the photos. I wanted to display the photos as if you were scanning the streetscape, so I decided to put them all side-by-side in a gallery that scrolled horizontally.

The first version used Leaflet to display the maps and, as noted above, had some problems where there were multiple photosets associated with a segment of road.

For the second version I decided to switch to MapLibre because it seems a bit more active and up-to-date. I’d already used MapLibre in the SLV Newspapers Explorer.

The interface first loads the cua-ways.geojson file to highlight the relevant roads. When you click on one of the roads, the way id is passed to a function that looks for associated photo sets in the cua-photos.json data. If there’s only one linked photoset, then the photos are displayed. However, if there’s more than one linked photoset, they’re displayed as a list. The user then selects from the list to display the related photos.

A couple of other things happen when you click on a way or select a photoset: the colour of the selected road segment changes, and the browser url is updated with the way or photoset identifier. You can bookmark or share these urls to go directly to a specific road or photoset. There’s also a button to reverse the order of the images – they scroll left to right, but sometimes they seem to have been photographed right to left.

Goodbye 2025! A brief summary of the highlights and lowlights…

Wed, 31 Dec 2025 17:14:14 +1100

My 2025 started badly and ended well. In the first few months of the year, battles with the gatekeepers at Trove sent me spiralling into a pretty dark place. But by year’s end I was having fun, working with the wonderful people at the State Library of Victoria. In between I caught up on some overdue project maintenance. Most of this is documented in the 37 blog posts I wrote this year, but here’s a quick summary.

Highlights

From September to December, I was Creative Technologist-in-Residence at the SLV LAB, exploring ways of opening up the Library’s place-based collections. There’s still a few things to finish off, but here’s list of the results so far.
As part of my SLV work, I created a fully searchable version of the 24 volumes of Sands & MacDougall directories digitised by the Library. This followed the pattern I’d used for 54 volumes of the NSW Post Office Directories, 44 volumes of the Sydney Telephone Directories, and 54 volumes of the Tasmanian Post Office Directories. So there’s now 176 volumes from the 1880s to the 1950s that can be easily explored for people and places— and all free to use of course.
In April, I added a new section to the GLAM Workbench documenting the Public Record Office Victoria’s collection API. I also used the API to create a Data Dashboard that provides an overview of PROV’s collection.
Also in April, I updated the GLAM Name Index Search to include an additional 6 million records from PROV. In total, the GLAM Name Index now includes more that 12 million records in 293 datasets from 10 Australian GLAM organisations — another free resource for Australian researchers.
In July I undertook some overdue maintenance on a variety of old apps and projects. In the process, I resurrected my old Wragge Labs domain and created a showcase of many of the websites, apps and experiments I’ve worked on over the past 30 years.
I was particularly pleased to get The Future of the Past working again, so once more you can create fridge magnet poetry from an odd collection of words harvested from Trove newspapers! I built FOTP back in 2012 when I was the Harold White Fellow at the NLA. Also this year I finally got around to transcribing my Harold White Lecture.
In June I wrote a short piece on the GLAM Workbench for the forthcoming publication Building User-Friendly Toolkits and Platforms for Digital Humanities. I think it provides a useful summary of what the GLAM Workbench is, and what I’d like it to be. I also wrote up the short but glorious history of Trove Twitter bots.

Lowlights

Saying goodbye to 15 years of work on Trove. It still hurts. And I still miss resources such as @TroveNewsBot and the Trove API Console which ran happily for more than a decade before being killed without warning by the NLA.
Saying goodbye to 17 years of work on the National Archives of Australia’s collections. This will be the first New Year’s Day in a decade when I haven’t updated my harvest of files with the access status of closed.

Next year

In 2026, I’m looking forward to starting work on the RAPID project, building on the work I’ve done on Commonwealth Hansard over the years to create new examples and documentation.
I’m honoured to be giving the closing keynote at the GLAM Labs Futures conference in Edinburgh in June — hoping we can pull together the funds to get there in person!

How you can help

Much of my work is unfunded, and keeping resources such as the GLAM Name Index running costs real money. I’ve been very grateful for the support of my GitHub sponsors over past years. Their contributions help cover a substantial proportion of my cloud hosting costs. But bidding farewell to Twitter and Trove has had an impact on my sponsorship income. If you use or value the things I build to help researchers make use of GLAM collections, you might like to sponsor me on GitHub, or Buy Me a Coffee. All contributions are greatly appreciated!

If you can’t afford a financial contribution, there are other ways you can help!

Let me know how you’re using my stuff! A bit of positive feedback does wonders when my enthusiasm is flagging. You can find my contact details at timsherratt.au.
Tell others how you use my stuff! Getting information about resources out to those who might benefit is really hard, so your help would be greatly appreciated.
The GLAM Workbench describes a few other ways you can get involved.

Goodbye 2025!

Exploring Victorian newspapers

Tue, 16 Dec 2025 13:03:48 +1100

Newspapers are a vital source for local history. That’s why, back in 2014, I created the Trove Places app – a map interface to help people find Trove’s digitised newspapers by their place of publication or distribution. Trove Places has proved very popular, and the State Libraries of South Australia, and Victoria, amongst others, point their users to it to help with their research. I’ve updated the data several times over the years, though the Trove’s new gatekeeping regime will make future updates difficult.

During my residency at the State Library of Victoria, one of the librarians noted how useful the app was, and asked whether it might be possible to include undigitised newspapers from the SLV catalogue as well as those in Trove. It was, and I did – here’s a brand new app to explore Victorian newspapers, both digitised and undigitised!

Just click on the map to find Victorian newspapers!

It’s pretty easy to use. You just click on the map in an area you’re interested in. The map will display the 20 nearest places where newspapers where published or distributed. The size of the markers indicates how many titles are associated with each place. In the sidebar, details of the newspapers are listed by place, ordered by their distance from your selected point.

You can also find local newspapers using the my place app. Once you enter an address, newspapers from your suburb or town will be displayed, as well as those from nearby locations.

newspapers from Geelong displayed in the my place app

Assembling the data

How do you find Victorian newspapers? The reference librarians at the SLV pointed me to the ‘Place newspaper published’ field in the catalogue. Searching this field for ‘Australia–Victoria’ returns 3,997 results, compared to the 460 digitised in Trove.

The first step in assembling the data was to harvest the newspaper records from the SLV catalogue. To do this I made use of the Primo JSON API. The method is documented in this notebook. The results was a newline-delimited JSON file, with one record per line.

The harvested metadata doesn’t include links to digitised versions of newspapers in Trove. To add these links I first looked in the 856 field of the newspaper’s MARC record. I also noticed that some Trove links were being loaded from an ‘edelivery’ JSON file, so I added these as well. I ended up with 344 unique links to Trove, but not all of these were to digitised newspapers as some more recent newspapers are available through eLegal deposit. In total there were 268 unique links to digitised newspapers. This is well short of the 460 Victorian newspapers in Trove. Why? It’s possible that the links haven’t been added into the SLV catalogue, or that the ‘place newspaper published’ field hasn’t been populated for records that include the links. It’s also possible that Trove links are hiding somewhere else in the SLV catalogue!

To try and fill this gap, I compared the catalogue metadata with my most recent harvest of Trove newspaper titles. If the Trove url was missing, I searched the catalogue data for the newspaper title. I then manually checked the results, making sure the dates and titles lined up, and added positive matches to a new CSV file which I merged back into the main dataset. This added another 152 Trove links.

The next step was to link the ‘place newspaper published’ values to places with known locations. The ‘place newspaper published’ information is included in the lds03 field of the harvested metadata. Records often contain references to multiple places, so I split all the newspaper/place combinations out into separate rows. I then matched the places against a list of Victorian place names and coordinates downloaded from the VicNames database. If there were no matches, I manually checked and adjusted the place names – for example, I changed ‘East Kew’ to ‘Kew East’, and ‘Bayside’ to ‘Bayside City’.

To add any Trove digitised newspapers that might still be missing, I made use of my existing Trove harvests. First I compared my Trove Places dataset with my latest harvest of newspaper titles. There were a few new titles, so I matched them to places using this notebook, based on my original Trove Places code. I then merged the Trove Places dataset with the new titles and checked it against the catalogue dataset. If any urls were missing, I added a record from the Trove data.

All of the processing steps are documented in this notebook.

Building the apps

To make the data easily searchable by its geospatial coordinates, I loaded all the data into an SQLite/Spatialite database and published it online using Datasette. The database contains linked tables for titles and places.

I also created a couple of canned queries which, together with Datasette’s built-in JSON API, made it possible to retrieve places and titles based on their distance from a given point. For example, this url retrieves places ordered by their distance from the point at latitude -36.815, longitude 144.965 : https://slv-places-481615284700.australia-southeast1.run.app/newspapers/places_from_point.json?longitude=144.965&latitude=-36.815&distance=100000&_shape=array

When you click on the map in the Victorian Newspapers Explorer, it fires off a request like this to find nearby places. It then makes a second request to find newspapers related to those places and displays the results.

The Victorian Newspapers Explorer was my first attempt at using MapLibre rather than Leaflet to display maps using Javascript. It’s more verbose, but more flexible, so I think I’ll gradually switch over my other apps, including Trove Places.

All the code of the Victorian Newspapers Explorer is in the slv-demo-apps repository.

Why bother?

Wed, 03 Dec 2025 16:18:22 +1100

This was the introduction to my talk on the results of my time as Creative Technologist-in-Residence at the State Library of Victoria. My slides, with my full notes are available online, but after a very strange year that has travelled from disappointment to exhilaration, I thought it was worth posting these words separately.

The work that I do, that I’ve been doing for the past 30 years, is focused on helping people find, use, and understand the wonderfully rich collections held by our libraries, archives, and museums – the GLAM sector. Much of it is quite practical, resulting in tools and applications that are used by a wide range of researchers. Some of it is playful, some of it is critical, and some of it is just weird.

You can browse through some of this history on wraggelabs.com. And if you’re interested in my current crop of tools you can head to the GLAM Workbench.

As some of you may know, I had a few setbacks at the beginning of this year which really made me wonder whether I wanted to continue doing this sort of work.

I mean, why bother?

I’m really grateful that this residency has given me a chance to refocus on the reasons why I do what I do.

I suppose my starting point is the fact that libraries can’t do everything themselves.

I’m thinking here specifically about the digital research space. There’s a lot that libraries, and other GLAM organisations, can do – provide search interfaces, APIs, downloadable datasets, documentation, and examples of how to access APIs and datasets using code. The sorts of things that the SLV LAB is doing.

I should pause here to unpack some acronyms. APIs deliver data in a form that machines can understand and process. Websites are for humans, APIs are for computers. APIs are also building blocks which can be connected up to create new applications – and I’ll be showing some examples of this later on.

So there is much that GLAM organisations can do to support digital research. But it will never be enough. Researchers – whether they be academics or family historians – will always want more. It is in the nature of research to ask new questions, to head off in new directions.

But rather than see this as a source of tension, I see it as an opportunity for collaboration. An opportunity to cultivate the in-between spaces where research methods, tools, and results can feed back into the contextual frameworks of GLAM collections. Where GLAM organisations can share and celebrate the work that’s done with their data. Where all can find inspiration, ideas and support.

We’ve tended to call this sort of stuff infrastructure, but I think that really downplays the human aspect. The research sector has started to develop the funding and career structures necessary to allow people to build and maintain these infrastructures, but we need more. We need to recognise that a single tool, developed by an individual without institutional support, can be just as important as a multi-million dollar platform. Passion is precious and needs to be protected.

Most of all, we need to keep a focus on the ethical imperatives – the reasons why we bother and why it matters. For me it boils down to openness and generosity. I have benefited greatly from the openness and generosity of others, and I want to pass that on. It’s the glue we need to hold those in-between spaces together; the sustenance we need to maintain our enthusiasm in the face of all the crap; the inspiration we need to try something new.

Initiatives like the SLV LAB are important, not just because they foster innovation, but because they invite new ideas in. They even give space for ageing hackers like me to spend some dedicated time doing what they love – crafting new pathways for people to explore our glorious GLAM collections.

Counting down... (to the end of my SLV residency)

Wed, 19 Nov 2025 16:01:22 +1100

My stint as Creative Technologist-in-Residence at the State Library of Victoria LAB comes to an end in a few weeks time and I’m frantically trying to pull things together. I’ll be back on-site at the Library from 1 to 5 December for a few events, and to report back to staff on what I’ve been doing.

On Tuesday 2 December, there’ll be a public workshop on using and contributing to the GLAM Workbench. Here’s the blurb:

More and more GLAM organisations are looking to share their data to foster creativity and support new types of research. But how can you help potential users understand the possibilities of your data? This workshop will explore how GLAM organisations can create and share resources that encourage experimentation.

The GLAM Workbench is a large collection of tools, hacks, and tutorials aimed at helping researchers make use of collection data. It uses platforms such as Jupyter notebooks to create live, working examples that run in your browser without additional software. Similar repositories of computational resources are being developed by GLAM organisations around the world.

This workshop will introduce the technologies and standards used in the GLAM Workbench, such as Jupyter notebooks. It will provide an overview of related activity around the world, including best practice guidelines for GLAM organisations developing computational resources. It will explain how organisations and individuals can contribute content to the GLAM Workbench, or use it as a model to create their own specialised workbenches.

Sharing data is important, but so is sharing skills, tools, and knowledge. Come along to find out how the GLAM Workbench can help.

It’s a free, hybrid event (in person and online) and will run from 1.00-3.00pm. A sign up page should be available soon.

On Wednesday 3 December I’m presenting the results of my residency in a ‘technologist’s talk’. It’s an internal event, but it’s in the public ‘Create quarter’ of the Library, so I think anyone can pop in. Hopefully there’ll be a video I can share.

To give you an idea of what I’ll be talking about, here’s some of the outcomes so far:

hacking the library workshop (slides, and blog post about urls)
bounding boxes for parish maps (blog post, code)
geolocating the Committee for Urban Action collection of photographs (prototype interface, still documenting the method)
a new fully-searchable version of the Sands & MacDougall’s directories (blog post, database, and another blog post)
georeferencing digitised maps – over 500 so far! (documentation, dashboard, data repository, blog post)
and as of yesterday, 3,000+ geolocated newspapers (documentation and interface coming!)

First attempt at mapping places of publication and distribution of Victorian newspapers

At the moment I’m trying to bring it all together in a new interface that let’s you type in an address and find collection materials relating to your home, your street, and your suburb. Only two weeks to go! Eeek!

Work in progress!

Some Sands & Mac tweaks thanks to ALTO and IIIF

Sun, 16 Nov 2025 15:17:26 +1100

I posted recently about my new fully-searchable version of the Sands & MacDougall directories. I’ve now moved on to try and pull together a number of the State Library of Victoria’s place-based collections into a new discovery interface. It’s going to be a busy couple of weeks as my residency ends in early December!

I wanted to incorporate Sands & Mac search results into the new interface. Getting the data was easy because Datasette has a JSON API baked in. But what about the images? I could just display a thumbnail of the whole page, but it would be better to show a snippet of the actual entry. Thanks to IIIF and ALTO, I now can.

IIIF makes it easy to cut small sections out of a larger image. You just put the coordinates of the desired section in the IIIF url. As I noted in my previous post, the ALTO files that contain the OCR data from Sands & Mac include the coordinates of every line, and every word. I just had to bring the two together.

All I did was update the code that extracts the data from the ALTO files to save the results as newline delimited JSON instead of a plain text file. Each line in each volume of Sands & Mac is now saved a JSON object that contains the text, as well as the height, width, vertical position, and horizontal position of the line within the page image. When I load up the SQLite database, I add the values for h, w, x, and y as well as the text for each line.

What does this make possible?

When you go to an individual entry, the page image now automatically pans and zooms so that the current entry is at the centre of the image viewer. I just updated the OpenSeadragon code to focus on the entry’s position.
If you share an entry on social media, a snipped out section of the page image showing the selected entry is displayed as there’s now an image META tag that points to an IIIF url.
You can retrieve entries via the API and use the coordinates to request snipped out images of them via IIIF.

Nice image snippets thanks to IIIF and ALTO (and a sneak prview of what's coming...)

A new way of searching Sands & Mac

Wed, 12 Nov 2025 22:25:21 +1100

In the fortnight I spent onsite at the State Library of Victoria, ‘Sands & Mac’ was mentioned many times. And no wonder. The Sands & McDougall’s directories are a goldmine for anyone researching family, local, or social history. They list thousands of names and addresses, enabling you to find individuals, and explore changing land use over time. When people ask the SLV’s librarians, ‘What can you tell me about the history of my house?’, Sands & Mac is one of the first resources consulted.

The SLV has digitised 24 volumes of Sands & Mac, one every five years from 1860 to 1974. You can browse the contents of each volume in the SLV image viewer, using the partial contents listing to help you find your way to sections of interest. To search the full text content you need to use the PDF version, either in the built-in viewer, or by downloading the PDF. There’s a handy guide to using Sands & Mac that explains the options.

However, there’s currently no way of searching across all 24 volumes, so as part of my residency at the SLV LAB, I thought I’d make one!

Try it now!

My new Sands & Mac database follows the pattern I’ve used previously to create fully-searchable versions of the NSW Post Office directories, Sydney telephone directories, and Tasmanian Post Office directories. Every line of text is saved to a database, so a single query searches for entries across all volumes. You can also use advanced search features like wildcards and boolean operators.

Search across all 24 volumes!

Once you’ve found a relevant entry you can view it in context, alongside a zoomable image of the page. You can even use Zotero to save individual entries to your own research database. This blog post from the Everyday Heritage project describes how the Tasmanian directories have been used to map Tasmania’s Chinese population.

View each entry in context! (Here's my Dad building his first house in Beaumaris in the 1950s.)

There’s still a few things I’d like to try, such as making use of the table of contents information for each volume. I’d also like to create some additional entry points to take users directly to listings for individual suburbs (maybe even streets!). Each volume has a directory of suburbs, so it would be a matter of extracting and cleaning the data and linking the entries to digitised pages. Certainly possible, but I don’t think I’ll have time to get it all done before the end of my residency. Perhaps I’ll try to get at least one volume done to demonstrate how it might work, and the value it would add. As I was writing this blog post I also realised there’s a dataset of businesses extracted from the Sands & Mac, so I need to think about how I can use that as well!

Technical information follows…

I’ve documented the process I used to create fully-searchable versions of the Tasmanian and NSW directories in the GLAM Workbench. I followed a similar method for Sands and Mac, though with a few dead-ends and discoveries along the way.

Downloading the PDFs

I assumed that it would be easiest to work from the PDF versions of each volume, as I’d done for Tasmania. So I set about finding a way to download them all. There’s only 24 volumes, so I could have downloaded them manually, but where’s the fun in that?

I started with a CSV file listing the Sands & Mac volumes that I downloaded from the catalogue. This gave me the Alma identifiers for each volume. To download the PDFs I needed two more identifiers, the IE identifier assigned to each digitised item, and a file identifier that points to the PDF version of the item. The IE identifier can be extracted from the item’s MARC record, as I described in my post on exploring urls. The PDF file identifier was a bit more difficult to track down. The PDF links in the image viewer are generated dynamically, so the data had to be coming from somewhere. Eventually I found that the viewer loaded a JSON file with all sorts of useful metadata in it!

The url to download the JSON file is: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. In the summary section I found identifiers for small_pdf and master_pdf.

I could then use these identifiers to construct urls to download the PDFs themselves: https://rosetta.slv.vic.gov.au/delivery/DeliveryManagerServlet?dps_func=stream&dps_pid=[PDF id]

Once I had the PDFs I used PyMuPDF to extract all the text and images. As I suspected the text wasn’t really fit for purpose. The OCR was ok, but the column structures were a mess. Because I wanted to index each entry individually, it was important to try and get the columns represented as accurately as possible. The images in the small PDFs were already bitonal, so I started feeding them to Tesseract to see if I could get better results. After a bit of tweaking, things were looking pretty good. But when I came to compile all the data, I realised there was a potential problem matching the PDF pages to the images available through IIIF. I found one case where some pages were missing from the PDF, and another couple where the page order was different.

As I was looking around for a solution, I realised that those JSON files I downloaded to get the PDF identifiers also included links to ALTO XML files that contain all the original OCR data (before it got mangled by the PDF formatting). There was one ALTO file for every page. Even better, the JSON linked the identifiers for the text and the image together – no more page mismatches!

Downloading the ALTO files

Let’s start this again shall we. After wasting several days futzing about with the PDFs, I decided to download all the ALTO files and extract the text from them. As I downloaded each XML file, I also grabbed the corresponding image identifier from the JSON and included both identifiers in the file name for safe keeping.

The ALTO files break the text down by block, line, and word. To extract the text, I just looped through every line, joining the words back together as a string, and writing the result to a new text file – one for each page.

It’s worth noting that the ALTO files include all the positional data generated by the OCR process, so you have the size and position of every word on every page. I just pulled out the text, but there are many more interesting things you could do…

Assembling and publishing the database

From here on everything pretty much followed the pattern of the NSW and Tasmanian directories. I looped through each volume, page, and line of text, adding the text and metadata to a SQLite database using sqlite_utils. I then indexed the text for full-text searching. At the same time I populated a metadata file with titles, urls, and few configuration details. The metadata file is used by Datasette to fill in parts of the interface.

I made some minor changes to the Datasette template I used for the other directories. In particular, I had to update the urls that loaded the IIIF images into the OpenSeadragon viewer. But it mostly just worked. It’s so nice to be able to reuse existing patterns!

Finally, I used Datasette’s publish command to push everything to Google Cloudrun. The final database contains details of more than 50,000 pages, and over 19 million lines of text! It weighs in at about 1.7gb. The Cloudrun service will ‘scale to zero’ when not in use. This saves some money and resources, but means it can take a little while to spin up. Once it’s loaded, it’s very fast. My original post on the Tasmanian directories included a little note on costs, if you’re interested.

More information

The notebooks I used are on GitHub:

Here are some posts about the NSW and Tasmanian directories:

Making NSW Postal Directories (and other digitised directories) easier to search with the GLAM Workbench and Datasette (September 2022)
From 48 PDFs to one searchable database – opening up the Tasmanian Post Office Directories with the GLAM Workbench (September 2022)
Where’s 1920? Missing volume added to Tasmanian Post Office Directories! (September 2024)
Six more volumes added to the searchable database of Tasmanian Post Office Directories! (November 2024)

Turning the SLV's maps into data with Allmaps and some GLAM plumbing

Tue, 04 Nov 2025 15:02:53 +1100

I often describe what I do as GLAM data plumbing. Most of the time I’m not creating new tools, I’m figuring out what data is available and how I can connect it up to existing tools. It’s rarely straightforward, but if I can get all the pipes connected and data flowing in the right direction, suddenly new things become possible.

Things like turning all the State Library of Victoria’s digitised maps into data.

I’ve just created a workflow that uses AllMaps and IIIF to georeference the SLV’s digitised maps. There’s some technical details below, but the idea is pretty simple. A userscript links the SLV image viewer to Allmaps – so you just click on a button, and the digitised map opens, ready for georeferencing.

Why is this useful? Georeferencing relates a digitised map to real world geography. It describes the map’s position and extent using geospatial coordinates – turning historic documents into geospatial data that can be indexed, visualised and manipulated. Georeferencing opens digitised maps to new research uses.

So, how many maps we can georeference before my residency finishes in December? Hundreds? Thousands? If you like maps and want to help, head to the documentation page to find out how to get started. And if you want to see how things are progressing, have a look at the project dashboard.

View the documentation to get started

A few technical details follow…

Early on in my time as Creative Technologist-in-Residence at the State Library of Victoria, I started playing around with Allmaps for georeferencing digitised maps. It’s a great tool (really a suite of tools and standards) because instead of constructing a whole new platform it integrates with existing IIIF services. The SLV provides digitised images through IIIF, so I thought it should be possible to use Allmaps to georeference the SLV’s map collection.

But I struck a problem that took some time to unravel. The IIIF urls in the SLV manifests include port numbers and that confused Allmaps. The manifests also sometimes contained references to image formats that weren’t actually accessible, generating errors when they were loaded. Hopefully these problems will be fixed by the SLV, but in the meantime I’ve created a proxy service that edits the manifest on the fly. The proxied urls can be loaded into the Allmaps Editor without errors. Pipes fixed, data flowing!

Using the manifest proxy

To generate a link to a proxied manifest, first grab the item's IE identifier from the url of the digitised item viewer. For example, the identifier in this url https://viewer.slv.vic.gov.au/?entity=IE15485265&mode=browse is IE15485265. Once you have the identifier, add it to the end of the url https://wraggelabs.com/slv_iiif/. For example, https://wraggelabs.com/slv_iiif/IE15485265. You can then supply this url to the Allmaps editor.

But having to fiddle around with proxies didn’t make a great user experience. I needed some way of integrating the two services, so that a user could just click a button in the SLV website and start editing in Allmaps. Userscripts to the rescue!

I wrote recently about hacking GLAM collection interfaces using userscripts. Since I started my residency at the SLV, I’ve also created a userscript to display the IIIF manifest url in the SLV image viewer, and run a Code Club workshop where we played around with an assortment of SLV website hacks.

As in a number of these examples, the georeferencing userscript adds new features to the SLV website, but there’s a fair bit more going on under the hood. It runs automatically every time you load the SLV image viewer, and then:

it checks the metadata of the digitised item to see it it’s a map (or something that contains maps, like an atlas or street directory)
if it looks like a map, it generates an Allmaps identifier using the item’s IIIF manifest url and checks with Allmaps to see whether the item has already been georeferenced
it adds a ‘Georeferencing’ section to the page, with a button to georeference the item (or edit the existing georeferencing)
if the item has already been georeferenced, it adds a button to view the item in the Allmaps Viewer, and embeds a live preview

Accessing metadata

The userscript gets the item metadata from a JSON file that's loaded by the image viewer. The JSON file includes a lot of extra, useful information about the digitised item. To access the JSON file, you just construct a url like this: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. The IE identifier is in the url of the image viewer.

Allmaps identifiers

Allmaps creates its identifiers by hash encoding the IIIF urls. The userscript borrows some code from the Allmaps id module to generate the ids, then sends a HEAD request to the Allmaps API to see whether an entry for the current manifest exists.

Example of an item that hasn't been georeferenced yet

Example of an item that has been georeferenced, displaying an embedded version of the Allmaps viewer

I’ve also created a GitHub repository to save copies of the data. Every two hours this notebook is run to query the Allmaps API for newly georeferenced maps. These are added to a dataset which is saved in three formats:

a CSV file
a CSV file that includes thumbnails and links for viewing in Datasette-Lite
a GeoJSON file, that can be viewed in services like geojson.io

At the same time, the data for each individual map is downloaded and saved as IIIF annotations (in JSON) and GeoJSON.

Finally, this notebook is run to generate a dashboard that provides an overview of the project’s progress.

The project dashboard is updated every two hours

One of the Allmaps developers described all my plumbing and workarounds as a ‘very cool lofi example of how you can set this up with little means’, and I think that’s pretty apt. It’s really just an experiment to demonstrate the possibilities, but by connecting up existing services it’s generating real data of long term value.

Me at 63...

Mon, 03 Nov 2025 17:55:17 +1100

Originally published in Pharos, newsletter of the Professional Historian’s Association (Vic & Tas), October-November 2025, in the ‘Member Profile’ section.

In the early 1990s I started working for a small self-funded organisation called the Australian Science Archives Project. Our mission was to preserve and raise awareness of Australia’s scientific past. When the web came along, we realised it provided an enormous opportunity to communicate history to the public. So I taught myself web development and created the first archives website in Australia. Since then my work has continued to explore what happens when we release GLAM collections into online spaces where people can see and use them differently.

What kind of work have you done? What are you working on now?

I’ve had a range of jobs in the GLAM and university sectors. While ‘history’ wasn’t often in my job title, I’ve always regarded myself as a historian first – whether I was coding, editing, writing, teaching, or managing, history was always the frame through which I understood my work. At the same time, I’ve maintained my own independent practice as a ‘historian and hacker’, developing tools and resources for other researchers, such as the GLAM Workbench. Much of this work is unfunded, but by sharing it openly I’ve created new opportunities for collaboration. For example, I’m currently the ‘Creative Technologist-in-Residence’ at the State Library of Victoria, bringing my years of GLAM hacking to bear on the Library’s place based collections.

Research or writing? (What do you enjoy more and why?)

Researching, or writing, or coding, or teaching, or outreaching (what is the correct verb?) – all have their joys and travails. For me, research is less about finding things in archives and libraries, and more about how we find things in archives and libraries. I poke about in online collections to try and understand how they work, what they reveal, and what they hide. This often leads to the development of new tools, the writing of documentation and blog posts, and sometimes even real, published articles. It’s a process that has consumed my life, for better or worse. Coding often slips into obsession when I have a gnarly problem to crack. Writing is a slog, but there’s nothing like the pleasure of a finely-turned sentence. Teaching is exhausting, but also exhilarating when you see the light bulb of understanding flick on.

What are the best and hardest things about the kind of work you do?

The best thing, the absolute hands-down best thing, is hearing from people who use, or have benefited from the tools and resources that I’ve created. I make things to help researchers see and use GLAM collections in new ways, so finding out what they’ve been doing with my stuff always provides a much-needed jolt of inspiration.

However, the flip side is that getting information about my tools and resources out to the people who might benefit most is hard and often frustrating work. I churn away in the social media mines, but people and organisations seem much more reluctant to share new work these days. There was a time (yeah, the good old days) when GLAM organisations actively engaged with researchers online, sharing the cool things people were doing with their collections. But not now. We all learn through the generosity of others, and I think its important that we find ways to support and enlarge the realm of generosity.

Creating bounding boxes for parish maps in the SLV collection

Mon, 06 Oct 2025 15:17:51 +1100

The State Library of Victoria holds a collection of 8,804 parish maps. As part of my residency at the SLV LAB, I’ve been poking around in the metadata.

SLV staff have geocoded many of the parish maps using the Composite Gazetteer of Australia, which provides coordinates for Victorian parishes and boroughs. These coordinates give us a point which should be roughly at the centre of each map, enabling us to visualise their locations and distribution. But how much area do they cover? To answer that question we need a bounding box that includes the coordinates of each corner of the map. We could create bounding boxes by using something like AllMaps or MapWarper to georeference each individual map, but that’s going to take a while! As a quick and dirty alternative, I wondered if it was possible to generate approximate bounding boxes from the available metadata. It seems we can!

The metadata

There are three pieces of metadata we need to construct bounding boxes:

the latitude and longitude of the centre point
the size of the physical map
the scale of the map (ie how the size of the map relates to real world)

The coordinates and scale can be included in a couple of different places in the map’s MARC record. The 034 field is specifically for ‘Coded Cartographic Mathematical Data’. The relevant subfields are:

$a: category of scale
$b: constant ratio linear horizontal scale (this is the most likely type of scale)
$d: westernmost longitude
$e: easternmost longitude
$f: northernmost latitude
$g: southernmost latitude

If the coordinates describe a point rather than a bounding box, then $d and $e will be the same, and $f and $g will be the same.

String representations of coordinates and scale can be found in the 255 field. The relevant subfields are:

$a: statement of scale, eg Scale [ca. 1:90,000].
$c: statement of coordinates, eg (E 142°18'/S 37°33')

The size of the map is recorded in 300 (physical description) field under the $c (dimensions) subfield. For example: on sheet 40 x 51 cm .

The method

I started with an existing dataset downloaded from the catalogue by SLV staff. This dataset included the scale and coordinate information in the 034 field, and the coordinate string in 255$c. At first I didn’t realise that the 034 held geo data, so I separately downloaded the scale information from 255:$a in each item’s MARC record (d’oh). If the maps were digitised, I also wanted their image identifiers so I could access them through the SLV’s IIIF service. The image id from the 956$e field of the MARC record can be used to construct an IIIF manifest url, so I extracted them as well.

Once I had all the catalogue data, I had to make sure everything was in a format I could work with. The coordinates in the MARC records are recorded as degrees/minutes/seconds, so I had to convert them to decimal values. The scale factor needed to be an integer, and I needed to extract the height and width as integers from the dimensions field.

I used lat_lon_parser to convert the coordinates to decimal, but needed a bit of regex string manipulation to get the values into a format that could be parsed. Regex also came to the rescue in getting the map dimensions. All the details are in this notebook.

Creating bounding boxes

After some searching I found this StackOverflow comment that described how to create a bounding box from a point, distance, and bearing. The point I already had, but the distance and bearing had to be calculated. Trigonometry to the rescue!

The distance from the point at the centre of the box to one of its corners is the hypotenuse of a right-angled triangle whose sides are equal to half the width and half the height of the map, and thanks to Pythagorus we know:

Once I had the distance in cm, I converted to inches, then multiplied by the scale factor, and finally converted the inches to miles. (It now occurs to me that there’s no need to convert to imperial measurements, but it doesn’t make any difference either way.)

The bearing that points to the corner of the box is the angle inside the same right-angled triangle, so can be calculated using:

With the point of origin, distance, and bearing I could use geopy to calculate the corners of the bounding box!

from geopy.distance import geodesic

destination = geodesic(miles=distance).destination(origin, bearing)
coords = destination.longitude, destination.latitude

See this notebook for the full details.

Limitations

Of course, this method is very rough and has a number of major limitations, in particular:

only about 38% of the maps have point coordinates
the point values don’t necessarily locate the centre of the map
not all the maps are oriented towards north
sometimes a parish includes multiple maps
the size of the margin around the map will affect the accuracy of the bounding box

But despite these problems the results seem pretty good. To test this I created a notebook to overlay the digitised maps on a modern basemap using the bounding boxes. Here’s an example.

You can see the map is slightly offset (presumably due to the second problem listed above). But the size seems about right. Certainly good enough to use the bounding boxes in some exploratory analyses!

Visualising the results

I’ve saved the processed data as a new dataset, and started playing around with a couple of ways of visualising the results. These are experiments, not discovery interfaces. But you can use them for a bit of exploration if you don’t mind a few bugs. They’re all in Jupyter notebooks that can be run using the Binder service.

The parish maps browser includes a dropdown list of parish maps with point coordinates. Select a map and:

if there’s a bounding box and an image identifier, the image of the parish map will be overlaid on the modern base map using the bounding box coordinates
if there’s a bounding box, but no image identifier, a rectangle will be drawn on the base map showing the dimensions of the bounding box
if there are point coordinates, but no bounding box, a marker will be placed on the base map

If the image of the map is displayed you can use the slider to adjust the opacity. Clicking on either the image, rectangle, or marker will display metadata about the parish map and a link to the SLV catalogue.

There’s also a visualisation of all the bounding boxes overlaid on a modern base map.

As you move your mouse over the bounding boxes the titles are displayed on the map, and if you click on a bounding box the metadata is displayed beneath the map, including a link to the SLV catalogue.

It’s obvious from the image above that some of the coordinates must be wrong! Visualisation is a great way of finding problems with your data. I now need to work through the results, documenting the problems, and thinking about how to make best use of the data. More to come!

Exploring SLV urls

Tue, 23 Sep 2025 17:22:45 +1100

I like urls. They take you places. And if you know how to read them, they can tell you things about the systems that created them. One of the first things I did when I started my residency at SLV LAB, was to try and understand how their collection urls work. There’s a couple of well-worn methods I use when digging into a new site.

The first is url hacking – this involves fiddling around with the parameters in a url and submitting the result to see what happens. The Trove Data Guide includes some examples of hacking Trove urls to change the delivery of search results.

The second method involves opening up the developer console in your web browser and watching the activity in the network tab as you click on links. This tells you where the information that gets loaded into your browser actually comes from – sometimes exposing handy urls that you can use to shortcut access to useful data.

Permalinks

The SLV uses Primo for its public-facing catalogue, as well as other systems such as Rosetta and IIIF to deliver digitised content. I’d noticed that Zotero gets some useful data from the catalogue using the default ‘Primo 2018’ translator, however, important things like the item url aren’t captured. The problem is that Primo’s ‘permalinks’ are generated as required by a browser click – they’re not embedded anywhere on the page. This makes it hard to Zotero to grab them. So I started wondering how Zotero could construct short, persistent(ish) links to items.

Here’s a link to an item in Primo: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636

It looks pretty long and messy, but if you start deleting parameters and resubmitting, you’ll find that only two parameters are essential, vid and docid. This means we can rewrite the url as: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&docid=alma9941325055707636 Much nicer.

The ‘permalink’ for the same item is: https://find.slv.vic.gov.au/permalink/61SLV_INST/1sev8ar/alma9941325055707636 If you look closely at the url path and compare it to the example above you’ll see the path is constructed from /vid/[some other id]/docid. One of the librarians explained to me that the other identifier in the permalink is an encoding of the view type, but given that the ‘fulldisplay’ view is the default, we don’t really need it. So the shortened url seems fine for use in Zotero and is easy to generate from the current url. Nice.

It’s also worth noting that the vid value doesn’t seem to change, so to construct catalogue urls in your code, all you really need is the ALMA identifier that’s in the docid parameter.

Structured data

Item pages in Primo include a link labelled ‘Display source record’. If you click on this you’re taken to a representation of the item’s metadata in MARC. Here’s what the urls look like: https://find.slv.vic.gov.au/discovery/sourceRecord?vid=61SLV_INST%3ASLV&docId=alma9941325055707636&recordOwner=61SLV_INST Notice that the ‘fulldisplay’ in the url path above has changed to ‘sourceRecord’. There’s also a new recordOwner parameter, but it seems you can delete this and still get the same result.

Having access to the MARC record is handy, because it delivers the metadata in a simple, structured plain text format. But while the ‘source record’ page looks like a plain text file, it’s actually a HTML page that embeds a plain text record. If you open up the network tab of your browser’s developer console and reload the ‘source record’ page, you’ll see a different url is loaded under the hood: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV&recordOwner=61SLV_INST&lang=en See how the url path has changed from /discovery/ to /primaws/rest/pub? This url does deliver a plain text version of the MARC record.

Once you have the plain text version you can parse the contents to extract the structured data. There are tools that can probably do this automatically, but it’s also pretty easy using regular expressions. Here’s an example of some code I used to parse map records.

def get_marc_value(marc, tag, subfield):
    """
    Gets the value of a tag/subfield from a text version of an item's MARC record.
    """
    try:
        tag = re.search(rf"^{tag}\t.+", marc, re.M).group(0)
        subfield = re.search(rf"\${subfield}([^\$]+)", tag).group(1)
    except AttributeError:
        return None
    return subfield.strip(" .,")

You can also access a JSON representation of the record by adding the parameter &showPnx=true to the catalogue url: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636&showPnx=true

Once again, this is a JSON representation embedded in a web page. Using the same developer console trick, you can identify the direct url is: https://find.slv.vic.gov.au/primaws/rest/pub/pnxs/L/alma9941325055707636?vid=61SLV_INST:SLV&lang=en&search_scope=slv_local&showPnx=true&lang=en You should be able to parse the response from this url as JSON and use it in your code. I think the Zotero translator makes use of this pnx data.

If you want to download the MARC or JSON representations in your code, all you really need is the alma identifier. Just use it to construct one of the direct urls, such as this: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV The recordOwner and lang parameters are not needed, and the vid parameter doesn’t change.

Librarians using Primo have documented a number of tricks like this and shared handy bookmarklets to rewrite urls and get catalogue data in different forms.

IIIF and images

SLV delivers digitised images using IIIF. The IIIF manifest urls are not directly exposed through the web interface, but you can construct your own.

IIIF manifest urls look like this: https://rosetta.slv.vic.gov.au/delivery/iiif/presentation/2.1/IE24074939/manifest.json All we need to construct them is the IE identifier, in this case IE24074939. But where do you find this identifier?

If you’re looking at an image in the SLV’s image viewer, the url will be something like this: https://viewer.slv.vic.gov.au/?entity=IE24074939&mode=browse Yep, the IE identifier is right there in the url. Just extract it from the viewer url, and plug it into the manifest url!

If you’re looking at a catalogue record, or starting with one of the alma identifiers, you can get the IE identifier from the 956$e field of the MARC record.

The IIIF manifest will, in turn, provide identifiers for individual images that can be requested using the standard IIIF syntax.

To save myself a bit of fiddling about, I created a userscript that exposes the IIIF manifest url within the image viewer. If you install it you’ll see something like this:

Handles

Links to digitised items sometimes come in the form of ‘handles’: http://handle.slv.vic.gov.au/10381/4338980 These urls are redirected to the image viewer.

If you want to construct one of these handles, the identifier can be found in 956$a field of the MARC record.

From old to new

I was looking at the datasets created about 8 years ago in the SLV open data repository and noticed they included urls from the previous catalogue. Fortunately, the old urls redirect to the new system.

For example, this url: http://search.slv.vic.gov.au/MAIN:Everything:SLV_VOYAGER1842440

Redirects to: https://find.slv.vic.gov.au/discovery/fulldisplay?context=L&vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&docid=alma9918424403607636

If you look closely at the urls you’ll see that the identifier from the old system is embedded in the new identifier: 1842440 is in 9918424403607636 – 99_1842440_3607636. This means if you have a lot of old urls, such as in the open datasets, you can easily rewrite them in your code.

The process of GLAM hacking

No doubt a lot of this is well-known to librarians, and there’s probably many subtleties or complexities that my poking about has missed. But I wanted to document the process as much as the results – to give an idea of what I do when I approach a new GLAM collection online. I suppose this is GLAM hacking 101.

Creative Technologist-in-Residence at the State Library of Victoria!

Tue, 23 Sep 2025 00:14:09 +1100

I’m very excited to be the new Creative Technologist-in-Residence at the SLV LAB. For the next few months I get to play around with metadata and images, think about online access, experiment with different technologies, and build things to help people to explore the State Library’s collections. In other words, I get to be in my happy place!

My group at the recent SLV WikiFest was thinking about ways of helping researchers find resources relating to particular locations – how do I find material about my suburb, or my street? Coincidentally, the main focus of my residency will also be place-based collections, so I get to really think through some of the possibilities. SLV staff have already pointed me to some amazing maps and photographs, such as the Committee for Urban Action collection, the Mahlstedt fire survey maps, the MMBW plans, and the Victorian parish maps.

At the same time, I’ll be using my usual GLAM hacking approach to poke around in the SLV website to try and understand what data is currently available, identify any roadblocks, and document opportunities for computational research.

The results of my residency will be shared on the SLV LAB site, in GitHub, in the SLV section of the GLAM Workbench, and of course here. As usual, I’ll be working in the open, documenting things as I go along, so please join me on the journey!

Although the residency was formally announced today, I’ve actually been working with SLV data for the last couple of weeks and I’ve already got a backlog of stuff I need to blog about. Here’s a taster – what happens when you generate bounding boxes for thousands of parish maps from the available metadata and throw them on a map…?

WikiFest at the State Library of Victoria

Fri, 29 Aug 2025 16:07:26 +1100

This week I was lucky enough to participate in WikiFest at the State Library of Victoria. Organised by the State Library’s new innovation LAB and Wikimedia Australia, Wikifest was a hands-on, participant-led workshop focused on the possibilities of connecting SLV’s collections to (and through!) Wikidata.

The day kicked off with a series of presentations demonstrating possible uses of Wikidata. I talked a bit about some of my recent GLAM/Wikidata experiments. My slides are online and contain plenty of links to code, demonstrations, and documentation. They’re openly-licensed, so feel free to take anything of use.

The rest of the day was spent in groups, working on particular projects and learning more about Wikidata in the process. My group was looking at providing placed-based entry points to SLV collections, and spent a lot of time exploring the representation of Victoria’s Local Government Areas (LGAs) in Wikidata. We realised there was quite a bit of work to do in adding things like dates and boundaries, but we could see some exciting future possibilities. We also made a start, adding an ‘inception’ date for the City of Moe, based on the Victorian Government Gazette, digitised by the SLV.

Bonus userscript

While I was preparing my presentation I was thinking about the the way entries for Australian people in Wikidata are linked to a range of different identifiers, such as DAAO, the Encyclopedia of Australian Science, and the Australian Dictionary of Biography (ADB). Often a single person can have multiple identifiers and this means that those identifiers themselves become connected through that person’s record. You can query Wikidata with one identifier, and get back links to a range of other information sources about that person.

To demonstrate this, I created a simple userscript that adds additional links to biographies in the ADB. The script grabs the ADB identifier from the url, queries Wikidata for additional identifiers, and writes the results into the page’s ‘Life Summary’. Basic, but useful!

For something more advanced, have a look at the Entity Explosion extension for Firefox.

GLAM hacking with userscripts

Thu, 17 Jul 2025 18:21:25 +1100

In teaching and workshops I used to get students to question the idea that websites are ‘published’. They’re not released into the world in a fixed, immutable form – they’re a set of blueprints which only reach their final form in your browser window. This makes it possible to change the way websites look and behave.

Mozilla used to have a nifty educational tool called X-Ray Googles. Using it you could explore the code underlying a web page and do fun things like inserting new text or images. I encouraged students to try hacking ASIO’s home page.

ASIO home page with some added First Dog on the Moon.

There are other ways you can fiddle with websites. For example, most browsers have a developer console that exposes the code and styling of a page. You can use the console to edit HTML elements or toggle styles, but your changes won’t be saved.

One way you can save and share your web site customisations is by creating userscripts. Userscripts are little bits of Javascript code that run in your browser after a web page loads. These scripts can change many aspects of a page – not just how it looks, but also how it works.

Some old userscripts

I’ve been playing around with userscripts for a long time. Back in 2008, I created a userscript that completely overhauled the way that digital files were presented in the National Archives of Australia’s online database, RecordSearch. My userscript added new options for navigating and printing the file, and even made it possible to view the complete file contents on a 3D zoomable wall.

This customised RecordSearch interface was created by a userscript.

The userscripts I’ve created over the years have tended to either be useful little hacks aimed at fixing annoying aspects of GLAM websites, or experiments in thinking about the sort of information that’s presented online by GLAM organisations, and how it might be different.

In the first category are hacks like my RecordSearch show pages userscript. In 2009, I got annoyed that there was no way of knowing how many pages were in a digitised file until you clicked on the link. So I fixed it. With my userscript running, the links to digitised files are rewritten to display the number of pages. I’ve updated the code numerous times over the years, adding new features, and dealing with changes to RecordSearch. The last update was just a few days ago.

In the second category is my userscript that inserts photos from The Real Face of White Australia into RecordSearch. The are many thousands of records in the National Archives of Australia that document the impact of the White Australia Policy on the lives of ordinary people. But it’s often hard to understand this from the file descriptions. The userscript displays portrait images extracted from the files alongside the metadata – it tells you there are people inside.

Amidst my recent self-archiving binge, I realised I’d never updated this userscript to work with the latest data from The Real Face of White Australia, so I spent some time getting it working again. In the process I realised that RecordSearch now included content security policies that made it a bit harder to insert new images. The solution was to use one of the special userscript functions, GM_addElement(), rather than plain old Javascript. But then I discovered that the if the show pages userscript ran after this one, it would trigger the security restrictions nonetheless! To avoid this I made sure that the two userscripts operated on separate elements. So now the show people userscript is working again!

And a new userscript to improve Trove lists

Fixing up the ‘people inside’ code reminded me of how much fun it was playing around with userscripts, so when David Coombe mentioned a problem he had using Trove lists on Mastodon last night, I had to have a go at fixing it.

The problem is that Trove lists display all the tags associated with each individual item. Some items have lots of tags, so this eats up the screen real estate, making it harder to browse the contents of a list. Notes attached to items can be hidden, but not tags. Why not?

My brand new userscript hides tags by default, and adds a new link to toggle their visibility for each individual item. The link also displays the number of tags attached to each item. This gives the user control over which tags are displayed and when.

The new userscript in action – toggle your tags!

The main difficulty in creating this userscript was knowing when the page had actually finished loading. The current version of Trove uses a lot of Javascript to load and manipulate content, so you have to tell the userscript to wait until everything has settled down. Otherwise the script could fire too soon and cause unexpected results. I tried a number of different approaches to handling this problem, but eventually settled on the waitForKeyElements script. (I just realised there’s a more recent version of this script that doesn’t require JQuery, so I might need to investigate this further.)

Another Trove problem fixed!

Using userscripts

In addition to the userscripts mentioned above, I’ve also created one that enables you to browse Trove newspaper pages using the arrows on your keyboard. Left and right arrows go to the next and previous pages, while up and down arrows jump between issues. Searching is great, but sometimes you just want to browse. Install this userscript for that old-time, authentic newspaper reading experience!

But how do you install userscripts? First of all you need a browser extension to manage your userscripts – I use TamperMonkey or ViolentMonkey. Just follow the instructions to add one of them to your browser.

To install one of my userscripts, you need to go to the script (saved as a GitHub Gist) and click on the ‘Raw’ button. Your userscript manager will then ask you if you want to add the userscript. Click install!

Click on the ‘Raw’ button to install.

Once they’re installed the userscripts will run automatically when specified pages are loaded. If you ever want to disable them, you can do that from your userscript manager’s dashboard.

For convenience, here are the Gist links to all the userscripts I’ve mentioned:

As with anything you install on your computer, you want to make sure that you trust the source of any userscripts you add.

The rebirth of Wragge Labs (and moving my Heroku apps)

Wed, 09 Jul 2025 17:48:23 +1100

It looks like some paid work I was counting on won’t be going ahead, so I’m trying to save a bit of money on cloud hosting. As I previously noted, this resulted in the resurrection of The future of the past, but I’ve also been continuing to slog away at migrating all my old Flask apps and experiments from Heroku to a single Digital Ocean droplet. As of today, I’ve migrated 11 apps. Here’s a few details…

A new (old) home

The first thing I had to figure out was how to group together a series of individual Flask apps so I could easily run and maintain them on a single server, without making major changes to the apps themselves. I decided to go with the application dispatching pattern described in the Flask documentation. This groups the apps within a single Python environment so I had to do some alignment of Python versions and packages, but it wasn’t too hard and having just one virtual environment to manage seems a lot easier in the long run.

The application dispatching pattern configures the server to run one application at the web root ('/'), with the other apps assigned individual sub-paths. This raised the question, what did I want sitting at the root address? Rather than selecting an existing application for the prime slot, I decided to take the opportunity to build a showcase that included details of many of the things I’ve created over the years.

The old Wragge Labs (circa 2012)

I also needed a new domain name. Or did I? Back in the old days, I had a site where I shared many of my tools and experiments – Wragge Labs. In the intervening years, I’d moved or migrated much of the content away and pointed the wraggelabs.com domain to my main site at timsherratt.au. But this seemed like a good opportunity to resurrect it. So if you’d like to have a play around with some of the things I’ve created over the last 30 years, head along to the all new wraggelabs.com!

The new Wragge Labs showcases websites, apps, and experiments from the past 30 years – some useful, some playful, and some creepy…

A surprising amount of the things I’ve built are still working. As I was compiling my list, I made a few running repairs – for example, fixing broken links in Linking history in place and Magic Squares to get them working again. However, some things only exist now in web archives, and others have been broken by the recent actions of the NLA and NAA.

To provide a bit of extra context, I’ve grouped together publications and presentations documenting many of the experiments. These are saved in Zotero, and tagged with the name of the app. When you click on a ‘Related’ link, the details of any linked resources are retrieved using the Zotero API and displayed on a new page. This means I can add new related resources simply by dropping them into Zotero.

Moving house

The process for moving the apps to their new home was pretty straightforward. On my local machine, I copied the code into the new aggregated structure, added any packages needed into a combined requirements file, and created a new top-level app to direct requests. And then I spun everything up and started fixing bugs…

All of the problems were easily resolved. Most involved fixing up paths to static assets, or in navigation links. The only significant changes to the Python code were caused by the deprecation of the .count() method in Pymongo.

To make life a little harder, I decided to take the opportunity to make sure that all the assets – javascript, css, and font files – were loaded from the local system, and not sitting in the cloud. Having everything local should make it easier to maintain the apps in the long term. It was a bit fiddly tracking down where everything was being loaded from, but not too hard.

The only other changes I made were to add some caching to most of the apps, particularly those that make calls to external databases or APIs. I used Flask-Caching with the local file system backend.

To get the new aggregated application working on a Digital Ocean droplet, I followed the instructions on how to serve Flask applications using uWSGI and Nginx. I think the only thing I did differently was to use pyenv to manage Python versions and the virtual environment. To update the app, I use rsync to copy across the code and systemctl to restart it. So far it’s all working pretty smoothly.

Redirecting Heroku

Once the apps were happy in their new home, I needed to redirect the Heroku addresses to wraggelabs.com. It was surprisingly hard to find good documentation on how to do this, so I’ll document my steps in detail in case its of use to others.

There are a few redirect apps for Heroku around, but I decided to use heroku-redirect because it basically just configures and runs Nginx without any additional processing. First I cloned heroku-redirect to my local system, and then for each app I wanted to migrate I followed these steps:

cd into the heroku-redirect directory
set the git remote for the app you want to redirect: heroku git:remote -a [app name]
if you’re not using the lastest Heroku stack, update it: heroku stack:set heroku-24 -a [app name]
set the url you want to redirect to (without trailing slash): heroku config:set LOCATION=[new url]
to add the full path to the redirected url: heroku config:set PRESERVE_PATH=true
I found I had to remove the Python buildpack from the app before updating, I did this using the Heroku dashboard, but no doubt there’s also a CLI command
I also used the dashboard to add a new nginx buildpack: https://github.com/heroku/heroku-buildpack-nginx.git
finally I pushed the new app, using --force to replace it completely: git push --force heroku master:main

Of course, you’ll need to have the Heroku CLI installed.

A number of my Heroku apps were using basic dynos (which cost US$7 a month), so once they were redirected, I changed them to used shared eco dynos. Yay – money saved! Hopefully, the redirects won’t push the eco dynos beyond their monthly limit.

More experiments to come?

One of the good things about all of this housekeeping is that it’s got me thinking about new experiments. I used to love Flask and Heroku because it they made it so easy to build and share things. Now I can do the same with wraggelabs.com!

The future of the past... in the present

Wed, 02 Jul 2025 13:26:45 +1100

I’ve been on a bit of a self-archiving binge lately. It started because I needed to cut back some of my web hosting costs, and was looking at ways of bringing together a group of separately hosted Heroku apps onto a single Digital Ocean droplet. While taking stock of my various apps and experiments, I remembered there were some that hadn’t survived earlier migrations – in particular, the future of the past.

The the future of the past was a weird little app built on top of a collection of 40,000 newspaper articles, harvested from Trove, that included the phrase ‘the future’. I created it as part of my Harold White Fellowship at the National Library of Australia in 2012, and told the story of its genesis in my fellowship lecture. In short, I extracted words with the highest TF-IDF values for each year in my dataset, and fell in love with them. The word groupings were so odd and evocative that I felt I had to find some way of sharing them.

The future of the past made words the primary means of navigating the collection of newspaper articles. At first you were presented with a random selection of words, sized according to their TF-IDF values. When you clicked on a word, you limited the results to years in which that word appeared. You kept clicking words until only one year matched. Then you were shown a random selection of words from that year, along with the words you’d followed to get to that point. Once you’d arrived at a year, you could click on words to display the content of articles that contained that word. But you could also make poetry.

The future of the past gestured towards fridge magnet poetry in its design and odd jumble of words. And when you finally landed on a single year, you could create your own poems by dragging words into the box at the bottom of the screen. Once you were happy with your poem you could share it on Twitter. And people did.

The most exciting and enjoyable part of the project was watching people create and share their poems. The future of the past even managed to win the ‘Best use of DH for fun’ in the 2012 DH Awards. As I said in my Harold White lecture, I wasn’t really sure what the future of the past was – a discovery interface? a game? a piece of art? But I suppose that’s one of the reasons why I liked it so much.

When I built the app, I was going through a stage of putting everything in Django. Only later did I realise that Flask was a much more suitable framework for the sort of small, experimental apps I was creating. Django was overkill, and the maintenance demands coupled with hosting issues made it difficult to keep things alive. At some point, the future of the past went dark and it just seemed too hard to get it going again…

But last week I had another look, and decided I could resurrect the app in a more maintenance-friendly form by converting it from Django to Flask, and migrating the data from MySQL to SQLite. Django and Flask are both Python frameworks, so it was mainly a matter of unpacking all the logic in Django’s views, models, and handlers and consolidating it into a couple of simple Flask functions. Fortunately, I managed to find an SQL dump of the original database in the backed-up downloads folder of an old laptop. It took a bit of fiddling, but I got the dumped data loaded into SQLite without too many problems.

I also realised I could use the new database to fix up another app I created during my fellowship – a word frequency browser. It’s just a static HTML page, so I added a couple of JSON APIs to the Flask app so it could access the data.

Both Django and Flask use Jinja2 templates, so I didn’t have to do anything much to the interface of the future of the past. I made sure that all the assets (fonts and javascript) were being loaded from local copies to avoid any future problems and, of course, I had to replace the Twitter integration. I decided to add options to share poems on both Mastodon and Bluesky. Mastodon was a little tricky because you need to know a user’s instance before you can post their toot. There are a number of solutions available, but I went with the pattern documented in this GitHub repository. It’s a little clunky because you need to enter your instance name each time you post, and you might also have to allow pop-ups for it to work properly, but it seems to do the job. I did think about updating some other aspects of the interface, but decided to preserve it in its original 2012 grey-toned glory.

So the future of the past lives again in the present!

Create your own poems!

Mining for meanings

Mon, 30 Jun 2025 18:36:42 +1100

In 2012, I was lucky enough to be awarded a Harold White Fellowship by the National Library of Australia. I used my time to explore ways of using Trove’s digitised newspapers as data, and presented my work at a public lecture in May 2012. I spoke from notes and never got round to writing it all up. The recording made by the NLA has disappeared from their website, but is still available in the Internet Archive. The text below is a transcription of the recording made in June 2025 with some minor editing.

You can also listen to the audio, browse the full set of slides, or download a PDF from Zenodo.

Photograph by Christopher Brothers, 2012, nla.gov.au/nla.obj-1…

I. Beyond discovery

Thanks Marie-Louise and thanks to the library for this great opportunity. And of course thanks to all of you for coming along on a night when I’m sure you’d rather be at home waiting for the budget speech. And this is the API working away here in the background. Okay, well, do I really need to introduce the newspaper database? I suspect I probably don’t for this sort of audience. You’re probably avid users of the digitized newspapers online. Are you? Yeah. I did my doctoral research back in the dark ages before Trove, and of course that meant spending many weeks, if not months, destroying my eyesight using microfilm readers. Using what are quite fragmentary printed indexes to try and find stuff which might be relevant to my study. But now of course more than 60 million newspaper articles online and most importantly, really the full text of these articles is searchable. It is something which we’re quite familiar with now, but it is something which is quite revolutionary in many ways.

This unprecedented access to a vast volume of material which documents the ordinary lives of Australians is already changing historical practice. We can now go beyond the well-known events, the big stories and explore the small stories, the fragments, the glimpses of lives which might not otherwise be recorded, but this access comes with a cost. What happens when we do a search and instead of getting 10 results or 100 results, we get 10,000 results or 100,000 results? How do we start to use or understand that sort of thing? What do we do when instead of the clarity and excitement of discovery, we end up with the anxiety and confusion that can come with overwhelming abundance?

Fortunately though, there are a growing number of digital tools which we can turn to. Tools and technologies which enable us to manage this deluge and to explore large volumes of text rather than sort of single search results. Tools that enable us to zoom out of our search results and have a look at the big picture to understand the trends and the patterns to see what’s going on. For example, perhaps we might want to try and track events over time. Have a look for example, this graph shows the prevalence of the words drought and floods in the newspaper database over time.

So we can actually look at that and use it as a way where we can map it to the specific events. And we can see here, of course, this is the federation drought at this point. We could also start to look for patterns that aren’t easy to see within your sort of normal list of search results.

I was interested in having a look at how the word decade might be used. So I searched for decade and I found, as you can see, that there’s these nice sort of regular peaks and I was wondering why have we got these such these regular peaks? And I did a bit more digging and I discovered why. That red line shows the usage of the word census. And you see here how the little peaks sit on top of each other?

So obviously the time we talk most about decades is when a census has come out. So again, this is a sort of pattern which would be very hard to find other ways by just sort of working through our list of search results. We can also use these sorts of technologies for exploring changes in language, the way we talk about things, the labels we use.

This is an example which I’ve taken from the National Library of New Zealand searching through New Zealand newspapers in this case. But what they’ve done is to look at the change in usage of the name for the south island, which was apparently, I didn’t know this originally called Middle Island and changed to the South Island. And so you can see here this sort of process of transition happening before South Island takes over completely.

We can also challenge our expectations. Now, I was always of the belief that the traditional name for people from English cultural background of that chap who wears a red suit and comes around at Christmas time was Father Christmas. And then in recent years that has been supplanted by the sort of Americanized Santa Claus, but it seems I’m wrong.

The red line here is Santa Claus, the blue line is Father Christmas. And so if we look here from the late 19th century to the early 20th century, Santa Claus is definitely winning. What’s interesting though, really interesting, is when we get the change over. Any guesses as to what’s going on there?

Audience: Coke advertising.

Pardon?

Audience: Coca-Cola advertising.

Well actually, I don’t know. My hypothesis, what’s happening, is this is around sort of 1914 and it seems that over the war period, Father Christmas starts to win over the top of Santa Claus. So whether, I mean this is pure hypothesis at this point, and it’s something which would be interesting to explore, whether it’s the Germanic sound of Santa Claus, it sort of lapses in popularity or perhaps there are other causes, completely other circumstances. But that’s the value of these sorts of things that they do allow you to ask some questions and to prompt you to do some other sorts of investigation.

Now these graphs which I’m showing you, were all created by a tool I developed called QueryPic. And we won’t just show you the slide, we’ll actually use it. I want a word. Anybody give a word?

Audience: Brooch.

Broach?

Audience: Because yours is nice.

I’m not sure it’s going to show anything. Yeah, brooch.

Audience: Know what we’ve done ‘automaton’, the one we talked about last week.

You have to spell it for me.

Audience: A-U-T-O-M-A-T-O-M. Actually, correct. That’s what you…

No, no, no, this is right. Okay. So what this is doing now, it’s actually going off to the Trove API and it’s getting the total results. I mean it’s actually a very simple tool. All it’s doing is it’s taking your query and it’s searching for each year across the span of the newspaper database, and it’s getting the total number of results for each year and it’s then presenting them in the form of the graph. As I say, it’s very simple, but it’s also quite effective as you can see. And it’s useful and it’s also quite fun. And what it gives you is the ability to quickly explore a hunch, to get a sort of sense of context or to start exploring, to start framing a more specific research question without spending… there we go… without spending days searching or tabulating as you would normally have to do. So you can see how easy it is to use and if you want to actually compare that to something else, you can just type in another word.

Okay, now there are obvious limitations to a tool like this. There’s a lot to unpack. I wouldn’t want to say that it’s evidence because there is so many assumptions built into the back end of it. Questions about what the search engine is actually giving you back, different usages of terms, obviously the contexts and things like the quality of the OCR itself. You know there are a whole lot of stuff. But despite all that, I think it is quite useful, as I said, in terms of allowing you to explore things quite quickly and to follow your hunches. I regard it as a starting point, not as an end.

Now, but there are some folks… let me see if it’s going to finish.. there are some folks who are a bit more confident about techniques such as this and who would suggest that not only can they provide evidence, but they can actually be used to develop mathematical representations of past behavior.

II. Finding formulas

You may have heard of the Culturomics project from Harvard University. These guys got access to the full corpus of Google’s digitized books. So 5 million books, the text of 5 million books. They pulled it all apart. They did a bit of cleaning up of the metadata, all sorts of stuff, and then they started searching it and they started to see what they, but they could pull out of it. And when they started searching, they noticed all sorts of patterns appearing and they argued that these patterns could actually form the basis for what they said was a new science of culture, hence culturomics.

There’s a lot I might say about that, but I just want to look at one example. Okay, this is an example which in their published paper in Science, they called ‘We forget’, and I generated it using an online tool called the Ngram Viewer. You can go and do this yourself if you like. And what it’s showing as you might be able to see is it’s searching for years used within the text. So 1883, 1910, 1950. It’s pulling out all the instances where those labels are used within the text, where those terms are used. And there does obviously seem to be some sort of pattern. And the research has noticed that the graphs have a characteristic shape, obviously rapid ascent and then a decline. But they also notice changes. Of course, the size of the peaks is changing over time, getting higher. They say that this is indicating a greater focus on the present and the rate of decay is increasing, so that the peak is actually dropping away faster. And they say from this, we are forgetting our past faster with each passing year.

I thought it would be interesting to repeat this experiment using QueryPic. So I did. It looks a bit different.

I mean, before we could interpret this difference, of course, there’s a lot that we would want to ask, a lot of, first of all, methodological questions. Again, exactly what are we searching in the two instances and how can we compare the searching, the books in one instance to the newspapers in others - dates obviously play a different role in newspapers than they do in books. But it was actually the conceptual issues, which really struck me in relation to this example and in particular the assumption that we can compare the past, present, and future uses of these labels as if we are talking about the same thing: as if the label 1950 means the same thing before 1950, in 1950 and after 1950. The names for events and periods that we assign, that we share, that we use are themselves the products of historical processes. They slip, they shift, they change.

We all know what we mean by the Great Depression. Where’s the Great Depression on this graph? So in terms of the usage at the time, the usage of the term ‘Great Depression’ was actually greater in the 1890s than in the 1930s.

We’re very familiar with the usage of the, we’re talking about ‘black’ days like Black Tuesdays. Black Fridays of course, is the one we’re most familiar with. And in Australia, these labels are generally attached to bushfires of course, and that’s the context where we generally understand them and use them and remember them. And over here, of course we have Black Friday. So what’s this big peak here? It’s not a bushfire. It refers to the Victorian government’s mass sackings of senior civil servants and judges in 1878. Obviously it was an extremely important event at the time, an extremely important event in government in Victoria, but it doesn’t quite figure in our collective memory in the same way as Black Friday does.

One of my early experiments with QueryPic was to look at the question of when did the Great War become the First World War? At what point did we stop thinking about the Great War as the war to end all wars and realize that it was one in a series of global conflicts? And the graph really does a nice job of confirming our expectations, I suppose, in that we see a nice crossover late in 1941, which if we were thinking about the passage of the war would be about when we probably would expect. But what’s missing from this, what’s missing of course is just the war.

Just as with the Great Depression and with the Black Wednesday, what’s hard is trying to recapture the moment as it was happening, the sense for want of a better word, of present-ness. Now, if we go back to ‘We forget’ what are we doing when we’re talking about one of these dates? I mean, if we think of the present as providing a line there - past, present, future - on the past side, what are we doing? We’re anticipating. We’re predicting. Perhaps, we’re dreading. And present, we’re experiencing, we’re enjoying, maybe suffering. In the future, we are remembering, we are regretting, perhaps reflecting. So instead of lumping all these together, it seems to me that we should be teasing them out and exploring their different interconnections.

We should be trying to give the past back its own sense of the present. And this in essence was the modest and thoroughly achievable goal of my Harold White Fellowship. I wanted to explore the possibilities of the digitized newspaper collection in supporting this sort of rich temporal contextualization using digital methods to recover the pasts, the presents and the futures of any moment in our history. I have to admit, I haven’t got very far yet, and Marie-Louise has been doing a good job of reassuring me that sometimes the fruits of these things take a while to develop. Now, there are a number of reasons why I haven’t gotten as far as I wanted, but I do have a few sort of sketches that I want to share with you.

III. The future of the past

Okay. What I decided to do is to try and create a sort of manageable sample set. So I decided to work with articles, which included the phrase, ‘the future’, in the heading or the first four lines of the newspaper articles, and that’s one of the facets you can use within Trove. Why did I limit it in this way? Well, I’ve been doing a lot of different work in Trove as Marie-Louise said. One project I’ve been working on was looking at ways of finding editorials within Trove and exploring the content of editorials over time. And in doing that, I discovered a number of frustrating things, one of which is sometimes the articles aren’t divided up as nicely as we want them to be. Particularly with editorials, editorials on different subjects are often joined together, so it’s difficult to separate out the specific ones that you want, but I thought by limiting my search in this way, that it increases my chance of relevance. And it also brought the number of matches down to what I thought was a reasonably manageable 60,000 or so.

So I started harvesting those 60,000 articles. I have over time been developing a number of tools working with Trove. One of which is a harvester, which enables you to get the data in bulk. And of course that’s necessary if you’re going to do this sort of large-scale analysis on it. I modified my existing harvesting tools to save the results directly into the database and when the API became available, I modified it to use the API, which makes a lot of things easier. Now, after about 40,000, I thought I probably had enough, and I decided I’d trust in Trove’s relevance ranking and just work with that set.

And then it was time to do some cleaning. Now, Trove’s crowdsourced OCR correction project has been a wonderful success, of course, but it’s worth noting that with the sample of articles that I harvested for this project, only 2% had any corrections at all. So 98% totally uncorrected, totally untouched. While I couldn’t hope to correct all of those articles myself, I could at least try to reduce some of the noise, which is created by these sorts of OCR errors. So I developed a series of scripts which would try to clean up some of that OCR output. First of all, it corrected some errors which are fairly consistent and hopefully fairly unambiguous OCR errors. And you can test yourself here. What’s that meant to be?

Audience: His.

Pardon?

Audience: His.

Nope. The.

What about this one? Nah, the. This one? No, of. Should get that one. Ah, yep. And you can check. There we go. Look of, ah, yep. So there are a series of these which I could through a script just fix up. I then checked each word in the text against a series of dictionaries and word lists, and this included a word list of places which I actually extracted from the places Gazetteer provided by Geoscience Australia. Anything which didn’t seem to match up, I marked up in a way that I could extract it later if I wanted to. And all of this, you’ve got to understand, went through a lot of trial and error, just trying stuff out, seeing what it produced, trying it again, fiddling with it lots and lots and lots of trial and error.

But after that, I could do some fun things. You’re all of course familiar with word clouds, but I bet you haven’t seen a non-word cloud. This is my non-word cloud.

Now, of course, the big question is what is ‘others’ doing there? I don’t know. For some reason my word list didn’t like the word others, but of course you can see here some more sort of consistent OCR errors. There’s another ‘the’ and another ‘the’, and that would be a ‘be’, in most cases. And we also see where words have been split up. We’ve got a ‘tralia’ down there. Oh, that’s a ‘which’ obviously. So it’s actually quite useful visualizing in this way because I can then feed that back into my process of cleaning. I can see where the common errors are, and I can start to feed that back into the process.

For each article that I processed in this way, I generated an accuracy score, which was simply the number of recognized words divided by the total number of words within the article. And I could use these scores to develop a couple of overviews.

So of my set, this is only just my sample of course, so this is OCR accuracy over time. There aren’t many articles in this earlier period, so it’s probably not worth worrying about too much. But what’s interesting is this decline here down to the 1920s where we’re going below the 80% mark. Why is that? I’ve got no idea. There are a whole lot of variables which could be certainly involved here, whether it’s the fonts, whether it’s the quality of the printing, the quality of the paper, the quality of the microfilming. I don’t know. It’s something which would be interesting to explore further and to and investigate.

We can also have a look at the poorest performing newspapers. So the Perth Gazette and West Australian Times didn’t do too well, and it got 58% in my scorecard. Again, this is only a select sample, so I’m not quite sure what you can read into any of this, but it’s sort of interesting. These figures weren’t particularly important for my work, but I do think that the general issue of OCR quality is really vitally important, particularly as we make more and more scholarly use of these sorts of collections in bulk. I mean, obviously we need to improve the quality, but we also need to expose our assumptions about the OCR quality that underlie our work so that when we are putting forward something, some sort of analysis of the text, we’ve got a way of communicating the quality of the material that we’re working with.

I then decided to make my sample set even more manageable by selecting just the first 10,000 articles, which had accuracy figures of over 80%. So I used my scores and I went through and I decided just to choose those ones which seemed to go pretty well. Of course, as any good digital humanities person does, I then started counting words.

As with most of this stuff that I’m showing you tonight, it’s online and you can go and play with it and use it yourself. So this shows the word frequencies over time, and there’s a time slider here, and you can just drag it along and see what’s happening in different years. Now, nothing really significant jumps out at me from looking at the word frequency clouds here. I mean, what is sort of interesting I suppose, is the preponderance of ‘would’ and ‘could’, which I suppose confirms the future orientation of the sample set that I’m working with. And there may well be other things within there that jump out at you. And so as I say, jump online and have a look and have a play with this and see what you can make of it.

I mean, word frequencies… Okay. So word frequencies can be interesting in getting a sort of overall picture of a large amount of text and starting to track some changes over time. But this sort of word frequency tells you what’s common. It doesn’t tell you what’s distinctive. It doesn’t tell you what’s interesting in an article. And another measure we can use to try and get at the distinctiveness of a piece of text is something called TF-IDF. It’s an acronym, term frequency, inverse document frequency. And what it does, it looks not just at the frequency of a word within a particular piece of text, it also looks at the frequency of that term across a collection of texts. So a word that is common in a particular article, but not very common in a collection of articles will appear as more significant, more heavily weighted in its TF-IDF value.

You use TF-IDF values all the time. They’re used by search engines in calculations of similarity. They can take the TF-IDF values, convert it into a sort of mathematical format and use it to calculate the similarity between two pieces of text. And the results of calculating TF-IDF values for collections like this are pretty interesting, and I’ll just show you a little comparison.

So on the left-hand side here, we have, this is 1939. The top 10 words on the left-hand side are just the plain frequency values, and here are the TF-IDF values. So you see we’re getting at something quite different and something quite interesting here. Obviously 1939, Hitler doesn’t figure in this list of terms. He’s at the top in this one. But we also get these really odd things like midget and roundabout. I found it really interesting producing these values, and I found them quite sort of evocative and interesting and wanted encouraging me to explore more, and I’m going to talk some more about this a bit later.

But I finally just wanted to show you one other way of understanding a collection of texts, and that’s through a thing called topic modeling. There’s a lot of topic modeling going on in the digital humanities at the moment, and there are a number of good blog posts, which I’ll put links to from here, which tell you about what topic modeling is. I’m just going to quickly race through it. Basically, I use a piece of software called Mallet. I pointed Mallet at my collection of texts, told that I wanted to define 10 topics, that is 10 clusters of articles within those texts, and it just did it.

And what it came back with is these lists of words which are grouped according to the topics which it believes existed. You can then go through and look at these lists of words and start to interpret them to try and understand what those topics are. And most of them are pretty clear. This of course, is the topic that tells me that I still didn’t clean up the OCR enough, but it’s interesting that it brought them all together.

We’ve got, I mean here we’ve got trade, here we’ve got technology, here we’ve got land/rural, here we’ve got international relations, here we’ve got government, and here we’ve got home and society. And it’s amazing once you run these things, how much sense the topics actually make to you.

And it also goes through and for each article in your collection, it weights them according to these topics. So you can then go and for each article you can see which is the most heavily weighted topic for that article, and you can calculate the number associated with each topic and you can do something like that. Okay, that’s not terribly instructive as it is, but you can, I won’t show you now, but if you click on that and go to the live version and you click on the legend down the bottom there, you can actually take away some of the lines so you can actually see what’s happening underneath, so you can just see the lines that you’re interested in.

But basically, I want to do a lot more work on these topics at this stage, and I haven’t really done a lot of interpretation of them. I want to see how I’m actually using those weightings and find better ways of actually looking at them. So anyway, here I am. Not a lot of interpretation at this stage. No great insights. I have some data. I have a data set, and I’m going to be continuing to play with it. And as I’ve said, all this stuff is available online, so you’re welcome to come and play with it too and see what you can make of it. Now, you may think that I’ve gone into a lot of tedious detail about what I did. Well, I’ve actually saved you from a lot of the gory details.

IV. Meanings for mining

The truth of much research in the digital humanities is that large amounts of time are spent yak shaving, and data munging. If you don’t know the term ‘yak shaving’, it’s that process that we’re all familiar with, when you start doing a particular task and you realize, in order to achieve that task, you have to actually do something else or research something else, and that actually continues into infinite regression until you find yourself doing something which seems totally unrelated to the task that you started with. I’ve had a lot of that recently. There were lots of issues just involved in using this data and starting to manipulate it. As I’ve said before, the issue of the OCR quality is crucial, and we have to be upfront about the problems and continue to look for the most effective solutions. We have to talk about questions of selection and completeness. What’s actually in Trove? How does it change and how does this sort of influence the results that we get?

One of my examples here is a thing called the Atomic Age Exhibition, which toured around Australia in 1948-49. It was a big thing. Many, many thousands of people visited. It was at the Easter Show in Sydney. If you search in Trove for Atomic Age Exhibition, you’ll find quite a lot of results coming from the Courier Mail in Brisbane. You’ll find virtually nothing from Sydney and Melbourne, and you might be inclined to think that the exhibition didn’t actually go to Sydney and Melbourne. Why is there nothing in Sydney and Melbourne? Because the exhibition was sponsored by the Herald in Melbourne and by the Daily Telegraph in Sydney, and both of those titles are currently not in the newspaper database.

So we’ve got to bring these sorts of questions and perspectives as we start to do this research. Another barrier, which I started to butt my head up against in doing this was that of computing power. Generating the TF-IDF values for my sample took about a day and a half on my laptop. And of course, then you realize, you did something stupid and you have to do the whole thing again. And I did wonder at various times whether I was reaching the limits of what’s practically possible for one bloke and his laptop and wondering whether my sort of piecemeal efforts will be blown away by academic teams with access to research funds, bright young graduate students, and time on a supercomputer.

Now, this list of problems and concerns might seem a bit depressing, and it might not be what you expected from this talk, but I want to reassure you, there are digital tools that make it easy to get started and start exploring the possibilities. QueryPic of course, there are other things like Voyant, which is a great online tool for starting to do text analysis, but sooner or later you’re going to have to confront some pretty hefty questions. But hey, that’s just history. The past is messy and it raises difficult questions about things like selection and interpretation. The issues aren’t necessarily new, it’s just that they’re raised in a bigger, more technically challenging context.

But what does really bug me is a nagging feeling that I should be taking statistics more seriously. That in constructing the sort of examples which I’ve been showing and the tools that I’ve been demonstrating, that I should actually be less impressionistic and more rigorous, as if I’m sort of not doing justice to the vast computing power that I have at my disposal. But I don’t want to do that. In January, I was at the American Historical Association meeting and I was actually able to see the culturomics guys live in person doing their spiel. And as they described their vision for a new science based on access to these huge cultural data sets, I tweeted, “Yeah, I want to use big data to tell better, more compelling, more human stories.”

The British historian Tim Hitchcock has similarly described his own unease that the demands of big data seem to be moving him towards a more positivist style of history. In the humanities, we’ve been really fortunate to make use of many decades of research into things like information retrieval. We’ve adopted many of their concepts, their tools, and their formulae, but we’ve also adopted some of their language. And so we talk about what we’re doing as mining. Mining is an extractive process. We dig stuff up, we pull it out of the ground. But this seems to be pretty much the opposite of what I want to do. I mean, I do want to find structures and separate them out for different types of analysis, but then again, I want to put them back together. I want to observe them in different contexts as rich and as complex as possible. How do we do that?

Well, first of all, we have to work out better ways of incorporating these sorts of big data perspectives into the narratives that we write. Just as QueryPic gives you that opportunity to sort of zoom out and get a big picture, I think we have to take control of the zoom and use it to our advantage. And this, by the way, probably means developing new forms of publication that allow easier and better integration of data and text. It’s challenging, but there’s not much point to dwelling on the dangers and problems of big data, and as Tim Hitchcock concludes, we simply need to get on with it.

V. Screwmeneutics and deformance

The second approach is to foster acts of creative subversion, to use digital tools in new ways. Literary scholars within the digital humanities talk about the possibilities of deformance, of using computational methods to change texts in ways that can open them up to new and new critical perspectives. Stephen Ramsey also talks about moving beyond traditional forms of search and browse and admitting ‘screwing around’ as a legitimate research methodology. Of course, historians don’t want to start deforming their sources or do they?

This is an experiment I created called The Real Face of White Australia. I always get a bit teary when I put this up. Well, what I’ve done here is actually use computer vision software to extract portrait photographs from certificates, which were used in the administration of the White Australia policy. These are records held by the National Archives of Australia. There’s several thousands of these, and this is just from one series, and you can just keep scrolling and scrolling forever, or almost forever. So by manipulating the sources in these ways, by extracting those photographs, I’ve created a new way of seeing these records and it’s quite powerful.

But we can also be playful. You may have seen this. This is a little game that I created using the newspaper database. Again, it’s very simple. It just picks a newspaper article at random from the database and asks you to try and guess the year in which it was published. So any guesses for this one? What would we say? Let’s say 1850… That’s a bit later than that… Let’s see, it’s earlier. Okay, so you can keep going like this. You can go and try it out yourself later. As I said, it’s very simple, but it’s also strangely addictive. And of course, it’s also a way of exploring the content of Trove by screwing around.

QueryPic, the Real Face of White Australia and newspaper roulette, my Headline Roulette, also have something else in common. They are public. I want people to use them. I want people to have fun. I want people to be moved. I want people to find things to be surprised and to do history.

Just yesterday I received an email from a self-confessed Australian history addict, oh no, Australian history fanatic, sorry. And she had become addicted to Headline Roulette. She wanted to know if I could add a facility for users to actually save their scores. So presumably they could go back and see if they’d improved or share them with their friends. So obviously the next step is the Facebook application. I mean, other people have described to me how scrolling through the Real Face of White Australia brought them to tears. And I’ve come to realize that these sorts of interactions really mean more to me than a footnote in an academic article. I’ve probably just killed my hopes of an academic career there. I want to use digital tools, I do want to use them to deform history. I want to use it to deform history in a way that makes it accessible to new audiences in new ways. And so I present to you in honor of my Howard White Fellowship, a new experiment.

Now, I described to you before the process involved in calculating the TF-IDF values. What I didn’t describe was the fun that I had when I was doing it. It was really quite exciting and amusing and funny and all sorts of things, watching the words fly past on the screen. And as I completed each year, I would had a little script which would show me the top 20 words for that year. And anybody who follows me on Twitter, we all have a good picture of what was going on there because I couldn’t help but share this. So I mean, it tells their own story. I mean, it was really like a sort of wonderful puzzle as I say there, as they all came up. And then I started tweeting some of them.

This was a nice one. I like the ‘hitler’ with ‘mudguards’, ‘duchess’, ‘opossum’, ‘hollywood’ and ‘canberra’.

And as I said here, of course, I mean there’s got to be a novel in ‘prince’, ‘pronunciation’, ‘keyboard’, ‘zulu’, ‘begged’, ‘unbent’, ‘diddle’, ‘candlesticks’, ‘virtuoso’, ‘highness’ and ‘pots’. This started me thinking, was there a way I could share this experience and use the TF-IDF values as a way of exploring my data set, a way of opening this experience to others, as creating a sort of shifting playful window on the future of the past. So this is my first attempt. Again, public, go play with it.

I’ve actually deliberately tried to keep most of the metadata away from this interface because I wanted the words to be the focus. And yeah, it does look a bit like that fridge poetry thing that you can get, and that’s quite deliberate. I mean, actually at some stage, I want to add a box down here where you can drag your words down here and make your own sort of collections and tweet them. What it’s showing you there is just a random selection of TF-IDF values from my sample, and you can click on any one of these, and it goes away and it sees, first of all, how many years have that value attached to it. If there’s only one year, then it’ll actually return that year. If there is, let’s see if we can find one that has more than one year. If it appears in more than one year, then it pulls out a random selection of those values from those years.

Okay. Oh no, we’ve got 1943. I’m not doing a good job of it this time. Anyway, you can have fun with it. And of course, if you want to actually see what’s going on, you can click on these and it will actually load the articles here, and you can explore the text of them there and see where the word’s popping up.

Okay, what is this? I’m not quite sure. It’s not really a discovery interface, although you can find interesting stuff. It’s not quite a game, but it is quite fun to explore. To me, I really, I’m sort of in love with it at the moment because it actually, I mean, you think about what I’m trying to do in terms of recapture the presentness of the past. Our experience is not about just the big stories of the day. Our experience of any moment includes a whole lot of trivial aspects. And I love the way that this sort of brings together Churchill and Corpuscle and Melvin, whoever Melvin is. I love the mix of words, and it just, to me is incredibly evocative. It makes you want to start imagining stories. It makes you want to explore, it makes you want to find out more, but it just has a wonderful, exciting aspect to it itself. So I’m not quite sure what I’m going to do with it or how I’m going to develop it, but really, as I say, I’m quite in love with it at the moment, and I hope you’ll have a play with it and see what you make of it.

Could it be a discovery interface? I don’t know. It does enable you to get into my dataset, but of course, it’s obviously from a rather indirect means and it includes lots of randomness as well. And I’m a big fan of randomness in actually developing new ways of discovery. So there you go. Please take it away, enjoy, play. I may not have conquered the meaning of time yet, but experiments like this actually make me think about the form in which I actually present those sorts of arguments and those sorts of ideas. How do we actually create resources which give that sort of sense of the disjunctions and the serendipity? So while I may not have achieved all I wanted to, I’ve come away with a better sense of what it is that I’m trying to do and what I want to do with this material. So thank you.

Questions

Marie-Louise Ayres:

Thanks very much, Tim. Just before we open up for questions, there was just a few things that I thought I wanted to say. One is, look, two Australians have corrected more than a million lines of text each. So if you think you couldn’t correct 40,000 editorials, you are not being ambitious enough. That’s the first thing. The second thing to say is that our own Trove team have found that the only surname that is not in Trove you could do things that are not is Kardashian. And the third is, I guess just thinking about how amazingly creative these visualizations are that Tim has been doing, and I hope you’ll ask him about them.

But the fourth thing I wanted to say is to pick you up on one of your early comments where you said, “I haven’t got as far as I wanted.” Now that’s a very interesting construction that includes the past, the present, the future, and a spatial term as well. So maybe you need to think about that. I think that we’d agree with the work that Tim’s has done. I don’t know where he wanted to be, but he’s gotten a long, long way and done things that the rest of us just probably haven’t even contemplated. So I’m hoping you’ll ask Tim some questions now and then we’ll have more opportunities afterwards. So it’s dark out there. So if you want to ask a question, can you just make sure you raise your hand and speak up? So don’t be shy. Yes.

Q&A 1:

How appropriate would your methodology be for spelling? Where I’m coming from is I know the Australian Labor Party and the British Labour Party spell different. And I can remember once going through microfilms that the Sydney Morning Herald in 1920s, and it dawned on me all those spellings were American. And so there must be things happening where you could compare how words are spelled or do all the correctors corrupt your data just by correcting.

Tim Sherratt:

I certainly think you could do that sort of analysis. And actually some of the nice examples that the culturomics guys used in the Google Books example was looking at, oh God, the name is… what irregular verbs, looking at changes in irregular verbs over time, which is quite interesting. But their data set goes back to quite a long way. And there are challenges. One of the challenges in working with Trove, I mean obviously the interface is geared towards discovery at the moment, making sure that people find what they’re after. But that means that sometimes if you want to find something exact, it can be a bit tricky. You’ve got to know sort of how to turn off the fuzziness in the searching. And sometimes you are foiled in that by the fact that people might’ve tagged something and the search by default also searches the tags and the comments.

So when I did my first, I don’t know whether you saw it in my World War I graph, you may have noted that there was a peak, little peak, for World War I actually during the First World War, which is sort of interesting if you think about it. And that’s because people had tagged those articles with World War I. So again, this is all about, and one thing which I would always emphasize as we start to do this research, we have to develop our literacy in terms of understanding search interfaces and how they work, and be prepared to go into the documentation and to look at the advanced searches and how they work and actually start experimenting a bit with what different searches bring back, so that you can actually have a good picture. And I think mean, obviously the institutions themselves have a role in communicating this and making it, exposing what’s going on behind the scenes. But I think it’s an important literacy for researchers going on into the future, being able to pull these things apart to understand exactly what’s going on. So you can do those sorts of quite detailed fine-grained comparisons.

Q&A 2:

I’m interested in what year was it the picture of White Australia and was that the people that were actually accepted or what?

Tim Sherratt:

Certainly not. Well, okay. I should say that this is part of a broader project called Invisible Australians. If you just go to InvisibleAustralians.org, there’s a lot more information about what we’re trying to do with these records. That particular set of records, as I think I said, those photographs were just pulled from one series within the National Archives of Australia. And there are many more series like that. They are a series of certificates. Basically, if a person deemed non-white was living in Australia, they wanted to travel overseas and get back into the country again…

Q&A 2:

Those people that lived in Australia rather than people that tried.

Tim Sherratt:

No, because… Yeah, there was no trying to it. So yes, it is people who are living in Australia. And this is what is particularly interesting about these records because what we want to do, and this isn’t Trove related, I’m sorry, but what we want to do with those records is to actually try and extract the biographical information which is contained within those certificates in order to sort of find out more about the community who was living under the White Australia policy, people who were living here, whose various activities were restricted in a number of ways by the White Australia policy in all its sort of legislative forms. And we’re bringing to bear a number of digital techniques to try and do that. As I said, with that particular case, it was a facial recognition script which pulled out as photographs, but we’re also harvesting material from the National Archives and doing some topic modeling. I’ll be doing some topic modeling as I showed there on some of the records to try and pull out clusters within those records. So anyway, check it out. It’s something I’m very passionate about.

Q&A 3:

I’m just wondering, it seems like the [inaudible 00:57:20] is a bit of issue as you talked about. And I guess there’s probably a couple of aspects to that. One is like a computer vision, technology is involved and the other part is what do you do after you’ve got that? Is there anything clever that you can do to guarantee IDF or whatever else to try and make better quality? I don’t know if you can describe how do you see the next few years panning out with that? Do you think that there’s a lot of improvement going on?

Tim Sherratt:

There’s certainly a lot of work going on and there are techniques which could be used. There are more developing specific language models that tell you that if you have a certain combination of letters, then after that combination you can expect a certain range of letters but not other letters, for example. And you can use those in sort of probabilistic techniques to sort of go across and see what’s likely to be at particular points. And specifically relating to digitized newspapers, there’s stuff going on. There’s a project in the US called Mapping Texts, which actually works with Chronicling America, the digitized newspapers from America. They actually went through and just actually very similar to what I did, but with a bigger budget and access to Stanford’s resources, did things like the topic modeling and word frequencies. It also did what’s called named entity recognition, which is pulling out people and places from the texts, but they also ran through their sample set and generated figures for OCR accuracy. And they’re fairly similar to my figures actually.

So there’s a lot of obviously recognition of this in Europe. There’s actually a particular research group which has been looking at methods of improving OCR. So I mean, and of course there are many more cases which are much more complex than this if you think about old Germanic scripts or something like that. And so there’s a lot of interest, a lot of concern, and a lot of work going on, I think, to see. And so I think it’s something that we have to be prepared to revisit over time that there are going to be more possibilities for doing stuff with computers as this comes online. And so we should need to assess constantly reassess what’s actually possible and see what we can do. But yeah, I, think, given the general awareness of the problem and the problems that it causes, I mean I think there’s certainly going to be a lot. And I think it’s really exciting the fact that we have now starting to get these collections all around the world of digitized newspapers and the possibilities that opens up for doing comparative stuff. What I didn’t mention with QueryPic is that you can also access New Zealand newspapers. It uses the DigitalNZ API and accesses papers past, so you can actually do graphs for New Zealand papers, but what you can’t do meaningfully is compare Australian and New Zealand results. And that’s because DigitalNZ, our research currently searches the titles of articles and not the full text. Wouldn’t it be really nice if we’re both searching the same things and we could do those sorts of comparisons and we could do it with the US and we could do it with Canada. I think there are some really interesting possibilities there.

Q&A 4:

I was just going to say ‘stine’ is very interesting there because in my opinion, it’s obviously a column break from Palestine. And so that’s a common sort OCR error, E and S being a sort of fragile combination. And not only that, but the rules of breaking, they tend to do chunks like that. And also it shows how the TF-IDF is working. Palestine itself as a whole doesn’t appear there because it’s not actually that important, but ‘stine’ got promoted because it’s extremely uncommon and ‘Pale’ got dropped off. And of course ‘Pale’ would be there… because it’s a very common word.

Tim Sherratt:

Yeah, that’s nice. But yes, I mean as you go through this, you will see other instances where the sort of OCR issue is coming up again. But that’s also another nice example of thinking about how using computational techniques we can start to improve some of the OCR because you are looking at the way words break and seeing if we can use that in some way. Thanks, that’s great.

Q&A 5:

As just before, you said the word evocative, Tim, I was saying to myself, evocative, and so I wanted to talk about that for a minute rather than talk about a technical thing. It seems to me this is a really interesting, I don’t know, I just want you to say more about is this a different kind of historical mode, this kind of desire to treat the past to evoke rather than to necessarily narratize or analyze, or define or pin down. What is it about, is there something distinctive about this evocative mode, which is to do with the digital techniques or, yeah, going on?

Tim Sherratt:

Yeah, I mean this is the thing I’ve actually just been trying to grapple with over the last few weeks as I started playing with this stuff. And I don’t know exactly what it is. All I know is what I feel as you do when you see it. And they do make you start to think in different ways and to imagine and make connections. I mean, I think with your work, with Cath’s work on Semble, there are possibilities for creating spaces which encourage people to make connections, to see relationships between things. And I think digital technologies do lend themselves to that because, I don’t know, as I said, I actually think randomness is something which is rather undervalued in terms of exploration and discovery. And as you know, there’s another project that we worked on called The History Wall at the National Museum of Australia, and that brought together material in quite a random fashion. And it was actually, again, quite evocative in terms of it being able to see the sort of possible relations between items there.

As I said, I don’t know what it lends itself to. What is the process? Is it discovery? Is it a prompt for research questions? I don’t know. But it just seems to me to be something which is worth exploring more and I find that it’s actually just something which I keep doing, so I must be interested in it in some way. And yeah, I mean obviously, I mean it’s something which would definitely be worth thinking about some more. I mean there are all sorts of ways in which you can develop a sort of evocative sense. I mean obviously historical photographs and they give you a different sort of feeling from seeing a text description. So yeah, I don’t know. Really interesting question.

Q&A 6:

Tim, I’d just like to congratulate you on your work. I just find this really interesting and I think it’s really great that researchers like yourself take our data and play around with it, as you said, because ultimately some of those ideas do lead to really useful actual applications. And I just wanted to say your OCR accuracy results are actually bang on because we did quite a bit of research on that five years ago before we launched, and it was 65% to 70%, which is of course low, which was why we said how can we change that and get the public to help?

But you’re quite right, as time moves on and as these big data sets are made more open and available, people develop technologies to improve that. So five years ago, that didn’t exist, an automated way to improve it. We now know of at least three other people like yourself have figured out how globally they could really increase that OCR accuracy rate and prove it. So I guess that’s questions that I would have, how some of this really fantastic research and ones you mentioned in Europe can be built back in to improve on services. But I just wanted to say, did you find it really useful having API being able to get that data? Because I know that was a dream we’d had for a long time and I know you waited a long time to go.

Tim Sherratt:

Well, I didn’t wait. I didn’t wait, did I? I actually just went ahead and did it myself.

Yeah, look, I mean background for those who didn’t know is that I built my own unofficial API at one point, which I used to do some experiments. But yeah, I mean it obviously makes a whole lot of things easier. First of all, the point of view of doing the large data dumps, getting that sort of material, and you’re not downloading the whole web page and all that stuff on it, but you’re just actually getting the data in a structured way. Great. And as anybody who also follows my work will know is that I had a number of frustrating experiences where things change on the web page and everything I created broke and I had to fix it.

So APIs do away with all that. It’s fantastic. But one of the really good things I like about having API access is how easy it makes it to do something like Headline Roulette. If you have an idea and you’ve got a bit of coding experience, you can act on it and you actually build something. And that to me is the most exciting aspect and encouraging people to actually experiment. That’s what it’s all about to me, is creating an environment where people do experiment with this stuff and build things.

A brief and biased history of Trove Twitter bots

Thu, 19 Jun 2025 12:08:35 +1100

The socials recently alerted me to an interesting article by Dominique Carlon, Jean Burgess, and Kateryna Kasianenko on the history of community-created Twitter bots. The article explores bot-making within the context of Twitter’s rise and fall, and provides a handy taxonomy of bot species. However, it doesn’t include any Australian bots amidst the examples. That’s a bit disappointing, as I remember the bot-building years as a time of great fun and creativity. My own contribution to the world of Twitter bots was mainly focused on Trove (what a surprise!), so I thought I might as well jot down a few incomplete and biased notes about the history of Trove Twitter bots.

Trove tweeting trends

It just so happens that I recently packaged up some data about Trove links shared on Twitter. Using this data, we can get a broad perspective on the activity of Trove Twitter bots between 2009 and 2020. The identification of bots is based on the Trove bots list I maintained on Twitter, so it’s possible I’ve missed some.

In total, 43 bots posted 318,767 tweets containing 270,474 unique Trove urls between June 2013 and December 2020.

Here’s a chart showing the total number of links to Trove shared by Twitter bots each year.

Most of the bots shared digitised newspaper articles, but some shared works from other Trove zones. This chart breaks the links down by the type of resource (‘article’ equals newspaper article).

And one final chart showing the number of active bots per year.

year	active bots
2013	4
2014	4
2015	6
2016	8
2017	12
2018	34
2019	38
2020	22

From the data above you can see that bot activity grew slowly between 2013 and 2017, before taking off dramatically in 2018. The peak year for Trove bots was 2019, when 38 individual bots shared more than 100,000 links to Trove. But a mass extinction event in 2020 almost halved the number of active bots. So what happened?

Build-a-bot begins

In June 2013, inspired by bot creators like Mark Sample, I hooked the Trove API up to Twitter to see what would happen when GLAM collections joined online social spaces. The result was @TroveNewsBot, sharing digitised newspaper articles from Trove.

Twitter bots started popping up around the world, sharing collection items from Europeana, the Digital Public Library of America, DigitalNZ, the Cooper Hewitt Museum, and the Brooklyn Museum, amongst others. But @TroveNewsBot was always a bit different. Instead of just sharing randomly selected resources, @TroveNewsBot helped people explore Trove without leaving Twitter. If you tweeted keywords at the bot, it would run a search using the API and tweet back the most relevant result. By adding hashtags, users could control a variety of search parameters – for example, if you included the hashtag #luckydip you’d get back a random article from your search results.

My favourite bot behaviour was its ‘opinionator’ mode. If you tweeted a url at @TroveNewsBot, it would retrieve the link, extract keywords from the text, and then search for those keywords in Trove’s newspapers. This enabled @TroveNewsBot to have conversations with other online resources – for example, it replied to tweets from DPLA and DigitalNZ, finding connections between different collections. I also used the ‘opinionator’ mode to set up a dialogue between past and present. Several times a day, the bot would grab keywords from the latest news items on the ABC (later the Guardian) website, search for historic newspaper articles, and then tweet both stories, old and new.

@TroveNewsBot’s opinionator mode in action – a slide from my keynote presentation ‘Life on the outside: connections, contexts, and the wild, wild web’ for the Annual Conference of the Japanese Association of Digital Humanities in 2014

As well as providing digitised content, such as the newspapers, Trove aggregates collection metadata from hundreds of organisations around Australia and makes it available through its own API. This meant that any organisation could use the Trove API to create a Twitter bot that shared items from their own collection. To encourage more of this sort of experimentation, I created the Build-a-Bot Workshop GitHub repository. This repository included instructions and code for anyone wanting to build their own collection bot on top of the Trove API. Like @TroveNewsBot, these collection bots could share random items and respond to user queries.

Before long, @CurtinLibBot was sharing photos from the Curtin University Library’s image collection, and @Kasparbot was tweeting about objects from the National Museum of Australia. By the end of 2013, I’d added to the family by creating @TroveBot. While @TroveNewsBot dug into the digitised newspaper articles, its younger sibling looked for inspiration amongst Trove’s other zones – sharing books, journals, photos, maps and more.

In 2015, Steve Leahy unleashed @TrovePenguinBot upon the world, searching for sardines amongst the digitised newspapers. In 2016, one of my students at the University of Canberra modified the NYPL Emoji Bot code to create @TroveEmojiBot – if you tweeted an emoji at the bot, it would respond with a suitably-themed newspaper article. In 2017, the Digitise the Dawn campaign bot-ified their Twitter account, posting an article each day from Louisa Lawson’s journal, The Dawn. Meanwhile, @astrove_bot started sharing newspaper articles relating to astronomy.

And then there was The Vintage Face Depot…

Things get weird

I’d been experimenting for a few years with faces as a way of connecting to GLAM collections –as alternative entry points, based not on metadata but the people inside. In 2015, this led me to create The Vintage Face Depot. If you tweeted a photo of yourself to @facedepot, the bot would select a face at random from a collection I’d compiled from Trove newspapers and superimpose that face over yours, tweeting you back the result and a link to the original article, so you could find out more about the person you’d been matched with.

@facedepot in action – a slide from my keynote presentation ‘Unremembering the forgotten’ for the Alliance of Digital Humanities Organizations Annual Conference in 2015

Now, in a time of deep fakes and AI generated images, @facedepot’s efforts seem quaint and kludgy. But that was always the point. I wanted to mess around with the barriers that put some people on the other side of this wall we call the past – to explore what historian Devon Elliot suggested on Twitter was an ‘uncanny temporal valley’. As I argued in The Perfect Face, a presentation at NDF2015:

The Vintage Face Depot tells you nothing about yourself. I built it at about the same time as Microsoft launched their How-Old bot that uses machine learning to estimate your age. Face Depot does nothing clever, and yet sometimes the results are uncanny, even unsettling. Microsoft might be able to tell you how old you are, but Face Depot asks who you are and pushes you in the direction of a past life, linked merely through chance.

Glitch bots for all

While I’d shared some bot-building code, rolling your own bot still required access to a web-connected server – a significant barrier for most would-be experimenters. This changed in 2017 with the arrival of Glitch, a platform that enabled anyone to build simple web apps for free. Perhaps most importantly, Glitch apps were remixable – simply by clicking a button, you could open an editor and create your own customised version of any app.

Glitch seemed like an ideal environment in which to experiment with bots, so I created four remixable Trove Twitter bot recipes:

trove-collection-bot – sharing resources from a partner collection
trove-list-bot – sharing items from a Trove list
trove-title-bot – sharing articles from specific newspapers
trove-tag-bot – sharing items with specific tags

These were supported by a detailed tutorial that walked through the process of customisation and suggested ways in which the basic recipes could be extended – for example, by adding a specific search query to a title bot.

This was the beginning of the bot explosion, with more than 30 Trove Twitter bots born between 2017 and 2019.

One of these, @NTTimesGazette, was created by curator and journalist Caddie Brain to tweet articles from the Northern Territory Times and Gazette. The bot was featured on ABC radio in Darwin under the headline: Twitter bot offers a rare look inside the Darwin’s forgotten first newspaper.

Historian Brett Holman created a series of bots related to aviation history. More than just a source of amusement, the bots became part of Brett’s research practice, as described in his History Australia article '@TroveAirRaidBot, a 24/7/365 research assistant'.

Perhaps the best part of this bot-making extravaganza was the number of self-professed ‘non coders’ who were able to take their first steps into the world of programming and actually create something. I have memories of sitting in the shade at Canberra’s now defunct Big Splash water park, troubleshooting someone’s Twitter bot on my phone, while the kids played on the water slides – it was fun, and it was exciting. Together, Trove, Twitter, and Glitch opened up new possibilities for learning and experimentation, and new ways of knowing Australia’s cultural heritage.

2019 bot roll call

As new bots emerged, I added them to my Trove bots Twitter list (here’s a partially archived copy).

You can get an idea of their diversity from the bot names – a mix of collections, subjects, and places. Here’s a list of Trove Twitter bots active in 2019:

astrove_bot
AustWWBot
BotCBR_QLD
CatsofTrove
digitisethedawn
DoSonTrove
facedepot
Kasparbot
KellyGangBot
LAAL_bot
NTTimesGazette
PenrithPictures
RemixHistorical
suthlib
TroveAirBot
TroveBot
TrovecakeBot
TroveCHIAbot
TroveDutchbot
TroveEmojiBot
trovefacesbot
TroveHoroscopes
Troveknitbot
Trovelandbot
trovelistbot
TroveMirrorBot
TroveNewsBot
TrovePenguinBot
TroveRefereeBot
trovesportsmel
trovetribunebot
TroveXmasBot
TsvBulletinBot
WomenAtWarBot

I also overhauled @TroveNewsBot in 2019, adding a number of new features, including article thumbnails.

Decline and fall

This golden age of bot-making came to an end late in 2019.

The first blow came when Trove updated its API. The bots needed some way of selecting random items from the millions available on Trove. This was fairly easy with version one of the API, but version two overhauled the way you accessed items within the result set, making random selections impossible. I eventually managed to hack together a random-ish method that added multiple facets to whittle down the results set until a selection could be made. Using this method, I created new versions of my Glitch bot recipes and updated the tutorial. But it seemed that the moment had passed, and many bot authors just let their creations die when version one of the API was switched off.

Surviving bots faced further challenges when Glitch started imposing limits on its free services. Glitch apps were designed to sleep when not in use, so to get your bot tweeting you had to fire regular web requests at it using a cron service. Glitch blocked access by these services and introduced a paid tier for ‘always on’ apps. More bots died as a result.

I was thinking about switching my recipes from Glitch to GitHub, making use of templates and scheduled actions. But while I prevaricated, Twitter started on its long, drawn out death spiral – first imposing new limits on API use, and later becoming the preferred networking site for nazis and transphobes. It was no place for creative bot-making.

The serious side of serendipity

Bot-making wasn’t just about fun – Trove Twitter bots had a serious purpose as well. In ‘Unremembering the forgotten’ I wrote:

Twitter bots can interrupt our social media meanderings with pinpoints of surprise, conflict, and meaning. And yet they are lightweight, almost disposable, in their development and implementation. No committees were formed, no grants were obtained—they are quick and creative: hacks in the best sense of the word. Bots are an example of how digital skills and tools allow us to try things, to build and play, without any expectation of significance or impact. We can experiment with the parameters of access.

A number of articles on the value of serendipity have considered how collection bots, like @TroveNewsBot, can puncture our research expectations. The random offerings of bots might offer new modes of discovery. In ‘Technologies of Serendipity’, Paul Fyfe argues:

For scholars or other readers, discovery results less from directed searching than from all the tangents encountered on the way. Thus, sources which are plural, redundant, and tangent-rich help promote discovery by the proliferating contingencies of their usage.

Similarly, Brett Holman notes that his own Trove bots help him make connections in his research:

By impinging on my consciousness when I am preoccupied by other things, @TroveAirRaidBot’s tweets draw my mind back to this research topic that is always sitting at the back of my mind somewhere, and it makes me make connections – randomly, haphazardly, but often very fruitfully leading me to think of something I hadn’t thought of before, or reminding me of something I’d forgotten, or juxtaposing some seemingly unrelated things. It’s a kind of directed serendipity.

Trove Twitter bots were also entry points and interventions – challenging our understanding of access. They offered playful demonstrations of how our experience of GLAM collections might be different. Mitchell Whitelaw suggested that such creations:

reflect an emerging interest in collections as active sites of meaning-making, and experimentation with how we might encounter such collections in an everyday digital environment.

In ‘Life on the outside’, I considered the lives that GLAM collections might lead beyond institutional confines:

These bots do not simply present collection items outside of the familiar context of discovery interfaces or online exhibitions, they move the encounter itself into a wholly new space. … Twitter bots loosen the institutional context of collections to allow them to participate in a space where people already congregate. They send collection items out into the wilds of the web, to find new meanings, new connections and perhaps even new love.

The promise of serendipitous discovery has now faded with the poisoning of social media spaces, and the retreat of many GLAM organisations from experimentation and openness. The need to control now carries more weight than the gift of creativity.

What remains

I migrated @TroveNewsBot to the Fediverse in May 2023, but sadly it was killed when NLA gatekeepers cancelled my Trove API keys without warning in January 2025.

A number of other Trove bots have survived the Twitter implosion and found their way to alternate platforms. @DigitiseTheDawn now shares articles on the Fediverse, while @TrovePenguinBot is pursuing sardines on Bluesky. Brett Holman has created new versions of his aviation-themed bots – @TroveAirBot, @TroveAirRaidBot, and @TroveUFOBot – on Bluesky. I’d be happy to add the details of any other survivors I might have missed.

In an odd coincidence, recent months have brought new restrictions on access to Trove API keys, and an announcement of the end of Glitch. There’s no going back.

ActivityPub and the Fediverse seem to offer new digital channels through which collections might flow and connect. See, for example, Aaron Straup Cope’s work at the SFO Museum. But how do we support and encourage this type of experimentation?

Personally speaking, this year’s been pretty shit so far, and I’ve been having trouble finding any motivation. But in pulling together these notes I found a section in ‘Unremembering the forgotten’ that reminded me of what’s at stake:

There is no open access to the past. There is no key we can enter to recall a life. I create these projects not because I want to contribute to some form of national memory, but because I want to unsettle what it means to remember: to go beyond the listing of names and the cataloging of files to develop modes of access that are confusing, challenging, inspiring, uncomfortable, and sometimes creepy.

There’s still plenty of work to do.

Some Archives Week goodies

Wed, 11 Jun 2025 17:40:11 +1100

It’s International Archives Week and I’m feeling a bit crook after being double-vaxxed yesterday, so instead of doing something productive, I’m just going to make a list of potentially handy archives-related resources from the Wonderful World of Wragge(TM).

The theme of Archives Week is #ArchivesAreAccessible, which you’d have to regard as rather aspirational given the various ways access is limited by law, policy, practice, technology, and history. But what the heck, discussions about the meaning of access are always welcome. It’s also a little jarring to see the #ArchivesAreAccessible theme being promoted by the National Archives of Australia just a few weeks after they implemented new restrictions that make it impossible to get machine-readable data out of their online database, RecordSearch. But I’m trying to move on, so…

Zotero

All Australian archives users should have Zotero installed. Through the magic of user-contributed ‘translators’, Zotero can capture structured data and digitised images from a variety of collections, including:

The first four are my work, so let me know if you have any suggestions or problems.

Indexes to records

Archives are well represented in the GLAM Workbench’s list of GLAM datasets shared through government open data portals. Many of these datasets are indexes that link records to people and places. They’re openly licensed and under used!

NSW State Archives has also compiled a lot of indexes. These aren’t shared through a portal, but you can harvest them from their website. To save you the effort, I’ve created a repository of the harvested indexes.

I’ve pulled many of these sources together to build a mega database of name indexes that lets you search for people across millions (yes millions) of records. As well as the sources described above, it also includes people-related records from the Public Record Office Victoria’s API. All together, the GLAM Name Index Search contains almost 13 million records in 293 datasets from 10 GLAM organisations. And unlike the commercial genealogical databases, it’s free! (If only Australian libraries and archives would link to it from their family history guides…)

Other datasets

Before my scrapers were scuppered by the NAA, I managed to compile a few datasets. Much of this data documents the way RecordSearch itself has changed, and while it might not be of use to researchers seeking particular records, it could help future researchers who are trying to understand the impact of online collections on the practice of history. This includes:

Summary data about all series in RecordSearch – a CSV file containing basic descriptive information about all the series currently registered on RecordSearch as well as the total number of items described, digitised, and in each access category. Harvests from May 2021 and April 2022 are currently available, and I’ll soon be adding May 2025.
Files digitised by the National Archives of Australia since 2021 – annual compilations of data harvested from RecordSearch’s list of recently digitised files. (The automated weekly harvests are now dead.)
Records held by the National Archives of Australia with the access status of ‘closed’ – a whole decade of annual harvests of records held by the NAA that have the access status of ‘closed’ (withheld from public access). The harvests were run on or about 1 January each year from 2016 to 2025. The aim in saving this data is to enable long-term analysis of the NAA’s access examination process.

Thanks to support from Wikimedia Australia, I’ve also added information about Australian government agencies from RecordSearch to WikiData. As a result you can get a list of Australian government departments since Federation using this Wikidata query. I’ve used the data to build this interactive visualisation of the relationships between government departments. There’s some more examples in the Wikidata section of the GLAM Workbench.

Also from the NAA is my collection of #redactionart found in ASIO surveillance files.

As part of the Real Face of White Australia project, we’ve been transcribing records created by the administration of the White Australia Policy, now held by the NAA. Some of the results are available in this data repository. (Note to self – I need to update this with the latest data!)

The ANU Archives section of the GLAM Workbench includes some datasets extracted from the Sydney Stock exchange stock and share lists. (Just noticed some CloudStor links in there that I need to fix…)

Public Record Office Victoria

PROV gets it’s own section because, as far as I know, they’re the only Australian archives with a functioning public API. (Brief moment of silence to remember the APIs that have come and gone over the years…).

There’s now a PROV section of the GLAM Workbench, that includes a ‘getting started’ notebook to document the basic functionality of the API. There’s some more information in this blog post.

I’ve also used the PROV API to create an automated data dashboard that provides an overview of their collection. It’s updated every Sunday.

Other things

RecordSearch users will understand the frustration of trying to share a url to a record, only to get an annoying error. There are a few ways around this (Zotero saves persistent links to things you save), but for a quick fix I created a simple tool to create persistent links in RecordSearch.

If you’re comfortable with a little browser hacking, you can also install this handy RecordSearch userscript (scroll to the bottom for installation instructions). It improves the functionality of RecordSearch in a few different ways, such as by indicating the number of pages in a digitised file.

Any ideas?

If you have any ideas for additional resources or datasets, or you’re having problems with an online collection, feel free to drop a note in the GLAM Workbench repository.

New dataset – Trove links shared on Twitter, 2009 to 2020

Tue, 10 Jun 2025 12:30:54 +1100

A few years ago, I harvested the details of tweets that included links to Trove. The data has just been sitting on my computer, so I thought I should package it up and share, in case it’s of use to anyone.

The story is that back in 2021, I was working on the article ‘More than newspapers’ for a special section of History Australia focusing on Trove. I was thinking that I might include something about the way Trove newspaper articles were mobilised within online discussions about history – a topic I first explored in ‘Life on the outside: connections, contexts, and the wild, wild web’, my keynote for the Annual Conference of the Japanese Association of Digital Humanities in 2014. In the end, the article went in another direction, so I didn’t use the data.

I remembered this recently and thought I should I should do something with it. I’ve now created a dataset and shared it on Zenodo. I’m not working on Trove any more, but I’m hoping that someone else might find the data useful!

The dataset contains information about tweets from 2009 to 2020 that include links to Trove. The tweet data was compiled using Twarc in May 2021, under Twitter’s academic access program. The search queries used were:

url:nla.gov.au/nla.news
url:trove.nla.gov.au
url:newspapers.nla.gov.au

Many of the tweets were produced by bots. Fortunately, I’d been maintaining a list of Trove bots on Twitter, so I used the list to separate the tweets into two files, one for bots and one for ordinary users.

To respect user intentions and comply with the Twitter API terms of use, I removed all the tweet information except for tweet_id and tweet_date from the files. If it hasn’t been deleted, the full data for each tweet can probably be obtained from the X API using the tweet_id, though you might need a paid subscription.

The two main files are:

trove_url_tweets.csv – links shared by human users (although it may include some unidentified bots)
trove_url_tweets_bots.csv – links shared by bots

I also created some additional data files:

trove_url_totals.csv – the number of times each Trove link was shared by users (not including bots)
active_users_per_year.csv – the number of unique users each year who shared a link to Trove
active_bots_per_year.csv – the number of active bots each year sharing links to Trove

There’s more information about the structure and contents of the data files in the Zenodo record.

Overview

I haven’t explored the data in detail, but here’s some quick summaries to give you a taste.

summary
number of unique users sharing Trove links	9,296
number of bots sharing Trove links	43
number of tweets by humans containing Trove links	48,323
number of tweets by bots containing Trove links	318,767
number of unique links shared by humans	36,906
number of unique links shared by bots	270,474

What types of links were people sharing?

types of link shared by humans	count
newspaper article	34,568
other (search queries, home page etc)	8,388
work (items other than newspapers – books, maps, photos etc)	4,856
newspaper page	1,378
newspaper title	406

How did the number of links shared by humans vary across time?

Which articles or pages were shared most often by humans? Here’s the top ten (click on the link to view).

trove_id	trove_type	tweets	retweets	quotes	total times shared
75869223	article	1,232	61	34	1,327
1298497	article	141	1,028	53	1,222
102074798	article	74	693	77	844
68141866	article	138	522	48	708
41602327	article	633	30	0	663
100645214	article	111	467	20	598
502650	page	1	513	12	526
60828173	article	48	444	19	511
4173156	article	53	321	10	384
79410604	article	2	303	69	374

The most shared article reports that PM Menzies had described Hitler as a ‘great man’ at a meeting in July 1939. However, most of the tweets sharing this link came from a single user. A number of the other articles relate to the weather, a reflection of the fact that Trove’s newspaper articles have been mobilised on both sides of the climate change debate.

How many Twitter users were sharing links to Trove each year?

I haven’t included any of the bot data in these summaries because I think I’ll write a second bot-themed post – coming soon!

Updates

I updated the data in this post on 19 June 2025, as I realised some Twitter accounts were originally run by humans before being bot-ified.

GLAM Workbench – preprint for 'Building User-Friendly Toolkits and Platforms for Digital Humanities'

Thu, 05 Jun 2025 16:16:53 +1100

This is a preprint of my contribution to the publication ‘Building User-Friendly Toolkits and Platforms for Digital Humanities’. It provides a brief overview of the GLAM Workbench. I had to leave a lot out, but hopefully it provides a useful summary of what the GLAM Workbench is, and what I’d like it to be.

The GLAM Workbench is a collection of tools and resources created to help researchers use and explore the digital collections of GLAM organisations (galleries, libraries, archives, and museums).¹ It’s mainly focused on collections from Australia and New Zealand, but some sections venture across international boundaries to explore topics such as web archives and Wikidata.

GLAM organisations make a lot of rich cultural data available online, but getting that data in a machine-readable form that can be aggregated and analysed is often difficult. The GLAM Workbench tries to fill this gap by providing code examples and API documentation, but data access alone is not enough. Researchers need to understand the history, structure, and extent of the data – both its limits and its possibilities. By sharing snapshots, building overviews, and exploring patterns and inconsistencies, the GLAM Workbench also attempts to contextualise GLAM collections and open them to new types of questions.²

History and motivation

I created the GLAM Workbench in 2017, but it incorporates the latest versions of tools, such as the Trove Newspaper Harvester, which I’ve been maintaining for more than 15 years.³ One of my motivations was simply to bring together useful snippets, notes, and doodles from a variety of blog posts, web applications, and code repositories, and make them available in a form that could be more easily navigated and maintained.

I was also keen to explore the way that Jupyter notebooks combine code and narrative. I wanted to find ways to support researchers as they developed their digital skills and confidence, not just dump them at the command line or point them to an app.

The ongoing development of the GLAM Workbench is also part of my own research. I’m interested in the meaning of access within the context of GLAM collections. What changes when you can download data and explore collections beyond the limitations of the web interface?⁴

Contents and technologies

At its heart, the GLAM Workbench comprises at least 171 Jupyter notebooks and 59 datasets shared through more than 70 GitHub repositories.⁵ Added to this are a number of web apps, online databases, and guides to related resources. Code from some notebooks has also been spun off into independent Python packages. All of this is brought together within a single documentation site, built using MkDocs Material.

The contents are mostly organised by institution, reflecting the idiosyncrasies of the data. I’ve partially implemented tags to draw together similar resources across institutions, but this needs to be made more consistent, ideally using the TaDiRAH taxonomy.⁶ Many of the notebooks describe methods for accessing data and building datasets. Others demonstrate techniques for visualisation and analysis, suggest workarounds for limits imposed by collection interfaces, or provide example-driven documentation for APIs and datasets.

There is no single platform or server underlying the GLAM Workbench. Instead, it follows a pattern described in the ARDC Community Data Lab’s architecture principles as ‘infrastructure at rest’.⁷ Notebooks can be run as required in a variety of contexts from cloud services to local computers. This is made possible by standardised configuration files and automated processes that build virtual computing environments from each GitHub repository.⁸

Impact and engagement

The GLAM Workbench has helped to expand understanding of the research possibilities of GLAM collection data. The list of publications citing the GLAM Workbench or one of its embedded tools now includes more than 100 entries.⁹ Some of these relate to individual research projects, while others survey the practices of GLAM organisations and the needs of research infrastructure around the world.

My work on the GLAM Workbench has helped inspire organisations such as the National Library of Scotland to explore new ways of supporting digital research.¹⁰ A recent report from the ‘Towards a National Collection’ project in the UK has mentioned the GLAM Workbench alongside a number of national libraries in Europe and the USA for ‘encouraging innovative research and expanding public engagement with heritage resources’.¹¹

And yet, there are disappointments. Most of the Australian GLAM organisations whose collections are featured in the GLAM Workbench have shown little interest in sharing or engaging with its resources. This makes it difficult to get tools to the people who could benefit from them. There’s some irony in the fact that the websites of the National Library of Scotland, the British Library, the UK National Archives, the V&A Museum, and DigitalNZ all include links to the GLAM Workbench, but the National Library of Australia (NLA) and the National Archives of Australia (NAA) do not.

Maintenance and sustainability

While a number of individuals have contributed notebooks and additions to the GLAM Workbench, it remains essentially a one man operation. Over the years, I’ve sought to ease the maintenance burden by automating processes, adding some basic testing frameworks, and generating machine-readable metadata that summarises the contents of each repository. For example, I created a GLAM Workbench repository template that makes it easy to start work on a new topic.¹²

Development of the web archives section of the GLAM Workbench was made possible by a grant from the International Internet Preservation Consortium, and the section’s ongoing maintenance is supported by the British Library.¹³ I’m grateful too for my GitHub sponsors who help cover some of my cloud hosting bills, and to the ARDC for funding to integrate RO-Crate metadata.¹⁴ But beyond this, the GLAM Workbench has received no dedicated funding or institutional support. It has, nonetheless, outlived some well-funded digital infrastructure projects in the HASS sector.

Sustainability means more than money, though. The GLAM Workbench doesn’t have to continue in its current form to have a long-term impact. My focus is on ensuring that its contents are open to future reuse and modification. Everything is openly licensed, published through GitHub, and preserved in Zenodo. If tools are useful they can live on, independent of me.

The future

I’m writing this at a difficult time. Changes wrought by the NLA and NAA in early 2025 have made it impossible for me to continue work on the Trove and RecordSearch sections of the GLAM Workbench.¹⁵ In the Trove section alone, there are more than 70 notebooks.

The GLAM Workbench is not my job, no-one pays me. I work on it because I think its useful and important, and because I enjoy the process of solving problems and helping researchers. The NLA’s actions, in particular, have robbed me of that joy, and made me consider whether I want to continue. Research infrastructure is people.

On the other hand, there are many more GLAM collections for me to explore. I’m also hoping to find new ways of collaborating with individuals and institutions. I’m often inspired to create new tools and resources by gnarly questions from researchers. While such questions continue, so the GLAM Workbench will grow.

References

Ames, Sarah, and Lucy Havens. “Exploring National Library of Scotland Datasets with Jupyter Notebooks.” IFLA Journal, December 27, 2021. doi.org/10.1177/0….

Bailey, Rebecca, Javier Pereda, Chris Michaels, and Tom Callahan. “Unlocking the Potential of Digital Collections. A Call to Action.” Towards a National Collection, November 21, 2024. doi.org/10.5281/z….

Candela, Gustavo, Sally Chambers, and Tim Sherratt. “An Approach to Assess the Quality of Jupyter Projects Published by GLAM Institutions.” Journal of the Association for Information Science and Technology 74, no. 13 (2023): 1550–64. doi.org/10.1002/a….

“GLAM Workbench (GitHub Organisation).” Accessed June 5, 2025. github.com/GLAM-Work….

IIPC. “Asking Questions with Web Archives – Introductory Notebooks for Historians.” Accessed June 5, 2025. netpreserve.org/projects/….

Jackson, Andy. “GLAM Workbench Update.” UK Web Archive Blog. Accessed June 2, 2025. blogs.bl.uk/webarchiv….

Sefton, Peter, Tom Honeyman, Tim Sherratt, and Conal Tuohy. “The ARDC Community Data Lab Architecture: Research Software Deployment Principles and Patterns for Integrity, Reproducibility and Sustainability,” May 10, 2024. zenodo.org/records/1….

Sherratt, Tim. “Develop a New GLAM Workbench Repository.” GLAM Workbench. Accessed June 5, 2025. glam-workbench.net/get-invol….

———. “Farewell Trove.” Tim Sherratt – Sharing Recent Updates and Work-in-Progress, May 7, 2025. updates.timsherratt.org/2025/05/0….

———. “GLAM Workbench.” Zenodo, June 5, 2025. doi.org/10.5281/z….

———. “GLAM Workbench.” Accessed June 5, 2025. glam-workbench.net/..

———. “GLAM Workbench Citations.” GLAM Workbench. Accessed June 5, 2025. glam-workbench.net/citations….

———. “Hacking Heritage: Understanding the Limits of Online Access.” In The Routledge International Handbook of New Digital Practices in Galleries, Libraries, Archives, Museums and Heritage Sites, edited by H Lewi, W Smith, S Cooke, and D vom Lehn, 116–30. London & New York: Routledge, 2020. doi.org/10.5281/z….

———. “No More Harvesting Data from the National Archives of Australia.” Tim Sherratt – Sharing Recent Updates and Work-in-Progress, May 19, 2025. updates.timsherratt.org/2025/05/1….

———. “Some Important Updates for the Trove Newspaper & Gazette Harvester.” Tim Sherratt – Sharing Recent Updates and Work-in-Progress, August 31, 2023. updates.timsherratt.org/2023/08/3….

———. “Supporters.” GLAM Workbench. Accessed June 5, 2025. glam-workbench.net/get-invol….

———. “Trove Newspapers: Data Dashboard.” Accessed June 5, 2025. wragge.github.io/trove-new….

———. “Trove-Newspaper-Harvester.” Python, October 23, 2023. doi.org/10.5281/z….

Sherratt, Tim, Harry Keightley, Ben Foley, and Michael Niemann. “GLAM-Workbench/Glam-Workbench-Template.” Python. GLAM Workbench, August 24, 2023. github.com/GLAM-Work….

“TaDiRAH The Taxonomy of Digital Research Activities in the Humanities.” Accessed June 5, 2025. tadirah.info/..

Talboom, Leontien, and Mark Bell. “Keeping It under Lock and Keywords: Exploring New Ways to Open up the Web Archives with Notebooks.” Archival Science, July 4, 2022. doi.org/10.1007/s….

“Trove Historical Data.” Accessed June 5, 2025. zenodo.org/communiti….

Sherratt, “GLAM Workbench.” ↩︎
See, for example: Sherratt, “Trove Newspapers: Data Dashboard.” and “Trove Historical Data.” ↩︎
Sherratt, “Trove-Newspaper-Harvester.” ↩︎
See, for example: Sherratt, “Hacking Heritage: Understanding the Limits of Online Access.” ↩︎
“GLAM Workbench (GitHub Organisation).” ↩︎
“TaDiRAH The Taxonomy of Digital Research Activities in the Humanities.” ↩︎
Sefton et al., “The ARDC Community Data Lab Architecture.” ↩︎
For more on best practices in sharing Jupyter projects, see: Candela, Chambers, and Sherratt, “An Approach to Assess the Quality of Jupyter Projects Published by GLAM Institutions.” ↩︎
Sherratt, “GLAM Workbench Citations.” ↩︎
Ames and Havens, “Exploring National Library of Scotland Datasets with Jupyter Notebooks.” For another example of the GLAM Workbench’s influence, see: Talboom and Bell, “Keeping It under Lock and Keywords.” ↩︎
Bailey et al., “Unlocking the Potential of Digital Collections. A Call to Action,” 58. ↩︎
Sherratt et al., “GLAM-Workbench/Glam-Workbench-Template.” For documentation see: Sherratt, “Develop a New Repository.” ↩︎
“Asking Questions with Web Archives – Introductory Notebooks for Historians.”; Jackson, “GLAM Workbench Update.” ↩︎
Sherratt, “Supporters.”; Sherratt, “Some Important Updates for the Trove Newspaper & Gazette Harvester.” ↩︎
Sherratt, “Farewell Trove.”; Sherratt, “No More Harvesting Data from the National Archives of Australia.” ↩︎

Tim Sherratt

Generosity in practice – a chat with Paula Bray at the State Library of Victoria

Zotero translator for Libraries Tasmania updated!

Exploring georeferenced maps from the SLV collection

The background

The interface

The future?

my place – exploring SLV collections through a street address

The collections

OpenStreetMap

Sands & MacDougall’s directories

Committee for Urban Action photographs

Georeferenced maps

Parish maps

Newspapers

Photographs

The interface

my house

my street

my suburb/town

nearby

The data

Next steps?

Geolocating photos from the SLV's Committee for Urban Action collection

The process

Finding intersections

Finding road segments

Problems finding intersections

Assembling the data

Constructing the interface

More information and links

Goodbye 2025! A brief summary of the highlights and lowlights…

Highlights

Lowlights

Next year

How you can help

Exploring Victorian newspapers

Assembling the data

Building the apps

Why bother?

Counting down... (to the end of my SLV residency)

Some Sands & Mac tweaks thanks to ALTO and IIIF

A new way of searching Sands & Mac

Technical information follows…

Downloading the PDFs

Downloading the ALTO files

Assembling and publishing the database

More information

Turning the SLV's maps into data with Allmaps and some GLAM plumbing

Me at 63...

What was your first history related job? What path have you taken since then?

What kind of work have you done? What are you working on now?

Research or writing? (What do you enjoy more and why?)

What are the best and hardest things about the kind of work you do?

Creating bounding boxes for parish maps in the SLV collection

The metadata

The method

Creating bounding boxes

Limitations

Visualising the results

Exploring SLV urls

Permalinks

Structured data

IIIF and images

Handles

From old to new

The process of GLAM hacking

Creative Technologist-in-Residence at the State Library of Victoria!

WikiFest at the State Library of Victoria

Bonus userscript

GLAM hacking with userscripts

Some old userscripts

And a new userscript to improve Trove lists

Using userscripts

The rebirth of Wragge Labs (and moving my Heroku apps)

A new (old) home

Moving house

Redirecting Heroku

More experiments to come?

The future of the past... in the present

GLAM Workbench – preprint for 'Building User-Friendly Toolkits and Platforms for Digital Humanities'