Tim Sherratt

Honoured to be 'the most dedicated Labber'

Wed, 15 Jul 2026 16:41:38 +1000

It can be pretty hard working on your own most of the time – unsure if what you’re doing is of any value or use. As I said in my keynote at the GLAM Labs Futures conference in Edinburgh recently:

the first half of last year was pretty bleak, and left me wondering whether I should just walk away from all of it – go bushwalking, catch up on gardening, perhaps just be a historian again.

So I was grateful for the opportunity to go to Edinburgh, find out what was happening in GLAM Labs around the world, connect with people I’d only met online, and rebuild my energy and enthusiasm.

What I didn’t expect was to be given a special award for being ‘the most dedicated Labber’!

The full text of the award reads:

We are thrilled to acknowledge and honour the incredible contributions of Dr. Tim Sherratt.

For many years, Tim has been a true pioneer, operating as a visionary ‘one-person lab.’ His GLAM Workbench Is an extraordinary repository of tools, tweaks, and thoughtful walkthroughs. It has served as sheer inspiration for anyone engaging in computational experiments with GLAM collections.

This award is a testament to his immense generosity in sharing his knowledge and skills, and for always championing the spirit of ‘good hacking!’

The award itself was hand-crafted in suitably experimental fashion by Olga Holownia.

It even glows in the dark! Olga assures me it’s only slightly poisonous.

After a difficult couple of years it means a lot to be recognised by my international peers in this way, and reminds me that I’m never really working alone.

Hacking the archive – GLAM Labs workshop at the University of Edinburgh

Wed, 15 Jul 2026 15:53:18 +1000

On 24 June I gave a ‘Hacking the archive’ workshop at the Edinburgh Futures Institute, University of Edinburgh. The workshop was organised by the EFI, the National Library of Scotland, and the IIPC. It preceded the GLAM Labs Futures conference, also held at the EFI.

The workshop was an expanded version of the ‘hacking the library’ session I ran for the State Library of Victoria’s Code Club in September last year. The aim of the workshop was first to give participants the confidence to poke around in websites – to play with urls, look under the hood to see how web pages are put together in the browser, and think about how they might be different. We then explored how you can change the way web pages look and work using bookmarklets and userscripts.

The slides are fairly detailed, and you can probably try a number of the activities yourself if you’re interested.

The bookmarklets and userscripts we experimented with all change aspects of the NLS website – such as inserting links or images. They’re all available in this GitHub repository and could be modified to work with other sites.

The warm weather and jet lag meant I needed to be revived at various points by ice blocks (thanks Olga!), but it was a fun session, and by the end it was great to see people planning and building their own website hacks!

Some stats describing the National Archives of Australia collection

Wed, 15 Jul 2026 14:20:48 +1000

What’s actually in the National Archives of Australia? And how much of that has been described, digitised, or opened to the public? I’ve made a few attempts at answering these questions over the years by harvesting summary statistics about every series in RecordSearch. I’ve now saved all this data in Zenodo. The dataset description reads:

This dataset contains information and statistics describing record series held by the National Archives of Australia (NAA). The data was harvested from RecordSearch, the NAA’s online database, using this notebook.

Under the Commonwealth Record Series System a ‘series’ is defined as ‘a group of records that has resulted from the same accumulation or filing process or that has a similar format or information content’. Series are created by government agencies, and contain any number of items. The NAA uses series to organise, describe, and manage its holdings. For example, Series A1 contains over 60,000 correspondence files created by the Department of External Affairs and its successors between 1903 and 1938.

This dataset aims to provide an overview of the NAA’s holdings by compiling basic information about each series. It contains four data harvests created in late 2016, May 2021, April 2022, and May 2025. By comparing these different data files it is possible to observe changes in the overall shape of the collection, such as how many items are described, open to the public, and digitised. This should help future researchers explore the impact of digital access on historical research. For example, the 2016 data was analysed in Sherratt, ‘Hacking Heritage: Understanding the Limits of Online Access’, 2019.

I’ve also created a Zenodo community for National Archives of Australia historical collection data which includes this dataset, as well as annual harvests of ‘closed’ files from 2016 to 2025, and details of files digitised since 2021.

Unfortunately, due to changes in RecordSearch in 2025 it is no longer possible to harvest this sort of data.

The future of online archives in 2009

Wed, 15 Jul 2026 12:23:12 +1000

Way back in 2009, I was working in the web content team at the National Archives of Australia (NAA). We’d been doing some pretty interesting work on projects like Mapping Our Anzacs, that brought together map-based finding aids and crowdsourced contributions. So when the NAA was asked to report on ‘emerging technologies for access’ by the Standards for Public Access Working Group of the Council of Australasian Archives and Records Authorities (CAARA) the job ended up with me.

In my usual way, I expanded the scope of the report until it was ridiculously over-ambitious, then had to cut it all back to try and get it finished, so I was never really satisfied with the end result. It also wasn’t a very happy time at the NAA, as the web content section was in the process of being disbanded (the usual combination of internal jealousies and fucked up management) and I was grappling with sarcoidosis. As soon as I submitted the report, I left the NAA for a placement at the National Museum of Australia and never went back.

I have no idea whether the report was ever submitted to CAARA. I published it on Scribd to get it out in the world and it’s sat there ever since. However, Scribd has become increasingly enshittified, so I thought I should finally put a copy in a real, public repository. Here it is in Knowledge Commons – Emerging Technologies for the Provision of Access to Archives: Issues, Challenges and Ideas.

Read/download on Knowledge commons

Many of the links are broken, of course, but quite a few of the issues still seem relevant, even if our optimism has faded over the years. In my recent GLAM Lab Futures keynote I suggested we should share and celebrate our own histories as a sort of counter balance to the relentless push of ‘progress’. So in that spirit, here’s one perspective on the future of online archives from 17 years in the past.

The future of the past: GLAM innovation and the responsibilities of history

Tue, 30 Jun 2026 00:08:58 +1000

Keynote presented to the GLAM Labs Futures conference, Edinburgh, 26 June 2026. View the full set of slides.

In 2012, I was lucky enough to be a Harold White Fellow at the National Library of Australia, working with data from digitised newspapers available through the Library’s innovative online service, Trove. I think it was the first time one of the fellowships had been awarded to a digital project.

I’d been playing around with Trove data for a couple of years, and had created tools to harvest and visualise searches in the digitised newspapers. There was no API back then, so everything was precariously balanced atop a series of screen scrapers. At one point, I actually pushed my scraper code to Google’s AppEngine to provide an ‘unofficial’ API.

My fellowship project drew on some of the themes of my history PhD, which had examined ideas of progress in 20th century Australia. My plan was to use the newspapers to explore how people over the past 150 years had imagined ‘the future’. I’m sure it’ll surprise no-one here to learn that I spent most of the three months of my fellowship trying to clean up OCR errors – few historical insights were forthcoming.

However, there was one experiment that I still enjoy. I harvested a collection of 40,000 articles that included the phrase ‘the future’, and grouped them by year. Then I found the words for each year that had the highest TF-IDF values – so not the most common words, but the words that were most distinctive when compared with the whole collection. I was running this process late at night, and started sharing the results over Twitter. The extracted words seemed evocative, almost poetic – so I decided to make something that people could use to craft their own odd little poems from the dataset.

This was ‘The Future of the Past’. The interface was obviously inspired by fridge magnet poetry. You drilled down through randomly selected, TF-IDF weighted words until you reached a year. Then you dragged words around to create your poems and share them on Twitter. It was also a way of exploring the collection. Words were linked to articles, which all linked back to Trove.

Over the years, the application gradually rusted, seized, and fell apart. It was running in Django and MySQL and I think I missed some updates or database migrations. When my webhost stopped supporting Python, getting it working again just seemed too hard.

Last year I was doing some housekeeping, trying to bring a lot of my old apps an experiments together to reduce both the maintenance burden and my cloud hosting bills. I realised I could convert the old app to run in Flask and access its data from SQLite. Of course, Twitter had by then congealed into a slimy hellhole of neo-nazis and transphobes, so I also changed the sharing options to include Mastodon and Bluesky.

I thought I should also publish my fellowship lecture somewhere to provide a bit of context. The lecture has long since disappeared from the National Library’s website, but I managed to find a recording in the Internet Archive that I could transcribe and pop into Zenodo.

So ‘The Future of the Past’ lives again! An experiment examining how the past imagined the future, itself disappeared into history, until resurrected it in the present to provide a record for the future.

GLAM Labs, and GLAM innovation in general, occupy a complex position with respect to time. We explore how new technologies can be used to mobilise the past in the present. We imagine future audiences, and reconstruct past lives. We think about what’s coming next, but also what needs to be preserved.

In my talk today I want to think a bit about how we navigate time.

While researching my PhD, I found out that an Atomic Age exhibition toured Australian cities in 1947 and 1948. The exhibition included a diorama that represented the first atomic bomb test in the New Mexico desert. Emerging from the fireball, a mysterious figure loomed over the scientists – the atomic genie had been released and awaited our command. Would we use its powers to foster progress or wreak destruction?

This choice was made even more explicit by a signpost in the middle of the exhibition. ‘Progress’ pointed to a display of the possibilities of atomic energy in industry, while ‘Destruction’ directed visitors to a scale model of Hiroshima with a recorded soundtrack and flashing lights for that authentic atomic annihilation experience. The idea that humankind was at some sort of ‘crossroads’ was a common way of representing the challenges of the atomic age. We had arrived at a critical moment in history, when our decisions would determine the fate of civilisation itself.

So what happened? Did we choose? This was not the first turning point that humankind had faced, nor the last. In 1966, Elizabeth Eisenstein, a historian whose major work focused on the impact of the printing press, wrote about history and our perceptions of time. She argued that linear, episodic structure of ‘history book time’ dumps us at the opening of ‘the most personally significant, densely packed, fact-crowded final chapter’. The past trails off into irrelevance as we confront an unknown future full of unprecedented challenges. We are, she says, ‘destined always to be poised as an adult on the threshold of a new age, where previous experience offers no sure guide’.

The genie is out of the bottle, there’s no turning back.

The question is not whether real crises exist, but whether our perception of time helps or hinders our efforts to address them. We imagine ourselves in an eternal present where the past is closed off, and the future empty. Solutions always lay ahead.

It took a couple of attempts to finally complete my PhD. In between, I worked for the Australian Science Archives Project (ASAP), a small, self-funded organisation attached to the University of Melbourne. Our mission was to preserve and make accessible the history of Australian science, but with limited funds for outreach we had to be a bit creative. In the early 1990s, I started converting finding aids to plain text files and loading them on to FTP and Gopher servers. Then in 1994 we took the leap to a new, exciting online platform – the web.

The ASAP website was one of the first Australian history sites on the web, and certainly the first archives site in Australia. As well as newsletters and finding aids, we published a database with information about hundreds of Australian scientists and any related archival holdings – it was originally called Bright SPARCS. In the days before things like MySQL, this meant I had to figure out how to use Visual Basic to convert the Microsoft Access database into what we would now call a ‘static’ site – lots and lots of little HTML files.

After my second, successful PhD attempt, and some time as a postdoc researching the history of meteorology, I ended up back in the archives world at the National Archives of Australia (NAA). I was a member of the small web content team, and in 2008 I had an idea for a web application to accompany a new physical exhibition on Australia’s involvement in World War I.

For a non-Australian audience, I feel I need to explain at this point that World War I is still a big deal in Australia. The qualities of Australia’s fighting men – the Anzacs – were mythologised and woven into a particular vision of national identity that still wields considerable political and cultural power. The exhibition I worked on was designed to highlight the digitisation of 376,000 WWI service records, funded through a special allocation from the federal government, and presented as ‘a gift to the nation’.

The archivists who described the service records had the foresight to embed some structured data, such as places of birth and enlistment, in the file titles. So I suggested we extract the place names, geolocate them, and create a map interface for users to explore the records by location. Sounds pretty standard these days, but there was nothing quite like it at the time. We also collected photos and stories from users by setting up a ‘scrapbook’ in Tumblr and linking it to the map interface through the Tumblr API.

The site, named ‘Mapping Our Anzacs’ (not the choice of the development team), was popular with users who added more than 1,000 scrapbook posts in the first six months. It was also popular with politicians and bureaucrats who trumpeted it as an example of how ‘web 2.0’ might transform government services through digital innovation and online engagement.

It lasted about 6 years in its original form. In 2014, the content was rolled into a new site called ‘Discovering Anzacs’ with some additional records. That site was suddenly decommissioned in 2023, breaking all the links that people had made to individual records, and discarding all their contributions. The media release announcing the change pointed people to an ‘archived version’ in the Australian Web Archive. It was headed: ‘Discovering Anzacs website decommissioned, making way for innovative new digital experiences’. These new digital experiences have yet to emerge.

The past is closed off and the future is empty. The media release made it seem as if the change was inevitable – technology had simply moved on. There’s no escaping the fact that long-term maintenance is hard, but there are always choices to be made. Let’s not simply shrug and point to the pace of change as a way of avoiding responsibility.

Bright Sparcs, on the other hand, is still online. The original site is archived and its urls preserved. The content and identifiers have been rolled forward into the ‘Encyclopedia of Australian Science and Innovation’. After 32 years, it still works. That’s mainly due to the efforts of Gavan McCarthy, the former director of ASAP, who created the original database in the 1980s and continues to maintain it. There are always choices to be made.

In 2009, I was asked to reflect on ‘Mapping Our Anzacs’ for a book edited by Kate Theimer on significance of ‘web 2.0’ for archives and local collections. When asked what advice I’d give to an organisation venturing down this path I suggested:

Start experimenting. The technology is developing so rapidly that if you spend 12 months planning a project it’s likely to be out-of-date even before you start. New web services and data sources are becoming available every day.

Obviously, I’m still a strong believer in the value of experimentation. But I look at the sentence on the speed of change now and think it could have come from the mouth of some corporate AI shill. Quick, don’t be left behind! You can’t afford to wait! Sign up now!

Maybe I’m getting old and slow, but I’m more inclined now to think about the range of timescales across which we work – about the traces we leave behind, as well as the short term impacts. If I was starting ‘Mapping Our Anzacs’ again, I think I’d be trying to make sure all the geolocated metadata was properly versioned and saved in an open repository. Similarly, I’d create an independent backup of the scrapbook posts. I was focused on meeting the deadline and getting it to work, but I also should’ve been thinking about what happens when the institution pulls the plug.

A lot of important work has been done since then on digital preservation, the value and ethics of maintenance, and planning for the death of projects. But I also wonder what the fate of our digital projects tells us about our orientation in time. For the NAA, ‘Mapping Our Anzacs’ was a burden inherited from a near-forgotten past. For me it was an example of what you can achieve on a tiny budget by hooking together existing services and opening yourself to the public. Even after 18 years, it still seems to address the future.

The work we do is embedded within its own histories – personal, institutional, technological. We find in those histories points of meaning and connection that help us make sense of where we are. I’m sure we can all point to projects or people that jolted our understanding of what was possible and sent us careening down new pathways. None of us start from scratch. For me, the period between 2007 and 2012 really helped to define what I do and why.

In 2008, Mitchell Whitelaw was granted a fellowship by the National Archives of Australia to undertake his ‘Visible Archive’ project. It was the first in a series of GLAM collection visualisation projects through which Mitchell developed his oft-cited concept of ‘generous interfaces’. I helped Mitchell wrangle some of the NAA data, and his work inspired me to look at collections as a whole, rather than as a series of individual items.

Also in 2008, I created my first Zotero translator for the National Archives online database, RecordSearch. It made me think about what happens when we liberate collection data from web interfaces. Zotero, along with Omeka, was the product of the Center for History and New Media at George Mason University – a site that bubbled with GLAM-related enthusiasm and encouraged us to embrace the constructive power of hacking.

In 2009, I visited CHNM to present ‘Mapping Our Anzacs’ at the American Association for History and Computing conference. On the same trip I spoke at the New York Public Library, which had started pushing out a series of groundbreaking digital projects, like ‘Map Warper,’ ‘Building Inspector’, and ‘What’s on the menu?’.

People weren’t just experimenting with code, they were building new structures to enlarge the space and meaning of innovation. CHNM gave us THATCamp, a series of DIY unconferences that connected Digital Humanities (DH) and GLAM practitioners across institutional and disciplinary boundaries. Sick of watching events from afar, I organised THATCamp Canberra in 2010 – it was probably the most fulfilling, and exhausting, thing I’ve ever done. And if you want to know what we discussed in 2010, or in the 2011 and 2014 sequels, you can – because I’ve archived the sites and continue to pay the hosting bills.

A few months after THATCamp Canberra, we were visited by Bethany Nowviskie, the Director of the Scholars' Lab in the University of Virginia Library. Bethany challenged us all to think about the institutional, human, and political contexts of DH and GLAM innovation. In her 2011 talk, ‘A skunk in the library’, Bethany described the Scholars' Lab as

a conscious experiment: an experiment in modeling effective relationships of research-and-development work by librarians & library IT both to the digital humanities as an exciting community of practice, & to our own future – the future of libraries within a scholarly communications ecosystem experiencing rapid reconfiguration.

By 2010, I’d left the Archives and was working part-time at the National Museum of Australia, where we created our own under-the-radar, skunky GLAM Lab.

There were also new ways to play around with data. GLAM institutions had started using the Flickr Commons to share their image collections, and Flickr had an API that could be used to extract data and make new connections. One of my early experiments was the Flickr Machine Tag Challenge, which encouraged people to annotate photos with machine-readable identifiers for subjects or creators. This put me in touch others exploring the potential of Linked Open Data, and John Voss invited me to be part of the first LOD-LAM summit in San Francisco in 2011.

Communities developed, online and in-person, to share ideas and enthusiasm. In 2011, I popped across to New Zealand for my first experience of the National Digital Forum. It was full of GLAM people doing cool digital stuff, and they were all so welcoming and generous. It felt liking coming home. The keynote speakers that year included Mitchell Whitelaw on ‘generous interfaces’, and Michael Lascarides on digital innovation at the NYPL.

The point of these potted histories isn’t to invoke nostalgia, or suggest that some magic has been lost. There’s no lessons to be learned. History is always a conversation between past and present. What might in some respects seem to be a positive story of my growth and development, is also a catalogue of my failings – the discomfort I felt in large institutions, my tendency to self-sabotage, my impatience with administration.

And not all these stories had happy endings. I created the Zotero translator for the National Archives database in my own time. When I released it, an alarmed email was circulated amongst the senior management titled ‘What has Tim done to Recordsearch?’. While ‘Mapping our Anzacs’ was a great success, the web content team that created it was seen as a problem. One senior manager thought we had too many PhDs. Our positions were redefined, and our roles limited to cutting and pasting content that others had created into the content management system. We all left.

I’m sure many of you have similar war stories. We’ve all seen GLAM Labs come and go – projects die, initiatives falter. But that’s all the more reason why we should remember.

History provides ballast to keep us upright amidst the storms. We can draw on the strength of past achievements, reflect on failures, marshal precedents to confront new challenges. History gives us the weight and resolve to stand against the assumption of inevitability, the fetishistic power of the ‘new’ – to ask the questions that need to be asked.

When the LinkedIn bros warn that GLAM organisations are being left behind by the latest AI developments, I think about how long we’ve been working with technologies like machine learning and computer vision. Dipping again into my own history, I remember 2008, when the Powerhouse Museum started using natural language processing to automatically tag collection items. I remember 2010, when Paul Hagon from the National Library of Australia gave a conference paper on using facial detection to explore image collections.

In 2011, Paul’s work inspired Kate Bagnall and me to use facial detection to find the people inside the records of Australia’s racist migration policies and expose ‘The Real Face of White Australia’.

A few years later, I started fiddling around with object detection to extract thousands of redactions from the surveillance files of Australia’s internal security organisation.

And if you want to erase yourself from history, try wrapping yourself in one of my #redactionart scarves, made from 100% recycled redactions.

Of course the technologies have changed, but there are continuities as well. The simplicity of turning points rarely withstands the scrutiny of history. Understanding is born from our struggle to reconcile the fact that everything is new, and yet nothing is new.

I spent a lot of time during my PhD destroying my eyesight with microfilm readers – trawling through newspapers year by year, decade by decade. If I’d started my research in the post-Trove era, my experience would have been very different. I wonder whether the questions I asked would have changed as well.

But Trove itself isn’t fixed in time, it has its own history that runs parallel to the explorations of its users. By a sort of happy accident, some of my early visualisations captured the state of the newspaper corpus as it was in 2011.

I repeated the same analysis at irregular intervals until 2022, when I set up an automated process in GitHub that captured weekly changes and displayed them on a dashboard. That continued until February 2025 when the National Library of Australia cancelled my API access.

I’ve often used these visualisations to encourage people to think about the way the online collections are constructed – about how their search results are affected by things like institutional policy, legislation, funding, and technology. Sometimes I’d compare visualisations from 2011 and 2022 and ask them how their research might have been different according to their own location in time.

There’s been a lot of useful thinking around how we measure the value and impact of digital resources in the GLAM sector – including detailed frameworks like Europeana’s Impact Playbook and Adrian Kingston’s Audience Impact Model. But the windows through which we observe impact are still pretty small.

I’m thinking of a researcher in 20 or 30 ‘years time who wants to understand how digital collections, like Trove, changed the practice of history – changed the types of questions we could ask about the past. They could mine the historical literature, extracting citations and analysing data use, but that only gives half of the picture. How can they examine the literature in the context of the digital collections as they were when the original research was conducted? Online collections grow as more material is digitised. Improvements in OCR make more items findable. Interface updates can affect access to the underlying data. How do we capture these sorts of changes?

This history – the history of digitisation, metadata enrichment, interface design, prototype construction, dataset documentation, tool development – asserts the value of what we do. It matters. It changes things. Our projects might disappear as institutional priorities shift, but they are not disposable. They should not be forgotten.

In a gesture towards that hypothetical future researcher, I’ve assembled an idiosyncratic collection of snapshots and datasets in Zenodo. They include things like the 2,495,958 public tags added to 10,403,650 resources in Trove from 2008 to 2024, lists of non-English newspapers in Trove, and the number of OCR corrections by year, article category, and newspaper title.

Perhaps my favourite set of collection snapshots comes from the National Archives of Australia’s RecordSearch database. The records of Australia’s federal government are supposed to available to the public after 20 years. However, some are withheld for reasons like national security and privacy. Each year, the NAA makes a big performance out of revealing newly-released cabinet records, which are duly reported by the media on 1 January. I thought it was only fair that the public should also see the list of records that were currently closed to public access. So every New Year’s Day for ten years, I harvested details of the files we weren’t allowed to see. I only stopped because the NAA introduced anti-bot measures that blocked my scraper script.

Beyond a little sly subversion, the data on closed files in the NAA is useful because it helps to document the workings of the access examination system – a point of much pain for researchers. Most of my work over the past 30 years has, in one way or another, explored the meaning of ‘access’ – how it is constructed, how that changes, and what it means for people using GLAM collections.

In January 1948, 13 year old Phyllis Nichols stood at the crossroads. She was visiting the Atomic Age Exhibition in Melbourne and according to the Sun newspaper:

She had covered the path of destruction and she turned with hope to the road to progress.

Such a weighty decision for a 13 year old. I must admit, there was a point where I seriously considered turning my thesis into a work of fiction focused on Phyllis’s adventures in atomic wonderland.

Phyllis chose well. But the crossroads metaphor was never really about choice. No-one was expected to pursue the path to nuclear annihilation. The crossroads demanded obedience to a specific vision of the future. It’s this way …or else.

‘Mapping Our Anzacs’ was created at a time of optimism, when it was thought that web technologies would open up the workings of government to new forms of public participation and transparency. But the dreams of ‘government 2.0’ have faded, as information becomes ever more tightly controlled. In the GLAM sector, APIs have come and gone. Datasets created for long past hack events linger without updates, almost forgotten. New defensive measures aimed at taming the onslaught of AI scraper bots have imposed extra limits on access. Meanwhile a handful of tech oligarchs tell us what our future will be. This is the reality of progress.

The work of GLAM Labs, of GLAM innovation, has always been focused on expanding the realm of the possible – encouraging people to see differently, to think differently. This work struggles constantly with the many meanings of ‘access’ – what use is data without good documentation, without permissive licences, without tools for analysis, without the skills of confidence to use those tools. It was this sort of struggle that motivated the GLAM Workbench.

I recently wrote a potted introduction to the GLAM Workbench for a forthcoming publication on tool-making in the digital humanities. I won’t read it all out, but I think it gives a pretty good overview of where things are.

I suppose I want to emphasise though that the aim of the GLAM Workbench has always been to document possibilities – to expose researchers to the richness of GLAM data, to the new types of questions they can ask, and to the methods that are available to connect everything up.

In a world that erects multiple barriers of expertise, ownership, participation, and authority, there’s power in simply knowing what’s possible.

Back in 2010, at the first THATCamp Canberra, someone thanked me and said ‘I’ve found my people’. That sense of belonging was always what made the National Digital Forum in New Zealand so special. I’m not great at organisations – meetings make me anxious, and my email is a bin fire – but I do draw a lot of strength from the passions of like-minded people.

I don’t really know what the future of the GLAM Workbench will be. I’d like to be confident and optimistic, but the first half of last year was pretty bleak, and left me wondering whether I should just walk away from all of it – go bushwalking, catch up on gardening, perhaps just be a historian again. I was saved by the GLAM Labs community. First of all, by the fabulous folk at the State Library of Victoria’s LAB, particularly Paula Bray and Sotirios Alpanis. They gave me what I needed – fun data and wicked challenges. For a few months, I was back in my happy place, creating new pathways through the SLV’s place-based collections.

The second boost was, of course, the invitation to be here today – to find out what’s happening in labs around the world, to finally meet people I’ve known online for years, to join in the excitement and, yes, to share the disappointments.

Perhaps the GLAM Workbench has done it’s job. I think it’s helped give people the confidence to dip a toe in the world of collections as data. It’s also provided a useful model for GLAM organisations seeking to encourage new types of research. Perhaps its main value was always as an intervention – an invocation of possibilities that filled a particular gap at a particular moment in time. I’ve always tried to wrap the GLAM Workbench in layers of documentation to enable any lasting value to be extracted as needed. Perhaps my focus should be to make sure that documentation is complete.

But it’s not a choice for me alone. The GLAM Workbench has always welcomed contributions, so if you’d like to carve out your own spaces, create your own sections, let me know! Perhaps the future of the GLAM Workbench will be shaped by the hands of others.

There’s also a lot of cool GLAM data out there to play around with, and it really doesn’t take much to get me excited about it. So perhaps I’ll just continue to follow my enthusiasms and see where that takes things.

All of these possible futures are good. The choices aren’t fixed – they leave the conversation with history open and constructive. I’m happy with that.

So greetings, thanks, and solidarity to all GLAM Labbers, past, present, and future. Despite all the setbacks and frustrations, your work matters. Take time to remember, to enjoy, and to celebrate.

The battle of the bots

Tue, 12 May 2026 17:16:04 +1000

I’ve spent a lot of time recently trying to protect my tools and resources from the relentless onslaught of AI scraper bots. I’ve documented much of the process here in case it’s of use to others (and to remind myself when I inevitably forget what I’ve done).

Over the years, I’ve often made use of cloud services such as Heroku and Google Cloudrun to share my odd assortment of tools and experiments. In particular, Cloudrun has offered an easy way of publishing interactive databases created using Datasette – one command and they’re online and open for exploration. This has worked (with a little fiddling) for databases of all sizes. The GLAM Name Index, for example, combines 10 databases totalling about 4 gigabytes of data. My only concern has been the amount of memory that Cloudrun demands to run Datasette databases. Throwing memory at things costs money, but because Cloudrun services can be configured to shut down when not in use, the overall hosting costs have remained manageable. Until now.

Earlier this year, I noticed that the Tung Wah Newspaper Index, that ran using Datasette on Heroku, was having trouble – it kept falling over because it exceeded Heroku’s CPU and memory limits. The cause was a dramatic increase in traffic from AI scraper bots. I’ve written my share of data scrapers over the years, but I’ve always tried to avoid placing any undue burden on systems. The AI scraper bots are different, they’re indiscriminate and unyielding – they want everything, they want it now, and they don’t care how much damage they do. In the case of the Tung Wah Newspaper Index, the problem was both the overall number of requests, and the number of requests for computationally-intensive queries, such as facets. The only good thing was that the Heroku instance had a fixed cost, so the influx was killing the resource, but not costing me extra.

I got the Tung Wah Newspaper Index running again by moving it to the Ubuntu server I run on Digital Ocean to host wraggelabs.com. I fiddled around with my nginx settings to add a block list of known bots, throttled the number of requests per second, and switched off facets in Datasette. I also analysed the nginx logs to identify the IP address ranges producing the most traffic and blocked them as well. I knew that this might affect legitimate users, but I just wanted to get it working again for most people.

Up until recently, the Datasette instances on Cloudrun didn’t seem to be attracting the same amount of traffic. But then I started getting alerts about unexpected billing increases. As I said above, Cloudrun instances can be configured to scale to zero when no requests are incoming – they switch themselves off when no one is using them. This conserves resources, and keeps costs down. But once the bots found my Datasette databases, the flood of requests was almost constant, so the services remained ‘on’ and my bills went up. All together, my Datasette instances used to cost me around $30 a month, this skyrocketed overnight and would’ve cost me several hundred per month if I left things as they were.

I was already using the datasette-block-robots plugin to generate a robots.txt file that told bots to go away. But the AI scraper bots just ignore these sorts of things. More drastic action was needed.

Sysadmin stuff makes me nervous, and I don’t like to fiddle with things that seem to be working. I was reluctant to move my databases from Cloudrun because it was so convenient, and because I was worried that other platforms would struggle with the amount of data I was sharing. As a result, I spent a number of days investigating Google’s own bot protection services – this seemed to involve putting a load balancer in front of my instances, then attaching their ‘Cloud Armor’ service to the load balancer. Of course, the more you paid, the more protection you got. I made a few attempts to get this going, but then gave up. The configuration itself was complex and confusing, but the main problem was that I felt I was just giving over more and more power to services that I didn’t really control or understand. The whole thing made me feel uneasy.

I did a bit of local testing and realised that actual memory requirements for my Datasette databases was much less than Cloudrun demanded. This meant I could conceivably get them all up and running on a smallish virtual server, such as I’m using for wraggelabs.com. But what about the bots? I decided to give Anubis a try. Some AI scraper bots identify themselves in the User-Agent header field of their HTTP requests – they tell you their names. This makes them relatively easy to block. But other bots masquerade as normal web browsers, so their requests seem to come from humans. Anubis sniffs out disguised bots by asking them to complete a challenge that’s only possible in a real web browser. It seemed like a much more complete and consistent approach than my ad hoc efforts to block IP ranges.

So the plan was:

create a new domain for my databases
create a new virtual server in a Digital Ocean droplet and attach the domain
set up nginx and Anubis to manage web traffic and repel bot attacks
migrate the databases, checking that they could comfortably fit within the server’s limits
redirect the Cloudrun services to the new server
monitor the bot traffic to make sure nothing was going to explode

A new home for my GLAM databases, with built-in bot protection

It’s all now done, the databases seem happy in their new home, Anubis is busy blocking bots, and my costs are contained. You can visit the databases at glam-databases.net. The gory details are below…

The technical details

I registered glam-databases.net and set up an Ubuntu server in a Sydney-located Digital Ocean droplet. I did the usual security stuff, like setting up a firewall and a non-root user, and then installed nginx and SSH using the Digital Ocean guides.

I installed Anubis using the instructions on the website. However, I wanted to use Unix sockets to connect everything up and ran into a few problems getting Anubis to write .sock files. This issue helped me understand that I needed to create a directory with the name of my Anubis service in /run/anubis. I eventually figured out that I also needed to add SOCKET_MODE=0777 to the Anubis config file to get the permissions right.

The other thing I struggled with a little was setting up logging. Once again it was mostly a permissions thing. The Anubis documentation explains that if you’re running Anubis via systemd you need to add a drop-in unit to allow Anubis to write to /var/log, however, I found that the drop-in needed to be saved to /etc/systemd/system/anubis@instance-name.service.d/ rather than /etc/systemd/anubis@instance-name.service.d/ as stated in the docs (notice the extra system directory).

Once the socket stuff was sorted, getting Anubis to talk to nginx was pretty straightforward, and I mostly just followed the Anubis documentation to make the necessary changes to the nginx config files. I’d set up a basic, static homepage for the site, so I tried accessing that to check that Anubis was in fact working. The Anubis challenge page was displayed as expected, and I could see the challenge details in the logs. Yay!

Next it was time to start moving my databases across. The first step was to ensure that each of my Datasette repositories included a requirements.in file, listing the necessary Python packages – this varied a bit across projects depending on the Datasette plugins they used, but the minimum was datasette and the datasette-block-robots plugin. To improve performance I used datasette inspect to calculate table row counts and write them to a json file in the repository. The counts file is loaded at run time using the --inspect-file parameter.

I created a directory for each of the databases on the new server, and used rsync to copy the database repositories across. Then in each of the database directories I created a virtual environment using pyenv, and installed all the necessary Python packages using pip-tools and the requirements.in file.

To get the databases running, I mostly just followed the Datasette documentation which provides information on running Datasette using systemd and configuring nginx. I used the --uds and --base_url settings to connect via unix sockets and deploy each database at a different url prefix. The only problem I found is that the base_url setting can be a little buggy – for example, links to json versions of pages sometimes included the base path prefix twice. Rather than fiddle about in the Datasette code, I decided to ‘fix’ this using rewrite directives in the nginx configuration file, for example: rewrite ^/gni/gni/(.+)$ /gni/$1 break; I’m sure there are better solutions, but I figured this would do the job until things got fixed upstream.

Here’s an example of the systemctl service files I created for each database (with the key redacted):

[Unit]
Description=Sand and MacDougall's Directories (Victoria) Datasette
After=network.target

[Service]
User=tim
Group=www-data
Environment=DATASETTE_SECRET=**********************************
WorkingDirectory=/home/tim/databases/victoria-sands-mac-dirs
ExecStart=/home/tim/.pyenv/versions/victoria-sands-mac-dirs/bin/datasette serve -i sands-mcdougalls-directories-victoria.db --uds /tmp/victoria-sands-mac-dirs.sock   --inspect-file=counts.json --template-dir templates --static static:static  -m metadata.json --setting base_url /vic-smd/  --setting facet_time_limit_ms 10000 --setting suggest_facets off --setting allow_csv_stream off
Restart=on-failure

[Install]
WantedBy=multi-user.target

The parameters used when starting Datasette vary a bit depending on the project (for example some don’t have custom templates) but in general they are:

-i run in immutable mode for better performance
--uds path to unix socket
--inspect-file path to precalculated row counts
--template-dir custom template directory
--static directory and url prefix for static files
-m path to metadata file
--setting base_url url prefix pointing to this database
--setting facet_time_limit_ms maximum time allowed for facet requests
--setting suggest_facets off don’t provide example facets (both for performance and to avoid providing new paths for bots to find)
--setting allow_csv_stream off for performance

I created a service file like this for every database and saved it to the /etc/systemd/system/ directory.

For each database I created two new sections in the nginx config file. First in the main server block I added a location entry that directed requests to the database’s url prefix. For example:

location /vic-smd {
        rewrite ^/vic-smd/vic-smd/sands-mcdougalls-directories-victoria/(.+)$ /vic-smd/sands-mcdougalls-directories-victoria/$1 break;
        proxy_pass http://victoria-sands-mac-dirs/vic-smd;
        proxy_set_header Host $host;
}

Then I created an upstream entry for each service that send these requests to the database socket.

upstream victoria-sands-mac-dirs {
	server unix:/tmp/victoria-sands-mac-dirs.sock;
}

Of course, after all the changes I ran sudo systemctl start or restart and checked that things were ok using sudo systemctl status.

Once I had migrated all the databases, I left things alone for a day to make sure they were comfortably settled into their new home and that the server didn’t explode. I monitored the system stats, and found that the memory consumption was much lower than it had been on Cloudrun. So all seemed good.

Then it was time to redirect the Cloudrun urls to the new database locations. When I created the Cloudrun instances, there was no option to add custom domains to instances in Australia. In most cases, I created redirects from the GLAM Workbench that I encouraged people to use and share, but I noticed that people were often saving the ugly Cloudrun urls instead. This meant that I couldn’t just switch the domain config, I had to replace the Cloudrun instances themselves with redirects. This ended up being much easier than I expected. Using docker-web-direct I could spin up little nginx proxies to replace the Cloudrun databases. For example to redirect the Tasmanian Post Office Directories, I used the Cloudrun CLI to send the following command:

gcloud run deploy tasmanian-post-office-directories --image morbz/docker-web-redirect --allow-unauthenticated --set-env-vars "REDIRECT_TARGET=https://glam-databases.net/tas-pod"

The key things are the service name, in this case tasmanian-post-office-directories, and the new url https://glam-databases.net/tas-pod.

I repeated this for each of the databases on Cloudrun. Of course, redirecting the urls to help people find the new locations also opened the door to the scraper bots, and they quickly muscled their way through. I was monitoring both the Anubis log file and the system stats as I redirected each database – the traffic went up, but everything was pretty stable, and Anubis seemed to be working.

A realtime snapshot of the Anubis log file

Checking the nginx log I noticed that one bot, meta-webindexer, was hitting one of the databases hard and wasn’t being blocked by Anubis. There’s an open issue in the Anubis GitHub repo to add this bot to the blocklist, but it was easy to modify the default bot policy file to repel it:

  - name: meta-webindexer
    user_agent_regex: meta-webindexer
    action: DENY

Datasette has a built-in JSON API and I use this to pull data into things like my SLV apps. For this to continue working I had to open a pathway in the Anubis bot policy file:

  - name: allow-api-requests
    action: ALLOW
    expression:
      all:
        - '"Accept" in headers'
        - 'headers["Accept"].contains("application/json")'
        - 'path.contains(".json")

This means that requests with .json in the url and application/json in the accept headers will not be challenged by Anubis. I then modified some of the apps to make sure they actually were sending the correct headers, and it worked!

Next steps

Things have gone pretty smoothly so far, but I’ll be keeping my on the stats. Now that I understand how Anubis works, I also want to install it on Wragge Labs and remove the ad hoc blocks I put in place earlier.

I’ll leave the redirects running on Cloudrun for a while, but not forever. If you use any of these databases, please update your bookmarks and links!

NSW Post Office Directories (54 volumes from 1886 to 1950)
Sydney Telephone Directories (44 volumes from 1926 to 1954)
Tasmanian Post Office Directories (54 volumes from 1890 to 1948)
Sands & McDougall’s Directories, Victoria (24 volumes from 1860 to 1974)
GLAM Name Index Search (12 million records from 293 datasets)

I’m relieved to have brought my Cloudrun costs under control, but all of this still costs money. Over recent years, I’ve been lucky to have the support of my GitHub sponsors to help me pay my cloud hosting bills. I’m planning on moving away from GitHub eventually and I’ve set up a brand new profile on LiberaPay. They’re an open source, community managed organisation that doesn’t take a cut of your donations, so if you want to help me keep things online you can set up a sponsorship there.

Generosity in practice – a chat with Paula Bray at the State Library of Victoria

Mon, 16 Mar 2026 14:14:17 +1000

While I was in Melbourne during my time as Creative Technologist-in-Residence at the State Library of Victorian LAB, I had a conversation with Paula Bray for the LAB’s podcast series. Paula is the SLV’s Chief Digital Officer, and has long championed the importance of digital innovation in the GLAM sector. It was fun to chat about stuff that I’ve been doing for the last 30 years, any why openness and generosity is important in working with GLAM collections. You can listen to our conversation on the LAB site.

Zotero translator for Libraries Tasmania updated!

Tue, 10 Mar 2026 09:31:36 +1000

The Zotero translator for Libraries Tasmania has been updated, fixing a problem with attaching images of digitised resources. The fix is in the main Zotero repository now, so it should find its way to your computer automatically.

I created the first version of the Libraries Tasmania translator back in 2022 – this post describes what it does. It works across all three sections of the catalogue, including the archives, and names index. The translator captures metadata, PDFs, and images from records, including things like digitised pages from convict records. This makes it easy for researchers to assemble their own datasets of Tasmanian records in Zotero, where they can add notes and annotations, or share with colleagues.

Capture images and metadata from the Libraries Tasmania catalogue using Zotero

The update was necessary because Libraries Tasmania changed the way some digitised resources were displayed and downloaded. Keeping Zotero translators working across system updates can take a bit of work! I also took the opportunity to update the code to meet current Zotero guidelines and clean up a few lingering problems. If you notice any oddities, please let me know.

There are now at least 8 custom translators to help you work with Australian GLAM collections.

Exploring georeferenced maps from the SLV collection

Thu, 12 Feb 2026 22:05:06 +1000

I’m in the process of tying up all the documentation relating to my time as Creative Technologist-in-Residence at the State Library of Victoria LAB. But as I was looking through the list of outputs, I realised I’d never written anything about the interface I created to explore georeferenced maps from the SLV collection.

I also remembered that there were a few improvements I wanted to make to the interface. So instead of spending a few hours writing up a blog post, I’ve spent several days completely overhauling the Georeferenced Maps Explorer. I’m pretty happy with how it’s working now. Have a play!

Wilson's Prom made up of a patchwork of georeferenced maps and aerial photographs using the Georefrenced Maps Explorer. Try it now!

To get started, just click on the basemap. Details of all georeferenced maps within 50km of your selected point will be displayed in the right-hand column. As you move your mouse over the list of results, the boundaries of the georeferenced maps will be displayed on the basemap. This gives you a preview of their location and size. Click on one of the results to display the georeferenced map as a layer on top of the modern basemap.

Hover over a result to see the map boundaries

You can add as many maps as you like. If your selected maps overlap, you can change the order in which they’re shown. Click on the layers icon in the top left of the basemap. You’ll see a list of the maps that are currently displayed. Use the arrow buttons to move a map backwards or forwards. You can also use the sliders to adjust the opacity of each map. This can make it easier to examine the relationship between maps. For example, you might want to compare the features of a historic map with those of the underlying basemap.

Stitch together multiple maps like this series of seven photomaps, and change the opacity to see the features underneath

The Explorer’s url updates with every selection you make, so you can bookmark or share a url to return to the same position and collection of maps. For example, this link will take you to the collection of maps of Wilson’s Prom shown above.

The background

If you missed the start of this journey back in November last year, you might be wondering what the georeferenced maps are and where they come from. During my SLV LAB residency, I found a way of hooking the SLV’s digitised maps up to a tool called Allmaps that helps you identify points that connect historic maps to our modern coordinate system. When enough points have been identified, the historic maps can be positioned on a modern basemap. This is known as georeferencing, georectifying, or ‘map warping’, as the results can often appear skewed or warped.

Once I had connected things up, I invited the world (or at least the tiny part of it that follows me on social media) to help turn the SLV’s maps into data. And they did! As of today, 1,447 of the SLV’s digitised maps have been georeferenced. This dashboard displays current georeferencing progress.

The total number of SLV maps georeferenced over time. It's still going up!

There’s still plenty more to do. If you’d like to help, the full instructions are available here. Georeferencing is pretty fun, so why not have a go?

You can explore the current collection of georeferenced maps in a few different ways. There’s a dataset you can download or search that gets updated every two hours. This data is loaded into a spatial database that’s used by the Georeferenced Maps Explorer. As part of my recent improvements, I’ve automated this process as well, so the database should be updated with the latest additions every 24 hours.

You can also search for georeferenced maps using the my place app. You just enter an address and my place pulls together data from a variety of sources – mixing the georeferenced maps up with parish maps, newspapers, photos, and entries from the Sands & MacDougall’s directories.

Georeferenced maps in my place results

The interface

The Georeferenced Maps Explorer uses MapLibre and the Allmaps MapLibre plugin to display the georeferenced maps. You might notice that it looks pretty similar to the Newspapers Explorer and the CUA Browser, both of which use MapLibre, as well as Bulma for CSS. I’ve been trying to settle on a fairly standard set of tools that I can use to create and maintain these sorts of interfaces without too much fuss. Basically I just cut and paste a lot of stuff, then modify as needed.

When you click on the basemap in the Explorer, the coordinates are sent off to the spatial database to retrieve details of georeferenced maps within 50km. The spatial database runs in Datasette, which has a built-in JSON API that I use with a set of predefined ‘canned’ queries to pull back the data I need. The results are displayed in the right-hand column, along with square thumbnails generated by the SLV’s IIIF service.

The metadata includes distance and area measures. These are used to find and sort the results. There are two distance measures, one from your selected point to the closest boundary of a map, and the other to the centre of a map. If the point is contained within a map’s boundaries, then the ‘bounds’ distance is zero. The search query finds maps whose closest boundaries are within 50km. Originally I sorted the results by this distance and the area of the maps. But this meant that large scale maps that included the selected point (such as maps of the whole of Victoria) appeared above nearby local maps. To make it easier to find maps within an area, I added the ‘centre’ distance and now sort the results using that. This allows nearby maps that don’t include the current point to bubble up towards the top of the search results, above many of the large scale maps. It’s far from perfect, but I think it strikes an ok balance.

The data also includes the boundaries of each map as GeoJSON. I use this to generate a MapLibre layer that contains all the boundaries as polygons. The boundaries are hidden until you hover over the corresponding search result, then the opacity of the boundary is flipped to 1 and it magically appears.

When you click on a search result, a request is fired off to Allmaps for the full georeferencing data. The Allmaps plugin uses this to retrieve the map image from the SLV’s IIIF service and display the warped map in MapLibre.

I looked around for quite a while to find a good way of changing the opacity and order of the warped maps in MapLibre. I eventually found the Map Libre GL Layer Manager which did a lot of what I wanted. I forked the repository and modified the code to get the opacity slider to work with warped map layers. Warped map layers already have a setOpacity method, it was just a matter of checking for ‘custom’ layers, then finding where the warped map was in the layer object.

if (type == "custom") {
        layer.implementation.setOpacity(opacity);

I also made a few cosmetic changes, such as renaming the tooltips on the reorder buttons from ‘move up’ and ‘move down’ to ‘send back’ and ‘bring forward’ – up and down just confused me.

I tried for a long time to find some way of adding tooltips or popups to the warped maps that would show their details when you moved the mouse over them. I found that if you were displaying multiple maps that looked similar, such as the photomaps above, it was difficult to know which map was which. After a chat with the Allmaps developers in their IIIF Slack channel, I realised that this approach wouldn’t work as the warped map layers don’t currently listen to mouse events. Instead I decided to add hover events to the list of results, rather than the maps, and use them to display the map boundaries as described above. This way I get the connection between the map and metadata that I wanted, as well as a useful way of previewing results.

I think I’ve probably stopped fiddling with the interface for now. I hope you find it useful!

The future?

There’s more that I’d like to do with the georeferenced maps. In particular, I’ve been thinking about an interface with a slider that showed the changing patchwork of maps over time…

Related resources:

the code for the Georeferenced Newspapers Explorer and all the other apps and sites I created during my residency is in this GitHub repository
the code to harvest the georeferenced data from Allmaps and build the dashboard is in this GitHub repository
there’s also the full list of all the apps, code, posts, and talks created during my residency

my place – exploring SLV collections through a street address

Mon, 02 Feb 2026 21:43:20 +1000

‘What can I find out about my house?' My work as Creative Technologist-in-Residence at the SLV LAB was inspired by questions like this that librarians at the SLV hear every day. I wanted to explore how the Library’s place-based collections could be used to provide new entry points for discovery and navigation – entry points based not on words, but locations.

At the end of my residency, I pulled all the different collections I’d been working with into a single interface – my place. It’s not polished or complete, but I think it’s a useful starting point to think about the possibilities. You just type in an address, street name, or place name and my place shows you maps, photos, newspapers, and even extracts from the Sands & MacDougall directories. Try it now!

Try my place! Just enter an address in the search box.

Search results in my place are bookmarkable. So save and share your discoveries!

The collections

my place draws its data from a number of different place-based collections that I’ve been working on during my residency.

OpenStreetMap

When you enter an address in the search box, my place looks it up in OpenStreetMap to get its geospatial coordinates. It then places a marker and re-centres the map at the top of the app.

Map centred on 149 Brunswick Street, Fitzroy

OpenStreetMap is also used to retrieve additional information about the suburb, including its boundaries.

Sands & MacDougall’s directories

my place queries the full-text searchable version of Sands & Mac for addresses. Results will vary based on the OCR quality and the nature of query, but it can give you a potted history of who has lived in your house. The search results are displayed in chronological order, and include an image snippet showing the actual printed entry as well as the text content and metadata.

Occupants of 149 Brunswick Street, Fitzroy from 1875 to 1925

Committee for Urban Action photographs

If you enter a full street address, my place will search the CUA collection for photos associated with the segment of road that includes the current address. It then displays the individual images from any matching photosets.

Photographs from CUA of the currently selected road

Otherwise my space will look for CUA photos that are near the current location, and display a randomly-selected image from each photoset.

Georeferenced maps

my place searches through digitised maps from the SLV collection that have been georeferenced by the public. It finds maps that either intersect with the currently selected location, or are nearby.

If you enter a full street address, the first 6 georeferenced maps will be positioned on a modern basemap with a marker indicating the currently selected point. This means you can see your address on a historical map. The number of georeferenced maps that can be displayed in this way is determined by the browser – so I’ve limited it to 6 to be safe.

Georeferenced maps positioned on a modern basemap, showing the location of the currently selected address

Parish maps

my place searches through parish maps in the SLV collection that have geospatial coordinates or approximate bounding boxes. It finds maps that either intersect with the currently selected location, or are nearby.

Newspapers

my place searches through my dataset of newspapers in the SLV collection that have a place of publication documented in the ‘Place newspaper published’ metadata field. It finds newspapers that are either associated with the current suburb/town, or a nearby suburb/town. This includes digitised and non-digitised titles. Digitised titles include a link to Trove.

Newspapers from the SLV collection published in Fitzroy

Photographs

I thought it would be cool to include a few photographs of the current suburb or town. To do this, I downloaded a list of place names from VicNames, then used the place names to search the SLV catalogue for photographs with relevant subject headings. A random selection of the harvested images is displayed in my place.

A few images of Fitzroy, displayed alongside a map of Fitzroy's current boundaries using data from OpenStreetMap

The interface

The interface is pretty simple. You type an address in the box and hit enter. If the geocoding process finds multiple matches, it’ll give you a list to choose from. Once the location is found, a marker is added and the main map re-centres. Then related resources are displayed below the map.

As you scroll down through the results you gradually zoom out from your initial starting point. This is reflected in the four bands or layers used to group resources: ‘my house’, ‘my street’, ‘my suburb’, and ‘nearby’. Each band contains a mix of resources from different collections.

When I started working on my place, I was thinking about a project from around 2010 called The History Wall. Like my place, The History Wall pulled many different types of resources together into a rich exploratory interface. As you scrolled through The History Wall you moved through time, with randomly selected items appearing from a range of sources including Trove newspapers, the ADB, and museum collections.

A version of The History Wall created for the National Museum of Australia's 'Irish in Australia' exhibition

I originally thought I’d inject some of the same randomness into my place, but I was worried it might just get too confusing. I thought it was important to keep the relationship between the starting point and the resources in focus even as you zoomed out. So my visual metaphor shifted to something more like a blast radius map, or a stratigraphic diagram, that displayed distinct groups and layers as you moved beyond the baseline. My limited CSS skills couldn’t make the vision in my head a reality, but there are lots of headings and colours instead to highlight the transitions!

The actual mix of groups and layers displayed depends on the nature of your query. If you’ve entered a complete street address, and there are results for that address in Sands & Mac, then you’ll see ‘my house’, ‘my suburb’, and ‘nearby’. If you’ve only entered a suburb or town, or your street address can’t be found, you’ll see two layers starting with ‘my suburb’.

Here’s an overview of what you might expect to see.

my house

Sands & MacDougall extracts (text search on full address)
georeferenced maps (search for maps that contain the base point)
parish maps (search for maps that contain the base point)

my street

CUA photos (search for matching street identifiers)

if there’s no ‘my house’ layer:

Sands & MacDougall extracts (text search on street name and suburb)
georeferenced maps (search for intersections between maps and street)
parish maps (search for intersections between maps and street)

my suburb/town

suburb boundaries from OSM
images (search for suburb name in metadata)
newspapers (search for suburb name in metadata)

if there’s no ‘my house’ or ‘my street’ layer:

georeferenced maps (search for intersections between maps and suburb boundaries)
parish maps (search for intersections between maps and suburb boundaries)

nearby

CUA photos (search for photosets within 5km of the base point, filtered to remove current street)
georeferenced maps (search for maps within 10km of base point, ordered by distance, max of 24 displayed)
parish maps (search for maps within 10km of base point, ordered by distance, max of 24 displayed)
newspapers (search for newspapers within 100km of base point, ordered by distance, max of 24 displayed)

The data

Most of the data used in my place is stored in two SQLite databases – one for Sands & Mac, and the other for CUA, georeferenced maps, parish maps, and newspapers. The metadata for the collection images is stored in a JSON file that is directly loaded by the interface.

I’ve published the SQLite databases online using Datasette and Spatialite. Spatialite makes it possible to find geospatial features that intersect, or are near, a given point. For example, you could find maps that include a specific set of coordinates.

Datasette has the ability to create ‘canned queries’ that feed url parameters into pre-defined SQL queries. This coupled with Datasette’s built-in JSON API makes it possible to construct query urls in my place and use them to retrieve JSON results data from my databases.

When you enter an address in my place, multiple queries are fired off to find intersecting or nearby resources. For example, this url finds georeferenced maps within 10km of a point at the centre of Fitzroy: slv-places-481615284700.australia-southeast1.run.app/georefere…&distance=10000&_shape=array.

In the case of Sands & Mac, my place uses a canned query that runs a full-text search across the OCRd content of a volume. Suburb names are often abbreviated in Sands & Mac, so the app first runs a query to find possible abbreviations, then adds them into the main query to inject a bit of fuzziness. This is repeated for all 24 digitised volumes.

Once the metadata is retrieved from the databases, images are loaded from the SLV’s IIIF service.

Next steps?

I’m not sure how much more work I’ll do on my place, but there are a few things I’d like to try. In particular, I’d like to help the user understand more about what data is being presented, or not presented, and why. Not all digitised maps have been georeferenced, not all parish maps have coordinates, street numbers have changed, and the OCR in Sands & Mac varies in quality. my place can only present a sample – a gesture towards the wealth of material available from the SLV. I feel that message needs to be made more explicit. Though I’m not sure how without overloading the interface.

There are additional data sources I’d like to play around with. my place already includes some code to query Wikidata for more information about a suburb. But I haven’t had a chance to do anything with it. I’d like to be able to provide additional contextual information from outside the SLV, such as electoral boundaries, populations, even election results. It would also be fun to display sightings of plants and animals from the Atlas of Living Australia.

What can I find out about my house? It would be great if my place could answer that question by taking the user on an open ended journey through our cultural, historical, and environmental landscape.

Geolocating photos from the SLV's Committee for Urban Action collection

Thu, 29 Jan 2026 17:08:06 +1000

Concerned about the loss of built heritage in the 1970s, the Committee for Urban Action photographed streetscapes across urban and regional Victoria. They compiled a remarkable collection of photographs that is now being digitised by the State Library of Victoria. More than 20,000 images are already available online!

The CUA worked systematically, capturing photos street by street, and recording the locations of each set of photographs. This information is used to prepare the title attached to each photo as it’s uploaded to the SLV catalogue. In general, titles include the name of the road where the photo was taken, the name of the suburb or town, and the names of two intersecting roads that define the boundaries of the current road segment. They can also tell you which side of the road the photo was taken on. For example, the title Gore Street, Fitzroy, from Gertrude Street to Webb Street - east side tells us the photo was taken on the east side of Gore Street, Fitzroy between the intersections with Gertrude Street and Webb Street.

Photos from Gore Street, Fitzroy displayed in the SLV image viewer

It’s great to have this sort of structured information linking photos to specific locations, but to navigate through the collection in space we need more. We need to link each photo to a set of geospatial coordinates by mapping each road segment. That was the challenge I took on as part of my residency in the SLV LAB.

When I started working on the collection I wasn’t really sure what was possible. I had to learn a lot, and ended up revising my processes multiple times as I got deeper into the data. But my aim was always to create some sort of map-based interface, that would allow users to click on a street and see any associated CUA photos. It’s still a bit buggy and incomplete, but here it is – explore the CUA collection street by street!

Gore Street, Fitzroy in the new CUA Browser

The process

My basic plan was to find the intersections using OpenStreetMap, then extract geospatial information about the segment of road between the two intersections. This involved much trial and error, but eventually I ended up with a process that:

parsed each item title to try and extract the names of the main road, the suburb, and the two intersecting roads
queried Nominatim for the suburb bounding box
for each intersecting road, queried OSM to find a node at, or around, its intersection with the main road, within the suburb bounding box
created a new bounding box from the coordinates of the two intersections
queried OSM for the main road within this bounding box
extracted the coordinates of the main road segment, removing any points outside of the bounding box

There’s more details below and in these notebooks: cua_finding_intersections.ipynb and cua_data_processing.ipynb.

Finding intersections

As described, the title of each photograph generally includes 4 pieces of information: the road, suburb, intersecting roads, and side. My plan was to find the intersections first to get the limits of the road segment. This is possible thanks to the awesome OpenStreetMap and its Overpass API. It took me a while to get my head around the Overpass query language, but there are lots of useful examples online. The query to find the intersection between Gore Street and Gertrude Street in Fitzroy looks like this:

[bbox:-37.8089071,144.9732006,-37.7929130,144.9851430];
way['highway'][name="Gore Street"];
node(w)->.n1;
way['highway'][name="Gertrude Street"];
node(w)->.n2;
node.n1.n2; 
out body;

You can try it out using Overpass Turbo’s web interface.

In OpenStreetMap, linear features, such as roads or rivers, are represented as ways. Each way is made up of a series of nodes or points with geospatial coordinates. Every way and node has its own unique identifier. Tags can be added to features to describe what type of things they are.

The query above looks for ways named ‘Gore Street’ and ‘Gertrude Street’ that are tagged as highway (a highway in OpenStreetMap is any road-like feature including things bike paths and foot trails).

way['highway'][name="Gore Street"];

It then extracts the nodes that make up each way and looks to see if there are any nodes in common between the two ways. A node shared between two ways indicates an intersection.

node(w)->.n2;
node.n1.n2;

The query is limited using a bounding box that encloses the suburb of Fitzroy. This avoids false positives and keeps down the query load.

[bbox:-37.8089071,144.9732006,-37.7929130,144.9851430];

The JSON result of this query gives as the latitude and longitude of the node at the intersection of the two roads.

{
  "version": 0.6,
  "generator": "Overpass API 0.7.62.10 2d4cfc48",
  "osm3s": {
    "timestamp_osm_base": "2026-01-27T03:11:45Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "node",
  "id": 224750459,
  "lat": -37.8062302,
  "lon": 144.9817848
}

  ]
}

After a bit of testing, I found this worked pretty well, except for roundabouts… In OpenStreetMap, roads don’t actually cross roundabouts – they end on one side, then begin anew on the other side. In cases like this, looking for shared nodes doesn’t work. Instead you have to look to see if the two roads have nodes that are less than a given distance apart. The query is similar to the one above, but uses around when comparing the nodes. In this case I’m looking for nodes that are within 20 metres of each other.

node(w.w2)(around.w1:20);

Finding road segments

Once I had the coordinates of the two intersections, I could look for the segment of road between between them. To do this I created a bounding box using the coordinates of the intersections, and then searched for ways by name within that defined area.

It’s important to note that there’s no one-to-one correspondence between roads and OSM ways. A single road might be represented in OSM as a series of separate, but connected, ways. For example, at a roundabout, or where a road divides, new ways might have been created to document the change. This means that when we query OSM for details of a road we often get back information about multiple ways. Some of these might be things like bike paths which we can filter using tags, but often they’ll be sections of the road that we want. For example, this query for Gore Street, within the bounds of its intersections with Gertrude Street and Webb Street, returns details of two ways.

way["highway"~"^(trunk|primary|secondary|tertiary|unclassified|residential|service|track|pedestrian|living_street)$"][name="Gore Street"](-37.8062302,144.98128480000003,-37.8040076,144.9826827);
out body;
>;
out body;

You can view the result in Overpass Turbo.

However, that doesn’t mean that the full extent of both ways is contained within the bounding box, just that some of the nodes of both ways are inside. Because of this, I filtered the results from all the ways and only kept nodes whose coordinates were within the desired region.

Problems finding intersections

The method described above works pretty well, and once I understood enough about the Overpass API to get out actual paths that I could display on a map, I fed all of the CUA photos through a script and got useful data for more than 80% of them.

One of my early tests.

Then I spent a lot of time trying to understand where the remainder were failing.

Some of them failed because the titles were missing information, or were formatted in a way I didn’t expect. For example, instead of a second intersecting road, some titles just said ‘to end’. This makes perfect sense to a human looking at a map, but it’s difficult to handle programmatically.

Some photos either recorded the wrong suburb, or the boundaries of the suburb had moved since the photos were taken. For example, many of the photos described as being from Eaglehawk are now in California Gully.

Similarly, some road names were wrong either because of documentation errors, or because the names have changed over time. There are also some variations in the way OSM records road names – in particular, I found that roads with hyphenated names sometimes had spaces around the hyphen and sometimes didn’t. There were also a couple of cases where names weren’t attached to the corresponding road segment in OSM, but I was able to edit these in OSM directly.

Other roads had multiple names, or change names along their path. I mean, what’s going on with Brunswick Street and St Georges Road in Fitzroy? Country towns seemed most prone to this – a highway might become ‘Main Road’ within the town boundaries, or the order of hyphenated places in road names might change. I found one road in Clunes that had four different names within the space of a few hundred metres.

One road, four names!

Finally, the routes of some roads had changed – intersections no longer intersected, roads were closed, or new parks had popped up to split a road in two.

My processing script logged the titles I couldn’t locate and I worked through the list manually, trying to identify what each problem was. I suppose there’s two ways I could’ve handled these problems – building more fuzziness into the process to check for things like alternative names, or by compiling a list of ‘corrected’ titles. I started off using the first approach, but as I worked through more and more anomalies, the checking logic became very complicated and inefficient. Just think about the knots you can tie yourself in trying to handle a title where the suburb is wrong and the main road changes names in between intersections.

I refactored the code multiple times, but it’s still pretty messy. In the end I created a list of ‘corrected’ titles as well, so it was a bit of a hybrid approach. I suspect I could have saved myself a lot of pain if I’d reversed the process – compiling ‘corrected’ titles first, then adapting the logic as patterns emerged.

There are still some photos I haven’t located. In some cases I just don’t have enough information. In others I need to manually record coordinates or way ids to feed into the process, and I haven’t worked out the best way to do this yet. You can see the titles that I’ve haven’t geolocated yet in the files: cua-not-found.txt and cua-not-parsed.txt.

In total, 18,603 out of 20,644 photos have been geolocated. That’s over 90%!

Assembling the data

I processed the data in a couple of phases to get it in the shape I wanted.

The first step was to group all the photos by title, so I could link each group to its location. But remember that titles often record which side of the road a photo was taken on. To bring all sides of a road segment together into a single group, I created a key from a normalised/slugified version of the title with the side value removed. I used this key to save information about each side within the same group.

I ended up with a dataset with this sort of structure (a truncated example):

"iffla-street-south-melbourne-coventry-street-normanby-street":
    {
        "title": "Iffla Street, South Melbourne, from Coventry Street to Normanby Street",
        "sides":
        {
            "east side":
            {
                "title": "Iffla Street, South Melbourne, from Coventry Street to Normanby Street - east side.",
                "images":
                [
                    {
                        "ie_id": "IE20321667",
                        "alma_id": 9939649155207636
                    }
                    more photos...
                ]
            },
            "west side":
            {
                "title": "Iffla Street, South Melbourne, from Normanby Street to Coventry Street - west side.",
                "images":
                [
                    {
                        "ie_id": "IE20320072",
                        "alma_id": 9939655629407636
                    },
					more photos...
                ]
            }
        },
        "ways":
        {
            "27631235":
            [
                [
                    144.9503379,
                    -37.835322
                ],
                more points...
            ]
        }
    },

You can see how the sides and matching ways have been brought together under the key value.

This structure was useful for grouping and processing the data, but to create a map interface I needed to bring the geospatial information to the surface. The first version of the interface used one big GeoJSON file in which the features were MultiLineStrings created from the paths of each road segment. The photo data was saved in the properties of each GeoJSON feature.

It sort of worked. The roads with photos were highlighted, and clicking on the roads displayed the photos. It was only when I changed the opacity of the lines that I realised that, in many cases, different road segments were being piled on top of each other. When the lines were opaque these piles were invisible, but add a bit of transparency and you could see that some lines were darker than others. Clicking on the lines only displayed the top layer, so some groups of photos were effectively invisible.

Version one of the interface showing how the colour of the highlighted roads varied once I decreased the opacity.

Why did this happen? I’d wrongly assumed that each segment of road would only have one group of photos associated with it. But it’s not hard to find cases where this is not true. Consider Moor Street, Fitzroy, between Nicholson Street and Brunswick Street. On the north side, there is a single group of photos that document the buildings between Nicholson Street and Brunswick Street. However, on the south side there’s two groups of photos. One covers the section between Nicholson Street and Fitzroy Street, the other covers Fitzroy Street to Brunswick Street. One section of road, three groups of photos…

Moor Street, Fitzroy, between Nicholson Street and Brunswick Street, in the new CUA Browser, showing the three photosets associated with the one section of road.

To make these layered groups more easily accessible through the interface I had to change the way the data was organised – separating the GeoJSON from the photosets so that multiple photosets could be associated with a single geospatial feature. I decided to create a GeoJSON feature for every OSM way in the dataset. However, I needed to prune the way’s coordinates to only include those that were part of the CUA road segments. To do this, I saved all the way data when I found the road segments. Then in the second processing phase, I grouped the way coordinates associated with the road segments and compared this list to the full way path. Any coordinate in the way path that wasn’t in the road segments was removed. It seems unnecessarily complex, but I wanted to make sure that only the parts of roads associated with photos were highlighted in the interface.

The result was two data files. The first, cua-ways.geojson, contains the pruned way paths and their OSM identifiers. The second, cua-photos.json, contains information about each photo set, including the sides, photos, paths, and associated way identifiers. The datasets are linked by the way identifiers.

Constructing the interface

My plan for the interface was pretty simple. There’d be a map on which all the road segments associated with CUA photos were highlighted. Clicking on a highlighted section would show the photos. I wanted to display the photos as if you were scanning the streetscape, so I decided to put them all side-by-side in a gallery that scrolled horizontally.

The first version used Leaflet to display the maps and, as noted above, had some problems where there were multiple photosets associated with a segment of road.

For the second version I decided to switch to MapLibre because it seems a bit more active and up-to-date. I’d already used MapLibre in the SLV Newspapers Explorer.

The interface first loads the cua-ways.geojson file to highlight the relevant roads. When you click on one of the roads, the way id is passed to a function that looks for associated photo sets in the cua-photos.json data. If there’s only one linked photoset, then the photos are displayed. However, if there’s more than one linked photoset, they’re displayed as a list. The user then selects from the list to display the related photos.

A couple of other things happen when you click on a way or select a photoset: the colour of the selected road segment changes, and the browser url is updated with the way or photoset identifier. You can bookmark or share these urls to go directly to a specific road or photoset. There’s also a button to reverse the order of the images – they scroll left to right, but sometimes they seem to have been photographed right to left.

Goodbye 2025! A brief summary of the highlights and lowlights…

Wed, 31 Dec 2025 16:14:14 +1000

My 2025 started badly and ended well. In the first few months of the year, battles with the gatekeepers at Trove sent me spiralling into a pretty dark place. But by year’s end I was having fun, working with the wonderful people at the State Library of Victoria. In between I caught up on some overdue project maintenance. Most of this is documented in the 37 blog posts I wrote this year, but here’s a quick summary.

Highlights

From September to December, I was Creative Technologist-in-Residence at the SLV LAB, exploring ways of opening up the Library’s place-based collections. There’s still a few things to finish off, but here’s list of the results so far.
As part of my SLV work, I created a fully searchable version of the 24 volumes of Sands & MacDougall directories digitised by the Library. This followed the pattern I’d used for 54 volumes of the NSW Post Office Directories, 44 volumes of the Sydney Telephone Directories, and 54 volumes of the Tasmanian Post Office Directories. So there’s now 176 volumes from the 1880s to the 1950s that can be easily explored for people and places— and all free to use of course.
In April, I added a new section to the GLAM Workbench documenting the Public Record Office Victoria’s collection API. I also used the API to create a Data Dashboard that provides an overview of PROV’s collection.
Also in April, I updated the GLAM Name Index Search to include an additional 6 million records from PROV. In total, the GLAM Name Index now includes more that 12 million records in 293 datasets from 10 Australian GLAM organisations — another free resource for Australian researchers.
In July I undertook some overdue maintenance on a variety of old apps and projects. In the process, I resurrected my old Wragge Labs domain and created a showcase of many of the websites, apps and experiments I’ve worked on over the past 30 years.
I was particularly pleased to get The Future of the Past working again, so once more you can create fridge magnet poetry from an odd collection of words harvested from Trove newspapers! I built FOTP back in 2012 when I was the Harold White Fellow at the NLA. Also this year I finally got around to transcribing my Harold White Lecture.
In June I wrote a short piece on the GLAM Workbench for the forthcoming publication Building User-Friendly Toolkits and Platforms for Digital Humanities. I think it provides a useful summary of what the GLAM Workbench is, and what I’d like it to be. I also wrote up the short but glorious history of Trove Twitter bots.

Lowlights

Saying goodbye to 15 years of work on Trove. It still hurts. And I still miss resources such as @TroveNewsBot and the Trove API Console which ran happily for more than a decade before being killed without warning by the NLA.
Saying goodbye to 17 years of work on the National Archives of Australia’s collections. This will be the first New Year’s Day in a decade when I haven’t updated my harvest of files with the access status of closed.

Next year

In 2026, I’m looking forward to starting work on the RAPID project, building on the work I’ve done on Commonwealth Hansard over the years to create new examples and documentation.
I’m honoured to be giving the closing keynote at the GLAM Labs Futures conference in Edinburgh in June — hoping we can pull together the funds to get there in person!

How you can help

Much of my work is unfunded, and keeping resources such as the GLAM Name Index running costs real money. I’ve been very grateful for the support of my GitHub sponsors over past years. Their contributions help cover a substantial proportion of my cloud hosting costs. But bidding farewell to Twitter and Trove has had an impact on my sponsorship income. If you use or value the things I build to help researchers make use of GLAM collections, you might like to sponsor me on GitHub, or Buy Me a Coffee. All contributions are greatly appreciated!

If you can’t afford a financial contribution, there are other ways you can help!

Let me know how you’re using my stuff! A bit of positive feedback does wonders when my enthusiasm is flagging. You can find my contact details at timsherratt.au.
Tell others how you use my stuff! Getting information about resources out to those who might benefit is really hard, so your help would be greatly appreciated.
The GLAM Workbench describes a few other ways you can get involved.

Goodbye 2025!

Exploring Victorian newspapers

Tue, 16 Dec 2025 12:03:48 +1000

Newspapers are a vital source for local history. That’s why, back in 2014, I created the Trove Places app – a map interface to help people find Trove’s digitised newspapers by their place of publication or distribution. Trove Places has proved very popular, and the State Libraries of South Australia, and Victoria, amongst others, point their users to it to help with their research. I’ve updated the data several times over the years, though the Trove’s new gatekeeping regime will make future updates difficult.

During my residency at the State Library of Victoria, one of the librarians noted how useful the app was, and asked whether it might be possible to include undigitised newspapers from the SLV catalogue as well as those in Trove. It was, and I did – here’s a brand new app to explore Victorian newspapers, both digitised and undigitised!

Just click on the map to find Victorian newspapers!

It’s pretty easy to use. You just click on the map in an area you’re interested in. The map will display the 20 nearest places where newspapers where published or distributed. The size of the markers indicates how many titles are associated with each place. In the sidebar, details of the newspapers are listed by place, ordered by their distance from your selected point.

You can also find local newspapers using the my place app. Once you enter an address, newspapers from your suburb or town will be displayed, as well as those from nearby locations.

newspapers from Geelong displayed in the my place app

Assembling the data

How do you find Victorian newspapers? The reference librarians at the SLV pointed me to the ‘Place newspaper published’ field in the catalogue. Searching this field for ‘Australia–Victoria’ returns 3,997 results, compared to the 460 digitised in Trove.

The first step in assembling the data was to harvest the newspaper records from the SLV catalogue. To do this I made use of the Primo JSON API. The method is documented in this notebook. The results was a newline-delimited JSON file, with one record per line.

The harvested metadata doesn’t include links to digitised versions of newspapers in Trove. To add these links I first looked in the 856 field of the newspaper’s MARC record. I also noticed that some Trove links were being loaded from an ‘edelivery’ JSON file, so I added these as well. I ended up with 344 unique links to Trove, but not all of these were to digitised newspapers as some more recent newspapers are available through eLegal deposit. In total there were 268 unique links to digitised newspapers. This is well short of the 460 Victorian newspapers in Trove. Why? It’s possible that the links haven’t been added into the SLV catalogue, or that the ‘place newspaper published’ field hasn’t been populated for records that include the links. It’s also possible that Trove links are hiding somewhere else in the SLV catalogue!

To try and fill this gap, I compared the catalogue metadata with my most recent harvest of Trove newspaper titles. If the Trove url was missing, I searched the catalogue data for the newspaper title. I then manually checked the results, making sure the dates and titles lined up, and added positive matches to a new CSV file which I merged back into the main dataset. This added another 152 Trove links.

The next step was to link the ‘place newspaper published’ values to places with known locations. The ‘place newspaper published’ information is included in the lds03 field of the harvested metadata. Records often contain references to multiple places, so I split all the newspaper/place combinations out into separate rows. I then matched the places against a list of Victorian place names and coordinates downloaded from the VicNames database. If there were no matches, I manually checked and adjusted the place names – for example, I changed ‘East Kew’ to ‘Kew East’, and ‘Bayside’ to ‘Bayside City’.

To add any Trove digitised newspapers that might still be missing, I made use of my existing Trove harvests. First I compared my Trove Places dataset with my latest harvest of newspaper titles. There were a few new titles, so I matched them to places using this notebook, based on my original Trove Places code. I then merged the Trove Places dataset with the new titles and checked it against the catalogue dataset. If any urls were missing, I added a record from the Trove data.

All of the processing steps are documented in this notebook.

Building the apps

To make the data easily searchable by its geospatial coordinates, I loaded all the data into an SQLite/Spatialite database and published it online using Datasette. The database contains linked tables for titles and places.

I also created a couple of canned queries which, together with Datasette’s built-in JSON API, made it possible to retrieve places and titles based on their distance from a given point. For example, this url retrieves places ordered by their distance from the point at latitude -36.815, longitude 144.965 : https://slv-places-481615284700.australia-southeast1.run.app/newspapers/places_from_point.json?longitude=144.965&latitude=-36.815&distance=100000&_shape=array

When you click on the map in the Victorian Newspapers Explorer, it fires off a request like this to find nearby places. It then makes a second request to find newspapers related to those places and displays the results.

The Victorian Newspapers Explorer was my first attempt at using MapLibre rather than Leaflet to display maps using Javascript. It’s more verbose, but more flexible, so I think I’ll gradually switch over my other apps, including Trove Places.

All the code of the Victorian Newspapers Explorer is in the slv-demo-apps repository.

Why bother?

Wed, 03 Dec 2025 15:18:22 +1000

This was the introduction to my talk on the results of my time as Creative Technologist-in-Residence at the State Library of Victoria. My slides, with my full notes are available online, but after a very strange year that has travelled from disappointment to exhilaration, I thought it was worth posting these words separately.

The work that I do, that I’ve been doing for the past 30 years, is focused on helping people find, use, and understand the wonderfully rich collections held by our libraries, archives, and museums – the GLAM sector. Much of it is quite practical, resulting in tools and applications that are used by a wide range of researchers. Some of it is playful, some of it is critical, and some of it is just weird.

You can browse through some of this history on wraggelabs.com. And if you’re interested in my current crop of tools you can head to the GLAM Workbench.

As some of you may know, I had a few setbacks at the beginning of this year which really made me wonder whether I wanted to continue doing this sort of work.

I mean, why bother?

I’m really grateful that this residency has given me a chance to refocus on the reasons why I do what I do.

I suppose my starting point is the fact that libraries can’t do everything themselves.

I’m thinking here specifically about the digital research space. There’s a lot that libraries, and other GLAM organisations, can do – provide search interfaces, APIs, downloadable datasets, documentation, and examples of how to access APIs and datasets using code. The sorts of things that the SLV LAB is doing.

I should pause here to unpack some acronyms. APIs deliver data in a form that machines can understand and process. Websites are for humans, APIs are for computers. APIs are also building blocks which can be connected up to create new applications – and I’ll be showing some examples of this later on.

So there is much that GLAM organisations can do to support digital research. But it will never be enough. Researchers – whether they be academics or family historians – will always want more. It is in the nature of research to ask new questions, to head off in new directions.

But rather than see this as a source of tension, I see it as an opportunity for collaboration. An opportunity to cultivate the in-between spaces where research methods, tools, and results can feed back into the contextual frameworks of GLAM collections. Where GLAM organisations can share and celebrate the work that’s done with their data. Where all can find inspiration, ideas and support.

We’ve tended to call this sort of stuff infrastructure, but I think that really downplays the human aspect. The research sector has started to develop the funding and career structures necessary to allow people to build and maintain these infrastructures, but we need more. We need to recognise that a single tool, developed by an individual without institutional support, can be just as important as a multi-million dollar platform. Passion is precious and needs to be protected.

Most of all, we need to keep a focus on the ethical imperatives – the reasons why we bother and why it matters. For me it boils down to openness and generosity. I have benefited greatly from the openness and generosity of others, and I want to pass that on. It’s the glue we need to hold those in-between spaces together; the sustenance we need to maintain our enthusiasm in the face of all the crap; the inspiration we need to try something new.

Initiatives like the SLV LAB are important, not just because they foster innovation, but because they invite new ideas in. They even give space for ageing hackers like me to spend some dedicated time doing what they love – crafting new pathways for people to explore our glorious GLAM collections.

Counting down... (to the end of my SLV residency)

Wed, 19 Nov 2025 15:01:22 +1000

My stint as Creative Technologist-in-Residence at the State Library of Victoria LAB comes to an end in a few weeks time and I’m frantically trying to pull things together. I’ll be back on-site at the Library from 1 to 5 December for a few events, and to report back to staff on what I’ve been doing.

On Tuesday 2 December, there’ll be a public workshop on using and contributing to the GLAM Workbench. Here’s the blurb:

More and more GLAM organisations are looking to share their data to foster creativity and support new types of research. But how can you help potential users understand the possibilities of your data? This workshop will explore how GLAM organisations can create and share resources that encourage experimentation.

The GLAM Workbench is a large collection of tools, hacks, and tutorials aimed at helping researchers make use of collection data. It uses platforms such as Jupyter notebooks to create live, working examples that run in your browser without additional software. Similar repositories of computational resources are being developed by GLAM organisations around the world.

This workshop will introduce the technologies and standards used in the GLAM Workbench, such as Jupyter notebooks. It will provide an overview of related activity around the world, including best practice guidelines for GLAM organisations developing computational resources. It will explain how organisations and individuals can contribute content to the GLAM Workbench, or use it as a model to create their own specialised workbenches.

Sharing data is important, but so is sharing skills, tools, and knowledge. Come along to find out how the GLAM Workbench can help.

It’s a free, hybrid event (in person and online) and will run from 1.00-3.00pm. A sign up page should be available soon.

On Wednesday 3 December I’m presenting the results of my residency in a ‘technologist’s talk’. It’s an internal event, but it’s in the public ‘Create quarter’ of the Library, so I think anyone can pop in. Hopefully there’ll be a video I can share.

To give you an idea of what I’ll be talking about, here’s some of the outcomes so far:

hacking the library workshop (slides, and blog post about urls)
bounding boxes for parish maps (blog post, code)
geolocating the Committee for Urban Action collection of photographs (prototype interface, still documenting the method)
a new fully-searchable version of the Sands & MacDougall’s directories (blog post, database, and another blog post)
georeferencing digitised maps – over 500 so far! (documentation, dashboard, data repository, blog post)
and as of yesterday, 3,000+ geolocated newspapers (documentation and interface coming!)

First attempt at mapping places of publication and distribution of Victorian newspapers

At the moment I’m trying to bring it all together in a new interface that let’s you type in an address and find collection materials relating to your home, your street, and your suburb. Only two weeks to go! Eeek!

Work in progress!

Some Sands & Mac tweaks thanks to ALTO and IIIF

Sun, 16 Nov 2025 14:17:26 +1000

I posted recently about my new fully-searchable version of the Sands & MacDougall directories. I’ve now moved on to try and pull together a number of the State Library of Victoria’s place-based collections into a new discovery interface. It’s going to be a busy couple of weeks as my residency ends in early December!

I wanted to incorporate Sands & Mac search results into the new interface. Getting the data was easy because Datasette has a JSON API baked in. But what about the images? I could just display a thumbnail of the whole page, but it would be better to show a snippet of the actual entry. Thanks to IIIF and ALTO, I now can.

IIIF makes it easy to cut small sections out of a larger image. You just put the coordinates of the desired section in the IIIF url. As I noted in my previous post, the ALTO files that contain the OCR data from Sands & Mac include the coordinates of every line, and every word. I just had to bring the two together.

All I did was update the code that extracts the data from the ALTO files to save the results as newline delimited JSON instead of a plain text file. Each line in each volume of Sands & Mac is now saved a JSON object that contains the text, as well as the height, width, vertical position, and horizontal position of the line within the page image. When I load up the SQLite database, I add the values for h, w, x, and y as well as the text for each line.

What does this make possible?

When you go to an individual entry, the page image now automatically pans and zooms so that the current entry is at the centre of the image viewer. I just updated the OpenSeadragon code to focus on the entry’s position.
If you share an entry on social media, a snipped out section of the page image showing the selected entry is displayed as there’s now an image META tag that points to an IIIF url.
You can retrieve entries via the API and use the coordinates to request snipped out images of them via IIIF.

Nice image snippets thanks to IIIF and ALTO (and a sneak prview of what's coming...)

A new way of searching Sands & Mac

Wed, 12 Nov 2025 21:25:21 +1000

In the fortnight I spent onsite at the State Library of Victoria, ‘Sands & Mac’ was mentioned many times. And no wonder. The Sands & McDougall’s directories are a goldmine for anyone researching family, local, or social history. They list thousands of names and addresses, enabling you to find individuals, and explore changing land use over time. When people ask the SLV’s librarians, ‘What can you tell me about the history of my house?’, Sands & Mac is one of the first resources consulted.

The SLV has digitised 24 volumes of Sands & Mac, one every five years from 1860 to 1974. You can browse the contents of each volume in the SLV image viewer, using the partial contents listing to help you find your way to sections of interest. To search the full text content you need to use the PDF version, either in the built-in viewer, or by downloading the PDF. There’s a handy guide to using Sands & Mac that explains the options.

However, there’s currently no way of searching across all 24 volumes, so as part of my residency at the SLV LAB, I thought I’d make one!

Try it now!

My new Sands & Mac database follows the pattern I’ve used previously to create fully-searchable versions of the NSW Post Office directories, Sydney telephone directories, and Tasmanian Post Office directories. Every line of text is saved to a database, so a single query searches for entries across all volumes. You can also use advanced search features like wildcards and boolean operators.

Search across all 24 volumes!

Once you’ve found a relevant entry you can view it in context, alongside a zoomable image of the page. You can even use Zotero to save individual entries to your own research database. This blog post from the Everyday Heritage project describes how the Tasmanian directories have been used to map Tasmania’s Chinese population.

View each entry in context! (Here's my Dad building his first house in Beaumaris in the 1950s.)

There’s still a few things I’d like to try, such as making use of the table of contents information for each volume. I’d also like to create some additional entry points to take users directly to listings for individual suburbs (maybe even streets!). Each volume has a directory of suburbs, so it would be a matter of extracting and cleaning the data and linking the entries to digitised pages. Certainly possible, but I don’t think I’ll have time to get it all done before the end of my residency. Perhaps I’ll try to get at least one volume done to demonstrate how it might work, and the value it would add. As I was writing this blog post I also realised there’s a dataset of businesses extracted from the Sands & Mac, so I need to think about how I can use that as well!

Technical information follows…

I’ve documented the process I used to create fully-searchable versions of the Tasmanian and NSW directories in the GLAM Workbench. I followed a similar method for Sands and Mac, though with a few dead-ends and discoveries along the way.

Downloading the PDFs

I assumed that it would be easiest to work from the PDF versions of each volume, as I’d done for Tasmania. So I set about finding a way to download them all. There’s only 24 volumes, so I could have downloaded them manually, but where’s the fun in that?

I started with a CSV file listing the Sands & Mac volumes that I downloaded from the catalogue. This gave me the Alma identifiers for each volume. To download the PDFs I needed two more identifiers, the IE identifier assigned to each digitised item, and a file identifier that points to the PDF version of the item. The IE identifier can be extracted from the item’s MARC record, as I described in my post on exploring urls. The PDF file identifier was a bit more difficult to track down. The PDF links in the image viewer are generated dynamically, so the data had to be coming from somewhere. Eventually I found that the viewer loaded a JSON file with all sorts of useful metadata in it!

The url to download the JSON file is: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. In the summary section I found identifiers for small_pdf and master_pdf.

I could then use these identifiers to construct urls to download the PDFs themselves: https://rosetta.slv.vic.gov.au/delivery/DeliveryManagerServlet?dps_func=stream&dps_pid=[PDF id]

Once I had the PDFs I used PyMuPDF to extract all the text and images. As I suspected the text wasn’t really fit for purpose. The OCR was ok, but the column structures were a mess. Because I wanted to index each entry individually, it was important to try and get the columns represented as accurately as possible. The images in the small PDFs were already bitonal, so I started feeding them to Tesseract to see if I could get better results. After a bit of tweaking, things were looking pretty good. But when I came to compile all the data, I realised there was a potential problem matching the PDF pages to the images available through IIIF. I found one case where some pages were missing from the PDF, and another couple where the page order was different.

As I was looking around for a solution, I realised that those JSON files I downloaded to get the PDF identifiers also included links to ALTO XML files that contain all the original OCR data (before it got mangled by the PDF formatting). There was one ALTO file for every page. Even better, the JSON linked the identifiers for the text and the image together – no more page mismatches!

Downloading the ALTO files

Let’s start this again shall we. After wasting several days futzing about with the PDFs, I decided to download all the ALTO files and extract the text from them. As I downloaded each XML file, I also grabbed the corresponding image identifier from the JSON and included both identifiers in the file name for safe keeping.

The ALTO files break the text down by block, line, and word. To extract the text, I just looped through every line, joining the words back together as a string, and writing the result to a new text file – one for each page.

It’s worth noting that the ALTO files include all the positional data generated by the OCR process, so you have the size and position of every word on every page. I just pulled out the text, but there are many more interesting things you could do…

Assembling and publishing the database

From here on everything pretty much followed the pattern of the NSW and Tasmanian directories. I looped through each volume, page, and line of text, adding the text and metadata to a SQLite database using sqlite_utils. I then indexed the text for full-text searching. At the same time I populated a metadata file with titles, urls, and few configuration details. The metadata file is used by Datasette to fill in parts of the interface.

I made some minor changes to the Datasette template I used for the other directories. In particular, I had to update the urls that loaded the IIIF images into the OpenSeadragon viewer. But it mostly just worked. It’s so nice to be able to reuse existing patterns!

Finally, I used Datasette’s publish command to push everything to Google Cloudrun. The final database contains details of more than 50,000 pages, and over 19 million lines of text! It weighs in at about 1.7gb. The Cloudrun service will ‘scale to zero’ when not in use. This saves some money and resources, but means it can take a little while to spin up. Once it’s loaded, it’s very fast. My original post on the Tasmanian directories included a little note on costs, if you’re interested.

More information

The notebooks I used are on GitHub:

Here are some posts about the NSW and Tasmanian directories:

Making NSW Postal Directories (and other digitised directories) easier to search with the GLAM Workbench and Datasette (September 2022)
From 48 PDFs to one searchable database – opening up the Tasmanian Post Office Directories with the GLAM Workbench (September 2022)
Where’s 1920? Missing volume added to Tasmanian Post Office Directories! (September 2024)
Six more volumes added to the searchable database of Tasmanian Post Office Directories! (November 2024)

Turning the SLV's maps into data with Allmaps and some GLAM plumbing

Tue, 04 Nov 2025 14:02:53 +1000

I often describe what I do as GLAM data plumbing. Most of the time I’m not creating new tools, I’m figuring out what data is available and how I can connect it up to existing tools. It’s rarely straightforward, but if I can get all the pipes connected and data flowing in the right direction, suddenly new things become possible.

Things like turning all the State Library of Victoria’s digitised maps into data.

I’ve just created a workflow that uses AllMaps and IIIF to georeference the SLV’s digitised maps. There’s some technical details below, but the idea is pretty simple. A userscript links the SLV image viewer to Allmaps – so you just click on a button, and the digitised map opens, ready for georeferencing.

Why is this useful? Georeferencing relates a digitised map to real world geography. It describes the map’s position and extent using geospatial coordinates – turning historic documents into geospatial data that can be indexed, visualised and manipulated. Georeferencing opens digitised maps to new research uses.

So, how many maps we can georeference before my residency finishes in December? Hundreds? Thousands? If you like maps and want to help, head to the documentation page to find out how to get started. And if you want to see how things are progressing, have a look at the project dashboard.

View the documentation to get started

A few technical details follow…

Early on in my time as Creative Technologist-in-Residence at the State Library of Victoria, I started playing around with Allmaps for georeferencing digitised maps. It’s a great tool (really a suite of tools and standards) because instead of constructing a whole new platform it integrates with existing IIIF services. The SLV provides digitised images through IIIF, so I thought it should be possible to use Allmaps to georeference the SLV’s map collection.

But I struck a problem that took some time to unravel. The IIIF urls in the SLV manifests include port numbers and that confused Allmaps. The manifests also sometimes contained references to image formats that weren’t actually accessible, generating errors when they were loaded. Hopefully these problems will be fixed by the SLV, but in the meantime I’ve created a proxy service that edits the manifest on the fly. The proxied urls can be loaded into the Allmaps Editor without errors. Pipes fixed, data flowing!

Using the manifest proxy

To generate a link to a proxied manifest, first grab the item's IE identifier from the url of the digitised item viewer. For example, the identifier in this url https://viewer.slv.vic.gov.au/?entity=IE15485265&mode=browse is IE15485265. Once you have the identifier, add it to the end of the url https://wraggelabs.com/slv_iiif/. For example, https://wraggelabs.com/slv_iiif/IE15485265. You can then supply this url to the Allmaps editor.

But having to fiddle around with proxies didn’t make a great user experience. I needed some way of integrating the two services, so that a user could just click a button in the SLV website and start editing in Allmaps. Userscripts to the rescue!

I wrote recently about hacking GLAM collection interfaces using userscripts. Since I started my residency at the SLV, I’ve also created a userscript to display the IIIF manifest url in the SLV image viewer, and run a Code Club workshop where we played around with an assortment of SLV website hacks.

As in a number of these examples, the georeferencing userscript adds new features to the SLV website, but there’s a fair bit more going on under the hood. It runs automatically every time you load the SLV image viewer, and then:

it checks the metadata of the digitised item to see it it’s a map (or something that contains maps, like an atlas or street directory)
if it looks like a map, it generates an Allmaps identifier using the item’s IIIF manifest url and checks with Allmaps to see whether the item has already been georeferenced
it adds a ‘Georeferencing’ section to the page, with a button to georeference the item (or edit the existing georeferencing)
if the item has already been georeferenced, it adds a button to view the item in the Allmaps Viewer, and embeds a live preview

Accessing metadata

The userscript gets the item metadata from a JSON file that's loaded by the image viewer. The JSON file includes a lot of extra, useful information about the digitised item. To access the JSON file, you just construct a url like this: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. The IE identifier is in the url of the image viewer.

Allmaps identifiers

Allmaps creates its identifiers by hash encoding the IIIF urls. The userscript borrows some code from the Allmaps id module to generate the ids, then sends a HEAD request to the Allmaps API to see whether an entry for the current manifest exists.

Example of an item that hasn't been georeferenced yet

Example of an item that has been georeferenced, displaying an embedded version of the Allmaps viewer

I’ve also created a GitHub repository to save copies of the data. Every two hours this notebook is run to query the Allmaps API for newly georeferenced maps. These are added to a dataset which is saved in three formats:

a CSV file
a CSV file that includes thumbnails and links for viewing in Datasette-Lite
a GeoJSON file, that can be viewed in services like geojson.io

At the same time, the data for each individual map is downloaded and saved as IIIF annotations (in JSON) and GeoJSON.

Finally, this notebook is run to generate a dashboard that provides an overview of the project’s progress.

The project dashboard is updated every two hours

One of the Allmaps developers described all my plumbing and workarounds as a ‘very cool lofi example of how you can set this up with little means’, and I think that’s pretty apt. It’s really just an experiment to demonstrate the possibilities, but by connecting up existing services it’s generating real data of long term value.

Me at 63...

Mon, 03 Nov 2025 16:55:17 +1000

Originally published in Pharos, newsletter of the Professional Historian’s Association (Vic & Tas), October-November 2025, in the ‘Member Profile’ section.

In the early 1990s I started working for a small self-funded organisation called the Australian Science Archives Project. Our mission was to preserve and raise awareness of Australia’s scientific past. When the web came along, we realised it provided an enormous opportunity to communicate history to the public. So I taught myself web development and created the first archives website in Australia. Since then my work has continued to explore what happens when we release GLAM collections into online spaces where people can see and use them differently.

What kind of work have you done? What are you working on now?

I’ve had a range of jobs in the GLAM and university sectors. While ‘history’ wasn’t often in my job title, I’ve always regarded myself as a historian first – whether I was coding, editing, writing, teaching, or managing, history was always the frame through which I understood my work. At the same time, I’ve maintained my own independent practice as a ‘historian and hacker’, developing tools and resources for other researchers, such as the GLAM Workbench. Much of this work is unfunded, but by sharing it openly I’ve created new opportunities for collaboration. For example, I’m currently the ‘Creative Technologist-in-Residence’ at the State Library of Victoria, bringing my years of GLAM hacking to bear on the Library’s place based collections.

Research or writing? (What do you enjoy more and why?)

Researching, or writing, or coding, or teaching, or outreaching (what is the correct verb?) – all have their joys and travails. For me, research is less about finding things in archives and libraries, and more about how we find things in archives and libraries. I poke about in online collections to try and understand how they work, what they reveal, and what they hide. This often leads to the development of new tools, the writing of documentation and blog posts, and sometimes even real, published articles. It’s a process that has consumed my life, for better or worse. Coding often slips into obsession when I have a gnarly problem to crack. Writing is a slog, but there’s nothing like the pleasure of a finely-turned sentence. Teaching is exhausting, but also exhilarating when you see the light bulb of understanding flick on.

What are the best and hardest things about the kind of work you do?

The best thing, the absolute hands-down best thing, is hearing from people who use, or have benefited from the tools and resources that I’ve created. I make things to help researchers see and use GLAM collections in new ways, so finding out what they’ve been doing with my stuff always provides a much-needed jolt of inspiration.

However, the flip side is that getting information about my tools and resources out to the people who might benefit most is hard and often frustrating work. I churn away in the social media mines, but people and organisations seem much more reluctant to share new work these days. There was a time (yeah, the good old days) when GLAM organisations actively engaged with researchers online, sharing the cool things people were doing with their collections. But not now. We all learn through the generosity of others, and I think its important that we find ways to support and enlarge the realm of generosity.

Creating bounding boxes for parish maps in the SLV collection

Mon, 06 Oct 2025 14:17:51 +1000

The State Library of Victoria holds a collection of 8,804 parish maps. As part of my residency at the SLV LAB, I’ve been poking around in the metadata.

SLV staff have geocoded many of the parish maps using the Composite Gazetteer of Australia, which provides coordinates for Victorian parishes and boroughs. These coordinates give us a point which should be roughly at the centre of each map, enabling us to visualise their locations and distribution. But how much area do they cover? To answer that question we need a bounding box that includes the coordinates of each corner of the map. We could create bounding boxes by using something like AllMaps or MapWarper to georeference each individual map, but that’s going to take a while! As a quick and dirty alternative, I wondered if it was possible to generate approximate bounding boxes from the available metadata. It seems we can!

The metadata

There are three pieces of metadata we need to construct bounding boxes:

the latitude and longitude of the centre point
the size of the physical map
the scale of the map (ie how the size of the map relates to real world)

The coordinates and scale can be included in a couple of different places in the map’s MARC record. The 034 field is specifically for ‘Coded Cartographic Mathematical Data’. The relevant subfields are:

$a: category of scale
$b: constant ratio linear horizontal scale (this is the most likely type of scale)
$d: westernmost longitude
$e: easternmost longitude
$f: northernmost latitude
$g: southernmost latitude

If the coordinates describe a point rather than a bounding box, then $d and $e will be the same, and $f and $g will be the same.

String representations of coordinates and scale can be found in the 255 field. The relevant subfields are:

$a: statement of scale, eg Scale [ca. 1:90,000].
$c: statement of coordinates, eg (E 142°18'/S 37°33')

The size of the map is recorded in 300 (physical description) field under the $c (dimensions) subfield. For example: on sheet 40 x 51 cm .

The method

I started with an existing dataset downloaded from the catalogue by SLV staff. This dataset included the scale and coordinate information in the 034 field, and the coordinate string in 255$c. At first I didn’t realise that the 034 held geo data, so I separately downloaded the scale information from 255:$a in each item’s MARC record (d’oh). If the maps were digitised, I also wanted their image identifiers so I could access them through the SLV’s IIIF service. The image id from the 956$e field of the MARC record can be used to construct an IIIF manifest url, so I extracted them as well.

Once I had all the catalogue data, I had to make sure everything was in a format I could work with. The coordinates in the MARC records are recorded as degrees/minutes/seconds, so I had to convert them to decimal values. The scale factor needed to be an integer, and I needed to extract the height and width as integers from the dimensions field.

I used lat_lon_parser to convert the coordinates to decimal, but needed a bit of regex string manipulation to get the values into a format that could be parsed. Regex also came to the rescue in getting the map dimensions. All the details are in this notebook.

Creating bounding boxes

After some searching I found this StackOverflow comment that described how to create a bounding box from a point, distance, and bearing. The point I already had, but the distance and bearing had to be calculated. Trigonometry to the rescue!

The distance from the point at the centre of the box to one of its corners is the hypotenuse of a right-angled triangle whose sides are equal to half the width and half the height of the map, and thanks to Pythagorus we know:

Once I had the distance in cm, I converted to inches, then multiplied by the scale factor, and finally converted the inches to miles. (It now occurs to me that there’s no need to convert to imperial measurements, but it doesn’t make any difference either way.)

The bearing that points to the corner of the box is the angle inside the same right-angled triangle, so can be calculated using:

With the point of origin, distance, and bearing I could use geopy to calculate the corners of the bounding box!

from geopy.distance import geodesic

destination = geodesic(miles=distance).destination(origin, bearing)
coords = destination.longitude, destination.latitude

See this notebook for the full details.

Limitations

Of course, this method is very rough and has a number of major limitations, in particular:

only about 38% of the maps have point coordinates
the point values don’t necessarily locate the centre of the map
not all the maps are oriented towards north
sometimes a parish includes multiple maps
the size of the margin around the map will affect the accuracy of the bounding box

But despite these problems the results seem pretty good. To test this I created a notebook to overlay the digitised maps on a modern basemap using the bounding boxes. Here’s an example.

You can see the map is slightly offset (presumably due to the second problem listed above). But the size seems about right. Certainly good enough to use the bounding boxes in some exploratory analyses!

Visualising the results

I’ve saved the processed data as a new dataset, and started playing around with a couple of ways of visualising the results. These are experiments, not discovery interfaces. But you can use them for a bit of exploration if you don’t mind a few bugs. They’re all in Jupyter notebooks that can be run using the Binder service.

The parish maps browser includes a dropdown list of parish maps with point coordinates. Select a map and:

if there’s a bounding box and an image identifier, the image of the parish map will be overlaid on the modern base map using the bounding box coordinates
if there’s a bounding box, but no image identifier, a rectangle will be drawn on the base map showing the dimensions of the bounding box
if there are point coordinates, but no bounding box, a marker will be placed on the base map

If the image of the map is displayed you can use the slider to adjust the opacity. Clicking on either the image, rectangle, or marker will display metadata about the parish map and a link to the SLV catalogue.

There’s also a visualisation of all the bounding boxes overlaid on a modern base map.

As you move your mouse over the bounding boxes the titles are displayed on the map, and if you click on a bounding box the metadata is displayed beneath the map, including a link to the SLV catalogue.

It’s obvious from the image above that some of the coordinates must be wrong! Visualisation is a great way of finding problems with your data. I now need to work through the results, documenting the problems, and thinking about how to make best use of the data. More to come!

Exploring SLV urls

Tue, 23 Sep 2025 16:22:45 +1000

I like urls. They take you places. And if you know how to read them, they can tell you things about the systems that created them. One of the first things I did when I started my residency at SLV LAB, was to try and understand how their collection urls work. There’s a couple of well-worn methods I use when digging into a new site.

The first is url hacking – this involves fiddling around with the parameters in a url and submitting the result to see what happens. The Trove Data Guide includes some examples of hacking Trove urls to change the delivery of search results.

The second method involves opening up the developer console in your web browser and watching the activity in the network tab as you click on links. This tells you where the information that gets loaded into your browser actually comes from – sometimes exposing handy urls that you can use to shortcut access to useful data.

Permalinks

The SLV uses Primo for its public-facing catalogue, as well as other systems such as Rosetta and IIIF to deliver digitised content. I’d noticed that Zotero gets some useful data from the catalogue using the default ‘Primo 2018’ translator, however, important things like the item url aren’t captured. The problem is that Primo’s ‘permalinks’ are generated as required by a browser click – they’re not embedded anywhere on the page. This makes it hard to Zotero to grab them. So I started wondering how Zotero could construct short, persistent(ish) links to items.

Here’s a link to an item in Primo: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636

It looks pretty long and messy, but if you start deleting parameters and resubmitting, you’ll find that only two parameters are essential, vid and docid. This means we can rewrite the url as: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&docid=alma9941325055707636 Much nicer.

The ‘permalink’ for the same item is: https://find.slv.vic.gov.au/permalink/61SLV_INST/1sev8ar/alma9941325055707636 If you look closely at the url path and compare it to the example above you’ll see the path is constructed from /vid/[some other id]/docid. One of the librarians explained to me that the other identifier in the permalink is an encoding of the view type, but given that the ‘fulldisplay’ view is the default, we don’t really need it. So the shortened url seems fine for use in Zotero and is easy to generate from the current url. Nice.

It’s also worth noting that the vid value doesn’t seem to change, so to construct catalogue urls in your code, all you really need is the ALMA identifier that’s in the docid parameter.

Structured data

Item pages in Primo include a link labelled ‘Display source record’. If you click on this you’re taken to a representation of the item’s metadata in MARC. Here’s what the urls look like: https://find.slv.vic.gov.au/discovery/sourceRecord?vid=61SLV_INST%3ASLV&docId=alma9941325055707636&recordOwner=61SLV_INST Notice that the ‘fulldisplay’ in the url path above has changed to ‘sourceRecord’. There’s also a new recordOwner parameter, but it seems you can delete this and still get the same result.

Having access to the MARC record is handy, because it delivers the metadata in a simple, structured plain text format. But while the ‘source record’ page looks like a plain text file, it’s actually a HTML page that embeds a plain text record. If you open up the network tab of your browser’s developer console and reload the ‘source record’ page, you’ll see a different url is loaded under the hood: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV&recordOwner=61SLV_INST&lang=en See how the url path has changed from /discovery/ to /primaws/rest/pub? This url does deliver a plain text version of the MARC record.

Once you have the plain text version you can parse the contents to extract the structured data. There are tools that can probably do this automatically, but it’s also pretty easy using regular expressions. Here’s an example of some code I used to parse map records.

def get_marc_value(marc, tag, subfield):
    """
    Gets the value of a tag/subfield from a text version of an item's MARC record.
    """
    try:
        tag = re.search(rf"^{tag}\t.+", marc, re.M).group(0)
        subfield = re.search(rf"\${subfield}([^\$]+)", tag).group(1)
    except AttributeError:
        return None
    return subfield.strip(" .,")

You can also access a JSON representation of the record by adding the parameter &showPnx=true to the catalogue url: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636&showPnx=true

Once again, this is a JSON representation embedded in a web page. Using the same developer console trick, you can identify the direct url is: https://find.slv.vic.gov.au/primaws/rest/pub/pnxs/L/alma9941325055707636?vid=61SLV_INST:SLV&lang=en&search_scope=slv_local&showPnx=true&lang=en You should be able to parse the response from this url as JSON and use it in your code. I think the Zotero translator makes use of this pnx data.

If you want to download the MARC or JSON representations in your code, all you really need is the alma identifier. Just use it to construct one of the direct urls, such as this: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV The recordOwner and lang parameters are not needed, and the vid parameter doesn’t change.

Librarians using Primo have documented a number of tricks like this and shared handy bookmarklets to rewrite urls and get catalogue data in different forms.

IIIF and images

SLV delivers digitised images using IIIF. The IIIF manifest urls are not directly exposed through the web interface, but you can construct your own.

IIIF manifest urls look like this: https://rosetta.slv.vic.gov.au/delivery/iiif/presentation/2.1/IE24074939/manifest.json All we need to construct them is the IE identifier, in this case IE24074939. But where do you find this identifier?

If you’re looking at an image in the SLV’s image viewer, the url will be something like this: https://viewer.slv.vic.gov.au/?entity=IE24074939&mode=browse Yep, the IE identifier is right there in the url. Just extract it from the viewer url, and plug it into the manifest url!

If you’re looking at a catalogue record, or starting with one of the alma identifiers, you can get the IE identifier from the 956$e field of the MARC record.

The IIIF manifest will, in turn, provide identifiers for individual images that can be requested using the standard IIIF syntax.

To save myself a bit of fiddling about, I created a userscript that exposes the IIIF manifest url within the image viewer. If you install it you’ll see something like this:

Handles

Links to digitised items sometimes come in the form of ‘handles’: http://handle.slv.vic.gov.au/10381/4338980 These urls are redirected to the image viewer.

If you want to construct one of these handles, the identifier can be found in 956$a field of the MARC record.

From old to new

I was looking at the datasets created about 8 years ago in the SLV open data repository and noticed they included urls from the previous catalogue. Fortunately, the old urls redirect to the new system.

For example, this url: http://search.slv.vic.gov.au/MAIN:Everything:SLV_VOYAGER1842440

Redirects to: https://find.slv.vic.gov.au/discovery/fulldisplay?context=L&vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&docid=alma9918424403607636

If you look closely at the urls you’ll see that the identifier from the old system is embedded in the new identifier: 1842440 is in 9918424403607636 – 99_1842440_3607636. This means if you have a lot of old urls, such as in the open datasets, you can easily rewrite them in your code.

The process of GLAM hacking

No doubt a lot of this is well-known to librarians, and there’s probably many subtleties or complexities that my poking about has missed. But I wanted to document the process as much as the results – to give an idea of what I do when I approach a new GLAM collection online. I suppose this is GLAM hacking 101.

Creative Technologist-in-Residence at the State Library of Victoria!

Mon, 22 Sep 2025 23:14:09 +1000

I’m very excited to be the new Creative Technologist-in-Residence at the SLV LAB. For the next few months I get to play around with metadata and images, think about online access, experiment with different technologies, and build things to help people to explore the State Library’s collections. In other words, I get to be in my happy place!

My group at the recent SLV WikiFest was thinking about ways of helping researchers find resources relating to particular locations – how do I find material about my suburb, or my street? Coincidentally, the main focus of my residency will also be place-based collections, so I get to really think through some of the possibilities. SLV staff have already pointed me to some amazing maps and photographs, such as the Committee for Urban Action collection, the Mahlstedt fire survey maps, the MMBW plans, and the Victorian parish maps.

At the same time, I’ll be using my usual GLAM hacking approach to poke around in the SLV website to try and understand what data is currently available, identify any roadblocks, and document opportunities for computational research.

The results of my residency will be shared on the SLV LAB site, in GitHub, in the SLV section of the GLAM Workbench, and of course here. As usual, I’ll be working in the open, documenting things as I go along, so please join me on the journey!

Although the residency was formally announced today, I’ve actually been working with SLV data for the last couple of weeks and I’ve already got a backlog of stuff I need to blog about. Here’s a taster – what happens when you generate bounding boxes for thousands of parish maps from the available metadata and throw them on a map…?

WikiFest at the State Library of Victoria

Fri, 29 Aug 2025 15:07:26 +1000

This week I was lucky enough to participate in WikiFest at the State Library of Victoria. Organised by the State Library’s new innovation LAB and Wikimedia Australia, Wikifest was a hands-on, participant-led workshop focused on the possibilities of connecting SLV’s collections to (and through!) Wikidata.

The day kicked off with a series of presentations demonstrating possible uses of Wikidata. I talked a bit about some of my recent GLAM/Wikidata experiments. My slides are online and contain plenty of links to code, demonstrations, and documentation. They’re openly-licensed, so feel free to take anything of use.

The rest of the day was spent in groups, working on particular projects and learning more about Wikidata in the process. My group was looking at providing placed-based entry points to SLV collections, and spent a lot of time exploring the representation of Victoria’s Local Government Areas (LGAs) in Wikidata. We realised there was quite a bit of work to do in adding things like dates and boundaries, but we could see some exciting future possibilities. We also made a start, adding an ‘inception’ date for the City of Moe, based on the Victorian Government Gazette, digitised by the SLV.

Bonus userscript

While I was preparing my presentation I was thinking about the the way entries for Australian people in Wikidata are linked to a range of different identifiers, such as DAAO, the Encyclopedia of Australian Science, and the Australian Dictionary of Biography (ADB). Often a single person can have multiple identifiers and this means that those identifiers themselves become connected through that person’s record. You can query Wikidata with one identifier, and get back links to a range of other information sources about that person.

To demonstrate this, I created a simple userscript that adds additional links to biographies in the ADB. The script grabs the ADB identifier from the url, queries Wikidata for additional identifiers, and writes the results into the page’s ‘Life Summary’. Basic, but useful!

For something more advanced, have a look at the Entity Explosion extension for Firefox.

GLAM hacking with userscripts

Thu, 17 Jul 2025 17:21:25 +1000

In teaching and workshops I used to get students to question the idea that websites are ‘published’. They’re not released into the world in a fixed, immutable form – they’re a set of blueprints which only reach their final form in your browser window. This makes it possible to change the way websites look and behave.

Mozilla used to have a nifty educational tool called X-Ray Googles. Using it you could explore the code underlying a web page and do fun things like inserting new text or images. I encouraged students to try hacking ASIO’s home page.

ASIO home page with some added First Dog on the Moon.

There are other ways you can fiddle with websites. For example, most browsers have a developer console that exposes the code and styling of a page. You can use the console to edit HTML elements or toggle styles, but your changes won’t be saved.

One way you can save and share your web site customisations is by creating userscripts. Userscripts are little bits of Javascript code that run in your browser after a web page loads. These scripts can change many aspects of a page – not just how it looks, but also how it works.

Some old userscripts

I’ve been playing around with userscripts for a long time. Back in 2008, I created a userscript that completely overhauled the way that digital files were presented in the National Archives of Australia’s online database, RecordSearch. My userscript added new options for navigating and printing the file, and even made it possible to view the complete file contents on a 3D zoomable wall.

This customised RecordSearch interface was created by a userscript.

The userscripts I’ve created over the years have tended to either be useful little hacks aimed at fixing annoying aspects of GLAM websites, or experiments in thinking about the sort of information that’s presented online by GLAM organisations, and how it might be different.

In the first category are hacks like my RecordSearch show pages userscript. In 2009, I got annoyed that there was no way of knowing how many pages were in a digitised file until you clicked on the link. So I fixed it. With my userscript running, the links to digitised files are rewritten to display the number of pages. I’ve updated the code numerous times over the years, adding new features, and dealing with changes to RecordSearch. The last update was just a few days ago.

In the second category is my userscript that inserts photos from The Real Face of White Australia into RecordSearch. The are many thousands of records in the National Archives of Australia that document the impact of the White Australia Policy on the lives of ordinary people. But it’s often hard to understand this from the file descriptions. The userscript displays portrait images extracted from the files alongside the metadata – it tells you there are people inside.

Amidst my recent self-archiving binge, I realised I’d never updated this userscript to work with the latest data from The Real Face of White Australia, so I spent some time getting it working again. In the process I realised that RecordSearch now included content security policies that made it a bit harder to insert new images. The solution was to use one of the special userscript functions, GM_addElement(), rather than plain old Javascript. But then I discovered that the if the show pages userscript ran after this one, it would trigger the security restrictions nonetheless! To avoid this I made sure that the two userscripts operated on separate elements. So now the show people userscript is working again!

And a new userscript to improve Trove lists

Fixing up the ‘people inside’ code reminded me of how much fun it was playing around with userscripts, so when David Coombe mentioned a problem he had using Trove lists on Mastodon last night, I had to have a go at fixing it.

The problem is that Trove lists display all the tags associated with each individual item. Some items have lots of tags, so this eats up the screen real estate, making it harder to browse the contents of a list. Notes attached to items can be hidden, but not tags. Why not?

My brand new userscript hides tags by default, and adds a new link to toggle their visibility for each individual item. The link also displays the number of tags attached to each item. This gives the user control over which tags are displayed and when.

The new userscript in action – toggle your tags!

The main difficulty in creating this userscript was knowing when the page had actually finished loading. The current version of Trove uses a lot of Javascript to load and manipulate content, so you have to tell the userscript to wait until everything has settled down. Otherwise the script could fire too soon and cause unexpected results. I tried a number of different approaches to handling this problem, but eventually settled on the waitForKeyElements script. (I just realised there’s a more recent version of this script that doesn’t require JQuery, so I might need to investigate this further.)

Another Trove problem fixed!

Using userscripts

In addition to the userscripts mentioned above, I’ve also created one that enables you to browse Trove newspaper pages using the arrows on your keyboard. Left and right arrows go to the next and previous pages, while up and down arrows jump between issues. Searching is great, but sometimes you just want to browse. Install this userscript for that old-time, authentic newspaper reading experience!

But how do you install userscripts? First of all you need a browser extension to manage your userscripts – I use TamperMonkey or ViolentMonkey. Just follow the instructions to add one of them to your browser.

To install one of my userscripts, you need to go to the script (saved as a GitHub Gist) and click on the ‘Raw’ button. Your userscript manager will then ask you if you want to add the userscript. Click install!

Click on the ‘Raw’ button to install.

Once they’re installed the userscripts will run automatically when specified pages are loaded. If you ever want to disable them, you can do that from your userscript manager’s dashboard.

For convenience, here are the Gist links to all the userscripts I’ve mentioned:

As with anything you install on your computer, you want to make sure that you trust the source of any userscripts you add.

The rebirth of Wragge Labs (and moving my Heroku apps)

Wed, 09 Jul 2025 16:48:23 +1000

It looks like some paid work I was counting on won’t be going ahead, so I’m trying to save a bit of money on cloud hosting. As I previously noted, this resulted in the resurrection of The future of the past, but I’ve also been continuing to slog away at migrating all my old Flask apps and experiments from Heroku to a single Digital Ocean droplet. As of today, I’ve migrated 11 apps. Here’s a few details…

A new (old) home

The first thing I had to figure out was how to group together a series of individual Flask apps so I could easily run and maintain them on a single server, without making major changes to the apps themselves. I decided to go with the application dispatching pattern described in the Flask documentation. This groups the apps within a single Python environment so I had to do some alignment of Python versions and packages, but it wasn’t too hard and having just one virtual environment to manage seems a lot easier in the long run.

The application dispatching pattern configures the server to run one application at the web root ('/'), with the other apps assigned individual sub-paths. This raised the question, what did I want sitting at the root address? Rather than selecting an existing application for the prime slot, I decided to take the opportunity to build a showcase that included details of many of the things I’ve created over the years.

The old Wragge Labs (circa 2012)

I also needed a new domain name. Or did I? Back in the old days, I had a site where I shared many of my tools and experiments – Wragge Labs. In the intervening years, I’d moved or migrated much of the content away and pointed the wraggelabs.com domain to my main site at timsherratt.au. But this seemed like a good opportunity to resurrect it. So if you’d like to have a play around with some of the things I’ve created over the last 30 years, head along to the all new wraggelabs.com!

The new Wragge Labs showcases websites, apps, and experiments from the past 30 years – some useful, some playful, and some creepy…

A surprising amount of the things I’ve built are still working. As I was compiling my list, I made a few running repairs – for example, fixing broken links in Linking history in place and Magic Squares to get them working again. However, some things only exist now in web archives, and others have been broken by the recent actions of the NLA and NAA.

To provide a bit of extra context, I’ve grouped together publications and presentations documenting many of the experiments. These are saved in Zotero, and tagged with the name of the app. When you click on a ‘Related’ link, the details of any linked resources are retrieved using the Zotero API and displayed on a new page. This means I can add new related resources simply by dropping them into Zotero.

Moving house

The process for moving the apps to their new home was pretty straightforward. On my local machine, I copied the code into the new aggregated structure, added any packages needed into a combined requirements file, and created a new top-level app to direct requests. And then I spun everything up and started fixing bugs…

All of the problems were easily resolved. Most involved fixing up paths to static assets, or in navigation links. The only significant changes to the Python code were caused by the deprecation of the .count() method in Pymongo.

To make life a little harder, I decided to take the opportunity to make sure that all the assets – javascript, css, and font files – were loaded from the local system, and not sitting in the cloud. Having everything local should make it easier to maintain the apps in the long term. It was a bit fiddly tracking down where everything was being loaded from, but not too hard.

The only other changes I made were to add some caching to most of the apps, particularly those that make calls to external databases or APIs. I used Flask-Caching with the local file system backend.

To get the new aggregated application working on a Digital Ocean droplet, I followed the instructions on how to serve Flask applications using uWSGI and Nginx. I think the only thing I did differently was to use pyenv to manage Python versions and the virtual environment. To update the app, I use rsync to copy across the code and systemctl to restart it. So far it’s all working pretty smoothly.

Redirecting Heroku

Once the apps were happy in their new home, I needed to redirect the Heroku addresses to wraggelabs.com. It was surprisingly hard to find good documentation on how to do this, so I’ll document my steps in detail in case its of use to others.

There are a few redirect apps for Heroku around, but I decided to use heroku-redirect because it basically just configures and runs Nginx without any additional processing. First I cloned heroku-redirect to my local system, and then for each app I wanted to migrate I followed these steps:

cd into the heroku-redirect directory
set the git remote for the app you want to redirect: heroku git:remote -a [app name]
if you’re not using the lastest Heroku stack, update it: heroku stack:set heroku-24 -a [app name]
set the url you want to redirect to (without trailing slash): heroku config:set LOCATION=[new url]
to add the full path to the redirected url: heroku config:set PRESERVE_PATH=true
I found I had to remove the Python buildpack from the app before updating, I did this using the Heroku dashboard, but no doubt there’s also a CLI command
I also used the dashboard to add a new nginx buildpack: https://github.com/heroku/heroku-buildpack-nginx.git
finally I pushed the new app, using --force to replace it completely: git push --force heroku master:main

Of course, you’ll need to have the Heroku CLI installed.

A number of my Heroku apps were using basic dynos (which cost US$7 a month), so once they were redirected, I changed them to used shared eco dynos. Yay – money saved! Hopefully, the redirects won’t push the eco dynos beyond their monthly limit.

More experiments to come?

One of the good things about all of this housekeeping is that it’s got me thinking about new experiments. I used to love Flask and Heroku because it they made it so easy to build and share things. Now I can do the same with wraggelabs.com!

Tim Sherratt

Honoured to be 'the most dedicated Labber'

Hacking the archive – GLAM Labs workshop at the University of Edinburgh

Some stats describing the National Archives of Australia collection

The future of online archives in 2009

The future of the past: GLAM innovation and the responsibilities of history

The battle of the bots

The technical details

Next steps

Generosity in practice – a chat with Paula Bray at the State Library of Victoria

Zotero translator for Libraries Tasmania updated!

Exploring georeferenced maps from the SLV collection

The background

The interface

The future?

my place – exploring SLV collections through a street address

The collections

OpenStreetMap

Sands & MacDougall’s directories

Committee for Urban Action photographs

Georeferenced maps

Parish maps

Newspapers

Photographs

The interface

my house

my street

my suburb/town

nearby

The data

Next steps?

Geolocating photos from the SLV's Committee for Urban Action collection

The process

Finding intersections

Finding road segments

Problems finding intersections

Assembling the data

Constructing the interface

More information and links

Goodbye 2025! A brief summary of the highlights and lowlights…

Highlights

Lowlights

Next year

How you can help

Exploring Victorian newspapers

Assembling the data

Building the apps

Why bother?

Counting down... (to the end of my SLV residency)

Some Sands & Mac tweaks thanks to ALTO and IIIF

A new way of searching Sands & Mac

Technical information follows…

Downloading the PDFs

Downloading the ALTO files

Assembling and publishing the database

More information

Turning the SLV's maps into data with Allmaps and some GLAM plumbing

Me at 63...

What was your first history related job? What path have you taken since then?

What kind of work have you done? What are you working on now?

Research or writing? (What do you enjoy more and why?)

What are the best and hardest things about the kind of work you do?

Creating bounding boxes for parish maps in the SLV collection

The metadata

The method

Creating bounding boxes

Limitations

Visualising the results

Exploring SLV urls

Permalinks

Structured data

IIIF and images

Handles

From old to new

The process of GLAM hacking

Creative Technologist-in-Residence at the State Library of Victoria!

WikiFest at the State Library of Victoria

Bonus userscript

GLAM hacking with userscripts

Some old userscripts