Jump to content

Operations/Minutes/2025-10-30

From OpenStreetMap Foundation

OpenStreetMap Foundation, Operations Meeting - Draft minutes

These minutes do not go through a formal acceptance process.
This is not strictly an Operations Working Group (OWG) meeting.

Thursday 30 October 2025, 19:00 London time
Location: Video room at https://osmvideo.cloud68.co

Participants

Absent

Minutes by Dorothea Kazazi.

New action items from this meeting

Grant to close the Chef ticket #807. [Topic: #807 nominatim.openstreetmap.org banned GNOME's AWS NAT GWs] (Done)

Reportage

Fastly post about OpenStreetMap

Grant and Tom reviewed Fastly's draft on the proposed post featuring OpenStreetMap, as part of their Customer Hero campaign. The draft was based on a call that the OWG had with Fastly and the draft seemed fine.

Post-meeting note: https://www.fastly.com/customers/openstreetmap

AWS Identity and Access Management Roles Anywhere

Related to action item 2025-10-16 Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert]

Grant was working on some improvements there and preparing for a change regarding moving accounts.

Reworking of Test Kitchen methods

2025-10-16 Grant to create a PR regarding refactoring some stuff. [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen Github actions]

Grant started the pull request.

Héctor joined 2' after start.

Gen10 (second-hand) server for Nominatim in the US

Related to action item 2025-10-02 Grant to go ahead with the purchase of the Gen10 (second-hand) server for Nominatim in the US. [Topic: Gen10 Nominatim purchase (USA)]

Grant still needs to speak to the Treasurer about the purchase, as the invoice needs to be paid soon after it gets raised.

  • Grant asked Héctor who should be the recipient of the GBP 5K invoice.
    • As it is in the OWG budget, the money can be spent. (Post-meeting note: per the [procurement policy](osmfoundation.org/wiki/Budgets_and_Procurement_Policy), approval is needed.)

Suggestion: Email the Treasurer and cc Michelle Heydon (OSMF Bookkeeper)

A link to the quotation for the server was shared during the meeting. The price will be close to quotation but a couple of hundred dollars different, as the quotation has expired and some small equipment parts need to be changed.

Mailing lists

Related to action item: 2025-07-24 Grant to do the Mailman 2 to 3 conversion [Topic: Mailing lists]

https://github.com/openstreetmap/operations/issues/1264

Some old mailing lists might have issues to be migrated to Mailman 3 and might need custom fixes. Grant might need to do some analytics.

  • Mailing lists which will be disabled and publicly archived, won't be migrated to Mailman 3.
  • The US mailing list might be disabled.

WAL archiving

Related operations issue: #1300 Replace wal-g for postgresql wal archiving.

Suggestions

  • Experiment with Postgres on the dev server with write-ahead log backups.
  • Use full functionality of the tools to do a full snapshot of the copies, which would allow point-in-time recovery, using the WAL logs.
    • This could be done with wal-g, but the options below look more developed.

Potential options

  • pgbackrest
  • pg barman (more modern)

Grant created the bucket for the backups of write-ahead logs on dev, plans to add the keys to the private Chef and will finish the AWS portion for dev. Will then discuss over IRC on the choice of the tool.

No outage announcement will probably be needed for the change of tool.

AWS access keys

Minutes redacted.

#807 nominatim.openstreetmap.org banned GNOME's AWS NAT GWs

Related to OSM chef ticket [https://github.com/openstreetmap/chef/issues/807 #807 nominatim.openstreetmap.org banned GNOME's AWS NAT GWs).

Background: Prior to 2022, there was an informal agreement between zeeshanak(GNOME Infrastructure) and [Sarah Hoffmann](https://github.com/lonvia) (Nominatim developer and maintainer) about having GNOME IP addresses whitelisted for access to nominatim.openstreetmap.org.

In October 2025 it was reported that Nominatim started blocking GNOME IPs. It seems that GNOME changed their user agent and started using a generic user agent, so they were getting blocked before hitting the IP whitelist. More information on the ticket https://github.com/openstreetmap/chef/issues/807

Related: Nominatim Usage Policy (aka Geocoding Policy).

Action item: Grant to close the Chef ticket #807. (Done)

#1294 Determine hardware needs for 2026

Related to OSM Operations ticket [https://github.com/openstreetmap/operations/issues/1294 #1294 Determine hardware needs for 2026).

Grant is not as eager to replace some of the hardware, as he is of the view that if hardware is still good we should keep it.

Norbert's disk space

Norbert (Backup server, Web server for planet.openstreetmap.org, Planet file generation server) has some old backups and logs that take space (Rails logs, Tile logs). Grant has been manually cleaning some of them, when he does the AWS sync.

Suggestions

  • Get more second-hand disks for Norbert
  • Expire some local planets/logs.

Other points mentioned during discussion

  • Unclear how the bays are connected to the controller.
  • Bying the same model second-hand disks: ~ GBP 600 per disk (8 TB).
  • We have two more disk bays.

#1284 OpenTelemetry Investigation

Related to OSM Operations ticket OpenTelemetry #1284 Investigation.

OpenTelemetry is self-hosted platform.

On multitude of options for the monitoring stack

Our open monitoring stack has at least two to three different options for each of its components.

Embed OpenTelemetry SDK -> Collect the reports (2-3 options) -> Send the reports to back-end storage (at least two options) -> View the traces (at least two options).

Aim: Figure out the monitoring stack.

Other points mentioned during discussion

  • Paying NewRelic or similar companies would be very expensive and not preferable, we can self host.

MediaWiki

4-5 tickets are blocked due to a piece of code related to normalising page titles. A search-related extension that Tom installed, is also blocked. Minh is investigating. Some work is needed to unblock them.

  • The Unicode issue is blocked on OPS doing a migration and not related to this piece of code.
  • MediaWiki upgrade: We got through half of it - there were questions regarding the encoding. Half of our MediaWikis are running the old version and some the new one.

On containerisation of MediaWiki

Is very difficult, because the config is in text files within the container, so

  • need a system that writes the config files on boot and
  • handling the migration on boot
  • multiple daemons need to run - if you follow the old model, where each daemon has its own container.

Swapping one bit of complexity for another.

Suggestion: Move to Naga (Subversion redirector, Trac redirector, IRC redirector, Blog aggregator, OSQA server for help.openstreetmap.org), which has most of the OSMF stuff and containers.

OPS meeting during State of the Map Europe 2025

Sunday slot booked for OPS meeting during the hackday after the SotM Europe 2025 conference.

Confirmed hackday attendees: Paul Norman, Sarah Hoffmann, Jochen Topf, Pablo Brasero.

Action items

  • 2025-10-16 Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert]
  • 2025-10-16 Grant to create a PR regarding refactoring some stuff. [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen Github actions]
  • 2025-10-16 Grant to create a PR about adding logic to Chef for retrying failed initial creation of Let's Encrypt certificates [Topic: Add logic to Chef for retrying failed initial creation of Let's Encrypt certificates]
  • 2025-10-16 Paul to ping Grant on the "Repurpose or return old tile servers" ticket https://github.com/openstreetmap/operations/issues/575 [Topic: #575 rhaegal - Repurpose or return old tile servers] Done
  • 2025-10-02 Grant to discuss with Paul Norman and flesh out his suggestion and determine the practicalities (e.g. key revocation). [Topic: AWS CA cert]
  • 2025-10-02 Grant to follow up with Paul. [Topic: Serving vector tile styles]
  • 2025-10-02 Grant to go ahead with the purchase of the Gen10 (second-hand) server for Nominatim in the US. [Topic: Gen10 Nominatim purchase (USA)]
  • 2025-09-18 Paul to look at potential issues related to the collation of indexes - Debian Postgres upgrade. [Topic: OSM DB upgrade to Postgres 17]
  • 2025-07-24 Grant to set-up a test for OWG's review [Topic: Switching www.osm.org to Fastly frontend]
  • 2025-07-24 Grant to do the Mailman 2 to 3 conversion [Topic: Mailing lists] - https://github.com/openstreetmap/operations/issues/1264
  • DONE first part, see the agenda: 2025-06-12 Tom still to run OSMDBT test. OPS then to plan a maintenance window for the OSM.org postgres database update. [Topic: OSM.org postgres database]
  • 2025-05-01 Progress, we need to form academic justification and then we should get something: Grant to follow-up with Australian hosting again. [Topic: OSUOSL funding / issues]
  • 2025-05-01 Grant to see if other University offers are still available and what hardware would be required. [Topic: OSUOSL funding / issues
  • 2025-03-20 Grant to follow-up with the South African contact about the potential hardware donation from a mobile network. [Topic: New offers of Servers Australia and South Africa]
  • 2025-03-20 Grant to run an SQL query to identify more email providers used by spammers. [Topic: Spam] #2025-05-01 Grant has created a small list of disposable email providers.

Action items that have been stricken-through are completed, removed, or have been moved to GitHub tickets.

Retrieved from "https://osmfoundation.org/w/index.php?title=Operations/Minutes/2025-10-30&oldid=14143"