Jump to content

Operations/Minutes/2026-08-20

From OpenStreetMap Foundation

OpenStreetMap Foundation, Operations Meeting - Draft minutes

These minutes do not go through a formal acceptance process.
This is not strictly an Operations Working Group (OWG) meeting.

Thursday 20 August 2026, 19:00 London time
Location: Video room at https://osmvideo.cloud68.co

Participants

Minutes by Dorothea Kazazi.

Reportage

Page checksums

Related to action item 2026-08-06 OPS to decide whether to enable page checksums. This will be decoupled from the pgbackrest upgrade, as it requires significant downtime. [Topic: pgbackrest upgrades]

If we're going to take one of the slave servers out of service, we could run a test on one of them to see how long it would take.

Storage comparison experiment using GPS traces of various sizes

Related to action item 2026-08-06 Rubén to conduct a storage comparison experiment using traces of various sizes, load them into a database in the current schema and as line strings, and compare table sizes. [Topic: GPS points database table]

There is a long discussion on the ticket of the website-repository.

MediaWiki OAuth extension work

Related to action item 2026-07-23 Grant and Minh to discuss the MediaWiki OAuth extension work. [Topic: cgimap maintenance]

Done and Grant hit a blocker: a PDF extension does not work correctly with the latest (46) Long-Term Support release of MediaWiki core, though the core itself is what is technically broken because of a missing language lookup function, not the extension. This function has already been fixed in MediaWiki core and is present on the release branch. Grant estimated it would take about three weeks until the next release.

Suggestion: switch the test wiki to follow the release branch so that the fix can be tested early, until the official release comes out.

Concern: running into version incompatibilities between extensions, where the main OSM wiki cannot compose a set of modules that works together for everything.

Other point mentioned during discussion: The "get language" issue might be related to a separate hack that Yuri had been asked to look into, which had also broken on the same function.

Decision seemed to be: Grant will discuss with Minh on IRC.

Test wiki: cursory testing of extensions

Related to action item 2026-07-23 Minh to do a cursory testing of extensions on the test wiki, once it is upgraded. [Topic: MediaWiki upgrade]

Done.

Looking into the Fastly timeout settings

Related to action item 2026-07-23 Grant to look into the Fastly timeout settings (connection timeout, first-byte timeout, and between-bytes timeout) and attempt to reproduce the GPX time-out issue using developer tools. [Topic: GPX uploads fail with 503]

Suggestion: Rubén to retest the slow GPX upload issue and report any changes.

Changes made by Grant to try to fix the timeout problem:

  • Increased the inter-byte timeout from around 10 seconds to 30 seconds.
  • Changed the traffic flow through Fastly. Previously, 100% of traffic passed through the next-generation firewall and then the CDN before reaching the backend web servers. Now, only 1% of traffic goes through the next-gen firewall, reducing it to a single hop, because the firewall has an uncontrollable, forced 60-second timeout that made it hard to identify which timeout was actually being hit during GPX uploads.

GPX trace conversion to LineStrings

Topic added by Rubén.

Issue: converting individual GPS points in the database to LineString geometry. This conversion is expected to reduce storage by approximately 500 to 600 GB. A ticket and pull request have been opened to track the work.

Concern

  • Converting the identifiable and trackable traces, and keeping the other ones in the current system.
    • It might be possible to convert all of them, thus we won't have to retain the old table.

Suggestions

  • Dropping public and private traces from the point cloud download, and removing those GPS points from the database, to switch to the cursor and will increase the efficiency.
    • Some users might still like to retain them. for aligning imagery.

Proposed plan by Tom

  • Merge the cursor PR. This drops public and private traces from the download and introduces cursors over the current schema.
  • Perform the LineString migration for trackable and identifiable traces.
  • Drop the GPS points for public and private traces, since they will no longer be used for anything at that point (the traces themselves remain downloadable as a whole for users with appropriate permissions).
  • Switch the cursors to use the LineStrings.

Grant offered to handle communications, provided he receives sufficient advance notice of approximately one to two weeks before the change is deployed.

Storage capacity

Issue: The secondary machines (Eddy and Karm) have only approximately 3 TB of storage each.

Suggestion: drop all public and private GPS points before starting the LineString migration in order to free up space.

Action item

Rubén to open a pull request for the LineString changes.

NTP servers and usage of public pool

DHCPD only resolves on ntp server on start-up, ilo then get pinned to IPs handed out by DHCP.

Issue: the current NTP configuration relies on the public pool (pool.ntp.org) resolved via DHCP. DHCP resolves pool hostnames to IP addresses only at startup and does not update them dynamically. This causes firmware devices such as iLO to continue querying stale IP addresses that may no longer be valid members of the NTP pool.

Suggestions

  • Switch to Google and Cloudflare anycast NTP addresses, which are stable, as proposed by the PR https://github.com/openstreetmap/chef/pull/864
  • Open the gateway NTP server, chrony, to serve the local network. The gateway machine already runs Chrony with Google and Cloudflare listed together with the public pool. The Google pool IPs were hardcoded by Grant as DNS queries were failing and we couldn't resolve the pool NTP.

Other points mentioned during discussion

  • Tom has been using the NTP pool for Linux desktops for decades without any real problems.
  • The NTP pool does not seem stable on firmware devices, as they over-query and then get the kiss of death packet.
  • Latency is very low: 0.8 millisecond from Amsterdam and approximately 4 milliseconds from London.

OPS meeting at SotM 2026

Meeting on Thursday evening (27 August).

Minh will be around from Tuesday and Rubén will also join, as he is already in Europe. Grant and Tom will travel on Thursday.

Review of OSM wiki tickets

List at https://github.com/openstreetmap/operations/issues?q=state%3Aopen%20label%3A%22service%3Awiki%22

Action items

  • 2026-08-06 Tom to update the notes on how to perform Postgres version upgrades. For major Postgres versions we have to run stanza-upgrade on the repository before we start up the new version of Postgres. Does not apply for point releases. [Topic: pgbackrest upgrades]
  • 2026-08-06 OPS to do a test on one of the slave servers and as part of that test a pgbackrest restore. [Topic: pgbackrest upgrades]
  • 2026-08-06 OPS to decide whether to enable page checksums. This will be decoupled from the pgbackrest upgrade, as it requires significant downtime. [Topic: pgbackrest upgrades]
  • 2026-08-06 Rubén to conduct a storage comparison experiment using traces of various sizes, load them into a database in the current schema and as line strings, and compare table sizes. [Topic: GPS points database table]
  • 2026-07-23 Grant and Minh to discuss the MediaWiki OAuth extension work. [Topic: cgimap maintenance]
  • 2026-07-23 ~~Grant to proceed with the MediaWiki upgrade on the test wiki in the next few days. [Topic: MediaWiki upgrade]~~
  • 2026-07-23 Minh to do a cursory testing of extensions on the test wiki, once it is upgraded. [Topic: MediaWiki upgrade]
  • 2026-07-23 Grant to look into the Fastly timeout settings (connection timeout, first-byte timeout, and between-bytes timeout) and attempt to reproduce the GPX time-out issue using developer tools. [Topic: GPX uploads fail with 503]
  • 2026-05-28 Minh to follow up on DWG non-disclosure agreements, then IP access for moderators in website repo. [DWG request for user info for moderators]
  • 2026-05-28 Paul to ask OSM France how do they view if we retire the HOT style from osm.org since it is not under active development** [AOB: Tracestrack topo and HOT layers]
  • 2026-05-14 Grant to check with Fastly team via Slack about 100% ngwaf traffic
  • 2026-03-19 Grant to research what triggers a large download from QGIS. [Topic: QGIS Tiles usage] # 2026-05-14 Not much progress; unable to find the new usage header in latest versions. # **2026-06-11 Action item to be parked.**
  • 2026-03-19 Paul and Grant to run some time limited experiments during non peak hours to test catching anonymous/fake-ua scrapers. Genuine Google Bot etc will continue to be permitted. [Topic: Fastly Client Challenges]
  • 2025-10-16 Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert] # 2026-05-14 Grant to chat to Tom about what CA + Signed Certs are required. Grant to read the AWS documentation on requirements.
  • 2025-10-16 Grant to create a PR regarding refactoring some stuff. Make kitchen handle selecting cookbook-OS combos. and only running tests on cookbooks that have changed [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen Github actions]
Retrieved from "https://osmfoundation.org/w/index.php?title=Operations/Minutes/2026-08-20&oldid=14844"