Jump to content

Operations/Minutes/2026-03-19

From OpenStreetMap Foundation

OpenStreetMap Foundation, Operations Meeting - Draft minutes

These minutes do not go through a formal acceptance process.
This is not strictly an Operations Working Group (OWG) meeting.

Thursday 19 March 2026, 19:00 London time
Location: Video room at https://osmvideo.cloud68.co

Participants

Minutes by Dorothea Kazazi, including some notes from Grant.

New action items from this meeting

  • Paul to create a breakdown of QGIS tile traffic statistics for different zoom levels. [Topic: QGIS Tiles usage]
  • Grant to research what triggers a large download from QGIS. [Topic: QGIS Tiles usage]
  • Paul to overhaul how we're doing the 404 tiles. [Topic: QGIS Tiles usage]
  • Paul to look into the typo on tile block message 403r [Topic: Typo on tile block message 403r?]
  • Paul and Grant will run some time limited experiments during non peak hours to test catching anonymous/fake-ua scrapers. Genuine Google Bot etc will continue to be permitted. [Topic: Fastly Client Challenges]

Reportage

Mailman conversion

Related to action item: 2026-03-05 Grant to do a dry run for the Mailman conversion, probably on Rhaegal in Croatia. [Topic: Upgrades: Machines on Ubuntu 22.04]

In progress.

2026 OWG Budget

Related to action item: 2026-02-05 [2026 OWG Budget] OWG to work out what is needed in 2026, and see if budget adjustments are required. Will come back to the board.

Ongoing planning. No need to track seperately.

MediaWiki

Related to action item: 2026-02-05 Grant to test some MediaWiki settings to improve size selection. [Topic: WikiCommons image resize]

Done and issue seems resolved. The current LTS MediaWiki version does not have the features we want for image sizing. We have turned on the options that allow us to do some resizing, but there are some edge cases. The ultimate fix is moving to a new long-term supported version of MediaWiki.

We still get occasionally rate-limited by MediaWiki Commons.

Mediawiki-related new errors

  • One related to .pdf handling
  • A wikibase one - needs fix by Yuri.

QGIS Tiles usage

Synopsis: We will produce some stats (tiles per zoom, peak rate, and tile usage heat-map)
We are likely to turn on TOTP validation for osm.org requests, might need extra set-cookie on osm.org

Issue: High use of OSMF tiles by QGIS.

  • It is unclear why the traffic is so high and there are limited things we can do.
  • We would like to cut traffic to 1/4-1/2.

Background

QGIS

  • uses the OSMF tileservers.
  • has removed OSMF tiles from the browse tile layers, but on startup it asks the user whether they want to start from a template and the template is an OSM base layer, with OSMF tiles.
  • supports arbitrary zooms
    • if someone zooms out just before they switch to another tile layer zoom, they can have 128 pixel tiles, which can fill a 4K screen. While this is not a load issue, it means we can't put any very effective low rate limiting.
  • does some oversampling by downloading a higher zoom level than it actually needs.
    • the set-up is good for imagery, and bad for raster maps.
    • could be only for high-definition monitors.

On traffic

  • Issue: The QGIS traffic peak during a European day is 2 to 4 times more than osm.org traffic, looking at zoom levels 13 and above.
  • Daily average: QGIS is using more tiles than osm.org.

Traffic could be caused by

  • Export function
  • Plugins

No header difference between the two cases, so we can't rate limit on the CDN.
We might not be able to figure out the reason for the high traffic.

On export function: The export to print tries to download tiles at 300 DPI.

Suggestions

  • One QGIS tile server: Send all the QGIS traffic to one tile server, and let the server become overloaded.
    • Sarah Hoffmann does something similar with Nominatim.
    • Use 2 servers.
    • Present a reasonable image tile error message to the users.
  • Rate-limit the tiles - we already doing that - we can't do it for anything over 4 kilobytes.
  • Rate-limit on the backend.
  • Create a breakdown of zoom-level statistics and 1) present the case to them and options they have or 2) remove a few zoom layers for QGIS.
  • Peak hours: do not allow download of tiles from high zoom levels and display a tile message "we don't have capacity" to the users.

On suggesting to QGIS to switch to vector tiles

  • Vector tiles would help because they're bigger area tiles, particularly when you get into over zoom.
  • They could download the tiles from Geofabrik.
  • The stylesheet support was not adequate, but this must be fixed by now.
  • Issues:
    • Time: If they release a new version, it would take 2 years for upgrade for most users.
    • Hosting the style: It would be nice if QGIS hosts it, as they could change it as needed.

On using Fail2Ban

  • Fail2Ban might be able to pick up multiple requests for tiles, if we scoped it very carefully.
  • We would have to take into account capacity weighted distribution among the five European tile servers. I.e. if someone gets two meta tiles on the same server, their load is going to be significantly higher than someone who gets meta tiles on different servers. We would have to set different limits for each server, in proportion to how much of the traffic they're serving.

On asking QGIS to buy a server

  • We don't give people that privilege - but they might be the exception to the rule.
  • Price:
    • ~ EUR 7K for a general purpose machine, similar to what we have.
    • The cost is about 70% more than the machines that we bought.
    • Prices are very high right now.
  • Would need 2 general purpose machines to support QGIS. Better to get something newer.

Other points mentioned during discussion

On urgency

  • They probably don't realise yet how urgent the situation is.
  • They can't do anything urgent, due to people upgrading very slowly.

On tarpiting

  • Applies to objects under 4 kilobytes (so, mostly empty tiles).
  • Delays transmission of a set of bytes by an integer number of seconds.

Nominatim

  • One queue for preferred people.
  • One for everyone else.

Decision

  • Create a breakdown of zoom-level statistics, a heatmap of what is accessed and present the case to QGIS and the options they have.
  • If the QGIS high tile traffic starts to cause significant harm to other people, we can change the directors to send the QGIS traffic to the Polish tile server.

Action items

  • Paul to create a breakdown of QGIS tile traffic statistics for different zoom levels.
  • Grant to research what triggers a large download from QGIS.
  • Paul to overhaul how we're doing the 404 tiles.

TOTP cookies for access control

Paul started work towards being able to use the TOTP cookie.

It's easy to change the website to set that cookie, which we send anyway.

  • Fastly gave us a code block, which should work.
  • Paul tried to do a validation by checking the TOTP cookie's presence, but 1) not all website pages set the cookie and 2) unclear if MapLibre requests send the cookie to tile.osm.org.

On several people getting tile access blocked notifications, after the recent referrer-related changes
Potential reasons

  • using a privacy-related browser extension, like uMatrix.
    • The default in uMatrix is spoof-referrer, which will send tile.openstreetmap.org as the referrer and will lead to blocks.
  • having the website security header turned on.
  • other browser extensions or using privacy mode.
  • overidding your default browser accept headers

Suggestions

  • Put a parameter on the request tile URL.
  • Change the website to set the cookie to any pages which need to access Overpass, as Overpass checks the cookie.
  • OWG to set an acceptable level of false positives.
  • OWG to document what people can do to fix the blocked tile access.
  • OWG can set TOTP so that when people visit osm.org in the last TOTP duration, this will carry over to other OSM-related sites.

Discussion on a specific case

Case: Someone visits example.org, which had blocked access to OSMF tiles. They then visit osm.org, see the tiles and get a TOTP cookie, and go back to example.org

  • If they view the same area, it will always work because successful tiles are cacheable and they will be in the browser's cache.
  • Depends on our configuration:
    • If the OWG explicitly blocks example.org, it would not work.
    • If we are serving stale tiles to example.org, we might not want to serve stale tiles to viewers from example.org who have also visited osm.org.

Other points mentioned during discussion

  • Some of the third-party layers (e.g. Tracestrack Topo) that we have don't work, because they insist on having OSM.org as the referrer.
  • The OWG needs to know who is using the tiles and can't cater for every single case.
  • The TOTP currently rolls over once an hour - there is a hard setting in the code.

Action item

Paul to make a PR on the website to send the TOTP to any page which has a map.

Paul was thanked for his changes to the error tiles, which now have codes.

Fastly client challenges

We can add a challenge to osm.org to prove those who access it are humans. Might help with scraping. Not to be used for the API.

On robots.txt

  • it was recently updated by the OWG.
  • it is mostly ignored by AI and modern scrapers - unless the scraper is explicitly named.

On scrapers

  • the "OpenAI scraper" traffic we see does not come from OpenAI's published IPs - so it could be fake.
  • OWG sees also a lot of ChatGPT traffic which is probably not from ChatGPT.

Suggestions

  • Leverage Fastly's tags, such as "suspected bots" and "official bots".
  • Add trackpoint rate-limiting on 408s.
  • Add a restriction so that ChatGPT can browse only from official ChatGPT IPs.
  • Put a trackpoint rate-limiting on 408s.

On spam

  • 150 accounts/day created from Pakistan, related to spam.
  • 200 signups within a few minutes with different email addresses.
  • The OWG thinks that the sign-ups are being created by people trying to hack people's mailboxes.
  • Some spammers report the emails from us as spam.

Other points mentioned during discussion

  • Fastly identifies "suspected bots", likely to be faking its user agents, and "official bots".
    • The "suspected bots" tag is not useful for the API.
  • It won't help with the trackpoints scrapers.

Action item

Paul and Grant will run some time limited experiments during non peak hours to test catching anonymous/fake user agent scrapers. Genuine Google Bot etc will continue to be permitted.

Any other business

Typo on tile block message 403r

Topic suggested by Dorothea.

https://cdn.masto.host/enosmtown/cache/media_attachments/files/116/256/790/787/629/574/original/a9cfec7f5f01878e.png "of to"

  • The tile message is manually line broken.
  • It is difficult to fit in the right number of words into a size-minimised tile, which then has to be small enough to get base64 encoded into our configuration.

Action item

Paul to look into the typo on tile block message 403r.

Funding a second sysadmin

Topic suggested by Craig.

The board, during the February board meeting, decided to try and fund a second systems operator and get that person in place as soon as possible.

Process

  • undefined
  • long. It will probably take 9 months to hire someone. Might see someone in Nov/Dec.
  • a lot of consultations with OWG expected to take place to figure out what will be needed.

Budget

  • we have funds which could probably fund 1/4 of a year.
  • we have to raise EUR 200,000 to make the budget balance this year- so pushing for fundraising.

Unclear if this is going to be a contractor or employee.

OPS comments

  • Paul is looking into contracting opportunities.
  • Immediate temporary contract: Paul could be contracted, and OSMF in parallel have the formal bureaucratic process of long-term hiring.

Creation of OSMF account on BTC exchange service for BTC donations

Topic raised by Grant, who asked Héctor (board) whether there was an update on his research for an BTC exchange service where OSMF could create a business account. Héctor has contacted some companies regarding their account rates, but non have answered so far.

  • OSMF has to cash out a 2 BTC donation ()1, 2)
  • OSMF needs to get its own account on a BTC exchange service (note: see the 2026-02-26 board disccusion). Otherwise, Grant would have a tax issue.

BTC down 40% over 6 months.

  • 75000 EUR in Jan
  • 60000 EUR now.

On converting BTC to fiat

  • We have been typically converting (big) BTC donations immediately.
  • Most NGOs cash out immediately.
  • For small BTC donations, we wait.

Other points mentioned during discussion

  • Risk of handling BTC is more.
  • Coinbase does not do business accounts and have 40% tax.

Action items

  • 2026-03-05Grant to do a dry run for the Mailman conversion, probably on Rhaegal in Croatia. [Topic: Upgrades: Machines on Ubuntu 22.04]
  • 2026-02-05 [2026 OWG Budget] OWG to work out what is needed in 2026, and see if budget adjustments are required. Will come back to the board.
  • 2026-02-05 Grant to test some mediawiki settings to improve size selection. [Topic: WikiCommons image resize]
  • 2026-01-22 Grant to get AWS S3 bucket credentials for the dev server. [Topic: Credativ consultancy on OSM.org Postgres database update]
  • 2026-01-22 Tom to draft follow up question on pgbackrest local backup required or can /JUST/ S3 be used. [Topic: Credativ consultancy on OSM.org Postgres database update]
  • 2025-10-16 Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert]
  • 2025-10-16 Grant to create a PR regarding refactoring some stuff. [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen GitHub actions]
  • 2025-10-16 Grant to create a PR about adding logic to Chef for retrying failed initial creation of Let's Encrypt certificates [Topic: Add logic to Chef for retrying failed initial creation of Let's Encrypt certificates]

Action items that have been stricken-through are completed, removed, or have been moved to GitHub tickets.

Retrieved from "https://osmfoundation.org/w/index.php?title=Operations/Minutes/2026-03-19&oldid=14347"