Operations/Minutes/2026-03-05

OpenStreetMap Foundation, Operations Meeting - Draft minutes

These minutes do not go through a formal acceptance process.
This is not strictly an Operations Working Group (OWG) meeting.

Thursday 5 March 2026, 19:00 London time
Location: Video room at https://osmvideo.cloud68.co

Participants

Tom Hughes (OWG volunteer)
Grant Slater (OWG, OSMF Senior Site Reliability Engineer)
Paul Norman (OWG volunteer, OSMF contractor)
Minh Nguyễn (OSMF core software development facilitator)
Rubén López Mendoza (OSM core software engineer)
Héctor Ochoa Ortiz (OSMF Board)

Minutes by Dorothea Kazazi, including some notes from Grant.

New action items from this meeting

Grant to do a dry run for the Mailman conversion, probably on Rhaegal (Croatia). [Topic: Upgrades - Machines on Ubuntu 22.04]
Paul and Grant to work on adding links to policy pages in error messages and Nominatim responses. [Topic: Response to vibe coding]

Reportage

OWG 2026 budget

Paul attended the February board meeting, in order to get a response from the board to his questions.

OWG should be preparing a budget including expansions.

Credativ consultancy on OSM.org Postgres database

Related to action item 2026-01-22 OPS to work on follow-up questions for the database consultant and then respond when ready. [Topic: Credativ consultancy on OSM.org Postgres database update]

Minh sent the existing OPS questions, almost verbatim, around a week ago. Has not heard back. Will ping Credativ.

Purchase of server for Nominatim in USA

Related to action item 2025-10-02 Grant to go ahead with the purchase of the Gen10 (second-hand) server for Nominatim in USA. PENDING: Blocked by OSMF treasurer. [Topic: Gen10 Nominatim purchase (USA)]

We have paid for the server. The supplier is waiting on some NVMes and hopes to ship out on the next week. Grant has notified the Oregon State University (OSU). OSU has asked us for a name for the machine, which can be provided by Sarah or Minh.

Collation of indexes

Related to action item 2025-09-18 Paul to look at potential issues related to the collation of indexes - Debian Postgres upgrade. [Topic: OSM DB upgrade to Postgres 17]

Action item to be removed.

Switching www.osm.org to Fastly frontend

Related to action item 2025-07-24 Grant to set-up a test for OWG's review [Topic: Switching www.osm.org to Fastly frontend] - Ready, pending some testing, to follow up via IRC and enable Fastly when ready.

We have a test instance. Blocking needs fixing, before we turn it on.

On blocking

Paul did some work on improving the Fastly config.
Suggestion: Keep current blocking on the backend and move it piece by piece to the CDN.

Multiple blockings

Weird user string related to old Chrome versions - blocked at Fastly.
>100,000 IP addresses blocked by Grant.

On IP addresses blocked by Grant

All IPs detected by Grant were using old user agents, but they have since updated their user agents to bypass the block.
Mixture of IP addresses, the majority being residential IP addresses. Many were related to Vietnamese ISPs and alsoa former governmental phone company.
This IP list needs to be removed, as the IP addresses keep changing and the list becomes outdated.
We need to find the equivalent signatures (e.g. user agents) to block them at Fastly - however, we can't see them, unless we let the traffic through.

A start-up company focusing on detecting residential proxy networks wants to talk with Grant. They can supply CSV dumps with new IPs and signatures for blocking. Tom and Paul are welcome to join the meeting.

Next step: Gather data and try and find the things to block on the frontend. During the initial period, blocking might be looser than we'd like.

Other points mentioned during discussion

Nominatim was getting 1800 requests per second from a single IP.
Many requests seem to be the result of vibe coding.

SQL query to identify additional email providers used by spammers

Related to action item 2025-03-20 Grant to run an SQL query to identify more email providers used by spammers. [Topic: Spam] #2025-05-01 Grant has created a small list now disposable email providers. 2025-09-18 parked

Action item to be removed.

Moving www.osm.org to Fastly

Tar-pitting

Chicken and egg issue with identify and block.
Best window is UK Saturday ~10:30am (GMT) morning for enabling, as there is very low traffic.

Suggestions

Select a time with low traffic, when we can put in the blocks, as we analyse the traffic.
Identify JA4H signatures from bad Nominatim traffic, and use it for blocks on the website.
- The overlap is probably going to be small.
Tarpiting or rate limiting API calls.
- We can't tar pit responses over 4,000 bytes.
- Tarpiting is not resource friendly - works well with error responses, as these are short.

Other points mentioned during discussion

Paul has looked into JA4T - TCP based, to identify traffic from mobile networks.
Paul has now access to the frontend servers.
Grant to upstream Tom's Apache changes into Chef.

Response to vibe coding

We should include verbose messages and links to policy pages that are hopefully been picked up LLMs.

Suggestions

Put messages about the need to comply to policies in the code files for the vibe code systems to read.
Be more verbose in the error messages and link to the policy pages.
- Tile.osm.org: we could return a header that is the link to the policy page or in the body of an error message.
API: provide structured responses.
Nominatim: include a policy link in normal responses, both for LLMs and humans.

Other points mentioned during discussion

The API can already provide responses in a variety of formats, depending on how you ask.
iD is switching to the JSON formats for the API endpoints.

Action item

Paul and Grant to work on adding links to policy pages in error messages and Nominatim responses.

Scraping osm.org webpages

There are scrapers that scrape osm.org web pages, including pages for nodes/ways/relations.

Grant has been contacting residential proxy companies about not being permitted to scrape the osm.org website. Grant had contacted the Licensing Working Group (LWG), to come up with appropriate wording. The LWG suggested that that they are doing it to get past our licensing terms, because if they go to planet.osm, it clearly states what the license of the data is.

Suggestion: Make the legal terms more prominent by including a licence link on the pages.
Presenting the terms makes it harder to deny seeing them and directly blocking the scrapers is not always easy.

On osm.org terms of use

The scrapers are violating our terms of use.
The terms of use are linked from osm.org (bottom right), however they are not explicitly saying that by using this site you are agreeing to the terms of use (post-meeting note: there is actually this phrasing in the terms "By using these Services, you agree to be bound by the following Terms of Service, as updated from time to time").

On agreement to osm.org terms of use

Documentation for osm.org account holders: Anton added a field to the user-record, showing the last time that an osm.org account holder agreed to the terms of use.
Logged-out users: Suggestion to have something to serve as a notice on the homepage. Preferably not a cookie banner.

Aim: have a stronger defense that we have given notice.

In terms of pursuing abuse, doesn't really matter what we're saying on their website, if we have notified them via email that they are not permitted to scrape the pages. This would supersede anything we put down for terms.
Maybe it would matter for damages by them before the communication.

On the company Bright Data

They have scraped/are scraping osm.org.
Israeli-based, backed by venture capitalists.
They have won legal cases against scraping Facebook and Twitter. They have also scraped Google Maps and got away with it.
Offices in New York, London and Israel.
Google could put measures to block any apps that have their SDK.

On the suggestion to escalate to the SDK makers or app-stores

Issue: attributing the scraping to any SDK from our point of view is not possible. as we get faked headers.

A company that approached us finds all the SDKs, installs them, monitors them and identifies the SDKs used for scraping and the resources they are scraping. They can supply us with some data and we could identify some patterns, but not necessarily identify which residential proxy networks have been used. There is also a lot of reselling going on.

Other points mentioned during discussion

The iD editor is changing to the json format.
Open Database License (ODbL) does not hold for a single item (node/way/relation).
The Licensing Working Group (LWG) discussion related to the terms of use is more about future changes to the terms or privacy, but not about abuse.
Some of these companies with residential proxy networks are very litigious.
The company selling the SDK may not be the same as the one operating the network or the one from which users purchase scraping services.
Grant has encouraged the OPS at Wikimedia to be more vocal in the problems that they're facing on a day-to-day basis.

2026/2027 growth ideas

community.osm.org: fine
Nominatim: reasonable levels

On upgrading frontends - G9 replacement on late 2027

Replacement time depends on the AI bubble and how much we can control the scraping abuse.
14 G9s in service, as we had put off upgrades for a long time. * Have 2016 CPUs and they mostly cope - until hit by abuse waves.

On upgrading the switches

They have very small CPUs and they would have issues if we added more logic into the firewall rules or decided to do dynamic routing.
Next generation have faster CPUs, x86 instead of arm, but the prices are still high.
Low priority.

On growing the OPS team

Hesitation to add more things on the OWG's to-do list, while the team is small.
Paul has asked the board whether they are considering working on the Strategic Plan's items 1.2, 1.3 and 1.4:
- Ensure high availability of operations team
- [[Strategic_Plan#Increase_paid_system_administration_staff|Increase paid system administration staff]
- [[Strategic_Plan#Increase_number_of_volunteer_administrators|Increase number of volunteer administrators]
- Paul was recluctant to push on that, as he noted that he would have a conflict of interest, as he would be applying.

Other points mentioned during discussion

Grant has started testing the mailman conversion, so that we can finally get rid of Shenron (Mailing lists server, hosted by Bytemark).
Grant has created an account on the dev server for Rubén.

Suggestion: Convey to the board that the team is spending most of its time on fire-fighting (scrapers, etc) and required short-term tasks

Any other business

Upgrades: Machines on Ubuntu 22.04

There are a couple of machines with Ubuntu 22.04 (databases, planet, mail server) and Ubuntu 20.04.

Ridley (Site gateway, Foundation related sites) has blockers: CiviCRM and potentially some old SotM websites.
Norbert (Backup server, Web server for planet.openstreetmap.org, Planet file generation server) would be an outage, while we do the planet migration - could have a temporary redirect.

Suggestions

Do a dry run for the Mailman conversion probably on Rhaegal (Croatia). We have 3 to 4 unused machines (Albi, Angor, Meraxes, Rhaegal).
Do Dublin planet server conversion on Saturday.
Do Horntail (Web server for planet.openstreetmap.org)

Other points mentioned during discussion

Grant had started Debian 13 testing - it doesn't affect mailman, as there is only a tiny difference between 12 and 13.

Action item

Grant to do a dry run for the Mailman conversion, probably on Rhaegal (Croatia).

Action items

Action items that have been stricken-through are completed, removed, or have been moved to GitHub tickets.