Operations/Minutes/2024-06-27
OpenStreetMap Foundation, Operations Meeting - Draft minutes
These minutes do not go through a formal acceptance process.
This is not strictly an Operations Working Group (OWG) meeting.
Thursday 27 June 2024, 19:00 London time
Location: Video room at https://osmvideo.cloud68.co
Participants
- Tom Hughes (OWG)
- Grant Slater (OWG)
- Paul Norman (OWG)
- Guillaume Rischard (OSMF board)
- Branko Kokanovic (Microsoft)
Minutes by Dorothea Kazazi.
Not present
New action items from this meeting
- Grant to send an announcement about the OAuth status on Monday. [Topic: OAuth]
- Paul to see if we're still appropriately balanced on the CDN and then OPS to decide on upgrading the RAM of Odin. [Topic: rhaegel usage?]
- OPS need to do capacity planning for tile.openstreetmap.org [Topic: rhaegel usage?]
Matomo - Microsoft access request
Background |
---|
About Matomo: Matomo (previously Piwik) is a website statistics and analytics tool, similar to Google Analytics, but open source, and deployable to your own server. OpenStreetMap uses this on the main osm:web front end, to gather stats on traffic and user flow. Read more on the OSM wiki.
Discussion on access request by Branko (Microsoft). Branko to ask LWG whether the privacy policy allows that. |
Branko Kokanovic (Microsoft) talked with Paul at SotM US 2024 and asked to do some stuff with Matomo. Microsoft started AB testing the single-sing on (SSO) and the results are suboptimal. Only 35% of people who encounter the OSM sign-in proceed to complete the sign-up process.The reason for this drop-off is unclear to MS, prompting a request for analytics from the OSMF side. They wish to use Matomo to see the number of visitors and sign-ups, to better understand the sign-up funnel. Also the MS team offered to do other analytics that the OPS lack the time to perform. Currently, OPS do not use Matomo.
Options include
- Provide answers.
- Grant them access to Matomo.
Concerns
- Setting a precedent, as access is not typically granted to others.
- Privacy/personal data issues.
- Corporate nature of the request.
Information logged by Matomo
- Account creation statistics are logged.
- Matomo does not display the full IP address, discarding the last two bytes for IPv4 addresses.
- Geolocation is performed before discarding the last two bytes of IPv4 addresses.
- Log-ins are shown in real-time.
- If a user is logged in, the UserID is anonymised in Matomo, showing only a hashed version.
- We do not log user progress through the sign-up process, as this feature is unavailable in the free version of Matomo.
Other Matomo info
- Matomo shows that the copyright page is the top landing and exit page, with a bounce rate close to 100%.
- Fewer than 1% of site visitors sign up.
- With the removal of the terms page, the sign-up funnel is minimal. Users must fill in the form and confirm their account.
Suggestions
- Require non-disclosure agreements (NDAs) to be signed.
- An NDA alone is insufficient. Changes would have to be made to the Privacy Policy https://osmfoundation.org/wiki/Privacy_Policy if Matomo access extends beyond internal use.
- It might boil down to trust, as OSMF cannot audit Microsofts's use of the data once access is granted and will not know who within Microsoft is accessing it.
- Access would be granted to individuals, not to Microsoft as an entity.
- Get input from the LWG.
- Have Microsoft lawyers review our privacy policy and provide an opinion on the appropriateness of granting access under an NDA.
- This review could be done, but it might take several months.
Decision seemed to be
- Microsoft to ask LWG whether our privacy policy allows granting Matomo access to Microsoft
- Branko offered to initiate the legal review process within Microsoft in parallel to assess the implications of OSMF granting access to Matomo under an NDA.
SotM EU 2024 attendance
Paul is attending SotM EU 2024 in Poland. Tom and Grant are currently undecided.
OpenMapTiles
Postpone to future call. We require additional time to review the detail.
They replied.
- Opposed to adding anything other than OSM Carto clone.
- There are no answers on the status of their licence change or the branding issue.
On uniqueness
- The cartography is intended to be identical to an existing layer.
- Technically it is unique, as vector tiles are inherently different from raster tiles.
Other points mentioned during discussion
- New vector style might be introduced.
Decision: Look at their answer in more detail.
OAuth Status
Brownouts continue. Editors are now generally good-to-go.
1st July go-date seems ok. Grant to push out an announcement.
- Tom ran it overnight from midnight to 07:00 UTC.
- Potlatch 3 ok now.
Action item
Grant to send an announcement about the OAuth status on Monday.
rhaegal usage?
Server | Located at | Description | System | Distro | Stats |
---|---|---|---|---|---|
rhaegal | Hosted by CARNet in Zagreb, Croatia | Supermicro X9DRW | Debian 12 | prometheus |
rhaegal as tile render server or keep for future vector?
Also discussed: https://GitHub.com/openstreetmap/operations/issues/1105
- We had reserved rhaegal for Vector Tiles, but Paul happy with Faffy (dev server).
- Preference to not have rhaegal as a raster tile server.
On the suggestion to add more rendering capacity in Europe
- rhaegal is less powerful than Odin and Ysera, which have queues. When they haven't got a mass dirty event, they're not dropping anything, just using the queues as a buffer.
- The increase in load is being driven by the fact that we're using more accurate expiry now. The increase was when we switched to pgsql expiry.
- Look at performance of Ysera vs Odin.
On Odin and Ysera
Server | Located at | Description | System | Distro | Stats |
---|---|---|---|---|---|
odin | Equinix Amsterdam | Tile server | Supermicro SYS-1029P-WTRT | Ubuntu 22.04 | prometheus |
Ysera | University College London | Tile server | Supermicro SYS-1029P-WTRT | Ubuntu 22.04 | prometheus |
- Odin and Ysera are past their predicted life-span. We had specced them out to last 5 years.
- Number of coses: 28 cores.
- CPUs: 6-7 years old.
Suggestions:
- Upgrade Odin and Ysera to the same clases as newer European render servers.
- OPS to look at performance of Ysera vs Odin, as Ysera now has more RAM.
- Some improvement ~ 10% extra performance.
- There's an open ticket. Cost: EUR 260+ to upgrade the ram + remote hands cost.
- Could double the number of calls in the system and go slightly higher clock rate, but it's probably about a GBP 1000 per machine.
- Not worth it.
Other points mentioned during discussion
- At worst we're serving about 5% dirty tiles of what hits the back end, and missing virtually none.
- Rams equally clocked at both machines.
Discussion on graphs
- Missed tiles: Where we served a 404 because we were not able to render it in time.
- Dropped tiles: where the tile was dirty, but the queue was really full, so we have failed to queue it.
Action items
- Paul to see if we're still appropriately balanced on the CDN and then OPS to decide on upgrading the RAM of Odin.
- OPS need to do capacity planning for tile.openstreetmap.org
"OSM PPA" work approval
Grant ok to build "OSM PPA" for Debian 12 using GitHub actions?
Will use GitHub actions private runner for ARM64 builds until 2025 (Catford hosted cluster of Raspberry Pi 5)
Estimate of work required 1 week.
Based on https://salsa.debian.org/salsa-ci-team/pipeline (builder) (debootstrap)
Florian https://wiki.openstreetmap.org/wiki/User:Flohoff had sent an email.
Suggestions
- Discuss the design offline.
- Use package.io - is a commercial repository, but they do have an open source option.
On arm building
- Discuss the design
- Currently it is a tricky requirement, as if you use emulation with GitHub actions, it is extremely slow (GDAL takes 6 hours to build and hits a timeout). It will probably be similar for Mapnik.
- Grant has some scripts for private testing.
Other points mentioned during discussion
- Build step is some wrapper shell scripts.
- The DSC tells it where the source is and it downloads the source from there and builds it.
- Estimated amount of work: 1 week.
- 5 packages to be built. Some are just available in backports. CGImap is blocking us moving with the six front ends.
Feedback
Tom and Paul in favour of Grant working on OSM PPA, as it will allow us to unblock a lot of Debian moves.
Action item
Grant to check whether the free version of package.io will cover us.
Editor policy public current draft?
Backgrouns |
---|
Editor Inclusion Policy for adding editors to the list of supported editors on OpenStreetMap.org
Initial draft policy (the current version is different), discussed on community.osm.org Proposed new editors that will be featured on the OSM website should follow the following guidelines. Anyone may propose new editors for consideration and any such proposed additional editor should be added to the proposals section of the Featured Editors 12 page. The acceptance of an additional editor is not automatic and must be assessed on a case-by-case basis using the following criteria as a guide. Definition Editor: A GUI application that provides its users with the capability of downloading, reviewing and modifying current OSM data, and committing changes to the data back to the OSM database. Criteria for possible inclusion A proposed editor must satisfy the following hard criteria: Internally supported. The editor’s maintainer must be in favor of having their editor included on the OpenStreetMap website. General Purpose and General Audience. The proposed editor must be a general purpose desktop GUI map editor aimed at a broad audience of (potential) mappers. Actively used by Mappers Proven. The proposed editor must have been in active public use for a while and demonstrated its usefulness by attracting an at least X active community of mappers. ? Or Consensus that it will match this criteria once launched ? Best Practices. The proposed editor should make a reasonable effort to support widely accepted best editing practices to discourage mappers from making harmful edits to OpenStreetMap data. Open Source. The proposed editor’s source code must be published under an open and permissive license. Community Feedback. The proposed editor must have an open community feedback channel where its maintainers actively respond. Reliable service. The proposed editor must be reliable in terms of uptime and availability. Privacy Policy. The proposed editor must publish a Privacy Policy that clearly lays out how any PII is stored and handled. A proposed editor should satisfy the following soft / arbitrary criteria: Open Contribution. The proposed editor’s maintainers should be open to receiving community contributions to the code and adding new people as maintainers. Internationalisation. The proposed editor should offer languages other than English for its graphical user interface. Help and Documentation. Help and / or documentation should be available to aid mappers in using the proposed editor. Unique. The proposed editor should offer something unique in approach or execution. Additional notes
Previous discussions: |
No push back from the Software dispute resolution panel (SDRP).
Suggestions
- Publish draft for review.
Other points mentioned during discussion
- Opinion expressed opposing the foundation putting a requirement that someone has to join the SRDP.
- Related comment by R. Fairhurst email pasted in the chat during the meeting.
Points to consider
- whether requiring to be a member, would be a conflict with the SDRP's mandate and,
- whether it would also be at odds with the foundation's mission to support but not control the project.
Action item
Grant to publish the updated draft for feedback and we will review at next Ops call. Feedback period 2 weeks.
Fastly
Fastly wants us to use things like their image tool processing tool "image optimizer" to serve webP when the browser supports it. Fastly's aim: For communication purposes.
On Fastly's "image optimizer"
- Can change the format of your image based on whether or not the client supports webP.
- Is not enabled on our account yet, but can be enabled in a couple of minutes.
On image file formats
- If we're going to serve webP and PNG, we should be generating webP and then converting it to PNG.
- webP can be faster to generate with mod-tile in some instances than a compressed PNG, depending on the compression.
Other points mentioned
- Most browsers can handle it well if they are instructed to load a PNG file but are actually served a WebP file.
- Most browsers since IE5 do content sniffing.
Concern: Would need to be open source, to power www.osm.org. Compute and the image processing stuff are different from the CDN.
On visualisation example mentioned
- Example of what could be created: map showing a percentage of our tiles, as flashing squares. Or a heat map.
- Tom had produced a nice map showing tile usage in Prometheus or Grafana. It was based on the metrics that Fastly provide of how many tiles they were serving from each cache.
- The visualisation probably does not have to be in real time, but might be nice to have.
Issues:
- We do not know exactly what they want.
- We probably do not have the bandwidth to take such a project on.
Fastly:
- They might be more interested in where the people were that were requesting tiles rather than what tiles they were requesting.
- Could do the sampling and send a percentage of logs.
- Do not want access to our logs.
- They have some visualisation people.
Other points mentioned
- Map on Mastodon showing the date shifting, posted by Paul. However, it is not suitable for real time.
Contract
- 9k/month pretend value, excluding the actual bandwidth.
Action items reviewed at the beginning of the meeting
- 2024-06-13 Grant to email SDRP and copy OPS and Guillaume
- 2024-05-30 OPS to add the Software dispute resolution panel requirement to the Editor Policy draft and see what feedback we receive. [Topic: Editor Policy]
- 2024-05-02 OPS to revisit the OpenMapTiles application.
- 2023-05-18 Paul to start an open document listing goals for longer-term planning. [Topic: Longer-term planning]
Action items that have been stricken-through are either completed, or have been moved to GitHub tickets.