Working Group Minutes/EWG 2012-02-13

Attendees

IRC nick	Real name
apmon	Kai Krueger
migurski	Michal Migurski
TomH	Tom Hughes
zere	Matt Amos
Summary

zere asked for feedback on a pseudo-ruby license-change algorithm [1].
There was discussion of vector tiles:
- What is a vector tile? There are a range of different possibilities at different points on a scale of generality (OSM XML) and ready-to-useness (SVG).
- It was a general consensus that a middle-way, with some geometry pre-processing and filtering, was probably the most useful. Something like GeoJSON, which can be output with KothicJS' engine or Tilestache.
Vector tiles discussion led into a discussion of how to get the best of both Imposm and Osm2pgsql together. A simple proposal of using views and partial indexes might produce benefits, but there were doubts over the performance.
IRC Log

18:01 < zere> minutes of the last meeting, for your enjoyment: http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-02-06
18:02 < zere> if there's anything wrong, please let me know.
18:04 < zere> last meeting i said i'd put together a code-y algorithm for the license change cleaning, and here it is: https://gist.github.com/1817152
18:05 < zere> it's sort of pseudo-ruby, might be a little verbose, but pretty close to working code
18:05 < zere> it's based on what frederik has told me of the OSMI license change layer process
18:09 < migurski> reads pretty clearly. I'd love something similar for osm2pgsql's polygon internals. =)
18:10 < zere> there's a few areas of uncertainty - particularly around what constitutes a "harmless" edit of a relation. and i'd welcome any feedback. i'm hoping to clean it up a bit and send it on to LWG. if they're happy that it's close to what they want then we can start building real code around it.
18:10 < zere> migurski: thanks :-)
18:14 < zere> last week RichardF asked to have vector tiles put on the agenda, but he's not about today. anyone want to lead off with something on that topic?
18:17 < migurski> Vector tiles == .osm XML?
18:17 < migurski> or SVG?
18:18 < apmon> For what purpose would the vector tiles be?
18:18 < zere> well, exactly... there's a range of "usefulness" - at one end, there's OSM XML which needs a lot of pre-processing and styling before it's presentable on-screen. on the other hand, there's SVG which needs little pre-processing and already contains styling information.
18:20 < apmon> KothicJS tiles are probably half way between them
18:20 < migurski> I've been talking to the DevSeed people about adding Mapnik support for a tiled datasource
18:21 < migurski> so tiled data is useful for that - it would be possible for people to be up and running with OSM + Mapnik without ever installing Postgres
18:21 < zere> the best reason i can see for having vector tiles is that they're potentially more useful as client-side engines become more powerful. if it's possible to style them then it could be that it's a very good alternative to setting up a completely new tileserver for each new style?
18:21 < migurski> exactly
18:21 < zere> which does rather rule out SVG as a candidate...
18:22 < zere> i guess we're talking about something closer to geoJSON?
18:22 < migurski> yeah something like that
18:22 < migurski> we've experimented heavily with this stuff in the past
18:23 < migurski> and found that geoJSON clipped to six digits of floating point precision + gzip was competitive in terms of file size with custom formats, and easy to deal with
18:23 < apmon> As far as I can tell kothicJS uses a form of geoJSON
18:24 < migurski> it was also very necessary to simplify geometries for tiles below z13 or so
18:24 < migurski> just became unmanageable
18:25 < zere> it's pretty easy to pull geoJSON out of postGIS in tiles, and simplify them en-route. could that be a solution, and allow re-use of the osm2pgsql expiry code?
18:25 < migurski> not familiar with the latter
18:25 < migurski> oh
18:25 < migurski> right
18:25 < apmon> Yes, it should be possible
18:25 < migurski> so, keep an osm2pgsql-to-postgis DB
18:26 < migurski> and run tiles out of that
18:26 < zere> just so we can cache the output, not hit the DB every time someone requests tile 0/0/0 ;-)
18:26 < migurski> yeah
18:26 < apmon> You can also use tirex with mod_tile for delivery
18:26 < zere> is what we're discussing similar to the tirex json backend you've been writing, apmon?
18:26 < apmon> mod_tile can already deliver js tiles (or other formats)
18:26 < migurski> these things can have long cache expiry times - days or more
18:26 < apmon> I think so, yes
18:27 < apmon> It should be very simple. I just haven't gotten around to it.
18:27 < apmon> Partly because I am not too familiar with python
18:27 < apmon> or perl, which is what tirex is written in
18:27 < migurski> *shudder*
18:28 < apmon> The communication between the tirex part and the "renderer" is actually very simple text over a socket.
18:29 < apmon> So you could write it in pretty much any language
18:29 < migurski> I can do a first pass of something like this in Python, using Tilestache
18:29 < apmon> It was just that I wanted to reuse the python code from the kothic server code
18:30 < migurski> ah
18:32 < zere> i wouldn't worry too much about implementation language at this point. just a proof-of-concept and then we can try integrating it with current server code - renderd / tirex / whatever.
18:32 < apmon> http://code.google.com/p/kothic/source/browse/src/ json_getter.py is the code for the backend
18:32 < apmon> It does caching and rendering, but no queuing (which is why I wanted to integrate it into tirex)
18:33 < migurski> I see that kothic dynamically simplifies geometries
18:33 < migurski> it's probably worth pre-simplifying geoms in PostGIS ahead of time
18:37 < zere> that kinda touches on another thing i know i've discussed before - getting something with the diff support of osm2pgsql and the flexibility and multiple table support of imposm.
18:39 < migurski> apmon: the built-in Tilestache Vector provider might serve the need we're talking FWIW http://tilestache.org/doc/TileStache.Vector.html
18:41 < migurski> zere: imposm does all it's own importing, but there's no reason it can't be implemented as a set of postgres queries on top of osm2pgsql based on my experiments with it
18:41 < migurski> so, osm2pgsql for the importy bits
18:41 < migurski> then some very based postgres queries to generate the imposm tables
18:42 < migurski> I've not yet seen anything in those tables that's not derivable from osm2pgsql tables
18:42 < zere> i'm not sure i'd want to rebuild the whole set of tables every minute, or whatever the diff import period is...
18:44 < migurski> they could be views
18:44 < zere> but it might be possible to hook into osm2pgsql's knowledge of which features have changed to limit the extent of the rebuilds.
18:48 < migurski> going to duck out - need to actually go to the office today =\
18:49 < zere> migurski: cool, thanks for coming :)
18:49 < migurski> bye! =)
18:49 -!- migurski [~migurski@dsl081-049-227.sfo1.dsl.speakeasy.net] has quit [Quit: migurski]
18:50 < zere> i just tried the views method, btw, but unfortunately it seems postgresql doesn't support creating indexes on views, which might make run-time query performance unusable.
18:51 < TomH> it won't create indexes on views no
18:52 < TomH> but it will use underlying indexes
18:52 < TomH> views are implemented by query rewriting in postgres you see
18:53 < zere> ah, so a query like "select way,name from planet_osm_lines where highway='residential' and way && bbox(...)" is the same as "select * from planet_osm_residentials where way && bbox" and planet_osm_residentials is a view "... where highway='residential'"?
18:54 < TomH> exactly
18:54 < TomH> postgres has a query rewriting engine that you can write rules for, and views are implemented on top of that
18:55 < zere> ah, and would require the same set of partial indexes to support any of these at low zoom with any speed.
18:56 < zere> for example, pulling capital cities out of a z4-7 tile, just from planet_osm_point, is pretty horrendous without a partial index.
18:56 < TomH> http://www.postgresql.org/docs/9.1/static/rules.html
18:56 < TomH> http://www.postgresql.org/docs/9.1/static/rules-views.html
18:58 < zere> yikes - rules look a bit like triggers on steroids. plenty of firepower to blow one's leg off ;-)
19:00 < TomH> BTW I plan to push tmcw's edits shortly unless anybody has any last minute objections
19:01 < zere> the whitespace/border changes?
19:01 < TomH> what you see at http://tomh.apis.dev.openstreetmap.org/ basically
19:02 < zere> yeah, i've got no objections
19:04 < zere> thanks to everyone for coming! see you next week :-)