Working Group Minutes/EWG 2012-03-05

Attendees

IRC nick	Real name
apmon	Kai Krueger
migurski	Michal Migurski
pnorman	Paul Norman
tmcw	Tom MacWright
TomH	Tom Hughes
zere	Matt Amos
Summary

Review of TTTs.
- OSB/notes - some progress on this, TomH to look this week.
- PL2 tutorial mode - get update from RichardF next week.
- PL2 i18n
  - Is now hooked up into translatewiki
  - Question over whether the features.xml file is translatable or localisable.
- i18n in OSQA (for help.osm.org) - seems that there's some upstream bugginness, will need someone to take this on.
- ODbL tools
  - The test suite for license changeover cases has had some extra input, but still needs more collaborators.
  - Would be useful to run it on an extract, e.g: from [1], which is small enough for people to play with locally.
- Clickable POIs
  - There was a wide-ranging discussion about what's available with the different render targets / metawriters in Mapnik and what will be available in upcoming work (GeoJSON renderer).
  - Database layouts / osm2pgsql / imposm were discussed in relation to challenges in transforming the OSM style XML (now called "standard") to Carto where it would be easier to edit.
IRC Log

18:05 < zere> minutes of the last meeting - it was a busy one! http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-02-27
18:06 < zere> as always, please let me know if there's anything wrong or missing
18:06 < zere> draft agenda for this meeting is to do a review of the TTTs on http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks and see if there's anything that needs updating / progress needs adding.
18:10 -!- migurski [~migurski@199.115.243.153] has joined #osm-ewg
18:10 < zere> first item on TTTs is the OSB/notes branch, which we discussed some last week. anything to add to the progress page http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks/Progress/OpenStreetBugs/notes_integration ?
18:10 < migurski> hi all.
18:10  * migurski splits attention with GDC presentation on javascript game development
18:11 < TomH> zere: afraid I didn't get time to look at it - will try and do so this week
18:11 -!- Milkshake [~836b0073@shenron.openstreetmap.org] has joined #osm-ewg
18:12 -!- Milkshake [~836b0073@shenron.openstreetmap.org] has quit []
18:12 < zere> TomH: cool, ok :-)
18:12 -!- Milkshake [~836b0073@shenron.openstreetmap.org] has joined #osm-ewg
18:13 < zere> next item is PL2's tutorial mode. http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks/Progress/Potlatch_2_tutorial_mode given that RichardF sent his apologies, i'll assume there's no update on this?
18:15 < zere> ok. the next one is supporting multiple languages on help.osm.org - http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks/Progress/Support_for_multiple_languages_on_help.osm.org
18:15 < zere> there's been no activity since last year. anyone a python/django fan and would like to take a look?
18:16 < apmon> zere: You skipped i18n of P2.
18:16 < apmon> Can this be marked as done?
18:17 < zere> yeah, that one i think is finished. at least, barring the bugs. i was going to leave that until last and see if we wanted to replace it with something else.
18:17 < TomH> well it's up and running on translatewiki now
18:17 < TomH> not sure if there are still strings in the code which need enabling for translation
18:18 < apmon> Yes, there are still a couple, as far as I can tell, but a good deal is now translateable
18:18 < apmon> The main thing is probably still missing is the features.xml (or what that file is called). But I am not sure if that falls into the same category
18:19 < TomH> well the question there is whether the whole thing (like what features are offered) should be varied
18:19 < zere> ah, so the features.xml contains strings in english that can't currently be translated?
18:21 < zere> ok, i'll make a note of that and we'll revisit when RichardF is around.
18:22 < apmon> But overall, I think translating p2 has been a success, with many translations coming in from translatewiki http://translatewiki.net/wiki/Translating:OpenStreetMap/stats/trunk
18:24 < zere> going back to the django / OSQA thing - surely we must have a pythonista here? migurski, tmcw, ajashton1?
18:25 < migurski> I am a python lover
18:25 < tmcw> I don't really fashion myself a *ista; not very familiar with OSQA though
18:25 < migurski> I think I missed the OSQA bit though
18:25 < tmcw> is this not a situation where osqa would be translated upstream?
18:26 < zere> ah, i don't think the problem was that OSQA isn't translatable - it seemed that one had to pick a single translation and use that, rather than being able to set a language on a per-session basis.
18:29 < TomH> ris looked at it, and add the normal django language selection stuff
18:29 < TomH> but it didn't seem to work - it would keep randomly using Japanese instead of the user's prefered language
18:30 < zere> hmmm... do you know if that code is up anywhere to collaborate on?
18:30 < TomH> all he had me do was add a couple of lines to the settings file to activate the standard django locale modules
18:30 < TomH> the osqa code is available sure
18:31 < zere> ok, so there's some bugginess there that could do with some attention.
18:31 < TomH> http://svn.osqa.net/svnroot/osqa/trunk
18:33 < zere> ok. let's move on or we won't get through these in time.
18:34 < zere> next item is ODbL migration tools. i've been working on some code to handle these and i'll put in an update for that.
18:34 < zere> does anyone else have anything they'd like to add to the update for ODbL tools?
18:35 < zere> from my side, work proceeds. i'm still looking for collaborators and both Frederik and Dermot have pledged to do some work too.
18:35 < migurski> what do you need in terms of collaboration? 
18:36 < zere> the main item is trying to pin down the license change as a set of unit tests, e.g: https://github.com/zerebubuth/openstreetmap-license-change/blob/master/test.rb
18:36 < zere> i think we've come close to being comprehensive on nodes, but not on ways or relations yet.
18:37 < migurski> this is the actual code that will be run next month? 
18:37 < zere> of course, the other side of the coin is that the tests must also pass, and the majority of the (now pretty fugly-looking) code is in the https://github.com/zerebubuth/openstreetmap-license-change/blob/master/change_bot.rb file
18:38 < zere> i don't know if this is the actual code, but so far there are no alternatives presenting themselves, so that looks likely.
18:38 < migurski> does it make actual queries against the API to live-check changesets? 
18:39 < zere> at this stage it's all stubbed out, but i hope stubbed out in a way that will make it easy to add interfaces to the API or direct to a DB.
18:39 -!- Milkshake [~836b0073@shenron.openstreetmap.org] has quit [Remote host closed the connection]
18:40 < migurski> is the idea that the code returns a thumbs-up or down for any given change? 
18:41 < apmon> Are there any estimates of how long the conversion will take?
18:41 < apmon> i.e. if you run it on the full database?
18:41 < zere> ah, it's more complicated than that. the code returns a set of changes to be made, where Edit[...] is an action to edit the element and Redact[...] is a hypothetical API call to mark a revision in the history as unsuitable for the change.
18:41 < zere> apmon: no, i've not got that far - hence the calls for help! :-)
18:42 < migurski> is it something that needs to be run against the production database? 
18:43 < zere> no, at the moment it doesn't run against any database - it's all stubbed out with in-memory structures. the plan is that it will be runnable against the API *or* any rails_port database.
18:43 < zere> and i know Dermot is interested in getting a test DB up on the dev server and loading a full-history dump (or extract) to start getting some concrete results from.
18:44 < migurski> it'd be interesting to make it possible to run this on a canned, smaller dataset
18:44 < migurski> just London, for example
18:45 < migurski> small enough to try on EC2 or a small box
18:45 < zere> yes, indeed. i think i remember hearing about some full-history-planet extracts. does anyone know anything more about that?
18:46 < apmon> http://lists.openstreetmap.org/pipermail/dev/2012-March/024456.html
18:48 < migurski> I see .osh files there
18:48 < zere> excellent! i'm suprised the berlin history is only 84MB to be honest!
18:48 < migurski> wow yeah, that's small - maybe because it grows in one direction? 
18:48 < zere> yeah, .osh is just another way of saying .osm, but letting you know that you should expect multiple versions of the same object.
18:49 < apmon> 47Mb as .osh.pbf
18:49 < apmon> there is a bigger selection of extracts as .pbf than as .bz2 it seems
18:50 < zere> i'm behind the times on PBF tools nowadays. does osmosis now handle all the extra flags & stuff to load a .osh.pbf into an apidb?
18:52 < zere> well, maybe straying a little too far from the agenda - we can talk about this on rebuild or dev...
18:53 < zere> the next item in the TTTs was clickable POIs. migurski, i know you had some ideas you wanted to try out?
18:53 < migurski> yeah, slowly. Been talking to tmcw offline about them, mostly around fiddling with the UTFGrid format
18:54 < migurski> I suspect that properly implementing this will involve deeper integration with the mapnik style
18:54 < tmcw> And mod_tile, etc.
18:54 < migurski> yeah
18:54 < tmcw> willing to help out with tips & some code, but it's not one of my major priorities
18:55 < apmon> tmcw: What needs doing on the mod_tile side?
18:55 < tmcw> apmon: there's a new mapnik API for rendering grids that would need to be exposed
18:55 < zere> yeah. that's what we had to do. https://github.com/MapQuest/MapQuest-Render-Stack/blob/master/py/renderer/mapnik.py#L141
18:55 < apmon> do you mean mod_tile, or renderd?
18:56 < zere> but we were using the inmem-metawriter, rather than the rendering grids. but i suspect the code would be quite similar
18:56 < tmcw> apmon: honestly don't know, haven't installed either. I'm assuming that whichever is doing the mapnik magic doesn't support this yet
18:56 < apmon> Not sure, but it might be easier to do this in tirex instead of renderd
18:57 < zere> tmcw: is it available via the python bindings?
18:57 < apmon> although if it is just an extension of mapnik, it perhaps doesn't really matter, as both use a c program to interface with mapnik
18:58 < tmcw> zere: y, render_grid
18:58 < migurski> FWIW, I'm also yak-shaving a pure-python implementation of UTFGrid
18:59 < zere> so, to continue to blow my own trumpet, it should be easy to add to the MQ renderer - the "worker" code there is all in python :-)
18:59 < migurski> yay, python =)
18:59 < zere> (although documentation is somewhat non-existent)
18:59 < migurski> the difficultly will be integration with the visible POIs
18:59 < tmcw> migurski: with mapnik?
18:59 < migurski> nah, just PIL - this is why I'm viewing my initial experiments in this direction as yak shaving
19:00 < tmcw> ah, okay
19:00 < tmcw> it was originally implemented in python/mapnik, and only recently became pure mapnik
19:00 < migurski> ultimately it'll be important for the active areas of the map to correspond to where mapnik placed the image
19:00 < tmcw> basically most 'iterate over all the things' techniques are dog-slow
19:00 < migurski> am I right that it's still necessary to specify the metawriter output file in XML for mapnik? 
19:00 < tmcw> the only reason the mapnik impl is passable is dane + agg magic
19:01 < tmcw> If you're using metawriters, yeah
19:01 < migurski> I imagine he's doing what I'm doing, which is outputting pixels into a buffer and turning them into data?
19:01 < migurski> re: XML, that's a real drag
19:01 < zere> no, no! use the inmem metawriter and you can get it out programmatically, no XML involved!
19:01 < migurski> totally the wrong place to put that feature; any plans to move it to the bindings like the grid?
19:02 < tmcw> the api hasn't gotten much love since utfgrid addressed the same concern and the geojson output concern will be implemented as a separate output format
19:02 < migurski> gotcha
19:02 < tmcw> basically no; I think metawriters are deadpool - the future work is in artem's geojson output work
19:02 < migurski> I've only seen UTFgrid demoed with polys
19:02 < migurski> does it work with lines, line-widths, and points images as well?
19:02 < tmcw> Yes, with any feature that mapnik outputs
19:03 < migurski> and it's a 2.0-only thing, yes? 
19:03 < tmcw> and correctly with layering, opacity, aliasing, etc.
19:03 < tmcw> due to dane-tears + agg
19:03 < migurski> ha
19:03 < tmcw> yes
19:03 < migurski> okay cool
19:03 < migurski> PPA's for 2.0 have been broken so I've not spent much time with it.
19:03 < migurski> but support for it is in Tiledrawer, which is my current bicycle for testing things mapnik
19:04 < tmcw> Yeah, PPAs are... awful.
19:04 < migurski> zere: inmem metawriter? 
19:04 < tmcw> We really weren't prepared for software distribution on linux being more painful than osx
19:04 < migurski> but generally, it's a single-pass affair in mapnik? 
19:04 < migurski> ugh, yeah
19:04 < tmcw> heck, I think even windows builds are easier
19:05 < zere> migurski: https://github.com/mapnik/mapnik/blob/master/bindings/python/mapnik_inmem_metawriter.cpp#L47
19:05 < migurski> I would like to purchase a small vial of dane tears btw
19:05 < migurski> ward off evil
19:05 < tmcw> Oh wow, MetaWriterInMem
19:05 < tmcw> guess that exists now
19:05 < migurski> zere, how does this manifest in python? 
19:05 < zere> it's like a metawriter, except that instead of putting the XML on a file on disk somewhere you have to read it back, it keeps it all as objects in memory which you can access.
19:07 < zere> you call map.find_inmem_metawriter("metawriter_name"), if that's not None then you can iterate over it, getting back objects with .box for the bbox and .properties for the properties map defined in the metawriter in the style XML.
19:09 < migurski> gotcha
19:09 < migurski> sounds orthogonal to utfgrid though, yes? 
19:09  * pnorman waves, from the ipad
19:09 < zere> it's far from perfect, and doesn't currently handle linear or polygon features, but it works for me doing clickable POIs
19:09 < zere> and yes, UTFgrid seems to be "for" a slightly different use-case than metawriters.
19:10 < TomH> so long as you don't want to handle polygon POIs you mean ;-)
19:10 < migurski> ha
19:10 < migurski> well
19:10 < TomH> aka the "mapzen poi collector" problem
19:10 < zere> i'm looking forward to seeing more of artem's geojson output, which might make the metawriter stuff obsolete...
19:10 < migurski> I was imagining that with a full-map hit grid and just one osm.xml, it would be possible to make everything clickable
19:11 < zere> TomH: solved - the centroids are drawn as icon + text where the icon + text are clickable and the polygon isn't ;-)
19:12 < zere> yeah. my use-case is people clicking on POIs to get directions, or view the info page. this was, after all, an MQ project. it wasn't looking at making the map completely interactive (unfortunately... maybe next time).
19:12 < migurski> anyway, goofing with it here
19:12 < migurski> http://linode.teczno.com/~migurski/osmgrid/tile.cgi/grid/preview.html#12/37.7817/-122.1963
19:12 < migurski> and http://linode.teczno.com/~migurski/osmgrid/tile.cgi/grid/16/10509/25324.json
19:15 < zere> nice! it's really interesting to see the actual hit-areas in the original "colours"
19:16 < migurski> yeah, kinda-sorta works =)
19:16 < zere> makes me wonder if the degree of freedom you have in colour selection could be used as a side-channel for other information too.
19:16 < migurski> interesting - what do you mean? 
19:17 < zere> well, let's say that you have your array of indexes - you can reorder them in whatever way you like. so let's say you chose the update time of the element and sort the colours from warm to cold by that. then it's easy to overlay a "recent activity" pseudo-heatmap over the map.
19:19 < migurski> ah yeah!
19:19 < migurski> I think Martijn recently did something similar with license change data
19:20 < migurski> but yeah ultimately it's just postgres queries for heatmaps
19:22 < migurski> tmcw: someone over there mentioned converting osm.xml to carto, maybe AJ?
19:22 < migurski> sounds relevant to clickable POIs
19:22 < migurski> via the grid
19:22 < tmcw> sorry, just looke dup
19:22 < zere> yeah. AJ mentioned it at the last EWG meeting
19:23 < zere> there was mention of "combinatorial explosions of tag intersections" :-)
19:23 < migurski> ha
19:23 < migurski> yeah, and using something like imposm to curb that
19:24 < zere> wasn't there a recent-ish patch to mapnik to do layer grouping that would help with that as well?
19:24 < tmcw> okay, read backscroll
19:25 < tmcw> converting osm.xml to carto wouldn't be necessary or really related to utfgrid at all
19:25 < tmcw> and yeah, a straight-up conversion is really gnarly; osm.xml is huge to start off with and very dependent on its queries
19:26 < tmcw> it does 'work' - osm.xml was the proof of concept for carto-generator
19:26 < tmcw> ( https://github.com/rundel/carto-generator )
19:29 < zere> yeah, here it is https://github.com/mapnik/mapnik/wiki/Grouped-Rendering - potentially the OSM style would be much less gnarly with this.
19:30  * migurski reads
19:31 < migurski> That's pretty hot, tmcw. Solves exactly the same issue as High Road.
19:32 < tmcw> Yeah
19:33 < tmcw> brb, lunchtime eastcoasttime
19:37 < zere> cool, maybe this is a good time to close the "official" meeting, but that's no reason to stop this interesting chat :-)
19:37 < zere> thanks to everyone for coming, and see you next week :-)