Working Group Minutes/EWG 2012-02-06

Attendees

IRC nick	Real name
apmon	Kai Krueger
migurski	Michal Migurski
pnorman	Paul Norman
RichardF	Richard Fairhurst
TomH	Tom Hughes
zere	Matt Amos

Summary

• Discussion of license change:
  • Current tools available (OSMI, cleanmap) seem to be adequate for nodes/ways, but many relation types are not visualised.
  • Having a coded-up version of "the rules" would be very helpful - zere to draft.
  • Releasing a cleaned planet file would also be useful.
• Routing front-end:
  • Current version: [1].
  • Would benefit from clearer coding guidelines being released.
  • Geocoder front-end (in rails port) would need some work to detect "A to B" type queries and engage routing mode rather than going to the geocoders.
  • Attempts to quantify the number of requests per second expected to the routing engine were unsuccessful, a guess is between 1 and 10 requests per second.
• pnorman is generating coastlines with the coastline checker, and has some documentation patches.
  • Might get this up on dev in the future.

IRC Log

18:04 < RichardF> evening!
18:05 < zere> minutes of the last meeting http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-01-30 for your please of reviewing. any mistakes or problems, please shout out.
18:05 < zere> the draft agenda for today is to discuss license change goals, routing, and anything else that may come up.
18:07 < apmon> sounds good
18:07 < RichardF>       /me nods
18:07  * pnorman nodes
18:07  * zere relations
18:07 < RichardF> no way!
18:07 -!- migurski [~migurski@173-164-158-185-SFBA.hfc.comcastbusiness.net] has joined #osm-ewg
18:08 < zere> sorry, i meant /me BANS POTLATCH ;-)
18:08 < RichardF>  harumph
18:08 < RichardF> ok, which first?
18:08 < zere> license change - that's likely to be quick, right?
18:09 < TomH> yeah - a few years should do
18:09 < RichardF> go for it...
18:11 < zere> we have the wonderful OSMI and cleanmap as tools, so the question is - do we need/want more tools? if not, what is the next step and how might EWG help out?
18:12 < apmon> how close are OSMI and particularly cleanmap to an eventual algorithm?
18:12 < apmon> Do they take the full history into account?
18:12 < RichardF> I think the only tool/visualisation currently missing is something to illustrate relation damage.
18:13 < RichardF> OSMI seems close to the algorithm; there are a few edge cases but nothing too worrying.
18:13 < RichardF> I suspect we're moving onto the stage now where the challenge is actually coding the changeover.
18:13 < pnorman> visualizing multipolygons should be easy once you can derive if they're tainted or not, not sure how you can visualize other relations
18:15 < zere> yes, a turn restriction relation which is "damaged" by having a tainted "via" node might be a hard thing to show on the map... is anyone showing these kinds of relations (other than route & multipolygon) on maps anywhere/
18:15 < RichardF> CM had something but I think it was only ever experimental.
18:16 < apmon> What other major relations are there? And who uses them?
18:16 < pnorman> Well, for relations that are tainted by tags and not their children, I expect that most will tend to have no previous versions
18:16 < apmon> Routing apps I guess mostly use the restriction relations
18:16 < pnorman> routes, multipolygons, turn restrictions. rendering, rendering, routing
18:17 < zere> i guess that begs the question of whether these relations are widespread in use and (not to be unfair, but) whether they're worth the time to figure out how to visualise them.
18:17 < pnorman> admin boundaries, but those are like multipolygons. they're also a mess
18:17 < migurski> I've been using route relations a lot lately, mostly the ones created by NE2
18:17 < migurski> had kind of an ah-ha moment and now they seem indispensable
18:17 < migurski> terrible timing.
18:18 < apmon> Can KothicJS render route relations?
18:18 < pnorman> taginfo reports: multipolygon, route, site, boundary, restriction, broad_leafed (not relations most likely), associatedStreet
18:18 < apmon> Perhaps we can set up something like a KothicJS render a la cleanmap. That way various different styles can be used
18:18 < RichardF> if we can think of a way of showing problematic relations then Frederik is, I think, willing to add it to OSMI.
18:18 < RichardF> (he asked for suggestions in a recent posting to rebuild@)
18:19 < pnorman> associatedStreet and site are the two I didn't list previously, and not using either, I'm not sure how to visualize them
18:19 < RichardF> but yes, as zere said, we probably need to concentrate on the big three - routes, multi
polygons, turn restrictions.
18:20 < apmon> Are multipolygons handled by cleanmap? I would guess yes, as they are part of the normal rendering rules. 
18:20 < zere> according to what pnorman found on taginfo, looks like turn restrictions are at number 5. so maybe not that important to show at this stage?
18:21 < apmon> My guess would be routing is likely going to be somewhat of a "nightmare" anyway due to connectivity interruptions
18:21 < pnorman> I don't see turn restrictions as particularly important to preserve. If they're tainted, it's likely irrevocably so
18:21 < zere> is the problem holding up development of these tools that "the rules" aren't well-known or well-defined?
18:21 < apmon> So connectivity is probably going to more of an issue than turn restrictions
18:22 < zere> yes, i strongly suspect connectivity will be a big issue.
18:23 < apmon> One thing one can perhaps at least do though, is just have a count (and percentage) of how many relations of each type are affected
18:23 < RichardF> zere: I think pretty much everyone's settled on the OSMI rules. I've made a suggestion on relations which seems to have gone down well (about assuming a clean, untagged, empty v0) and that seems to be pretty much it.
18:23 < pnorman> I see routes as being particularly difficult since they tend to be large high-revision relations
18:23 < migurski> is it possible to put those rules into code somehow and release a planet file that reflects their application?
18:24 < zere> RichardF: can we "code-ify" (or codify) them a little more? at the moment (afaik) they're all written in a very imprecise language (english) and spread across multiple emails.
18:25 < RichardF> zere: absolutely. do you want me to encourage that via the rebuild@ list?
18:25 < pnorman> If we can get rules into psudo-code, give them to the LWG to be vetted, and have certainty it would help. 
18:26 < zere> pnorman: exactly that. and something that covers all cases - ready to be implemented in tools.
18:26 < apmon> That would be very helpful and reduce a lot of uncertainty
18:26 < zere> i suspect such code must already exist inside OSMI and cleanmap, but i don't know to what extent it's possible to isolate it.
18:27 < zere> RichardF: perhaps... i could put together something based on my understanding and i'll stick it up as a gist, then we can collaboratively hack on it?
18:27 < RichardF> zere: excellent idea.
18:27 < apmon> Having clear pseudo code for lawyers and everyone else to read would be useful though even if it already exists in code
18:28 < zere> excellent - didn't i say this would be really quick? ;-)
18:30 < apmon> Moving on to the next point then?
18:31 < zere> assuming we had pseduo-code for the "taintedness" of a particular item, is there anything else that anyone wants to discuss with the license change?
18:31 < zere> if there's anything else that's unclear - we can offer to help LWG out with this stuff.
18:31 < RichardF> or AWG
18:33 < RichardF> routing next?
18:34 < apmon> OK
18:34 < zere> last week apmon reported some progress on routing. how goes it? anything to discuss and help out with?
18:36 < apmon> No further progress on the frontend side since last week. But imho, I think the current version would mostly be ok as a first pass ( apmon.dev.openstreetmap.org/routing/ )
18:36 < TomH> the code is horrendous as I recall
18:36 < apmon> SteveC had suggested to add a functionality of typing "London to Birmingham" into the search box
18:36 < TomH> from the brief look I had at it while moving it to rails 3
18:19 < RichardF> but yes, as zere said, we probably need to concentrate on the big three - routes, multi
polygons, turn restrictions.
18:20 < apmon> Are multipolygons handled by cleanmap? I would guess yes, as they are part of the normal rendering rules. 
18:20 < zere> according to what pnorman found on taginfo, looks like turn restrictions are at number 5. so maybe not that important to show at this stage?
18:21 < apmon> My guess would be routing is likely going to be somewhat of a "nightmare" anyway due to connectivity interruptions
18:21 < pnorman> I don't see turn restrictions as particularly important to preserve. If they're tainted, it's likely irrevocably so
18:21 < zere> is the problem holding up development of these tools that "the rules" aren't well-known or well-defined?
18:21 < apmon> So connectivity is probably going to more of an issue than turn restrictions
18:22 < zere> yes, i strongly suspect connectivity will be a big issue.
18:23 < apmon> One thing one can perhaps at least do though, is just have a count (and percentage) of how many relations of each type are affected
18:23 < RichardF> zere: I think pretty much everyone's settled on the OSMI rules. I've made a suggestion on relations which seems to have gone down well (about assuming a clean, untagged, empty v0) and that seems to be pretty much it.
18:23 < pnorman> I see routes as being particularly difficult since they tend to be large high-revision relations
18:23 < migurski> is it possible to put those rules into code somehow and release a planet file that reflects their application?
18:24 < zere> RichardF: can we "code-ify" (or codify) them a little more? at the moment (afaik) they're all written in a very imprecise language (english) and spread across multiple emails.
18:25 < RichardF> zere: absolutely. do you want me to encourage that via the rebuild@ list?
18:25 < pnorman> If we can get rules into psudo-code, give them to the LWG to be vetted, and have certainty it would help. 
18:26 < zere> pnorman: exactly that. and something that covers all cases - ready to be implemented in tools.
18:26 < apmon> That would be very helpful and reduce a lot of uncertainty
18:26 < zere> i suspect such code must already exist inside OSMI and cleanmap, but i don't know to what extent it's possible to isolate it.
18:27 < zere> RichardF: perhaps... i could put together something based on my understanding and i'll stick it up as a gist, then we can collaboratively hack on it?
18:27 < RichardF> zere: excellent idea.
18:27 < apmon> Having clear pseudo code for lawyers and everyone else to read would be useful though even if it already exists in code
18:28 < zere> excellent - didn't i say this would be really quick? ;-)
18:30 < apmon> Moving on to the next point then?
18:31 < zere> assuming we had pseduo-code for the "taintedness" of a particular item, is there anything else that anyone wants to discuss with the license change?
18:31 < zere> if there's anything else that's unclear - we can offer to help LWG out with this stuff.
18:31 < RichardF> or AWG
18:33 < RichardF> routing next?
18:34 < apmon> OK
18:34 < zere> last week apmon reported some progress on routing. how goes it? anything to discuss and help out with?
18:36 < apmon> No further progress on the frontend side since last week. But imho, I think the current version would mostly be ok as a first pass ( apmon.dev.openstreetmap.org/routing/ )
18:36 < TomH> the code is horrendous as I recall
18:36 < apmon> SteveC had suggested to add a functionality of typing "London to Birmingham" into the search box
18:36 < TomH> from the brief look I had at it while moving it to rails 3
18:37 < apmon> It is pretty short, so it shouldn't be too difficult to clean up
18:37 < apmon> If someone tells me what clean rails code should look like I can do (parts of) it
18:37 < TomH> really? my memory is of pages of javascript that was all pointlessly embedded in erb tempates and things
18:37 < RichardF> could we throw it at rails-dev as a project? many hands make light work etc.
18:38 < TomH> but like I say I haven't looked at it seriously
18:38 < RichardF> esp. if we could say "these are our coding standards, so this is what we'd like to see cleared up before it goes live"
18:38 < TomH> I just remember getting a bit of a sinking feeling as I saw what the code was like
18:38 < zere> argh, yes. thanks RichardF for reminding me - i should get that done too.
18:38 < RichardF> TomH: oh come come, you can't be that horrified by anything in a world where amf_controller still exists
18:39  * RichardF giggles at the little "Calculating route" animations
18:39 < TomH> RichardF: yes, but would it get added now ;-)
18:40 < RichardF> apmon: that's pretty nice. don't think I'll let you get away with adding another tab ;) but apart from that, the panel works really well
18:40 < apmon> RichardF: If you have better suggestions...
18:40 < TomH> oh it looks fine, it's the code behind it that's nasty
18:40 < pnorman> I had a few UI comments that I'll compile
18:40 < RichardF> apmon: well - the stuff I was working on for the redesign branch was to boost the search box
18:41 < apmon> So basically what SteveC was suggesting?
18:41 < TomH> apmon: your routing2 branch is the current state of the art right?
18:41 < RichardF> don't know
18:42 < apmon> TomH: Yes it is.
18:42 < zere> i'll see what twain47 thinks about how difficult adding detection of "A to B"-type queries is...
18:42 < pnorman> apmon: OSRM silently fails for me
18:42 < TomH> apmon: I'll put that up on the newly rebuilt http://apis.dev.openstreetmap.org/ then
18:43 < apmon> TomH: Thanks
18:43 < TomH> zere: no need for him to be involvee
18:43 < RichardF> to be honest, I think you all you need to do is replace "Where am I?" (which no-one uses) with "Get directions", and have that open up the panel rather than the tab
18:43 < TomH> zere: the geocoder controller parse queries and decides where to send them
18:43 < apmon> pnorman: The demo backend of OSRM isn't very reliable
18:43 < RichardF> UI-wise, the tabs should really be reserved for views on the current bbox
18:43 < pnorman> apmon: ah
18:43 < zere> TomH: ah, ok. i didn't know that.
18:44 < apmon> but better error messages are on the todo list
18:44 < pnorman> is there support for routing via a third point or dragging on the route to modify?
18:44 < apmon> No support of via points as of yet. You can drag the end point on the map though
18:45 < apmon> With respect to the routing backend, what kind of load would people expect it has to handle?
18:45 < RichardF> doubt we'll know that until we try it!
18:45 < pnorman> how do we quantify routing load? is queries/second the best way?
18:45 < apmon> My initial guess would be in the range of 1 - 10 req / sec
18:46 < apmon> Distance will also play a significant role depending on the backend
18:46 < apmon> RichardF: My guess was derived from the statistics I saw on nominatim a while ago
18:47 < zere> yes. in "real life" people mostly make very short routes, but i think that with OSM they'd be using it more for debugging, which could involve some very long routes.
18:47 < apmon> But looking at the numbers on poldi now, I am wondering if it might be higher
18:47 < zere> or, at least, what we did at CM for debugging involved some very long routes ;-)
18:47 < pnorman> I think nominatim was lower usage than it could be since it's been so slow and out of date for some time
18:48 < apmon> Debugging the US interstate network is a little bit of a special case
18:48 < apmon> imho debugging mostly involves fairly short routes to identify at which intersection the problem begins...
18:49 < apmon> but again, it is hard to come up with good numbers of what it will be like.
18:49 < zere> i dunno about that - there'll be plenty of people interested in the connectivity (and monitoring the connectivity) of their area. or figuring out how complete the routable road network in Africa or Siberia is...
18:49 < pnorman> maybe track req/sec and km (either routed, or straight-line distance) / sec
18:50 < apmon> I can try and write a little java program to do load testing on a routing backend.
18:50 < apmon> And then tune the values
18:51 < apmon> With regard to OSRM, I sent Dennis an email last week. His estimate of resource requirement for world wide routing was 192Gb of ram and 500 Gb of SSD
18:51 < apmon> That is a pretty heavy resource requirement!
18:51 < pnorman> that's a lot of ram
18:51 < TomH> maybe it would be best to say that the backend (at least any OSM run one) is not for direct use and is only to support the web site?
18:52 < pnorman> I like the multiple backends the demo has
18:52 < apmon> TomH: Yes, I would suggest that as an initial policy. If it turns out it can handle a lot more, one can relax it
18:53 < apmon> Given the high resource requirements, I would like to evalutate gosmore a bit more seriously.
18:53  * TomH grits teeth
18:53 < apmon> The resource requirements nic gave were on the order of 16Gb of RAM and disk space
18:54 < zere> well, come on. if gosmore is way less resource-intensive than OSRM then it's worth considering.
18:54 < apmon> Which would be much more manageable.
18:54 < zere> indeed.
18:54 < TomH> yes but he has completely failed to demonstrate that it is workable on errol
18:54 < apmon> Well, on errol, pretty much everything is slow
18:54 < zere> it was taking more than 16GB ram on errol, though, wasn't it?
18:54 < TomH> errol has 48Gb or rRAM
18:54 < apmon> Loading the rails front page can sometimes take nearly a minute
18:54 < zere> although, iirc, he was running several in parallel or something like that.
18:55 < apmon> on errol
18:55 < TomH> and he was killing it every night by creating dozens of multi Gb processes
18:55 < zere> apologies - that's likely OWL.
18:55 < TomH> mostly it's just the idea of having to deal with Nic though...
18:55 < pnorman> oh, errol appears to of disappeared off of munin, maybe it's munin-node has stopped responding?
18:56 < TomH> he manages to rub me up the wrong way
18:56 < apmon> zere: which is fine. It just makes it difficult to do any benchmarking if the server is overloaded to start with
18:56 < zere> apmon: yes, i'm working to get it off there asap, but there have been some hiccups (aren't there always?)
18:56 < apmon> yes there are ;-)
18:57 < zere> if you want to have some time dedicated for testing, i can shut down OWL for a few days if that would help?
18:57 < TomH> pnorman: looks fine to me
18:57 < TomH> http://munin.openstreetmap.org/openstreetmap/errol.openstreetmap/index.html
18:57 < pnorman> TomH: Do you see it on the main munin page?
18:57 < TomH> pnorman: what is the main munin page?
18:57 < pnorman> TomH: check the date on those
18:57 < zere> TomH: timesttamp on those images is feb 2nd...
18:57 < pnorman> http://munin.openstreetmap.org/
18:58 < TomH> hmm
18:58 < TomH> munin-node is running
18:58 < TomH> and it hasn't been complaining
18:58 < pnorman> If a server stops responding and no graphs are manually defined in munin.conf it drops off of the pages generated by munin-html
18:58 < apmon> zere: I guess I will get it installed locally on my laptop first to figure out the processes involved.
18:58 < apmon> Once that is done, we can see about what causes the most load on errol and if we can shut those down for a little while
18:59 < zere> i think melaskia has been doing some work with gosmore recently - she might be able to help, if you need any.
18:59  * TomH adds munin to his hate list
18:59 < TomH> oh wait, it's already there
18:59 < pnorman> Can we get the munin_update plugin enabled maybe? That tells you how long hosts take
18:59 < TomH> pnorman: ha ha ha
18:59 < apmon> zere: Ah, great
18:59 < TomH> that's simple (as anybody who suffers from all the munin email knows...) too long!
19:00 < TomH> because munin's shit
19:00 < pnorman> Ya, all the fork()'ing sucks
19:00 < TomH> I was more thinking the lack of parallelism 
19:00 < TomH> so one tiny thing wrong blocks everything and causes the world to go tits up
19:01 < pnorman> there are settings that can help that - I run munin locally and have written some plugins, anything you want me to take a look at?
19:01 < zere> ok. might be time for anything else anyone wants to discuss? if not, let me know and i'll get it on the agenda for next week.
19:02 < RichardF> might be good at some point to discuss possibility of supplying vector tiles
19:02 < RichardF> which I think Simon Poole had suggested as "this would be a good way of fulfilling the 'community map' role once performed by t@h"
19:02 < RichardF> but no great urgency on that
19:03 < apmon> RichardF: Adding the KothicJS backend stuff to tirex is also still on my todo list...
19:03 < RichardF> :)
19:03 < zere> yeah, sure. we can agenda that for next week. i figure it's a pretty interesting topic ;-)
19:03 < apmon> That should (assuming it works out) make it fairly easy to provide vector tiles
19:04 < pnorman> coastlines: just sent out a message, I have the generation working
19:04 < zere> pnorman: excellent!
19:04  * apmon starts too many things and then doesn't finish any of them... :-S
19:04 < pnorman> I might add my script to svn if I can figure out svn
19:04 < RichardF> apmon: you and the rest of us!
19:05 < apmon> Yes, which brings me to the point of that we might want to try and priorotize the top ten tasks
19:05 < zere> pnorman: ping me if you need any help. are you running this on the dev server?
19:05 < pnorman> zere: I'm running the generation locally and uploading to the dev server
19:05 < apmon> to make sure some of them actually get completed, rather than all of them begun
19:06 < zere> pnorman: cool. i'd love to help out if you want to try running them regularly on dev and getting the coastline-checker up & running again.
19:07 < pnorman> I've found the coastline generation process is badly documented and it's not clear who handles them
19:07 < zere> i think the answer is; nobody does it particularly regularly anymore. at least, as far as i'm aware.
19:09 < pnorman> coastlines are never going to be instant feedback with the current system, but imo weekly or better should be the target
19:09 < apmon> The ones on tile are from the 23rd of January
19:10 < pnorman> that's more recent than I thought
19:10 < apmon> but I think before that the last one was at the beginning of December
19:10 < apmon> So nearly only once every two months
19:14 < pnorman> Maybe the best option is to use either my shapefiles or migurski's?
19:16 < pnorman> TomH: email best for requesting svn access? I want to supply some documentation patches to coastcheck
19:17 < zere> i think the key is to get the old coastline-checker error display up, then people can judge for themselves what the quality of a particular file might be.
19:17 < TomH> pnorman: yes please
19:18 < apmon> I need to go now, bye
19:18 < RichardF>  bye!
19:18 -!- apmon [~kai@ntct41-92-dhcp.resnet.colorado.edu] has left #osm-ewg []
19:18 < pnorman> zere: I actually found only about 25 errors last run world-wide