Attendees
| IRC nick
|
Real name
|
| apmon |
Kai Krueger
|
| pnorman |
Paul Norman
|
| RichardF |
Richard Fairhurst
|
| shaunmcdonald |
Shaun McDonald
|
| TomH |
Tom Hughes
|
| zere |
Matt Amos
|
Summary
- Routing branch
- apmon submitted a PR for i18n support.
- zere contacted the operators of the GraphHopper and MapQuest routing services; both said they were happy with their services being included on the OSM home page, as long as they were given advanced notice of the "go live" date and their attribution requirements were met.
- DennisL, operator of the OSRM routing service, was also contacted, but a reply has not yet been received.
- ACTION apmon to contact CloudMade about their routing service.
- An attempt was made to figure out a sensible target date, but much is still unknown about how long the review & bug-fix process will take.
- There was some discussion about whether requests from the OSM homepage would meet the AUP of OSRM's KIT-hosted instance.
- Still unknown how much RAM, disk space, processing power, etc... the primary options (GraphHopper, OSRM) take.
- ACTION zere to run some benchmarks.
IRC Log
17:30:50 * pnorman waves
17:31:12 <zere> welcome! minutes of the last meeting: http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2014-01-27 please let me know if anything needs changing.
17:31:41 <zere> the only update from last meeting, i think, is that we discussed routing backends, and getting permission, last meeting. but there wasn't an action.
17:32:15 <zere> apmon mentioned it again yesterday, and we asked DennisL (who usually hangs out in #osm-dev), but haven't yet received a reply.
17:32:17 <pnorman> but you went ahead and got permission anyways, right? :)
17:33:09 <zere> i also emailed MQ and graphhopper, both of whom were happy for us to use their backend as long as we let them know ahead of time so that they could be prepared. and follow their respective ToS/attribution requirements.
17:33:24 <zere> pnorman: indeed :-)
17:33:42 <RichardF> zere: wow, fantastic, thank you
17:33:55 <zere> so that leaves CM, and following up with DennisL about OSRM... anyone want to volunteer for that?
17:33:58 <pnorman> Assuming we get an OK from DennisL, what else needs doing?
17:34:10 <RichardF> I have a few little code tweaks I need to do (haven't had any time since last week's meeting)
17:34:19 <RichardF> i18n, obviously
17:34:30 <pnorman> apmon had a pull for that
17:34:47 * RichardF checks
17:34:54 <RichardF> \o/
17:34:58 <RichardF> thanks apmon!
17:35:08 * RichardF has turned off github mails because it sends so fricking many of them
17:35:22 <apmon> I noticed I forgot to pass the language parameter to the nominatim results, but otherwise I think I have covered all the I18n
17:35:55 <RichardF> that's excellent.
17:36:52 <apmon> zere: I could volunteer to contact CM, but I don't know if it is better if someone more "official" does it
17:37:46 <pnorman> well we could assign it as a task to you in this EWG meeting which is probably official enough
17:37:50 <apmon> RichardF: Do you want improvement suggestions?
17:38:02 <apmon> I guess that works
17:38:10 <RichardF> apmon: I'll listen to them but I'm more interested in code :)
17:38:30 <apmon> I'll see if I can get around to some of them...
17:38:53 <apmon> I find the way the drag and drop markers behave slightly unintuitive.
17:39:12 <apmon> The drop where the hand is and not where the point of the marker is.
17:39:23 <pnorman> oh yes I had noted that
17:39:26 <RichardF> apmon: yep, fixing that would be nice for someone with more Leaflet sk1llz than me
17:39:48 <apmon> that is probably not me then... ;-)
17:40:04 <apmon> Although, I think you only need to set an offset somewhere. At least that is how it was in OL
17:40:32 <apmon> Another thing is, it would be good to have to total distance and time noted at the top
17:40:32 <RichardF> yeah, L.icon has an offset property, I'm just not 100% sure how it interacts with drag and drop
17:40:41 <apmon> I could send a patch for that if you want
17:40:42 <RichardF> yep, that's definitely a todo
17:40:51 <zere> apmon: yeah, if you're volunteering here, then you can say "on behalf of OSMF EWG" or whatever, if you feel the need to make it "official". generally, i've not found it necessary, and if it makes you uncomfortable, feel free to CC / send responses back to "engineering@osmfoundation.org"
17:41:24 <apmon> zere: OK, put me down for that then
17:41:31 <apmon> Do we have a rough timeframe?
17:41:41 <zere> #action apmon to contact CM for permission to use backend
17:41:49 <RichardF> I'd like to get the existing todos sorted this week.
17:42:33 <zere> uhh... i don't think so. but if all the major TODOs are small enough that RichardF thinks they can be done in a week, then i guess it should be ready for merge in a couple of weeks, and we could go live the week after that?
17:42:38 <zere> does that sound sensible?
17:42:45 <pnorman> Maybe semi-finalize it by the end of the week and send out an announcement to get more eyes?
17:43:12 <zere> so a "rough estimate date" of the 24th? or more hand-wavy "end of the month"?
17:43:27 <apmon> Nice! I'll see if I can pitch in with things towards the end of the week, if there is still things to do
17:43:27 <RichardF> yeeees. speaking personally, the danger with sending out any announcement is that I don't want to become the OSM community's bikeshedding bitch
17:43:27 <pnorman> I can do an announcement when RichardF says the todos are done
17:43:39 <pnorman> hand-wavy is better
17:43:46 <TomH> errr seriously?
17:43:56 <TomH> I haven't even started a review yet
17:44:02 <TomH> a think a couple of weeks is very optimistic
17:44:24 <TomH> there are clear UI issues before I even start looking at the code itself
17:44:52 <zere> fair enough. if we have a look next week, do you think you'll have a handle on how long / large the review process is going to be?
17:45:54 <TomH> well in terms of UI issues I suspect it will be more a question of how long they take to fix
17:46:00 <apmon> I'll probably wait with contacting CM then until we have a slightly better handle on how long things will take
17:46:10 <TomH> code will depend on how horrid it is when I start looking at it ;-)
17:46:15 <RichardF> the code is lovely.
17:46:18 <RichardF> at least as lovely as Potlatch 1.
17:46:20 <apmon> If we don't get the permission (in time) we can always just drop them from the menu as we have alternatives
17:46:43 <TomH> then we need to consider the operational question of whether we want to go live entirely reliant on third party routing engines
17:47:16 <zere> apmon: yup, there's no rush - partly because i suspect people want to be included.
17:48:03 <zere> TomH: the problem with waiting on that is that it sends us around another cycle of "which routing engine", "what hardware requiremetns", etc...
17:48:23 <apmon> Imho, given we have the permission of the backend maintainers I would suggest to go live without an OSMF backend, as that allows to gain some experience of how to provision the OSMF one.
17:49:05 <TomH> and me some experience of fending off the baying hoards when everything collapses
17:49:08 <TomH> thanks for that
17:49:09 <apmon> That should then allow to make more informed decision of which backend and what hardware is needed.
17:49:39 <zere> why do you think everything would collapse?
17:50:10 <TomH> OSRM goes down? we swamp one or other of the providers with traffic they can't handle?
17:50:11 <apmon> If things struggle with performance, there are a couple things one can do immediately to improve things
17:50:17 <apmon> e.g. turn off draggable routes
17:50:40 <apmon> switch the default to a backend that can handle it better, e.g. likely the mapquest one won't have too much troubles
17:50:56 <zere> if OSRM goes down then we'd be screwed whether it's us hosting it or DennisL / KIT.
17:51:51 <zere> at least with multiple 3rd party backends we can switch the default from one to another relatively easily if they fail.
17:52:02 <TomH> well yes, but in one case fixing it is in our control and in the other it isn't
17:52:30 <TomH> I'll probably give in I guess but it does scare me somewhat
17:52:38 <zere> right, but in one case fixing it is our responsibility, and in the other case it isn't ;-)
17:52:44 <shaunmcdonald> Could we initially require users to be logged in to access the routing so that demand can be gauged and catered for?
17:53:16 <zere> that's going to give a pretty inaccurate gauge of demand, i think.
17:53:37 <RichardF> an OSMF routing box would be nice and I think is certainly something to aim for - presumably an OSRM worldwide car instance. I'd be happy to "maintain" the Lua profile for that, though obviously I couldn't do the box sysadminry. but I don't think it's essential for release 0.
17:54:11 <pnorman> I don't like the idea of holding up jsrouting until we have an OSRM (or other) routing box
17:54:24 <zere> sure, i agree. but, as TomH pointed out, it's not you who'll be getting the complaints *if* it falls over.
17:54:49 <RichardF> we can add a little "experimental" text to it somewhere.
17:54:56 <apmon> I would say it is fairly likely the OSMF instance will either be OSRM or Graphhopper, given that those are the main opensource ones
17:55:05 <zere> hahaha, yeah, because that fig leaf works wonders.
17:55:19 <zere> ^^ (RichardF)
17:55:43 <RichardF> no, it doesn't, but it does say "if this falls over, don't say you weren't warned"
17:55:58 <pnorman> well, if it falls over from load, we're going to have issues, if it falls over because of a random failure, at worst, we can switch the default to something else and pull it from the menu
17:56:09 <zere> well, does someone want to have a look at a worldwide car OSRM instance and come back with numbers on how long it took to import, convert, peak RAM usage, disk space, etc..?
17:56:45 <TomH> RichardF: anyway, let me know when you're ready for me to start looking at it
17:56:51 <RichardF> TomH: absolutely, will do
17:57:03 <apmon> TomH: You mentioned UI issues, what are those?
17:57:06 <pnorman> In a way we've already crossed the bridge on relying on third-party servers, iD relies on taginfo
17:57:42 <apmon> taginfo in iD is a little less prominant than a routing feature on the main page...
17:57:54 <zere> s/taginfo/bing imagery servers/
17:58:24 <TomH> apmon: er well lots of little things really - it's all a bit rough and ready at the moment but I've been assuming that's just prototype
17:59:05 <TomH> though I see the "get directions" link which I had been treating as a temporary hack has got more sophisticated so maybe that is expected to be permanent
17:59:29 <zere> anyway, TomH has some genuine concerns, and this isn't a decision we need to make right now - there's still TODOs on the UI software, and there's still time in parallel to check out whether we can feasibly run a routing server (and UCL doesn't go down / block us / whatever).
18:00:18 <zere> so, again: is someone willing to have a look at the requirements of a worldwide car OSRM import / running instance?
18:00:39 <pnorman> I can ask DennisL, but we might need some idea of load
18:01:06 <RichardF> yep, Dennis would be the person to ask
18:01:29 <zere> at this point, i'm more concerned with the import / CH step - i think they have significant mem / time requirements.
18:01:43 <RichardF> next OSRM release is reputed to have ~30% less memory requirement
18:01:47 <apmon> Without draggable routes, it seems <10 req/s seems a reasonable estimate from the unique visits figure zere mentioned the other day
18:02:28 <apmon> From the documentation of Graphhopper, their CH step supposedly can be done on a 32GB machine
18:02:30 <zere> yeah, assuming ~1 route/UV. which i have no data, or even theory, to back up ;-)
18:02:58 <apmon> how unique is a unique visitor?
18:03:21 <zere> per day. i don't know exactly how piwik does it, but i assume it's via cookies.
18:03:23 <apmon> I.e. is it unique per session, per day, per month...
18:04:28 <pnorman> well it's definately >32GB RAM for OSRM
18:04:35 <apmon> if per day, then I still think from an "introspective perspective" that 10 routes per UV on average seems reasonable
18:05:34 <apmon> https://github.com/graphhopper/graphhopper/wiki/World-Wide-Road-Network
18:05:49 <apmon> are the numbers I have seen for graphhopper
18:05:59 <pnorman> The OSRM demo server has 128GB RAM (https://github.com/DennisOSRM/Project-OSRM/wiki/Demo-server)
18:06:16 <RichardF> apmon: jsrouting is currently telling me to "Turn leeeeft onto Eureka Road". don't know if that's an i18n issue but if so it's a very silly one
18:06:16 <apmon> and given it is written in Java, it is surprising it would have less memory requirements than OSRM
18:06:36 <apmon> Oh, damm, I forgot to undo that
18:06:52 <apmon> That was for debuging purposes to make sure it is going through the I18n code
18:06:55 <pnorman> apmon: it's the algorithms used. My understanding is that what OSRM does is very fast, but very memory intensive for the graph building
18:06:56 * RichardF laughs
18:07:02 <apmon> can you quickly fix that... ;-)
18:07:02 * zere greps the patch for "swim across the Atlantic ocean"
18:07:16 <RichardF> apmon: not right now, haven't got my OSM instance running
18:07:29 <apmon> you just need to fix the en.yml
18:07:31 <zere> pnorman: actually, OSRM and graphhopper are both CH-based, so the algorithms should be very similar.
18:07:49 <TomH> "drive to greenham common; strap yourself to ICBM; set target and initiate launch"
18:08:19 <RichardF> apmon: aha
18:08:31 <zere> oh, that's what all those people were doing there in the 80s? must have been cheap - they all looked quite bedraggled
18:08:50 <apmon> graphhopper does support other algorithms like A*, but it also supports CH, which is what you would want in this case
18:09:21 <RichardF> apmon: fixed
18:09:42 <zere> the only thing i can think of for why graphhopper would have lower memory requirements is that it produces a smaller graph from OSM in the first place. but given that the intermediate formats are different, it might be hard to track down
18:10:12 <apmon> RichardF: Thanks. I had put my self a not to make sure I don't forget and then I did anyway :-(
18:10:20 * RichardF knows the feeling
18:11:03 <apmon> One reason might be that I am not sure if graphhopper actually supports turn restrictions.
18:11:14 <RichardF> ideal for a cycling router then!
18:11:26 <apmon> There seem to have been a bunch of commits regarding that recently, but I don't know the state of things
18:11:37 <apmon> and if that did change the memory requirements drastically
18:12:39 <apmon> So it still seems that using OSRM for the default car mode is best and then potentially using the others for the other modes
18:13:14 <apmon> unless I'll have to leave now
18:13:33 <apmon> scratch that unless, I do have to leave
18:15:10 <zere> weird... i would have thought that supporting turn restrictions would, if anything, add to the memory load.
18:15:22 <zere> oh, wait
18:15:25 <pnorman> re: osrm, I think we meet https://github.com/DennisOSRM/Project-OSRM/wiki/Api-usage-policy
18:16:00 <zere> i should read things properly before replying to them... apmon was saying they might *not* support restrictions...
18:16:19 <pnorman> RichardF: are you implementing coordinate hinting for OSRM?
18:16:34 <apmon> Do we send a osm specific user agent, or just the standard browser one?
18:16:54 <zere> yup, i think we do, and i think DennisL would be happy to be on osm.org, but we should check anyway.
18:17:29 <zere> we can't override the browser UA, can we? even if so, then the referer will be osm.org, so would be identifying.
18:18:06 <shaunmcdonald> zere: remember the some browsers drop the referrer
18:18:20 <shaunmcdonald> or certain proxies block the referrer.
18:19:13 <RichardF> pnorman: will be doing so but that code isn't finished yet (have a look at osrm_car.js if you like, there are bits of it in there)
18:20:31 <zere> shaunmcdonald: urgh, really? none of the major browsers do that, do they?
18:21:24 <shaunmcdonald> zere: not by default, but can be turned on.
18:21:35 <apmon> I think I have disabled referrer for privacy reasons. But I do it through a local squid proxy, so I doubt many people do it this way
18:23:04 <zere> any idea what proportion of people? i mean, DennisL can block no-referrer requests if it's a big deal. but if we can't control the requests that browsers are making, then it makes lift very difficult
18:23:26 <zere> not just for us, but also for anyone who's trying to follow our tile usage policy ;-)
18:23:45 <TomH> FF has network.http.sendRefererHeader
18:23:55 <TomH> though I don't think it is exposed in the UI these days
18:24:27 <pnorman> I really doubt many people are disabling the referrer in such a way. Maybe an ad-blocking extension might hit more people, but very few people use those sorts of things
18:24:32 <shaunmcdonald> There's this extension for chrome https://chrome.google.com/webstore/detail/referer-control/hnkcfpcejkafcihlgbojoidoihckciin?hl=en
18:25:16 <shaunmcdonald> Could send an extra parameter in the requests to say where it's coming from?
18:26:36 <zere> if it's only a few people who've set up their own squid firewalls, or added a chrome extension, or fiddled with their internal firefox settings, then i don't think it's a big deal
18:26:54 <pnorman> OSRM requires at least 55GB of free RAM
18:27:02 <zere> of course, that's for DennisL / other backend providers to decide, but i don't think we need to worry about it right now
18:27:21 <zere> pnorman: that's what it says on the web page... but it's dated last year.
18:27:46 <zere> and the planet has got bigger since then, and apparently OSRM has got more efficient.
18:28:00 <RichardF> I don't think it's got more efficient _yet_, but next release, apparently
18:28:15 <zere> we really need someone to run it, but in the absence of volunteers, i guess i'll see what i can do.
18:28:33 <zere> #action zere to run OSRM worldwide car import and report resource requirements
18:28:44 <pnorman> I'm asking him
18:28:59 <zere> so... we can continue talking about routing next week. was there anything else anyone wanted to talk about this week?
18:30:39 <pnorman> working on the https://github.com/openstreetmap/osm2pgsql/compare/9fc4fd49789067951bb7c309756310588fae96c4...a5755430392af3459d06ede91e9498b10bd18ae1#diff-8647d42f728e3cd81a88ef0c2b3151ecL878 issue now
18:31:37 <zere> pnorman: cool :-)
18:32:59 <zere> ok. thanks, everyone, for coming. hope to see you next week for more routing fun :-)