Attendees
| IRC nick
|
Real name
|
| apmon |
Kai Krueger
|
| BigJimTheClown, SteveC |
Steve Coast
|
| migurski |
Michal Migurski
|
| RichardF |
Richard Fairhurst
|
| TomH |
Tom Hughes
|
| zere |
Matt Amos
|
Summary
- zere put out a call for help / collaborators to finish writing a unit test suite / example algorithm for license change see github.
- apmon has a demo of Gosmore running on his laptop
- Scaling this up to planet-size implies around 15GB resident RAM usage.
- Large (1000km) route takes around 10sec, on a laptop. We expect server-grade hardware to do better.
- apmon will try a larger example (Europe?) on the dev server.
- TomH deployed rails 3.2 the previous morning.
- Some discussion of coding standard / style conventions, following a proposed README and the Rails coding conventions.
IRC Log
18:01 < zere> minutes of the last meeting http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-02-13
18:02 < zere> please let me know if there are any problems or omissions in there.
18:05 * RichardF wasn't here, but looks ok to me...
18:05 < zere> arising from past minutes, i'd like to point out https://github.com/zerebubuth/openstreetmap-license-change - in a sense, it's an evolution of the gist from last week.
18:07 < zere> but the algorithm was getting very hard to verify by eye, and more difficult for non-coders to understand. instead, i thought it would be more understandable to talk in concrete terms, so the code there has a number of tests which should be pretty readable
18:07 < RichardF> excellent"
18:07 < zere> and the code then arises simply in a TDD fashion from meeting the requirements in the tests
18:08 -!- BigJimTheClown [~836b0073@shenron.openstreetmap.org] has quit [Quit: CGI:IRC (Ping timeout)]
18:08 < zere> i would very much like any help that anyone is willing to give on this - be it writing tests, code, or even test-cases in plain english.
18:09 < zere> it should be a relatively simple pipeline from plain english comment/tests -> code tests -> working code.
18:09 < RichardF> I'm on deadline this week, but remind me next week and I'll take a look :)
18:09 < zere> for some values of "relatively simple", of course.
18:12 < zere> the draft agenda has us talking about routing some more - apmon, is there anything you'd like to discuss?
18:13 < apmon> I have a demo backend of gosmore running on my laptop
18:13 < zere> awesome!
18:13 -!- BigJimTheClown [~836b0073@shenron.openstreetmap.org] has joined #osm-ewg
18:13 < apmon> It can handle Germany routing fairly well (well for some definition of fairly well)
18:14 < zere> i know it's a small installation, but what are the resource requirements like?
18:14 < apmon> In order to test larger extracts I would need to use something more powerful than my laptop though
18:14 < apmon> The resulting .pac file is about 900Mb in size
18:14 < apmon> so slightly smaller than the .pbf file
18:14 < zere> does gosmore mmap() that when it's serving routes?
18:14 < apmon> I think it relies on the disk cache
18:15 < zere> oh, it fseek()s around in the fiel?
18:15 < apmon> I haven't found that part of the code yet.
18:15 < zere> i'm just trying to figure out if it needs enough memory to hold the whole .pac file in RAM at once
18:15 < apmon> But looking at the memory requirements in top, my guess would be yes, it fseeks() around the file
18:16 < apmon> The whole planet would be roughly 15Gb would be my guess
18:16 -!- BigJimTheClown [~836b0073@shenron.openstreetmap.org] has quit []
18:16 < zere> obviously, that would be the best for performance, but it would rule out doing any sort of testing where the machine size wasn't quite big enough.
18:16 -!- BigJimTheClown [~836b0073@shenron.openstreetmap.org] has joined #osm-ewg
18:16 < apmon> Given that it doesn't mmap the file. It could work with less RAM, but performance would quickly degrade
18:16 < TomH> mmap doesn't require you can hold it all in RAM
18:17 < TomH> the advantage of mmap would be sharing between multiple process
18:17 < apmon> but it would show up as virtual ram in top
18:17 < zere> yes, i know. i was hoping it was mmap() or fseek(), not malloc() + read() ;-)
18:18 < apmon> I'll try and understand the code to get a better idea of what it actually does (and if there are optimisation potential)
18:18 < zere> either way, if the whole world is likely to be ~15GB then it might be worth testing on the dev machine. if you want to try it out, just ping me and i'll stop OWL updates so that the machine isn't so loaded.
18:18 < apmon> A route across Germany (1000km) took about 10 sec on my laptop (once cached)
18:19 < apmon> Yes, a 48Gb machine should work fairly well from what I have seen
18:20 < apmon> zere: Actually, I think it does a bunch of malloc as well. The per route RAM requirement does go up significantly for longer routes as well
18:20 < zere> i guess while that's hardly ideal, it's likely to be maybe 2-3x faster on server hardware (larger caches, faster ram)... and that's certainly not terrible in terms of performance.
18:20 < apmon> Yes, on a 16 Core server, you would still be able to do a 1000km route per second
18:21 < zere> yeah, it's just dijkstra, so the memory requirement should be O(n^2) where n is (roughly) the route length
18:21 < apmon> which would probably be sufficient for our purposes
18:21 < apmon> zere: Do you know if it is at least A*?
18:21 < zere> i'd expect that most routes would be short, though...
18:22 < apmon> Short routes are usually calculated in a few 100ms
18:22 < zere> i think it's simple dijkstra with no extra heuristics, but it's been a long time since i read the code and i didn't fully understand it even then...
18:23 < RichardF> we could potentially refuse routes where start and end are more than n km apart (by Pythagoras)
18:23 < zere> i know Melaskia has been doing some work with/using/in it recently, so she would know better than i (and nroets, of course)
18:23 < apmon> RichardF: Yes, that would be a possibility. With a suggestion to use either the mapquest or cloudmade engine if you really need these long routes
18:24 < RichardF> yep.
18:24 < apmon> Although a 2000km route through Afrika or Siberia would probably be fine...
18:24 < RichardF> or any other routing engines that may become available ;)
18:24 < zere> i think we'd have to have it in production a while before we could figure out exactly what types of limits, and what values, were effective.
18:25 < apmon> The server code also sets ulimits on CPU and RAM usage, so it doesn't overload things
18:26 < apmon> Without better statistics of what kind of usage it will see, it is difficult to evaluate it further.
18:26 < zere> yes, exactly.
18:26 < apmon> My guess would be that gosmore is likely on the verge of feasible. So the balance might tipp either way.
18:27 < apmon> I'll try and compile a europe routing file on the dev server and see if that changes things in any way.
18:29 < zere> excellent :-)
18:30 < zere> that was the only other thing i had on the agenda - so does anyone else have anything they'd like to discuss?
18:30 < BigJimTheClown> yeah
18:30 -!- BigJimTheClown is now known as SteveC
18:30 < SteveC> thats better
18:31 < SteveC> Andy's email about the 50 groups, OWG, EWG, sysadmins etc - worth discussing something that you guys would be happier with?
18:31 < SteveC> I think all that fell out of the meeting in london 1+ years ago
18:34 < apmon> Do you have a link to the email?
18:35 < SteveC> not here unfortunately, I think it was to owg/board, but the issue has come up repeatedly
18:36 < apmon> can you describe the issues a bit?
18:36 < zere> the issue, i assume, is the confusion that some people have over which groups have which roles.
18:36 < SteveC> there are a number of groups, perhaps too many, with overlapping and not entirely clear goals. Thus if you want to communicate, you don't know which group or which method to use
18:37 < zere> is it possible that http://www.osmfoundation.org/wiki/Working_Groups simply needs to be improved?
18:37 < SteveC> so we could try and figure it out bottom up here, ignore it, or something else
18:37 -!- migurski [~migurski@dsl081-049-227.sfo1.dsl.speakeasy.net] has joined #osm-ewg
18:38 < SteveC> zere: yeah so perhaps sysadmins becomes part of OWG or something, I dunno
18:38 < zere> i'm not really sure that EWG is the place to discuss issues that span all working groups
18:39 < SteveC> what would be the right place? MT?
18:39 < zere> or that now is the time to start heavily introspecting on the whole issue of roles, responsibilities, lines of command, bureaucracy, etc...
18:39 < apmon> I thought OWG is the new sysadmins? Just a rename
18:39 < TomH> OWG is a rename of TWG
18:40 < TomH> but [OT]WG != the sysadmins
18:40 < zere> you said, "if you want to communicate", so maybe it could be that CWG is the appropriate place to do this?
18:41 < SteveC> ok, I give up, it's your show after all
18:42 < SteveC> Tom - any chance you can reply to the various emails you have from the board?
18:42 < TomH> yes Steve I plan to do so tonight
18:43 < SteveC> thx
18:43 < zere> that's not a terribly helpful thing to say. this is an EWG meeting, and works best then we stick to engineering-related items. if anything, discussing that sort of thing is not only for the wider community / MT, but also making the problem of overlapping and unclear remits worse...
18:43 < SteveC> you said there was nothing else to discuss :-)
18:43 < zere> sure, i guess as long as we're filling dead air ;-)
18:43 < apmon> Something else to mention: It seems there are a number of different options for vector tiles
18:44 < RichardF> for the format or the delivery?
18:44 < apmon> The delivery
18:44 < zere> but discussing OWG/sysadmins here would be a bit like trying to organise a SOTM venue in LWG... ;-)
18:44 < apmon> iandees mentioned http://tile.osm.osuosl.org/tiles/osmus_vector_point/13/1972/2948.geojson which runs with tilestach
18:44 < TomH> zere: do we want to get feedback on the new README?
18:45 -!- SteveC [~836b0073@shenron.openstreetmap.org] has quit [Quit: CGI:IRC]
18:45 < TomH> I did fork it and make some changes on my version
18:45 -!- SteveC [~836b0073@shenron.openstreetmap.org] has joined #osm-ewg
18:45 < zere> anyway, sorry, that's a digression on top of a digression... back to the topic :-)
18:45 < SteveC> zere: that's clearly nonsense given the big overlap in the groups
18:45 < apmon> I have also made some progress on a geojson backend for tirex ( https://github.com/apmon/tirex )
18:46 < RichardF> excellent.
18:46 < TomH> oh rails 3.2 was deployed yesterday morning BTW
18:46 < zere> TomH: you mean https://gist.github.com/1817927 ? yes - i think it would be worth soliciting comments on that.
18:47 < zere> ^^ the background for this (several meetings ago) was to make it clearer how to contribute to the rails_port and what is generally considered to be best practices in terms of code quality, etc... before submitting.
18:47 < zere> apmon: that's great! good work :-)
18:48 < apmon> TomH: Are there any good tutorials on best practices in rails? It might be useful to link to them if there are any good ones
18:49 < TomH> no idea - shaun might have some ideas
18:49 < zere> there's this: http://guides.rubyonrails.org/contributing_to_ruby_on_rails.html
18:49 < apmon> E.g. I usually approach things more in a C/Java way. Which perhaps is not always the best thing with rails
18:49 < TomH> though I suspect there will be multiple conflicting ideas of wha is best practice ;-)
18:50 < zere> how much we would want to follow rails' own guidelines, i don't know...
18:51 < TomH> http://guides.rubyonrails.org/ in general may be a good pointer per apmon's suggestion
18:52 < zere> hmm... there's also nothing in the gist about code style, a la http://guides.rubyonrails.org/contributing_to_ruby_on_rails.html#follow-the-coding-conventions
18:52 < zere> do we want to add that sort of thing?
18:52 < zere> of course, the more we add, the more it's likely to clash with the code that's already there ;-)
18:53 < apmon> Still, having some conventions would be good
18:54 < apmon> If you work on code that doesn't meet the conventions, you might end up updating it so that it does.
18:54 < zere> yes, that's the way we've handled changes to code style conventions in teams i've worked on before.
18:54 < TomH> would be good to advice on formatijng
18:54 < TomH> no tabs (yes RichardF that means you)
18:55 < TomH> try and follow the same style as the file you are editing
18:55 < TomH> unless you're going to reformat it of course
18:56 < zere> TomH: do you want to add something to that effect in the file? shall we try to iterate towards something and review it again next week?
18:57 < TomH> yeah ok I'll try and add something
18:57 < zere> thanks :-)
18:58 < zere> i notice we're coming up on the hour - does anyone have any comments, announcements, or other stuff quickly?
18:59 < SteveC> im curious if any of the stats went up significantly due to the observer
18:59 < TomH> don't think so
18:59 < SteveC> or, which of the 3,000 stats pages I should look at to see for myself :-)
18:59 < TomH> but I'll have a quick look
19:00 < zere> is there a rough time of day for the observer release?
19:00 < zere> i guess it went out in print as well as online, so probably not....
19:00 < RichardF> yesterday morning UK time
19:01 < TomH> nothing showing on piwik that I can see
19:02 < SteveC> can I log in to piwik?
19:02 < TomH> 663 referals from guardian.co.uk yesterday by the looks of it
19:02 < TomH> SteveC: I'll have to create you an acount - hang on a sec
19:03 < SteveC> thx
19:05 < SteveC> ha, piwik has a funnel feature
19:06 < TomH> it's a plugin, but it's borked at the moment
19:07 < zere> there's a small spike around the 16th-22nd jan, but so far the stats for yesterday look pretty typical.
19:09 < zere> 869 referrals from guardian.co.uk today - maybe it's a slow-burner?
19:09 < SteveC> bang!
19:10 < RichardF> nah, it's not the sort of feature that's going to lead to vast traffic. it's more a positioning/brand-building thing, that's why we did it.
19:10 < RichardF> offline->online conversions are usually pretty minimal, I wouldn't expect much from the paper publication.
19:11 < zere> and with that, before this becomes a CWG meeting, thank you all for coming! hope to see you next week :-)
