Working Group Minutes/EWG 2011-10-17

From OpenStreetMap Foundation


IRC nick Real name
apmon Kai Krueger
pnorman Paul Norman
RichardF Richard Fairhurst
TomH Tom Hughes
zere Matt Amos


  • TomH's rails3 branch is almost ready to go - just needs some changes to Osmosis.
  • apmon started a thread on talk-de about development. Main items discussed:
    • Better documentation would be good.
    • VMs may not be as useful as having good installation documents.
    • More client-side libraries for interacting with OSM would be good.
      • apmon volunteered to start a wiki page for client library information.
    • There is significant interest in having a German hack weekend.
    • A list of small projects ("low-hanging fruit") would be useful.
    • Reporting "success stories" of people tackling bugs would provide more motivation.
  • RichardF reported his experiences trying to choose a database schema from the many available to serve his needs.
    • Osmosis has comprehensive, but not necessarily easy-to-follow, documentation. It would be better to have a "recipes" doc for common uses.
    • zere volunteered to write a page summarising the database schemas available and their potential uses.
    • pnorman volunteered to start a wiki page on Osmosis recipes.


18:00 < zere> last meeting minutes
18:01 < zere> if there are any problems with it, or corrections needed, please let me know
18:03 < TomH> looks fine to me
18:04 < apmon> hi
18:05 < zere> ok, so items from the last meeting
18:05 < zere> anyone tried TomH's rails3 branch?
18:05 < zere> for the second week, i have to admit to shame and failure.
18:05 < TomH> a couple of people forked it
18:06 < apmon> TomH: do you have a time frame in which you plan to deploy it? Or is it not that far yet?
18:06 < TomH> well I'd like to do it as soon as possible
18:06 < TomH> though I do need to talk to brett about osmosis
18:08 < apmon> I haven't so far, but I will see if I can try it out this week
18:08 < apmon> Then I can also see how much porting work it is if or how the rails3 effects the OSB branch
18:09 < apmon> With respect to my task, I have gotten around to sending an email to talk-de regarding ideas where the EWG could help ( )
18:09 < TomH> well one reason I want it is that I ran into a dead end where what I wanted to do on that branch would be easier with tails 4
18:10 < zere> apmon: great, thanks :-)
18:10 < zere> TomH: rails4? is that coming soon?
18:10 < TomH> typo
18:11 < apmon> I got quite a bit of response to the thread, some of which insightful, and unsurprisingly a lot off topic
18:12 < apmon> Lets see if I can get together some of the key points. Most of them are views of individuals, so it is hard to judge if they represent the majority view
18:12 < apmon> In no particular order:
18:13 < zere> yeah, that's a pretty hefty thread. key points (on-topic ;-) would be useful...
18:13 < apmon> Documentation is useful and worth the effort putting more into it
18:14 < apmon> I don't know if people are aware of the current rails_port docu, as I thought it was not too bad in its current state
18:15 < apmon> VMs might not be the best way. Providing easy installation instructions and pre bundeled packages might be easier. One can still set up a VM locally then and install the packages
18:15 < apmon> Not rails_port related, but general ideas of OSM development. Having more libraries to do some of the core parts (and make people aware of the current ones)
18:16 < zere> client-side libraries?
18:16 < apmon> There is some interest in Germany having hack weekends where people can come and learn more about how to develop
18:16 < apmon> Too be honest, I am not entirely sure what kind of libraries.
18:17 < apmon> Partly client libraries like rendering or routing engines, but also libraries to make it easier to parse and process osm xml / pbf
18:17 < apmon> So far I have not engaged in the thread, as I wanted to just observe what ideas people come up with.
18:18 < apmon> The next stage would be to try and figure out in more detail what and how the useful ideas would help
18:18 < pnorman> I know that I used a python library which made .osm  xml parsing a  lot easier
18:18 < zere> well, woodpeck isn't here, but i'll see if i can persuade him to run a hack weekend.

18:18 < apmon> Jochen was suggesting I would do it, which might be a little difficult given I am in Boulder at the moment ;-)
18:19 < apmon> Other useful ideas were:
18:19 < zere> is there a page on the wiki which collects the libraries (however many there are) for client programming?
18:20 < apmon> Having a list of "most wanted features" and "low hanging fruit", where potential new hackers can find ideas with which to get their feet wet
18:21 < apmon> Another aspect was motivation. Have a few "success stories" of how someone was bothered by a bug / feature request, went and implemented and got it deployed shortly after and how the community thought it was a great thing
18:21 < apmon> I think those are roughly the key ideas I can remember from that thread
18:22 < zere> the same thing as we used to do with the lolcat of awesomeness?
18:22 < pnorman> zere: no wiki  list that i can find
18:22 < apmon> Oh, and Frederik helpfully suggested rails development is very hard and those people who need hand holding through installation and documentation want produce useful patches anyway so no point in trying to cater to them
18:22 < apmon> s/want/ won't/
18:23 < RichardF> I'm not sure how many client libraries are actively developed. I started using osmlib (Ruby) and put in a couple of feature requests... Jochen has now asked me if I want to maintain it ;)
18:23 < zere> hmm... ok. so maybe that's the first step then - collect all known libraries for doing OSM xml/pbf parsing and other useful stuff together on one page?
18:23 < RichardF> which given that I know precisely 0.0 about Ruby...
18:23 < apmon> Oh, yes that was another issue that came up. No one knows who is maintaing what and where to send patches or requests or discussion
18:24 < apmon> zere: Yes, collecting available developer resources would be a great start imho
18:24 < apmon> I think we do have a bunch, but they are somewhat unknown and difficult to find
18:26 < zere> there's a fine balance between trying to make it easier for motivated people to get started and suggesting that everyone is given free lessons in rails / OSM development all the time.
18:27 < zere> i think we agreed at a previous meeting that there simply isn't enough manpower to do 1:1 programming and hand-holding.
18:27 < zere> but that's not to say that we can't still massively lower the barrier to entry for a motivated individual.
18:27 < RichardF> absolutely.
18:28 < RichardF> just an example - I'm currently working on a project that involves feeding a planet extract into a database, and querying it. I actually can't even begin to find a canonical answer as to what sort of database I should use for it
18:30 < zere> there's a law of diminishing returns for any educational attempt. but i don't think that means we shouldn't try. just need to find the point at which the returns aren't worth it any more.
18:30 < zere> and we're far away from that at the moment (imho)
18:31 < zere> RichardF: that's because the sort of database and the schema are pretty much dependently driven by the queries you're going to be asking. ;-)
18:31 < RichardF> zere: oh, sure. but I can't find any reference as to "this sort of db is what you need for this sort of query" :)
18:32 < apmon> zere: Given that at the moment we don't have too many people lining up to learn how to code for the rails_port, I am not sure I am too worried about getting swamped
18:33 < zere> apmon: yes, quite.
18:33 < zere> RichardF: what sort of queries are you doing?

18:33 < apmon> RichardF: Don't say we are lacking good documentation... ;-)
18:33 < RichardF> zere: map calls on a GB planet extract.
18:34 < apmon> Install the rails_port and cgi_map?
18:34 < apmon> jxapi db would be another option
18:34 < zere> use osmosis to create an apidb database (diff-updatable, in case you care) and run cgimap.
18:35 < zere> or, as apmon just said, run jxapi (again, osmosis, but a different schema this time)
18:35 < RichardF> apmon: that's a lot of overhead - I'd like it to be something I can run on a simple vm. Have been playing with the postgis snapshot schema but it has a load of stuff I don't really need, and makes it a bit too big for my vm.
18:36 < RichardF> thank you for the suggestions, will investigate...
18:36 < apmon> OSM has too much data. Makes it very resource intense to work with it :-S
18:36 < RichardF> I'm sure the GB extract can be done in less than 12Gb... just not sure how :(
18:37 < pnorman> a page that says apidb is good for this but has these downsides, pgsnapshot is good for this, etc would be useful
18:37 < zere> and they all have the downside that OSM data is really big ;-)
18:38 < apmon> brings up another point, do we have good tools to filter OSM data?
18:38 < zere> does anyone want to go and write either 1) the wiki page for the client / OSM parsing / interaction libraries, and/or 2) the comparison of different DB schemas?
18:38 < apmon> E.g. get rid of all the landuse and building data, if one doesn't need it?
18:39 < zere> yeah, isn't there an osmosis plugin for that?
18:39 < pnorman> it can be done in osmosis, but i've found jxapi + josm w/ gigs of ram was easier than figuring out osmosis
18:43 < zere> hmm... the osmosis docs need to be better too, then?
18:43 < pnorman> yes.
18:43 < pnorman> they are accurate but hard to understand
18:43 < zere> is what's needed more an "osmosis recipes" page?
18:44 < pnorman> yes
18:45 < apmon> Isn't there a osmosis gui by now? Osmembrane or something like that?
18:45 < zere> ok. so 3 pages to write: 1) client/osm libraries, 2) DB schemas, 3) osmosis recipes.
18:45 < zere> any volunteers for the above ^^^
18:45 < zere> anyone?
18:45 < apmon>
18:46 < zere> that's pretty cool!
18:46 < apmon> I can start a page gathering the different libraries
18:46 < zere> apmon: thanks! :-)
18:47 < pnorman> i can take on 3, although not an ewg member
18:47 < pnorman> toebee wrote a blog post about 2
18:47 < zere> pnorman: thanks :-)
18:47 < zere> and don't worry about the "membership" thing. i don't think anyone's a formal member of EWG yet.
18:48 < apmon> There isn't even really a reason atm for it to be a formal working group, rather than a bunch of people sitting together
18:48 < zere> i'll see if toebee minds me borrowing liberally from his blog post for #2 then.

18:48 < zere> apmon: my thoughts exactly.
18:51 < zere> last 10 minutes. anyone have anything they'd like to discuss?
18:53 < apmon> I have to go now. I don't have any more to discuss for now
18:53 < RichardF> no suggestions here
18:53 -!- apmon [] has quit [Quit: apmon]
18:55 < zere> ok. thanks to everyone for coming & hope to see you next week :-)