Working Group Minutes/EWG 2013-06-03
Attendees
| IRC nick | Real name |
|---|---|
| apmon_ | Kai Krueger |
| Firefishy | Grant Slater |
| gravitystorm | Andy Allan |
| pnorman | Paul Norman |
| zere | Matt Amos |
Summary
- rails_port README
- gravitystorm has been making more progress.
- Carto style
- Firefishy has installed hardware: 80GB RAM, 2x 512GB SSD, 4x 450GB 15kRPM SAS.
- Should be ready for software setup and testing.
- To begin with, gravitystorm is maintainer of new style. Prefers patches via pull requests.
- OWL
- Development has slowed since the sole developer has less time. Needs to be made easier to install - there are few docs at the moment.
- "there were a couple of people who seemed interested in contributing but I think they got stuck at the setup stage and/or understanding the code"
- ACTION: zere to look at writing install instructions for OWL
- Routing
- apmon_ wants to look at getting this branch merged in, but it will need some modernisation.
- Discuss at hack-day post SOTM-US.
IRC Log
17:04:27 <zere> minutes of the last meeting: http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2013-05-20 17:04:52 <zere> and apologies about last week - between illness and the national holiday here i clean forgot. sorry. 17:05:14 <zere> gravitystorm: anything new on the rails_port README stuff? 17:05:34 <gravitystorm> Nope, but I am again working on it during the meeting 17:05:57 <gravitystorm> I've had an offer from RichardF to write OSX instructions when the base ones have been finished 17:08:17 * pnorman is grabbing a drink 17:08:46 <zere> excellent. where by "base", you mean "linux"? ;-) 17:11:55 <gravitystorm> Ubuntu 12.04 LTS, if you want to be specific, but I'm aiming for "any supported Ubuntu". We'll need to port the instructions to the various linuxen too. 17:12:51 <zere> yeah, that sounds about right 17:14:20 <zere> much of it is surely common, though? there's the package management bits, but then the rest is basically postgres + ruby setup. 17:14:50 <zere> assuming user-installed gems, all the ruby stuff should be the same across platforms. 17:15:06 <gravitystorm> yes, it's the prerequists that are the gnarly bits 17:15:23 <gravitystorm> On Unix-like systems as root do: 17:15:23 <gravitystorm> # rails server 17:15:35 <gravitystorm> ^^^ another nasty bit from the current docs... 17:16:23 <zere> as root? gah... 17:17:31 <gravitystorm> indeed 17:18:17 <zere> Firefishy: what's the word on orm? it's got the new disks + ram, right? 17:18:47 <Firefishy> 80GB ram installed. 2x 512GB SSD installed (raid1), checking quick on tile disks... 17:19:34 <Firefishy> 4x 450GB 15kRPM SAS (raid?) 17:19:55 <Firefishy> The disks I will firm up once testing has been done. 17:21:08 <zere> cool. and is the plan to set stuff up on there with the carto style? 17:23:24 <Firefishy> I will raid up the 4x 450GB disks later tonight. Then over to gravitystorm to remove the tile.osm.org dependency and then I think TomH can finish up the chef script. 17:23:45 <gravitystorm> Firefishy: I'll remove the dependency tomorrow morning 17:24:11 <apmon_> Firefishy: Raid 10 on tile disks? 17:24:12 <zere> nice. it sounds like we're pretty close to having and editable style ;-) 17:24:41 <pnorman> RAID10 would give us 900GB of cache, is that enough? 17:24:48 <zere> (i had an "again" on the end of my last line, then i realised it probably hasn't ever really been editable without great XMLintestinal fortitude) 17:25:11 <apmon_> It used to be editable to steve8 17:25:28 <Firefishy> 3 Phases: 1) Testing under direct orm.osm.org/ url, 2) point Australia or other small cache to orm 3) move all to orm. 17:25:45 <gravitystorm> Firefishy: sounds like a good plan 17:26:09 <Firefishy> pnorman: no. I need to get better disks. The 4x 450GB disks are for testing. I'll add more or replace. 17:26:16 <apmon_> Firefishy: For warm up, can you set it up that all rendering requests that go to yevaud also go to orm? 17:26:24 <apmon_> That would also give a good speed comparison 17:26:26 <zere> apmon_: yup. and kudos to him for being able to handle that many <>s 17:26:54 <apmon_> Do we have a contribution model to the new carto style? 17:27:02 <zere> it wouldn't be a fair comparison until the caches warm up on orm, surely? 17:27:19 <Firefishy> apmon_: My plan is slowly flick over caches, building up load. 17:27:21 <apmon_> I.e. what is the social model of who gets commit rights to the style? 17:27:26 <gravitystorm> apmon_: sure, via pull requests that I'm handling at the moment 17:27:37 <apmon_> gravitystorm: So you will be the new maintainer? 17:27:43 <apmon_> OK, sounds good 17:27:57 <apmon_> zere: Why wouldn't it be a fair comparison? 17:28:20 <apmon_> I.e. you basically do what pnorman did for speed testing, just in a continous way 17:28:35 <apmon_> i.e. feed yevauds renderd log into orm's renderd 17:29:11 <gravitystorm> apmon_: yes, I'd be the maintainer in the short term. I'm happy to handle pull requests, although I'd prefer more make-this-easy or make-this-better requests, rather than just add-another-very-rare-poi type thing 17:29:31 <pnorman> do we have an easy to get that done? also, do we know how we're going to share caches between the two servers yet? 17:29:35 <zere> because orm would be rendering some requests which yevaud wouldn't be, and that would mean yevaud would have a smaller subset of the postgres data in memory. i'm not saying it's a completely invalid comparison, just that it's another variable on top of different hardware to consider. 17:30:07 <apmon_> not if you feed in yevaud's renderd logs. Then they render exactly the same amount 17:31:32 <pnorman> gravitystorm: probably a good idea to classify the github issues as 1.0 vs 2.0 etc, and maybe time to start versioning 17:32:31 <gravitystorm> I can start assigning them to milestones - there's already versioning in place. We're on v2.2.0 already 17:32:49 <zere> apmon_: ok, that works for benchmarking, assuming that the disk load of serving tiles isn't too onerous for yevaud. 17:33:25 <pnorman> gravitystorm: I don't see any branches or tags 17:33:38 <apmon_> I think the tile disks are pretty much maxed out on yevaud. But they are completely separate to the db disks 17:33:52 <apmon_> the write load on the tile disks, should hopefully not be a factor 17:33:55 <gravitystorm> pnorman: https://github.com/gravitystorm/openstreetmap-carto/tags 17:34:06 <pnorman> ah, there it is 17:34:08 <apmon_> as the writes should be cached in the OS buffer cache 17:35:34 <pnorman> back to the maintainer issue, who's going to do it long term? do we need to recruit someone for that? 17:36:10 <zere> sure. sounds like it could work, but there are so many variables we're not controlling for, it would be hard to pick out the cause of any particular reggression. but maybe we only care about the overall anyway. 17:36:38 * Firefishy needs to pop out. Back in 15mins 17:36:49 <apmon_> On a top level, yes we would first of all care about the overall performance 17:37:07 <apmon_> only if that is much worse than expected, would one need more detailed analysis of why 17:37:56 <apmon_> and hopefully it should be fairly representative of what to expect in production 17:38:12 <zere> cool. it seems that's all going to happen pretty soon now. that's great :-) 17:38:35 <apmon_> :-) 17:38:41 <gravitystorm> pnorman: I have no plans to stop doing it, if that's what you mean, but I'm happy to hand it over to other people if they are doing a better job. Of course, if 'teh community' doesn't like me as maintainer, it's dead easy to fork it and move on :-) 17:38:53 <zere> i don't think i have anything extra from the last meeting. was there anything else anyone wanted to discuss? 17:38:58 <gravitystorm> pnorman: i.e., we should treat it like any other software project. 17:39:10 <pnorman> ya 17:39:38 <pnorman> Routing - how are we doing on that? I looked at apmon's branch and it needs refactoring against leaflet at the very least 17:39:53 <gravitystorm> zere: briefly, I think pawel's OWL stuff might need some getting-started TLC, as per a recent conversation on dev. I'll be looking at that eventually but others are free to jump in of course. 17:41:30 <zere> pnorman: sure, i think there was some discussion in a previous EWG meeting, but that might well all be out of date. i'm sure apmon_ would welcome any patches. 17:41:36 <apmon_> gravitystorm: They can fork it all they like, but that won't make the changes appear on osm.org tiles 17:41:49 <apmon_> so who is maintainer of osm.orgs version does matter 17:42:30 <zere> well... perhaps not - "osm.org's version" can be switched without needing the maintainer's permission. i think that's what gravitystorm is saying. 17:42:35 <apmon_> Would SotM-US be a good time to all get together and plan how to proceed with routing? 17:42:37 <gravitystorm> zere: indeed 17:43:15 <zere> so if gravitystorm stops maintaining it, or doesn't want to do it any more, then someone can fork it and we can discuss when that happens. 17:43:43 <zere> otherwise, i say let's try it out and see how it goes. 17:44:22 <apmon_> Yes, as long as we have someone in the short to medium term (which with gravitystorm we do) that should be fine. 17:44:45 <zere> apmon_: yep. i won't be there, but i think most other people will be. iirc, the only problem with the branch is that it's out of date and there was lots of copy & paste that TomH thought should be cleaned up. 17:44:57 <apmon_> With respect to routing, I suspect there are two main issues. 17:45:01 <apmon_> 1) Design a good UI 17:45:21 <pnorman> this will bring us up to 3/4 of the osm.org styles maintained by andy. not that i'm objecting to his cartography and I know we have a lack of cartographers to maintain styles 17:45:29 <apmon_> 2) How strongly should it be server based, or how much should be done in js? 17:45:44 <apmon_> I.e. should all requests go through the rails port, or should the client contact the routing APIs directly 17:47:19 <zere> pnorman: sure, we have a lack of contributors in general, and maintainers in particular. hopefully gravitystorm will be able to find someone with the drive and skill to hand over to. 17:47:39 <zere> (* and free time) 17:48:19 <apmon_> The original routing UI by nic roets did most of the thing client side in JS. 17:48:31 <zere> apmon_: i kinda lean towards doing it all in JS... just to reduce load on the servers. but there's probably a disadvantage to that that i'm not seeing right now. 17:48:36 <apmon_> Back then TomH strongly objected to that and wanted things to go through the rails_port 17:49:01 <apmon_> but more recently, with the editing notes on the browse page, TomH seems to be leaning towards client side again 17:49:13 <zere> i guess we'll have to find out from him what his objection was. 17:49:27 <gravitystorm> probably something for rails-dev@ then 17:49:37 <apmon_> Yes, I also lean towards client side 17:49:41 <zere> all of the JS can be dynamically loaded, right? so it's not a bloating issue? 17:50:31 <zere> apmon_, gravitystorm, pnorman: maybe one of you wants to volunteer to ask this question on dev@, so you can get some ground-work laid for discussion at SOTM-US + hackday? 17:52:16 <zere> ... and suddenly there's silence ;-) 17:52:21 <pnorman> javascript and rails are two languages I don't really know 17:53:26 <apmon_> I can post the question on dev 17:53:45 <apmon_> I am trying to see if I can dig up the previous discussion on it 17:54:15 <zere> pnorman: there's no time like the present to learn ;-) and i'm sure there's plenty of other jobs which need doing. translations, perhaps, since it's a major bit of functionality... 17:54:52 <zere> gravitystorm: re OWL, what's the summary of that discussion on dev@? 17:56:07 <gravitystorm> zere: summary is: pawel isn't doing more development at the moment, and nobody else has started 17:56:10 <gravitystorm> "there were a couple of people who seemed interested in contributing but I think they got stuck at the setup stage and/or understanding the code" 17:56:47 <gravitystorm> that suggests an approach for EWG to help get developers, or at least, make sure there's nothing stopping developers - by improving the getting-started experience for developers 17:56:57 <apmon_> Well pnorman is probably busy rewriting the API for his summer of code project? 17:57:08 <zere> ok. i have to admit, after i handed over the repo to pawel, i stopped following what was going on. 17:57:22 <zere> so probably the only one who has a full idea of it is pawel himself 17:57:31 <apmon_> Do you why he isn't doing more development? 17:57:42 <zere> having said that, i should probably at least try to reaquaint myself with it. 17:57:58 <pnorman> right now I'm collecting some data/benchmarking some pgsnapshot options, haven't really gotten into the coding yet. 17:58:09 <pnorman> this does bring up another issue - cgimap testcases 17:58:33 <zere> pnorman: what's the issue with the testcases? 17:59:05 <gravitystorm> apmon_: I think it's just general lack of time, since his ML post sounds like he still wants to finish it (i.e. it's not the case that it's abandoned for technical reasons) 17:59:19 <pnorman> well, I'm wondering if we need some way to compare cgimap to rails port for the non-map? calls 17:59:26 <gravitystorm> http://lists.openstreetmap.org/pipermail/dev/2013-May/026978.html && http://lists.openstreetmap.org/pipermail/dev/2013-May/026998.html for the record 17:59:40 <zere> pnorman: apmon_ wrote something exactly for that. 18:00:31 <zere> at the last count (a very, very long time ago), the behaviour for the map call was the same. but i've only tested other stuff by hand. 18:00:54 <pnorman> also, I'm wondering how well suited the existing ruby test framework is for what I'm doing, mainly with the DB loading (https://github.com/zerebubuth/openstreetmap-cgimap/blob/master/test/test_functions.rb#L71) 18:00:55 <zere> ideally, there'd be some easy way to compare, but also some "offline" tests which didn't need a running rails_port. 18:01:45 <zere> pnorman: sure, the existing test "framework" was something i threw together over a weekend. i'd be very glad to see it improved! 18:02:05 <zere> if you want to replace it entirely with something else - that's totally ok by me. 18:02:16 <zere> as long as i can still "make check", then i'm happy enough 18:02:52 <apmon_> pnorman: https://trac.openstreetmap.org/browser/subversion/applications/utils/OsmMapCallValidator 18:03:41 <zere> gravitystorm: so, first step w.r.t OWL is probably install instructions, do you think? 18:03:56 <pnorman> I think it works for apidb, and overall makes sense because the api is in ruby. the problem is that pgsnapshot+cgimap doesn't involve any ruby but involves postgis 18:03:58 <apmon_> It did a statistical sampling based comparison between two map call implementations 18:04:16 <gravitystorm> zere: I suspect so, but that's just going on hunch. I haven't looked at it yet and won't get around to it before I've finished the rails port docs. 18:04:55 <pnorman> for comparing two API endpoints my thought was to diff them with osmosis, because you can have two differing XML files that are the same OSM data 18:05:40 <apmon_> That is roughly what the OsmMapCallValidator did (although it did its own diff and didn't use osmosis_ 18:06:24 <pnorman> for ogr2osm tests I passed the XML through xmllint --format but when I changed XML libraries I had to re-do the results files to change " to ' (or the other way) 18:06:53 <zere> well, i don't want the only testing framework in cgimap to require rails_port. it doesn't at the moment, and could be pretty easily extended to pgsnapshot (it just uses the pg gem). 18:07:44 <zere> #action zere to look at writing install instructions for OWL 18:07:59 <pnorman> it's the data loading that I'm concerned about, although I suppose requiring osmosis if you're building and testing with a pgsnapshot backend is reasonable 18:10:03 <zere> sure. another potential is loading the test cases with osmosis and pg_dump them, then just load the dumps for each test. 18:10:14 <zere> but whatever works ;-) 18:10:36 <zere> did anyone have anything else, quickly? 18:11:55 <pnorman> anon notes - does anyone know when that was discussed? I'm trying to remember if it was okay to add data to the map solely based on an anon note, and where the documentation of that is 18:12:47 <zere> iirc, we figured it wasn't OK, as there was no CT-agreement from an anon user. 18:12:59 <zere> should be in previous minutes somewhere.... 18:13:07 <gravitystorm> zere: nothing from me 18:13:16 <pnorman> that's what I recalled but I'm damned if I can find it 18:13:22 <apmon_> Isn't it up to the person who adds data from the notes if it is OK? 18:13:38 <zere> pnorman: http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-11-19 ? 18:13:42 <pnorman> but I know that plenty of people are adding information solely from anon notes 18:14:45 <pnorman> the warning is "This note includes comments from anonymous users which should be independently verified." which people are interpreting as a comment about being sure that it's accurate 18:15:22 <zere> same thing, really. if you verify data then you've independently collected it? 18:15:26 <apmon_> Isn't that a valid interpretation? 18:16:07 <gravitystorm> I'm not sure this is an EWG topic any more :-) 18:16:07 <zere> or, if you haven't independently collected it, then you can't be sure it's accurate? 18:16:07 <apmon_> A different question regarding notes is, should there be categories? 18:16:16 <apmon_> and other filtering features? 18:16:41 <apmon_> And should there be an equivalent of created_by tag? 18:16:52 <pnorman> probably not ewg, but we did intend to look at anon notes post-release 18:17:30 <zere> hmmm... as long as it can be done without confusing anyone wanting to put a note in. the point was to lower the bar for contributing, i'd hate to see it raised any more than it is already. 18:17:51 <zere> perhaps a stack overflow like tags system which could be added/edited? 18:17:59 <apmon_> Is that refering to categories or created_by? 18:18:11 <zere> i can feel myself moving upwards in the atmosphere. 18:18:19 <zere> perhaps a discussion for next time? 18:18:30 <apmon_> When is the next time? 18:18:59 <zere> next monday? 18:19:01 <zere> oh 18:19:02 <zere> right 18:19:18 <zere> that's the hack day post SOTM-US, is it? 18:19:24 <pnorman> yes 18:19:41 <zere> i guess the 17th, then 18:22:21 <gravitystorm> sounds good to me 18:22:43 <pnorman> okay. see many of you on monday, and i'm off for breakfast and then to work on my presentation... because I'm leaving in 2 days 18:23:00 <zere> awesome. thank you all for coming, and see you on the 17th. have fun in SF!