Working Group Minutes/EWG 2013-05-13

From OpenStreetMap Foundation

Attendees

IRC nick Real name
apmon Kai Krueger
gravitystorm Andy Allan
pnorman Paul Norman
tmcw Tom MacWright
TomH Tom Hughes
zere Matt Amos

Summary

  • rails_port README
    • gravitystorm showed an edited version of the README, with much verbiage moved out to other files.
    • ACTION gravitystorm continue working on rails port documentation
  • Carto benchmarking
    • pnorman presented the results of some of the benchmarking he's been doing.
    • ACTION pnorman to check how EC2 osm2pgsql ebs snapshots used to be done

IRC Log

17:03:04 <zere> minutes of the last meeting: http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2013-05-06
17:03:11 <zere> #topic actions
17:03:26 <zere> now we get to ask gravitystorm how the README stuff is going :-)
17:03:44 <gravitystorm> git push
17:03:44 <gravitystorm> ;-)
17:04:28 <gravitystorm> Probably best if we come back to that in a few minutes
17:05:16 <zere> the demo effect...
17:05:32 <zere> ok, i had a couple of actions, and i have to say that i didn't do either of them. apologies - and i'll try harder this week.
17:05:46 <pnorman> i actually have progress on benchmarking.
17:06:06 <zere> apmon isn't around, so i guess we'll catch up with him next time re: translatewiki.
17:06:13 <zere> pnorman: please go ahead :-)
17:06:54 <pnorman> gravitystorm's instructions work, but carto + millstone + npm is... interesting
17:08:13 <pnorman> so it'd be nice if there was an easier way of getting carto + millstone running for those of us who aren't using tilemill or for those of us who use a desktop machine for editing and a server with the databases+mapnik+etc
17:08:44 <apmon_> Just for the record, I also so far failed to get carto working (Ubuntu 13.04)
17:09:14 <apmon_> I think it is due to a screwed up nodejs environment, that I think happened in an attempt to get tilemill working from the PPA
17:09:23 <TomH> clearly my node fu is very strong then
17:09:44 <zere> interesting... so the issue really is with npm?
17:09:56 <apmon_> In my case yes
17:09:59 <pnorman> yes, and I think me and apmon were managing to get different errors
17:10:34 <apmon_> Mine first failed in opening an SSL connection to github for some reason
17:10:45 <zere> hmmm... i'm wondering if it makes sense to port carto so something with a working package manager, then? or is that too much effort?
17:10:55 <zere> s/port carto/port the carto compiler/
17:11:00 <gravitystorm> zere: I'd say no chance on that
17:11:00 <apmon_> and then after taking TomH's suggestion of deleting some node files, it gave up completely on installing carto... ;-)
17:11:23 <pnorman> I know ubuntu has a carto package but it's really outdated
17:11:35 <TomH> fedora has one now!
17:11:42 <TomH> (guess who packaged it ;-))
17:12:02 <apmon_> In response to asking debian gis to package tilemill, they said they wouldn't due to broken nodejs packaging
17:12:35 <gravitystorm> zere: my reasoning being that it's a moving target, and 99.5% of the development is done by nodejs developers (i.e. MapBox). So I wouldn't want to try developing e.g. carto-ruby or something like that.
17:12:50 <pnorman> i can confidently say that after i reinstall I'm never letting npm anywhere near my hardware and will only ever touch it in a vm
17:14:06 <gravitystorm> pnorman: I felt the same at first, but now that there are working ubuntu packages for nodejs + npm, it's fine with me. And I avoid any 'global' installs with npm so any damage is per-project :-)
17:14:15 <apmon_> gravitystorm: Could you always check in a compiled version in addition to the carto source?
17:14:28 <pnorman> apmon_: not an option, it bakes in the shapefile paths
17:14:40 <apmon_> which shapefile paths?
17:14:45 <pnorman> coastlines, etc
17:15:04 <apmon_> they are backed into carto as well, aren't they?
17:15:13 <tmcw> zere: you're welcome to port carto, g'luck :)
17:15:34 <gravitystorm> apmon_: no. The relative paths in the project.mml are expanded to full paths when compiled, IIRC
17:15:38 <tmcw> there's a half-finished c++ port you can follow-up on called carto-parser
17:15:43 <pnorman> no - they're relative in carto, they become absolute in the XML
17:16:33 <gravitystorm> anyway, I'm happy to work on anything that makes it easier to compile. I suspect that a lot of the hurt is that teh internets are full of brain-damaged instructions on how to install nodejs + npm on ubuntu
17:16:35 <apmon_> Well, as mentioned on osm-dev the other day db name, table prefix and all the other things are baked in as well
17:16:42 <pnorman> perhaps part of this could be eased with better install instructions?
17:17:07 <apmon_> so if you do one more find and replace doesn't matter too much
17:17:33 <gravitystorm> apmon_: the lack of configurability on dbname etc is something I hope to fix, not to use as an excuse for baking in more stuff :-)
17:19:44 <apmon_> :-)
17:19:44 <gravitystorm> tmcw: do you know if there is any movement towards cascadenik-style datasource templates within the project.mml file? Even if the parameters are baked in, it was nice to have them only written once rather than per-layer repitition.
17:19:44 <tmcw> isn't there an issue open for this?
17:21:01 <gravitystorm> https://github.com/mapnik/Cascadenik/wiki/Managing-Data-Sources#datasource-templates is how cascadenik did it
17:21:46 <gravitystorm> tmcw: not that I see (with a brief check in the carto issues list)
17:22:47 <zere> gravitystorm: https://github.com/mapbox/carto/issues/189 ?
17:23:57 <gravitystorm> zere: no, that's not it really. There could be connection templates defined in json, so it's orthogonal to any XML issues
17:24:22 <gravitystorm> pnorman: with your shiny new working-carto setup, have you managed any benchmarking?
17:25:01 <tmcw> huh coulda sworn there was a ticked for named datasource support. there should be.
17:25:33 <pnorman> I have some results, none very profound.
17:25:57 <pnorman> lua is slower than non-lua, by 60% in the PBF reading phase on nodes.
17:26:21 <pnorman> warm ram caches matter hugely.
17:26:40 <pnorman> 9k tiles is enough to push stuff out of cache with 34GB ram
17:27:02 <pnorman> 1k tiles uses 24GB of RAM in caches
17:27:56 <zere> when you say "stuff", you mean db indexes & data?
17:28:09 <pnorman> and that's it, unless you want numbers. nothing with carto yet, I shut off all the EC2 stuff and I'm starting stuff up again.
17:28:13 <pnorman> stuff meaning whatever is normally cached, yes
17:28:51 <pnorman> er, or maybe 18k tiles is enough to push stuff out.
17:30:11 <pnorman> alternating between a 9k set to dirty the caches and a 9k set to benchmark, I got 2.28 meta/sec on the benchmark set. if I ran the 9k set to benchmark repeatedly without the dirty cache set I got 3.12 meta/sec
17:30:52 <zere> 37% faster doing the single 9k set.
17:32:08 <pnorman> oh yes - is the mapnik that comes when you install renderd from apmon's PPA a recent enough version for everything?
17:33:12 <gravitystorm> got a link to the ppa, or a version number for mapnik?
17:35:01 <pnorman> https://launchpad.net/~kakrueger/+archive/openstreetmap
17:35:20 <pnorman> ah, 2.1.0. i expect that's enough
17:36:22 <gravitystorm> Yes, that's fine. I'm itching to move to a recent 2.2.x commit (for some halo-based stuff) but I'm going to try to wrestle the mapnik guys into making a release when I see them at sotm-us
17:36:48 <gravitystorm> so for now, I think 2.0.0 or better is the dependency
17:37:07 <zere> gravitystorm: i also notice https://github.com/gravitystorm/openstreetmap-website/tree/docs
17:37:19 <gravitystorm> zere: dude, that was quick
17:37:43 <zere> bah, it's taken me 10 minutes to notice that ;-)
17:38:00 <gravitystorm> zere: less than that, I only pushed about 2 minutes ago :-)
17:38:27 <pnorman> i recall seeing this error at one point, I don't think 2.0.0 works: http://support.mapbox.com/discussions/tilemill/1347-how-do-i-use-the-xml-file-created-by-export-to-mapnik-xml
17:39:07 <TomH> no you need 2.1 for carto
17:39:59 <TomH> 2.0 will barf on the param block that it generates
17:40:17 <pnorman> gravitystorm: want to update the osm-carto readme or have me do it?
17:40:39 <gravitystorm> pnorman: I'll do it
17:41:50 <zere> speaking of branches, i've been working on https://github.com/zerebubuth/openstreetmap-website/tree/json - adding JSON support to the API
17:42:19 <zere> i've been meaning to add a TODO so that what still needs doing is more apparent, and i'll try and get that done
17:42:39 <zere> if anyone wants to help out, i'd welcome it
17:43:03 <TomH> zere: I take it you saw that we have jsonify in now, with the notes branch?
17:43:43 <zere> well, that was one of the things i was wondering
17:44:27 <zere> if we move everything to views, as is the The Right Way To Do It, do you think that would be terrible in terms of speed?
17:46:02 <zere> also, there's a JSON document question - maybe tmcw would know better than i - but it seems from some other JSON APIs on the internet that where you have an array of objects with only one element, it's the done thing to collapse that down to just the object.
17:46:02 <apmon> gravitystorm: Can you get the mapnik guys to get 2.2 into the distro repositories as well then, if the main osm style depends on it?
17:46:56 <tmcw> zere: endpoints should be consistent on what they serve
17:47:05 <tmcw> so if it's like /foo/singleelem it should be a single element
17:47:08 <zere> e.g: instead of {"foo":[{"bar":1}]} to just have {"foo":{"bar":1}}, but still {"foo":[{"bar":1},{"baz":2}]}?
17:47:21 <gravitystorm> apmon: that would be nice, but one step at a time - getting tagged releases (e.g. 2.1.1 , 2.2.0 ) is hard enough.
17:47:31 <tmcw> zere: no, in that instance: always arrays
17:47:35 <TomH> apmon: 2.2 isn't even out yet!
17:47:45 <tmcw> if there can be an array with a call, it should always be that way
17:48:12 <zere> tmcw: cool. i will adjust the code/tests accordingly.
17:48:22 <apmon> which is why I'd be in favour of sticking to mapnik 2.1 as long as possible
17:49:31 <TomH> well nothing needs 2.2 as far as I know
17:49:36 <TomH> idris is using 2.1 just fine
17:49:40 <pnorman> the delays with packaging suck - postgis 1.5 is still what's packaged, and 2.0 was released over a year ago
17:50:07 <apmon> It was in response to "< gravitystorm> Yes, that's fine. I'm itching to move to a recent 2.2.x commit" but maybe I missunderstood
17:50:21 <gravitystorm> apmon: pnorman: I think the dire situation on packing (mapnik, postgis to name two) is a separate topic
17:50:50 <apmon> pnorman: We can do something about improving packaging. It is open source after all... ;-)
17:51:11 <apmon> and we should make more tools available through packaging in up stream repositories
17:51:18 <gravitystorm> apmon: no, you didn't misunderstand. But I think the packaging problems are more to do with Debain GIS team than the maintainers of the upstream projects
17:51:31 <apmon> imho that is one of the biggest things in reducing the hurdles of contributing
17:51:39 <apmon> (which is the main purpose of ewg)
17:51:57 <TomH> well now Debian is unfrozen that might improve
17:52:09 <gravitystorm> here's hoping.
17:52:20 <pnorman> yes. i do not enjoy ages trying to get versions that aren't 1-2 years old running
17:52:30 <apmon> Yes, in response to my request to get mod_tile and renderd in, as well as update mapnik, the response was they are intending to do that after the freeze
17:52:36 <apmon> which should indeed now be over
17:52:50 <apmon> so we should do another push to get mapnik updated as soon as possible in debian
17:53:08 <apmon> but it is unlikely it will get updated to 2.2 unless mapnik actually releases 2.2 ;-)
17:53:56 <zere> anything we can do directly, with this, or are we just hoping?
17:53:59 <gravitystorm> apmon: and I'll keep encouraging mapnik to release more often, in order to reduce the net age of the packages within each distro
17:55:14 <apmon> Imho, we should aim to get to a stage where all of the core osm tools can be used straight from the distros repo
17:55:20 <apmon> for at least debian, ubuntu and red hat
17:55:50 <zere> tmcw: would you rather see {"foo":[]} or just {} if the array is zero-size?
17:55:52 <gravitystorm> zere: making sure our software have reasonable release cadencies (i.e. tagging releases) will help downstream packagers. But apart from that, the only concrete action is to get someone to become a downstream package maintainer directly
17:56:53 <apmon> As we have PPA's for a fair number, we should probably be able to get people to maintain it officially in debian
17:57:00 <tmcw> zere: rather see {foo:[]}
17:57:15 <tmcw> you never know when you want to add more structure to the thing, like totalResults:0 or so on
17:57:16 <gravitystorm> apmon: here's looking at you :-)
17:57:25 <zere> gravitystorm: would it be possible to fork mapnik and add the tags? 2.1.x.0, etc...?
17:57:48 <apmon> I will try and get mod_tile, renderd and osm2pgsql (updated) into debian
17:58:37 <gravitystorm> apmon: I would certainly concentrate on getting out-of-date packages updated - it's even more offputting to newbies that it not being packaged at all imho
17:59:15 <pnorman> +1, and it makes it confusing when you've got the same in the PPA and a hopelessly out-of-date version packaged
17:59:55 <apmon> yes indeed
18:00:12 <pnorman> oh, I remember what happened to my last benchmark. I ran out of rendered tile disk space :)
18:01:10 <apmon> paravoid: can you "save" the tiles to /dev/null ?
18:01:41 <gravitystorm> zere: let me take this up with the mapnik guys again - maybe there's stuff that can be done to make releasing less of a hassle (with builds or testing or whatever) in order to encourage them to release more often. There's certainly stuff in 2.2.0-pre that I want to use for openstreetmap-carto
18:02:08 <pnorman> can you change renderd's tile location? there's a big DOES NOT WORK YET in renderd.conf
18:02:09 <apmon> Yes you can
18:02:18 <apmon> you can even store it in memcached if you want
18:02:43 <apmon> I thought I removed the DOES NOT WORK YET
18:03:25 <pnorman> gravitystorm: it'd be nice to have some kind of design to the style as opposed to everything possible thrown at the map to see what sticks through the label placement
18:05:16 <pnorman> apmon: would /dev/null work?
18:05:21 <gravitystorm> pnorman: oh indeed. At the moment I'm working through trying to make it easier to hack on though - a good example is secondary road casings, which are a dark brown colour but mysteriously thin so you get only a slight effect. Untangling this kind of thing will take a while.
18:06:14 <zere> awesome. i notice we're at the top of the hour. was there anything else anyone wanted to discuss?
18:06:16 <apmon> pnorman: Maybe not, as I think it does a rename at the end, which would fail on dev/null
18:06:47 <apmon> TomH: What is the situation with respect to the various osm tools in Red Hat?
18:06:52 <gravitystorm> zere: not from my end. I'll continue to work on the rails port documentation, and I look forward to seeing the outcome of any (and all) openstreetmap-carto benchmarks
18:06:57 <pnorman> as a general note, spot-priced us-east m2.2xlarge instances are nice. they have 34GB of RAM and the local storage gets 1.6k iops read. spot-priced they're 0.07 USD/hr
18:07:19 <apmon> Is it worthwhile to e.g. add a page to switch2osm for red hat from packages?
18:07:26 <gravitystorm> zere: how can I add actions for myself to meeting bot?
18:07:57 <pnorman> switch2osm from source could use a refresh
18:07:58 <gravitystorm> #action gravitystorm continue working on rails port documentation
18:08:26 <zere> gravitystorm: yup, like that :-)
18:08:28 <TomH> apmon: mapnik is 2.0 but I am now maintainer and working to update to 2.1 once the agg bundling issues are resolved
18:08:29 <apmon> pnorman: That is pretty decent pricing. I should probably learn how to use EC2 after all
18:08:45 <TomH> carto is now in, working towards tilemill
18:08:46 <apmon> TomH: Sweet!
18:09:15 <apmon> is mod_tile / renderd packaged?
18:09:22 <TomH> not AFAIK
18:09:47 <pnorman> apmon: i bid 0.12, but you pay the current price and get terminated if current > bid. history shows that current sticks at the minimum, it never goes up (except maybe at christmas?)
18:10:35 <pnorman> apmon: something I'd like to do is get EBS snapshots with the latest planet at the very least - it saves downloading it
18:11:15 <pnorman> #action pnorman to check how EC2 osm2pgsql ebs snapshots used to be done
18:11:43 <apmon> terminated, or suspended?
18:11:50 <pnorman> terminated
18:12:03 <apmon> hence the bidding higher?
18:12:23 <pnorman> yes. but you don't pay higher until demand goes up
18:15:48 <pnorman> from what i've read it sounds like ec2 spot is amazon investing in enough power to handle its website at christmas and selling it on the rest of the year
18:17:07 <zere> thanks everyone for coming! see you again next week, i hope :-)
Retrieved from "https://osmfoundation.org/w/index.php?title=Working_Group_Minutes/EWG_2013-05-13&oldid=2146"