Distribution of offline maps
asked 2017-11-05 13:44:41 +0300
I am looking for server(s) that could be used for distribution of the offline maps data.
The offline maps are generated to be used by OSM Scout Server. There are several types of datasets that are generated to be used by Mapnik (drawing), Valhalla (routing), Geocoder-NLP (search), and LibOsmscout (all-in-one) backends. One more backend is in works (MapboxGL) that would expand the amount of data needed to be distributed to about 200 GB.
At present, we use data.modrana.org for distribution of the maps, thanks to @MartinK! This server has a decent uptime, but sometimes is down. Downtimes account for about 1-2% and are usually for short periods of time, but, in unfortunate circumstances downtime could last for few days. So, we are looking for fallback servers or maybe the servers that could be used in parallel to data.modrana.org. The easier the solution, the better it is since it would allow me to focus on other map-related issues.
Professional CDN is an option as well. But then there should be a solution for paying for it, including taxation issues. I would prefer to avoid having money transferred via myself to avoid discussions with tax department and not investing time into this as well. For CDN, ideally, financing should be provided via some non-profit organization or (as a charity?) via some interested company(ies).
While I am looking for data distribution for my own software, it maybe more general problem that is faced by map applications and, possibly, some other applications that require larger datasets to be streamed to the devices.
Requirements:
About 200 GB for storage that would peak to 400 GB during new dataset upload (we upload first and later remove the older version of the maps)
No idea about traffic and its probably more expensive part. According to Jolla Store, OSM Scout Server has less than 1000 users at present. Some users prefer OpenRepos, these have to be added.
I Use OSM Scout Server on my Sailfish OS device(s).
vattuvarg ( 2017-11-05 14:10:56 +0300 )editHow to you envision the management of the server to take place? Assuming I have a server with 400GB free space and 5-10TB/month traffic. Would I just setup a mirror and data.modrana.org would do some load balancing?
bomo ( 2017-11-05 14:21:43 +0300 )editFYI, @rinigus is the author of OSM Scout server.
objectifnul ( 2017-11-05 14:29:30 +0300 )edit@bomo, good question. So far, we distributed maps with me uploading them to one CDN provider and MartinK syncing from there (automatically). I think we can find solution that would fit us all as soon as we have servers to balance :)
Most probably some load-balancing will have to be done by app. Otherwise we'll be hit by modrana downtime as well. Probably we can expand https://github.com/rinigus/osmscout-server/blob/master/scripts/import/provided/url.json to include more servers and build some logic around it.
rinigus ( 2017-11-05 14:44:54 +0300 )edit@rinigus thanks for your reply. Doing the balancing on the device could be beneficial. But what about renting a cheap but reliable server for balancing only (low traffic, low space)? Alternatively, it would be possible to configure the DNS to do round robin.
bomo ( 2017-11-05 16:18:21 +0300 )edit