We have moved to a new Sailfish OS Forum. Please start new discussions there.
6

Distribution of offline maps

asked 2017-11-05 13:44:41 +0300

rinigus gravatar image

I am looking for server(s) that could be used for distribution of the offline maps data.

The offline maps are generated to be used by OSM Scout Server. There are several types of datasets that are generated to be used by Mapnik (drawing), Valhalla (routing), Geocoder-NLP (search), and LibOsmscout (all-in-one) backends. One more backend is in works (MapboxGL) that would expand the amount of data needed to be distributed to about 200 GB.

At present, we use data.modrana.org for distribution of the maps, thanks to @MartinK! This server has a decent uptime, but sometimes is down. Downtimes account for about 1-2% and are usually for short periods of time, but, in unfortunate circumstances downtime could last for few days. So, we are looking for fallback servers or maybe the servers that could be used in parallel to data.modrana.org. The easier the solution, the better it is since it would allow me to focus on other map-related issues.

Professional CDN is an option as well. But then there should be a solution for paying for it, including taxation issues. I would prefer to avoid having money transferred via myself to avoid discussions with tax department and not investing time into this as well. For CDN, ideally, financing should be provided via some non-profit organization or (as a charity?) via some interested company(ies).

While I am looking for data distribution for my own software, it maybe more general problem that is faced by map applications and, possibly, some other applications that require larger datasets to be streamed to the devices.

Requirements:

  • About 200 GB for storage that would peak to 400 GB during new dataset upload (we upload first and later remove the older version of the maps)

  • No idea about traffic and its probably more expensive part. According to Jolla Store, OSM Scout Server has less than 1000 users at present. Some users prefer OpenRepos, these have to be added.

edit retag flag offensive close delete

Comments

I Use OSM Scout Server on my Sailfish OS device(s).

vattuvarg ( 2017-11-05 14:10:56 +0300 )edit

How to you envision the management of the server to take place? Assuming I have a server with 400GB free space and 5-10TB/month traffic. Would I just setup a mirror and data.modrana.org would do some load balancing?

bomo ( 2017-11-05 14:21:43 +0300 )edit
1

FYI, @rinigus is the author of OSM Scout server.

objectifnul ( 2017-11-05 14:29:30 +0300 )edit

@bomo, good question. So far, we distributed maps with me uploading them to one CDN provider and MartinK syncing from there (automatically). I think we can find solution that would fit us all as soon as we have servers to balance :)

Most probably some load-balancing will have to be done by app. Otherwise we'll be hit by modrana downtime as well. Probably we can expand https://github.com/rinigus/osmscout-server/blob/master/scripts/import/provided/url.json to include more servers and build some logic around it.

rinigus ( 2017-11-05 14:44:54 +0300 )edit

@rinigus thanks for your reply. Doing the balancing on the device could be beneficial. But what about renting a cheap but reliable server for balancing only (low traffic, low space)? Alternatively, it would be possible to configure the DNS to do round robin.

bomo ( 2017-11-05 16:18:21 +0300 )edit

1 Answer

Sort by » oldest newest most voted
2

answered 2017-11-05 14:07:01 +0300

vattuvarg gravatar image

updated 2017-11-05 14:07:44 +0300

I think it would be possible to create a solution for the maps using IPFS.

edit flag offensive delete publish link more

Comments

1

Interesting. Note that it has to be simple and not suck the battery out our devices either. From the quick look: its build around GO and P2P - not sure we can use it on SFOS without P2P. If there is a simple API that I can use to push the data out on my server side when distributing maps and download data by devices - why not. This API should be C++, I presume, to make it usable without requiring too much. Note that OSM Scout Server can be used on plain Linux as a console application/server as well => don't want to start requiring PyOtherSide, for example.

rinigus ( 2017-11-05 14:48:45 +0300 )edit

There is no need for an API. The URL would look like a normal web page.

Example
https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/I/m/Japanese_ryakuji_KAN.png

vattuvarg ( 2017-11-05 14:53:15 +0300 )edit

An Introduction to IPFS

vattuvarg ( 2017-11-05 15:01:32 +0300 )edit

Serving a website over IPFS

vattuvarg ( 2017-11-05 15:05:14 +0300 )edit

@vattuvarg, just few questions before diving into it in details. I am asking you with the hope to be efficient and spend my time mainly on map import issues :) . Hopefully, you know or can find the answers:

I expect that someone needs to provide bandwidth. In our case, devices will not contribute to data transfer, they will only get data from URL. Do I need then to have IPFS installed on the servers providing the bandwidth? I presume that installation of IPFS would make it harder to find the servers sending the data out.

rinigus ( 2017-11-05 15:25:37 +0300 )edit
Login/Signup to Answer

Question tools

Follow
5 followers

Stats

Asked: 2017-11-05 13:44:41 +0300

Seen: 413 times

Last updated: Nov 05 '17