[closed] Store Area Function - Practical Limits To Offline Maps?

[Yemble]

I have a decision to make with regard to a 64Gb microSD that I have purchased for my new Note tablet, which may be influenced by any practical limitations of AQs tile cache.

The 64Gb SD card will arrive pre-formatted with Microsoft’s proprietary exFAT file system. This is OK on the default tablet firmware as Samsung have licensed support for exFAT, however, other Android OS’s, like CyanogenMod, do not support exFAT. Windows 7/8 obviously do have native support exFAT.

I could reformat the SD card as FAT32, as I have done on my three Galaxy S3s running CyanogenMod, however this comes at a price… namely that the maximum single file size becomes limited to 4Gb.

My questions relate to the maximum practical size of the AQ cache. From what I can see, AQ stores all of its cached data in a single database file. This differs from Memory-Map (PDA/PC version) which holds its data in a large number of medium sized files. With Memory-Map, I can store GB-OS offline maps for the entire UK at both 1:25K and 1:50K scales in around 24Gb. I attempted to create an offline map GB-OS for the UK using Back Country Navigator, but I gave up after discovering:

1) That it would consume well over 64Gb (based upon a sample area)
2) That on FAT32 that DB breaks completely at 4Gb (as predicted)

So, in AQ, given that there is now a temptation to build a full off-line GB-OS map of the UK, 5000 tiles at a time, using the area downloader, are there any practical reasons why this could not be done?

1) Does AQ store its data in a more efficient manner than BCN, ie. is 64Gb enough for the UK at both 1:50K & 1:25K scales?
2) Does AQ work well with a very large database, or does performance suffer to a significant degree?
3) On FAT32, what happens when the AQ DB file size reaches 4Gb?
4) Is there an operational size limit for the AQ DB on exFAT?

[Psyberia-Support]

Hi,
These are interesting questions...
AlpineQuest does not actually store all data in a single file, but in multiple 2GB files (FAT32 can be limited to 2GB when using small sector that's why this size has been chosen). This is a quite large limit so for a standard usage the data will remain in a single file.
So:
1) I don't know how BCN works, but in AlpineQuest there is a size overhead of approximately 20%. This overhead is added to allow map updates to be stored efficiently, even if growing time after time (I am considering dropping this overhead to 10%). In addition to that, you must take the size of the index file (.IDX) into account (usually between 2% and 2.5% of the data size, you can check that using your existing stored data). Considering an identical data compression with MM (AQ uses JPG so it should even be better), your data should not be much more than 30GB;
2) You should not see (significant) performance loss even with very large cache thanks to the indexation of the data;
3) As explained above, as soon as a cache reaches 2GB, an additional cache file is created;
4) As a result of 3), on exFAT it will behave like on a FAT32 storage.
However I must admit that I have never stressed a lot AlpineQuest with very large cache data.
Best regards

[Yemble]

Many thanks for the detailed explanation. I am encouraged to learn that you span the DB in 2Gb segments, as BCN definitely does not! The SD card arrived today and I've decided to leave it as exFAT, even though FAT32 would obviously be OK with AQ. I guess that I should not be surprised that you have done the job properly

30Gb seem OK for the UK OS maps, if it can be achieved... I guess that is down to my patience as it will take a while to get there... but at least it sounds feasible

[Psyberia-Support]

Thanks!
I'll try to add an user setting to rise the queue limit.

[Yemble]

Thanks!
I'll try to add an user setting to rise the queue limit.

Yeh, I really could do with an unlimited option. I am experimenting with a stripe across Northern England and I have already reached over 42000 tiles, with no end yet in sight! I would also appreciate some indication in AQ of how many files are covered by a selected area at the scales selected.

Thanks.

[Psyberia-Support]

I would also appreciate some indication in AQ of how many files are covered by a selected area at the scales selected.

Isn't it the number displayed close to the number 6 in the screen-shot on the right or is it something else you would like to see ? In this example "402" is the total number of file for all zoom levels until the selected one.

[Yemble]

No, you misunderstand what I am referring to.

If you select a large map area, say of 50000 tiles, on the first pass, AQ will show a total number of tiles as just 5000. On the second run it will show 10000, third 15000, etc. What it does not show is that the total number of tiles in the selected area is 50000, so you never actually know how close you are to completing the entire task.

[Psyberia-Support]

While adjusting the area and zoom level, the displayed number of tiles (402 in the above example, 50 000 for yours) is the total number for the selected area.
So you would like to see this number also displayed during the download process, am I right?
The thing is that even if you can start the download process with more than the selected number of tiles (zoom bar displayed in red), it's not thought for that. As soon as you only start the process with less that the authorized limit, I guess that the process progress is the one you would like to see.

[Yemble]

Yes, precisely that. There are two totals here, the first being the one currently displayed, which is the total for the job, restricted by the 5000 tile limit. The second is the grand total of available tiles within the selected area (at the chosen scales). It would also be nice to know the grand total of tiles within the selected area that are not currently in cache, ie. still outstanding for download.

[Psyberia-Support]

Ok but it's a moot point (I just learned this expression right now...) since these totals are only different if you start the process even if the queue limit has been exceeded.
Anyway I have made two updates in the last 1.4.4 version:
- the queue limit is now 7500 (instead of 5000);
- you can increase the queue limit using the user setting 'maps.features.areadownloader.max_queue_size=[value]'.

[Yemble]

Brilliant, l'll give it a go and get back to you...

[Chris]

Hello,
Where is the user setting to increase the queue limit?
I'm having trouble downloading large amounts of data (15gb) mainly google satellite with google map overlay.
I get large amounts of errors, et it seems to freeze.

Thanks

[Yemble]

Hello,
Where is the user setting to increase the queue limit?
I'm having trouble downloading large amounts of data (15gb) mainly google satellite with google map overlay.
I get large amounts of errors, et it seems to freeze.

Thanks

Chris, you need to create a text file called user_settings.conf in the alpinequest folder, then add the line mentioned on the previous page.

[Psyberia-Support]

I'll add a page on the online help about user settings.

I get large amounts of errors, et it seems to freeze.

Do you know what errors you encountered? (you can long-press a map process, and select 'Details' to have the list of errors).
Thanks

[Chris]

Thanks Yemble,

AlpineQuest: it seem s to freeze on "process running"

I get about 75% of tiles skipped

errors:

26xUnknownHostException
3xSocketException