Guide: Externalising android_storage and other directories / files to SD-card

Question

25

Guide: Externalising android_storage and other directories / files to SD-card

asked 2019-04-08 00:07:25 +0300

olf
10393 ●104 ●201 ●158

updated 2020-06-17 17:01:00 +0300

Although there are a couple of guides for extending the internal mass storage (eMMC) of devices running SailfishOS (in general; this is not addressing the "tiny system partition issue" of SailfishOS on non-BTRFS devices, i.e. all but the Jolla 1 phone) with an SD-card (on TMO and here on TJC), IMO none of them is completely satisfactory (technically, by functionality and usability), but they provided valuable ideas.

This is an enhanced, updated and streamlined guide created from the original "[How-to] Externalising android_storage and other directories / files to SD-card". It only covers SailfishOS 2.2.0 and newer.
For SailfishOS versions before 2.2.0 and the technical background of externalising android_storage, please consult the original how-to. The section topics and numbers are mostly the same for both guides.

1. Externalising regular files and directories in /home/nemo
2. Externalising /home/nemo/android_storage

This is part 2 "Externalising android_storage and other directories / files to SD-card", you may want to read part 1 "Creating partitions on SD-card, optionally encrypted" first.

All this was tested for years on Jolla 1 phones, Xperia Xs, XA2s and 10s, plus Gemini PDAs with SailfishOS versions from 2.2.0 to 3.2.1 and various SD-cards; if things should really turn out to look different on other devices (which is very unlikely, except for applying section 2 to the new, Android 8 based runtime environment, e.g. on an Xperia XA2 or 10), you may have to adapt the following sections (thus then some understanding of the commands issued is crucial); and last but not least this guide still may have flaws. Please report your experiences and adaptations in these cases, but do not blame anyone (e.g. me) for any mishaps.
Contributions and constructive suggestions are welcome.

1. Externalising regular files and directories in /home/nemo

For externalising android_storage, see section 2.

Mind using "hidden" directory names (starting with a ".") on the SD-card for externalised directories not to be indexed by the SailfishOS' Media Tracker service twice (there and in their original location, symlinked later in this guide).
Use regular directory names for all data, which solely resides on the SD-card (i.e. without being symlinked somewhere under /home/nemo/) and should be indexed by the Media Tracker.

As user nemo, execute the following commands:

cd /run/media/nemo/XXXXXXXX/
echo 'Data externalised from /home/nemo/ to SD-card resides in "hidden" directories starting with a ".".' > README-data.txt
mkdir .nemo-sdcard 
cd
ln -s /run/media/nemo/XXXXXXXX/.nemo-sdcard SD-card

You may move existing, regular directories or subdirectories under /home/nemo/ (but not android_storage or anything under it, and one also shall not do that with "hidden" files or directories) onto SD-card by executing (you may want to back them up, first):

cd Foo
mkdir /run/media/nemo/XXXXXXXX/.foo
cp -av . /run/media/nemo/XXXXXXXX/.foo/  # Only continue, if copying has been successful 
cd ..
rm -r Foo
ln -s /run/media/nemo/XXXXXXXX/.foo Foo

You can also externalise specific files (instead of whole directory trees) by executing (you may want to back them up, first):

cd Foo
mkdir /run/media/nemo/XXXXXXXX/.foo
cp -pv Bar /run/media/nemo/XXXXXXXX/.foo/  # Only continue, if copying has been successful 
rm Bar
ln -s /run/media/nemo/XXXXXXXX/.foo/Bar Bar

Note that ".foo" and "Foo" are just a placeholders for specific directory names, "XXXXXXXX" is a placeholder for a specific mounted partition on the SD-card (rsp. technically: its UUID) and "Bar" is a placeholder for a specific file.

Arbitrary examples:
~nemo/Music --> /run/media/nemo/XXXXXXXX/.music
~nemo/Videos --> /run/media/nemo/XXXXXXXX/.videos
~nemo/Documents/Maps --> /run/media/nemo/XXXXXXXX/maps/osmscout/osmscout
~nemo/Maps/map.navit.bin --> /run/media/nemo/XXXXXXXX/maps/osm-maps/osm_bbox_X,Y,U,V.bin

2. Externalising /home/nemo/android_storage

Because AlienDalvik needs a filesystem providing classic UNIX access rights and ownerships, this section 2 does not work using a partition on SD-card formatted with (v)FAT.
You might utilise the "Guide: Creating partitions on SD-card, optionally encrypted" to employ a (at least one) proper, "native" filesystem on your SD-card before continuing.

Devices with BTRFS as their native filesystem (e.g. Jolla 1 phones) use /data/media as AlienDalvik's media directory (i.e., the "virtual SD-card", which becomes bind-mounted at /home/nemo/android_storage) for their AOSP4.1.2 based Android runtime environment.
"Second generation devices" employing EXT4 on LVM (e.g. Xperia X, Intex Aquafish, Jolla C, Jolla Tablet) with their AOSP4.4 based AlienDalvik all seem to use /home/nemo/android_storage directly (symlinked from /home/.android/data/media). Currently only tested with an Xperia X, please report differences on your "second generation device".
"Third generation devices" with their AOSP8 based Android runtime environment (e.g. the Xperia 10 and XA2 families) are not analysed yet, currently lacking such a device.

Install mount-sdcard (e.g. per Storeman) to make automatic mounting on boot-up working reliably for AlienDalvik (i.e., to be race-free by a "Before=alien-service-manager.service" statement).
For automatic mounting of encrypted SD-cards, which have to be prepared as described in part 1 for externalising android_storage to them, have crypto-sdcard installed. Also running mount-sdcard is not really necessary in this case (when the content of android_storage resides on an encrypted partition), but it still provides some additional benefits.

2.1 Copying data to SD-card

Execute the following commands:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
cd /home/nemo  # cd /data for BTRFS-using devices
cp -av android_storage /run/media/nemo/XXXXXXXX/.android-sdcard  # Pick the right UUID!
# cp -av media /run/media/nemo/XXXXXXXX/.android-sdcard  # for BTRFS-using devices
touch /run/media/nemo/XXXXXXXX/.android-sdcard/ANDROID-SDCARD  # Set a marker
chmod a= /run/media/nemo/XXXXXXXX/.android-sdcard/ANDROID-SDCARD  # Make it persistent

Optionally check via checksums, that the data has been correctly written (needs rsync installed, per pkcon install rsync):

rsync -nc -avvhh /home/nemo/android_storage/ /run/media/nemo/XXXXXXXX/.android-sdcard/ | fgrep -v ' is uptodate'  # Should detect no files not being up-to-date on "2nd gen" devices
rsync -nc -avvhh /data/media/ /run/media/nemo/XXXXXXXX/.android-sdcard/ | fgrep -v ' is uptodate'  # Should detect no files not being up-to-date on BTRFS-using devices

2.2 Integrating an externalised android_storage when booting

Unfortunately there seems to be no proper way of integrating an externalised android_storage seamlessly into SailfishOS with AlienDalvik without altering one of its scripts.

Alter /opt/alien/system/script/platform_envsetup.sh by executing:

devel-su
cd /opt/alien/system/script/
cp -p platform_envsetup.sh platform_envsetup.sh.orig  # Backup

Edit this file by adapting the MEDIA_STORAGE path variable, so that a diff platform_envsetup.sh.orig platform_envsetup.sh outputs (replace XXXXXXXX with the right UUID):

* On "2nd gen" devices:

20c20
< MEDIA_STORAGE=/home/nemo/android_storage
---
> MEDIA_STORAGE=/run/media/nemo/XXXXXXXX/.android-sdcard

* On BTRFS-using devices:

16c16
< MEDIA_STORAGE=$ANDROID_DATA/media
---
> MEDIA_STORAGE=/run/media/nemo/XXXXXXXX/.android-sdcard

Check (user- and group-) ownership and access rights with ls -l platform_envsetup.sh* (compared to the backed up original) and adapt them with chown and chmod, if necessary.
Finally backup your altered version with cp -p platform_envsetup.sh platform_envsetup.sh.patched!

Mind that the altered shell script may be overwritten by SailfishOS upgrades (as it happened when upgrading to SailfishOS 3.2.0, for the first time since 2.2.0), hence check for this then (at least when Android apps are failing to find their data after a SailfishOS upgrade) and reestablish the change (in the updated script file!), if overwritten (see timestamps with ls -l).

2.2.1 Additional step for devices with AOSP4.4 based AlienDalvik

On "2nd gen" devices (e.g. Xperia X, Jolla C, Intex Aquafish, Jolla Tablet) perform this additional step (i.e., not on BTRFS-using devices as the Jolla 1!), which can be done as user nemo (root user should also work):

cd /home/nemo
mv -v android_storage android_storage.bak
ln -s /run/media/nemo/XXXXXXXX/.android-sdcard android_storage

2.3 Finishing up

Reboot to see, if everything is working as intended: Check with your Android apps, if they access their data as before, as from their perspective everything should be the same as before externalising their data.

If in doubt, check if the relevant mounts succeeded:
Issue as user nemo a mount | fgrep mmcblk1 for unencrypted SD-cards respectively mount | fgrep crypto for encrypted SD-cards (employed per part 1): You should see each partition being mounted twice when AlienDalvik is running, once regularly at /run/media/nemo/XXXXXXXX as the mounted target (i.e., in the second field after the "on") and a second time at /opt/alien/run/media/nemo/XXXXXXXX for AlienDalvik.

Additional check for BTRFS-using devices
Issue a mount -t fuse: Again, you should see /dev/fuse being mounted twice when AlienDalvik is running, once regularly at /home/nemo/android_storage as the mounted target and at /opt/alien/home/nemo/android_storage for AlienDalvik.

If this does not yield the expected outcome, debug with journalctl -r as root user, while having less installed (per pkcon install less): Search with "/" for mmcblk1 rsp. crypto (if using an encrypted SD-card) (additionally on BTRFS-using devices: look for fuse), use "n" to jump to the next hit and press PageUp / PageDown to look at adjacent messages. Report your findings along with the SailfishOS version and the device you use in a comment here by describing your experience and pasting the output generated by above commands, which does not look as described.

2.3.1 Removing the old content of android_storage residing on internal eMMC

When working well, delete the old content of android_storage on internal eMMC, as this is the purpose of the whole procedure:

* On "2nd gen" devices:

devel-su
cd /home/nemo
rm -rf android_storage.bak

* On BTRFS-using devices:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
cd /data/media
ls -la | more  # Check: You should *not* see ANDROID-SDCARD
pwd  # Check twice that you are really in the directory you cd'ed to two lines above!
rm -rf * .[^.]*  # !!!

Done.

edit retag flag offensive close delete

Comments

1

note that, like said, the AOSP 8.1 Oreo layer on the XA2, works in a completely different way:

unlike X and J1, ~nemo/android_storage is actually empty.
like them, the android data is in /home/.android
on XA2, like on a real android, storage is mounted un /home/.android/data/media/0
it's fuse mounted to ~nemo/android_storage while alien dalvik is running (and before running, any data in nemo/android_storage is imported in media/0 - for backward compatibility ?)
i can presume that, once supported, SD in compatibility layer should appear in media/1 like on a real android ?

DrYak ( 2019-04-08 00:22:35 +0300 )edit

1

Oh, that is half J1 and half "2nd gen" style:

On a J1, ~nemo/android_storage is actually empty, too.
But the J1 has its android data in /data
/data/media is fuse mounted to ~nemo/android_storage while alien dalvik is running on a J1
There already are artifacts of the Android SD compatibility layer on "2nd gen" devices

olf ( 2019-04-11 00:17:31 +0300 )edit

you can just link the userdata storage dir from android emulator :

instead of moving android dir you copy the android storage dir to leave a backupdir on internal phone storage

   sudo mv /home/nemo/android_storage $SDcardPath
   rm /home/.android/data/media
   ln -s /home/.android/data/media $SDcardPath/android_storage

if sudo is not installed use devel-su

speefak ( 2019-08-05 15:41:57 +0300 )edit

@speefak, no, solely establishing a symlink does not do the job.

And why would one want to leave the (then stale) data on internal eMMC, when everything is working fine?
The primary reason for externalising data to SD-card is to free the internal eMMC of this data.
Side note: IMO backups should be carried out with a little thought, i.e. regularly and to external media (e.g. a USB stick per OTG, a PC, some cloud storage). Defining some left over, old data as a backup may lead to sorrow.

P.S.: Moving (per mv) across filesystems performs exactly the same actions as copying (per cp) and removing later (per rm), but is much more error prone (in case of a power loss, kernel panic, sudden reboot, inadvertently closing the Shell window etc.).

olf ( 2020-06-17 17:18:25 +0300 )edit

note that, like said, the AOSP 8.1 Oreo layer on the XA2, works in a completely different way:
unlike X and J1, ~nemo/android_storage is actually empty.
like them, the android data is in /home/.android
on XA2, like on a real android, storage is mounted un /home/.android/data/media/0
it's fuse mounted to ~nemo/android_storage while alien dalvik is running (and before running, any data in nemo/android_storage is imported in media/0 - for backward compatibility ?)
i can presume that, once supported, SD in compatibility layer should appear in media/1 like on a real android ?
DrYak ( 2019-04-08 00:22:35 +0300 )edit
Oh, that is half J1 and half "2nd gen" style:
On a J1, ~nemo/android_storage is actually empty, too.
But the J1 has its android data in /data
/data/media is fuse mounted to ~nemo/android_storage while alien dalvik is running on a J1
There already are artifacts of the Android SD compatibility layer on "2nd gen" devices
olf ( 2019-04-11 00:17:31 +0300 )edit
you can just link the userdata storage dir from android emulator :
instead of moving android dir you copy the android storage dir to leave a backupdir on internal phone storage
sudo mv /home/nemo/android_storage $SDcardPath rm /home/.android/data/media ln -s /home/.android/data/media $SDcardPath/android_storage
if sudo is not installed use devel-su
speefak ( 2019-08-05 15:41:57 +0300 )edit
@speefak, no, solely establishing a symlink does not do the job.
And why would one want to leave the (then stale) data on internal eMMC, when everything is working fine?
The primary reason for externalising data to SD-card is to free the internal eMMC of this data.
Side note: IMO backups should be carried out with a little thought, i.e. regularly and to external media (e.g. a USB stick per OTG, a PC, some cloud storage). Defining some left over, old data as a backup may lead to sorrow.
P.S.: Moving (per mv) across filesystems performs exactly the same actions as copying (per cp) and removing later (per rm), but is much more error prone (in case of a power loss, kernel panic, sudden reboot, inadvertently closing the Shell window etc.).
olf ( 2020-06-17 17:18:25 +0300 )edit

Answer 1

3

answered 2019-04-08 00:21:31 +0300

M.Bln.
4289 ●76 ●117 ●113

First: Great work. And thanks for the guide!

But even if I would like to have more options for storage (swapping folders etc.) I am afraid to break something with the next SF-update. So I would like more to see an option in SF to do so instead of writing scripts for it.

Anyway it's nice to have the possibility and a guide to do so myself! :-)

edit flag offensive delete publish link

Comments

1

@M.Bln., this is a comment, not an answer, so please convert it.

olf ( 2019-04-10 16:05:01 +0300 )edit

Answer 2

0

answered 2019-04-23 14:57:18 +0300

campeon
141 ●4 ●11 ●13

Thanks a lot for this procedure. The question i got is... Does this procedure work for a XA2.... Is there any change needed? I´, a bit clumsy and i dont know exacty what to change.

And... is there any prevention needed in case of S.O. update ?

I think its a major inconvenience the S.O. inhability to handle the Micro SD card for Andorid apps.

Thanks for your comments.

edit flag offensive delete publish link

Comments

2

As already noted in above guide: Not yet.

Edit: Take a look at the contribution of @mynameisnotimportant.

olf ( 2019-04-23 15:16:30 +0300 )edit

1

Thanks anyway por your contribution. Xperia X is a more beautiful device than Xperia XA2, which looks cheaper. And i have had both to compere.

campeon ( 2019-04-25 15:52:40 +0300 )edit

Answer 3

6

answered 2019-05-19 05:33:01 +0300

mynameisnotimportant
397 ●10 ●18 ●23

I am not sure I understood everything in the guide, but it seems very involved. I just wanted to share what I did, since it may be a bit easier to follow. All it is is just two new folders, and an extra line in a file. First, I made a folder on my SD card called Android (the path to which is /media/sdcard/Name_of_Card/Android) and added a folder to /home/nemo/android_storage with the same name as my SD card (but it could be any name). After that just edit /var/lib/lxc/aliendalvik/config to add another bind mount to the LXC script. There's already one in there from Jolla (lxc.mount.entry = /home/nemo/android_storage data/media/0 none bind,create=dir 0 0), so all I did was add another undernearth it:

lxc.mount.entry = /media/sdcard/Name_of_Card/Android data/media/0/Name_of_Card none bind,create=dir 0 0.

This lets me access the SD card under a subfolder of the 'internal memory' on Android at least. Technically you can completely replace android_storage here if you wanted to. Yes it could be replaced with system updates, but I have a feeling the frequency of updates which edit said file is gonna be manageable :)

edit flag offensive delete publish link

Comments

That is only on "Third generation devices" with their AOSP8 based Android runtime environment (e.g. the Xperia XA2 family), which are not yet covered by the guide above.

Thanks for researching a way to put android_storage (or even sub-directories of it; that was not possible with the old, AOSP 4 based AlienDalvik) onto SD-card on these devices.

olf ( 2019-05-19 20:19:29 +0300 )edit

1

@mynameisnotimportant This works really great. Many thanks for your hint! I put android_storage completely onto the sd card on a XA2 Plus device.

The important lines in the config were changed to:

# android_storage (changed to be located on sd card)
#lxc.mount.entry = /home/nemo/android_storage data/media/0 none bind,create=dir 0 0
lxc.mount.entry = /run/media/nemo/<sd_card_name>/android_storage data/media/0 none bind,create=dir 0 0

jobe-m ( 2019-10-13 00:44:51 +0300 )edit

Awesome. Really nice. I updated to 3.2.0 and did not work anymore for me. Did a flash and update to 3.2.0 and did not work. Flash and stay on 3.1.0 and works. Anybody on 3.2.0 using this?

BLB ( 2019-11-05 15:44:21 +0300 )edit

I guess the SELINUX is the culprit I will look into it from that angle

BLB ( 2019-11-07 18:17:03 +0300 )edit

1

Well :) I hit a 40 cm form screen problem ... with the upgrade the file gets updated ... as mentioned in this thread. /dev/mmcblk1p2 89355160 312904 89025872 1% /run/media/nemo/Android

lxc.mount.entry = /run/media/nemo/Android data/media/0 none bind,create=dir 0 0

I add that and start hitting issues with aliendalvik start up

systemctl status aliendalvik.service ● aliendalvik.service - Alien Dalvik Loaded: loaded (/lib/systemd/system/aliendalvik.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Fri 2019-11-08 07:37:50 EET; 11min ago Process: 8060 ExecStopPost=/usr/bin/stop-aliendalvik.sh (code=exited, status=0/SUCCESS) Process: 7733 ExecStart=/usr/bin/start-aliendalvik.sh (code=exited, status=0/SUCCESS) Process: 7727 ExecStartPre=/usr/bin/start-aliendalvik-preinit.sh (code=exited, status=203/EXEC) Main PID: 7733 (code=exited, status=0/SUCCESS)

/usr/bin/start-aliendalvik-preinit.sh gets on an infinite wait for a file that does not get there ... need to look more

BLB ( 2019-11-08 07:58:41 +0300 )edit

Answer 4

0

answered 2020-06-16 22:48:43 +0300

speefak
306 ●12 ●24 ●21

updated 2020-06-18 00:29:26 +0300

For 2nd generation emulation ( Sony Xperia X, Jolla C ) and EXT4 formatted sd cards just copy and paste the code to move android storage to sd card

if [[ ! $(rpm -qa | /bin/grep -w sudo) ]]; then sudoCMD=devel-su ; else sudoCMD=sudo; fi
$sudoCMD systemctl stop aliendalvik.service
SDCardPath=$(mount -l | /bin/grep "/dev/mmcblk1p1" |  cut -d " " -f3 | head -n1)
$sudoCMD chown $USER.$USER $SDCardPath
mv android_storage  $SDCardPath/
ln -s $SDCardPath/android_storage .
$sudoCMD cp /opt/alien/system/script/platform_envsetup.sh /opt/alien/system/script/platform_envsetup.sh_bck
$sudoCMD sed 's|MEDIA_STORAGE=.*|MEDIA_STORAGE='$SDCardPath'/android_storage|' -i /opt/alien/system/script/platform_envsetup.sh

Setting a Symlink without edit path in the android env config file causes erros in some apps.

running SFOS 3.0.3 on Sony Xperia X Compact ( F5321 ) => https://speefak.spdns.de/oss_lifestyle/sailfish-os-speicherpfade-auf-sd-karte-mappen/

edit flag offensive delete publish link

Comments

Altering EXTERNAL_STORAGE to point to a directory on an SD-card (instead of altering MEDIA_STORAGE) in /opt/alien/system/script/platform_envsetup.shdoes not (and IMO, cannot) work (at least for me on a XperiaX@SFOS3.2.1):
Android's predefined "Media"-directories (Alarms, DCIM, Download, Movies, Music, Notifications, Pictures, Podcasts, Ringtones) are not accessible for Android apps, then. E.g., Firefox cannot download files.

Furthermore, the Shell script snippet you cut and pasted from your web site needs some prerequisites fulfilled, or it will fail. The most prominent ones are:

A "second generation device": Jolla C and derivates, Jolla Tablet, Xperia X series
It implicitly uses /dev/mmcblk1p1 (no other partitions, e.g. encrypted ones, whole devices, etc.)

@speefak, what are you trying to achieve, i.e. what did above guide not provide for you (or, which steps failed for you, if any)?
And / or, which Android apps did you have issues with, when letting MEDIA_STORAGE point to a directory on SD-card?

Maybe this is just a consequence of the classic misconception, what EXTERNAL_STORAGE means (since Android 3). For details, see e.g.:

P.S.: BTW, the Shell script for the Jolla 1 on your webpage looks broken to me (IMO it will not run): Has it ever been tested on a Jolla 1?

olf ( 2020-06-17 15:48:44 +0300 )edit

Answer 5

0

answered 2020-06-18 00:24:13 +0300

speefak
306 ●12 ●24 ●21

updated 2020-06-18 00:31:23 +0300

I altered Media_Storage, External_Storage was from another testing script version. I changed this yet - sry.

My goal is a copy paste solution and you are right, this script has some prerequisites ( 2nd Gen and EXT4 formatted sdcard )

edit flag offensive delete publish link

Guide: Externalising android_storage and other directories / files to SD-card

1. Externalising regular files and directories in /home/nemo

2. Externalising /home/nemo/android_storage

2.1 Copying data to SD-card

2.2 Integrating an externalised android_storage when booting

2.2.1 Additional step for devices with AOSP4.4 based AlienDalvik

2.3 Finishing up

2.3.1 Removing the old content of android_storage residing on internal eMMC

Comments

5 Answers

Comments

Comments

Comments

Comments

Question tools

Stats

Related questions

Guide: Externalising android_storage and other directories / files to SD-card

1. Externalising regular files and directories in /home/nemo

2. Externalising /home/nemo/android_storage

2.1 Copying data to SD-card

2.2 Integrating an externalised android_storage when booting

2.2.1 Additional step for devices with AOSP4.4 based AlienDalvik

2.3 Finishing up

2.3.1 Removing the old content of android_storage residing on internal eMMC

Comments

5 Answers

Comments

Comments

Comments

Comments

Question tools

Public thread

Stats

Related questions