[How-to] Externalising android_storage and other directories / files to SD-card

asked 2018-02-12 00:26:04 +0300

olf
10393 ●104 ●201 ●158

updated 2019-07-13 18:27:38 +0300

Although there are a couple of guides for extending the internal mass storage (eMMC) of devices running SailfishOS (in general; this is not addressing the "tiny system partition issue" of SailfishOS on the Xperia X and Jolla C / Intex Aquafish) with a SD-card (on TMO and here on TJC), IMO none of them is completely satisfactory (technically, by functionality and usability), but they provided valuable ideas.

Note: For SailfishOS 2.2.0 and newer, a reworked, updated and more streamlined version of this how-to exists, adapting all occurrences of /media/sdcard to /run/media/nemo (and consequently /media/sdcard/XXXXXXXX to /run/media/nemo/XXXXXXXX ), as this guide was originally written for SailfishOS 2.1.x.

• 1. Externalising regular files and directories in /home/nemo
• 2. Externalising /home/nemo/android_storage
  • 2.1 Copying data to SD-card
  • 2.2 Integrating externalised android_storage for automatic startup
  • 2.3 Finishing up
  • 2.3.1 Removing data residing on internal eMMC
  • 2.4 Updating externalised data to internal eMMC or vice versa

This is part 2 "Externalising android_storage and other directories / files to SD-card", you may want to read part 1 "Creating partitions on SD-card, optionally encrypted" first.

This was tested on two Jolla 1 phones under SailfishOS versions 2.1.3 and 2.1.4 with various SD-cards; if things turn out to look different with other devices, you may have to adapt the following sections (thus then some understanding of the commands issued is crucial); and last but not least this description may be faulty. Please report your experiences and adaptations in these cases, but do not blame anyone (e.g. me) for any mishaps.
Contributions and constructive suggestions are welcome.

1. Externalising regular files and directories in /home/nemo

For externalising android_storage, see section 2.

Note using "hidden" directory names (starting with a ".") on the SD-card for files not being indexed by Tracker twice (there and in their original location).
Use regular directory names for all data, which solely resides on the SD-card (i.e. without a link somewhere under /home/nemo/) and should be indexed by the media tracker.

As user nemo, execute the following commands:

cd /media/sdcard/XXXXXXXX/
echo 'Data externalised from /home/nemo/ to SD-card resides in "hidden" directories starting with a ".".' > README-data.txt
mkdir .nemo-sdcard 
cd
ln -s /media/sdcard/XXXXXXXX/.nemo-sdcard SD-card

You may move existing, regular directories or subdirectories under /home/nemo/ (but not android_storage or "hidden" directories) onto SD-card by executing (you may want to back them up, first):

cd Foo
mkdir /media/sdcard/XXXXXXXX/.foo
cp -av . /media/sdcard/XXXXXXXX/.foo/  # Only continue, if copying has been successful 
cd ..
rm -r Foo
ln -s /media/sdcard/XXXXXXXX/.foo Foo

You can also externalise specific files (instead of whole directory trees) by executing (you may want to back them up, first):

cd Foo
mkdir  /media/sdcard/XXXXXXXX/.foo
cp -pv Bar /media/sdcard/XXXXXXXX/.foo/  # Only continue, if copying has been successful 
rm Bar
ln -s /media/sdcard/XXXXXXXX/.foo/Bar Bar

Note that ".foo" and "Foo" are just a placeholders for specific directory names, "XXXXXXXX" is a placeholder for a specific mounted partition on the SD-card and "Bar" is a placeholder for a specific file.

Arbitrary examples:
~nemo/Music --> /media/sdcard/XXXXXXXX/.music
~nemo/Videos --> /media/sdcard/XXXXXXXX/.videos
~nemo/Documents/Maps --> /media/sdcard/XXXXXXXX/maps/osmscout/osmscout
~nemo/Maps/map.navit.bin --> /media/sdcard/XXXXXXXX/maps/osm-maps/osm_bbox_X,Y,U,V.bin

2. Externalising /home/nemo/android_storage

Because AlienDalvik needs a filesystem providing classic UNIX access rights and ownerships, this section 2 does not work using a partition on SD-card formatted with (v)FAT.
Utilise the "[How-to] Creating partitions on SD-card, optionally encrypted" to employ a (at least one) proper, "native" filesystem on your SD-card before continuing. Also see the shortcuts provided for this lengthy how-to.

Mind that devices with BTRFS as their native filesystem (e.g. Jolla 1 phone) use /data as AlienDalvik's data directory, while all other devices (e.g. SailfishX, Intex Aquafish, Jolla C, Jolla Tablet) use /home/.android/data

Furthermore, the instructions in this section 2 are primarily suitable for BTRFS-using devices (i.e. Jolla 1) under SailfishOS versions before 2.2.0!
For all other devices and SailfishOS versions since 2.2.0, use the reworked, updated and streamlined version of this how-to and its newer section 2.

2.1 Copying data to SD-card

Execute the following commands:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
cd /data  # /home/.android/data for non-BTRFS devices
cp -av media /media/sdcard/XXXXXXXX/.android-sdcard  # Pick the right UUID!
touch /media/sdcard/XXXXXXXX/.android-sdcard/ANDROID-SDCARD  # Set a marker
chmod a= /media/sdcard/XXXXXXXX/.android-sdcard/ANDROID-SDCARD  # Make it persistent

In order to check via checksums, that the data has been correctly written (needs rsync installed):

rsync -nc -avvhh media/ /media/sdcard/XXXXXXXX/.android-sdcard/ | grep -vF " is uptodate"  # Should detect no files not being up-to-date

2.2 Integrating externalised android_storage for automatic startup

Unfortunately there seems to be no proper way of integrating an externalised /data/media (or /home/.android/data/media) directory (which later on becomes mounted on android_storage) seamlessly into SailfishOS with AlienDalvik without altering one of its scripts.
Mind that the altered shell script (a single one for either of the two methods following) may be overwritten by SailfishOS updates, hence check for this then and reestablish the changes (in the newly updated files!), if overwritten (see timestamps with ls -l).

2.2.a Method A: With a bind mount unit

This method falls back to the original content of /data/media/ (or /home/.android/data/media), if mounting it on the SD-card fails (this SD-card not present or died, unable to unlock encrypted partition etc.). So you will still be able to use Android apps with the older data on internal eMMC in this situation, but then the data on SD-card becomes "out of sync" with what the apps last wrote, hence failing may be preferred (see method B for this).

Note: For SailfishOS 2.2.0 and newer, please use method B, as method A would need to be adapted for these SailfishOS versions.
Furthermore, while both methods have been working fine for months, method B is easier to employ.

Alter /usr/sbin/apkd-mount by executing:

devel-su
cd /usr/sbin/
cp -p apkd-mount apkd-mount.orig  # Backup

Edit the second last line (insert "r") and add a new line after it (before the final "fi"), so that diff apkd-mount.orig apkd-mount outputs:

41c41,42
<     /bin/mount --bind "$DATADIR" "$MOUNTDIR"
---
>     /bin/mount --rbind "$DATADIR" "$MOUNTDIR"
>     /bin/mount --make-rslave "$MOUNTDIR"  # Without "rslave" it will not cleanly unmount

(See on using "rbind" and mount manpage.)

Backup your altered version with cp -p apkd-mount apkd-mount.patched
Check (user- and group-) ownership and access rights with ls -l apkd-mount* (compare to the backed up original) and adapt with chown and chmod, if necessary.

Create a new mount unit data-media.mount (for non-BTRFS devices: home-.android-data-media.mount) in /etc/systemd/system/, containing (you need to change =/data/media/ to =/home/.android/data/media/ twice for non-BTRFS devices):

[Unit]
Description=SD-card bind mount for AlienDalvik
ConditionPathExists=/dev/mmcblk1pX
ConditionPathExists=!/tmp/os-update-running
ConditionPathIsDirectory=/data/media/
ConditionPathIsMountPoint=/media/sdcard/XXXXXXXX/
Requires=local-fs.target systemd-user-sessions.service
Before=alien-service-manager.service
After=mount-sd@mmcblk1pX.service
BindsTo=mount-sd@mmcblk1pX.service

[Mount]
What=/media/sdcard/XXXXXXXX/.android-sdcard/
Where=/data/media/
Options=bind

[Install]
WantedBy=graphical.target aliendalvik.service

Mind to substitute all "X" with appropriate values for your configuration.
Check with ls -l that the (user- and group-) ownership and access rights for data-media.mount are as for the other unit files here (i.e. "-rw-r--r-- root root data-media.mount" on a Jolla 1 under SFOS 2.1.3.7) and adapt with chown and chmod, if necessary.
Finally activate this mount unit with: systemctl enable data-media.mount # home-.android-data-media.mount for non-BTRFS devices

Note that "Before=alien-service-manager.service" starts this mount unit quite early (possibly prolonging boot time), because alien-service-manager.service calls (among other things) /opt/alien/system/script/alien_service_manager.sh, which in turn calls the binary executable $ANDROID_ROOT/bin/servicemanager ($ANDROID_ROOT = /opt/alien/system): As this is a binary file, there is no easy way to determine, if it relies on /opt/alien/data/media to be static.
While by simple testing it appears to be O.K. to use "Before=aliendalvik.service" (which is significantly later), between "alien-service-manager.service" and "aliendalvik.service" "alien-settings.service" is started, which poses a similar issue by calling /opt/alien/system/script/alien_settings_server.sh, which in turn calls the binary executable $ANDROID_ROOT/genv/bin/alien_settings_server: Also unable to determine if it relies on /opt/alien/data/media to be static.
By reverse-engineering the somewhat convoluted call-chains from the systemd unit files to the scripts in /opt/alien/system/script/ and further, using "Before=aliendalvik.service" in above mount unit is O.K., except for the aforementioned two binaries, for which it is unknown. So I prefer to be on the safe side by starting above mount unit early.

2.2.b Method B: By setting MEDIA_STORAGE in /opt/alien/system/script/platform_envsetup.sh

This alternative method redirects Android's "data/media" directory directly to SD-card, so it will not fall back to the original content of /data/media/ (or /home/.android/data/media/), if mounting it on the SD-card fails (i.e. specific SD-card not present or died, unable to unlock encrypted partition etc.): In this case a new, empty, "hidden" directory .android-sdcard is created in the (then not mounted) mount-directory /media/sdcard/XXXXXXXX/ on internal eMMC.
This is the preferable method, if you plan to delete the original content of /data/media/ (or /home/.android/data/media/) completely (see section 2.4.1) or to configure a freshly installed device.

Alter /opt/alien/system/script/platform_envsetup.sh by executing:

devel-su
cd /opt/alien/system/script/
cp -p platform_envsetup.sh platform_envsetup.sh.orig  # Backup

Edit line 16, so that diff platform_envsetup.sh.orig platform_envsetup.sh outputs (replace XXXXXXXX with the right UUID):

16c16
< MEDIA_STORAGE=$ANDROID_DATA/media
---
> MEDIA_STORAGE=/media/sdcard/XXXXXXXX/.android-sdcard

Backup your altered version with cp -p platform_envsetup.sh platform_envsetup.sh.patched
Check (user- and group-) ownership and access rights with ls -l platform_envsetup.sh* (compared to the backed up original) and adapt with chown and chmod, if necessary.

If you are using SailfishOS 2.2.0 or newer and an unencrypted SD-card, please install mount-sdcard (e.g., per Storeman) to make automatic mounting on boot-up working reliably (i.e., to be race-free by a "Before=alien-service-manager.service" statement) for method B.
For automatic mounting of encrypted SD-cards, install (additionally or alternatively) crypto-sdcard, when using SailfishOS 2.2.0 or newer. For older versions of SailfishOS one has to employ these changes manually, as described in section 4.3.5 of "[How-to] Creating partitions on SD-card, optionally encrypted" for either method (A or B) to use encrypted SD-cards.

2.3 Finishing up

Reboot to see, if everything working as intended. If in doubt, issue as user nemo mount | grep mmcblk1pX as a first check.
If needed, debug with journalctl -r as root user, while having less installed (e.g. per pkcon install less): Search with "/" (e.g. for card bind or mmcblk1pX), use "n" to jump to the next hit and press PageUp / PageDown to look at adjacent messages.
When using method A, also check output of systemctl status data-media.mount # home-.android-data-media.mount for non-BTRFS devices

2.3.1 Removing data residing on internal eMMC

When working well, you may delete large files / directories (or everything) in the original /data/media/:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
umount /data/media  # /home/.android/data/media/ for non-BTRFS devices
cd /data/media/  # Ditto!
ls -la | more  # Check: You should *not* see ANDROID-SDCARD
# Delete large files, downloads etc., or empty completely with: rm -rf * .[^.]*

Before emptying /data/media/ (or /home/.android/data/media/) completely, mind that this is the fallback data for method A, when the partition on the SD-card is not available. So a deleting everything there is fine, as long you do not rely on Android apps (or yourself) to have this data available without SD-card.

When done, reboot.

2.4 Updating externalised data to internal eMMC or vice versa

To update subdirectories or files in the original /data/media/ (or /home/.android/data/media/) with newer data from /media/sdcard/XXXXXXXX/.android-sdcard/ (or vice versa) you may use:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
umount /data/media  # /home/.android/data/media for non-BTRFS devices 
cp -auv /media/sdcard/XXXXXXXX/.android-sdcard/osmand /data/media/osmand  # Just an example; /home/.android/data/media/osmand for non-BTRFS devices; may also take a look at the option -n in cp's manpage; switch source and destination directories for updating the other way around

When done, reboot.