[How-to] Externalising android_storage and other directories / files to SD-card

asked 2018-02-12 00:26:04 +0200

olf gravatar image

updated 2018-02-19 03:24:20 +0200

Although there are a couple of guides for extending the internal mass storage (eMMC) of devices running SailfishOS (in general; this is not addressing the "tiny system partition issue" of SailfishOS on the Xperia X and Jolla C / Intex Aquafish) with a SD-card (on TMO and here on TJC), IMO none of them is completely satisfactory (technically, by functionality and usability), but they provided valuable ideas.

This is part 2 "Externalising android_storage and other directories / files to SD-card", you may want to read part 1 "Creating partitions on SD-card, optionally encrypted" first.

This was tested on two Jolla 1 phones under SailfishOS 2.1.3.7 with two different SD-cards; if things turn out to look different with other devices, you may have to adapt the following sections (thus then some understanding of the commands issued is crucial); and last but not least this description may be faulty. Please report your experiences and adaptations in these cases, but do not blame anyone (e.g. me) for any mishaps.
Contributions and constructive suggestions are welcome.

1. Externalising regular files and directories in /home/nemo

For externalising android_storage, see section 2.

Note using "hidden" directory names (starting with a ".") on the SD-card for files not being indexed by Tracker twice (there and in their original location).
Use regular directory names for all data, which solely resides on the SD-card (i.e. without a link somewhere under /home/nemo/) and should be indexed by the media tracker.

As user nemo, execute the following commands:

cd /media/sdcard/XXXXXXXX/
echo 'Data externalised from /home/nemo/ to SD-card resides in "hidden" directories starting with a ".".' > README-data.txt
mkdir .nemo-sdcard 
cd
ln -s /media/sdcard/XXXXXXXX/.nemo-sdcard SD-card

You may move existing, regular directories or subdirectories under /home/nemo/ (but not android_storage or "hidden" directories) onto SD-card by executing (you may want to back them up, first):

cd Foo
mkdir /media/sdcard/XXXXXXXX/.foo
cp -av . /media/sdcard/XXXXXXXX/.foo/  # Only continue, if copying has been successful 
cd ..
rm -r Foo
ln -s /media/sdcard/XXXXXXXX/.foo Foo

You can also externalise specific files (instead of whole directory trees) by executing (you may want to back them up, first):

cd Foo
mkdir  /media/sdcard/XXXXXXXX/.foo
cp -pv Bar /media/sdcard/XXXXXXXX/.foo/  # Only continue, if copying has been successful 
rm Bar
ln -s /media/sdcard/XXXXXXXX/.foo/Bar Bar

Note that ".foo" and "Foo" are just a placeholders for the specific directory names, "XXXXXXXX" is a placeholder a specific mounted partition on the SD-card and "Bar" is a placeholder for a specific file.

Arbitrary examples:
~nemo/Music --> /media/sdcard/XXXXXXXX/.music
~nemo/Videos --> /media/sdcard/XXXXXXXX/.videos
~nemo/Documents/Maps --> /media/sdcard/XXXXXXXX/maps/osmscout/osmscout
~nemo/Maps/map.navit.bin --> /media/sdcard/XXXXXXXX/maps/osm-maps/osm_bbox_X,Y,U,V.bin

2. Externalising ~nemo/android_storage

Mind that devices with BTRFS as their native filesystem (e.g. Jolla 1 phone) use /data as AlienDalvik's data directory, while all other devices (e.g. SailfishX, Intex Aquafish, Jolla C, Jolla Tablet) use /home/.android/data

2.1 Copy data to SD-card

Execute the following commands:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
cd /data  # /home/.android/data for non-BTRFS devices
cp -av media /media/sdcard/XXXXXXXX/.android-sdcard  # Pick the right UUID!
touch /media/sdcard/XXXXXXXX/.android-sdcard/ANDROID-SDCARD  # Set a marker
chmod a= /media/sdcard/XXXXXXXX/.android-sdcard/ANDROID-SDCARD  # Make it persistent

2.2 Integrating externalised android_storage for automatic startup

Unfortunately there seems to be no proper way of integrating an externalised /data/media (or /home/.android/data/media) directory (which later on becomes mounted on android_storage) seamlessly into SailfishOS with AlienDalvik without altering one of its scripts.
Mind that the altered shell script (a single one for either of the two methods following) may be overwritten by SailfishOS updates, hence check for this then and reestablish the changes (in the newly updated files!), if overwritten (see timestamps with ls -l).

2.2.a Method A: With a bind mount unit

This method falls back to the original content of /data/media/ (or /home/.android/data/media), if mounting it on the SD-card fails (this SD-card not present or died, unable to unlock encrypted partition etc.). So you will still be able to use Android apps with the older data on internal eMMC in this situation, but then the data on SD-card becomes "out of sync" with what the apps last wrote, hence failing may be preferred (see method B for this).

Alter /usr/sbin/apkd-mount by executing:

devel-su
cd /usr/sbin/
cp -p apkd-mount apkd-mount.orig  # Backup

Edit the second last line (insert "r") and add a new line after it (before the final "fi"), so that diff apkd-mount.orig apkd-mount outputs:

41c41,42
<     /bin/mount --bind "$DATADIR" "$MOUNTDIR"
---
>     /bin/mount --rbind "$DATADIR" "$MOUNTDIR"
>     /bin/mount --make-rslave "$MOUNTDIR"  # Without "rslave" it will not cleanly unmount

(See on using "rbind" and mount manpage.)

Backup your altered version with cp -p apkd-mount apkd-mount.patched
Check (user- and group-) ownership and access rights with ls -l apkd-mount* (compare to the backed up original) and adapt with chown and chmod, if necessary.

Create a new mount unit data-media.mount (for non-BTRFS devices: home-.android-data-media.mount) in /etc/systemd/system/, containing (you need to change =/data/media/ to =/home/.android/data/media/ twice for non-BTRFS devices):

[Unit]
Description=SD-card bind mount for AlienDalvik
ConditionPathExists=/dev/mmcblk1pX
ConditionPathExists=!/tmp/os-update-running
ConditionPathIsDirectory=/data/media/
ConditionPathIsMountPoint=/media/sdcard/XXXXXXXX/
Requires=local-fs.target systemd-user-sessions.service
Before=alien-service-manager.service
After=mount-sd@mmcblk1pX.service
BindsTo=mount-sd@mmcblk1pX.service

[Mount]
What=/media/sdcard/XXXXXXXX/.android-sdcard/
Where=/data/media/
Options=bind

[Install]
WantedBy=graphical.target aliendalvik.service

Mind to substitute all "X" with appropriate values for your configuration.
Check with ls -l that the (user- and group-) ownership and access rights for data-media.mount are as for the other unit files here (i.e. "-rw-r--r-- root root data-media.mount" on a Jolla 1 under SFOS 2.1.3.7) and adapt with chown and chmod, if necessary).
Finally activate this mount unit with: systemctl enable data-media.mount # home-.android-data-media.mount for non-BTRFS devices

Note that "Before=alien-service-manager.service" starts this mount unit quite early (possibly prolonging boot time), because alien-service-manager.service calls (among other things) /opt/alien/system/script/alien_service_manager.sh, which in turn calls the binary executable $ANDROID_ROOT/bin/servicemanager ($ANDROID_ROOT = /opt/alien/system): As this is a binary file, there is no easy way to determine, if it relies on /opt/alien/data/media to be static.
While by simple testing it appears to be O.K. to use "Before=aliendalvik.service" (which is significantly later), between "alien-service-manager.service" and "aliendalvik.service" "alien-settings.service" is started, which poses a similar issue by calling /opt/alien/system/script/alien_settings_server.sh, which in turn calls the binary executable $ANDROID_ROOT/genv/bin/alien_settings_server: Also unable to determine if it relies on /opt/alien/data/media to be static.
By reverse-engineering the somewhat convoluted call-chains from the systemd unit files to the scripts in /opt/alien/system/script/ and further, using "Before=aliendalvik.service" in above mount unit is O.K., except for the aforementioned two binaries, for which it is unknown. So I prefer to be on the safe side by starting above mount unit early.

2.2. Method B: Setting MEDIA_STORAGE in /opt/alien/system/script/platform_envsetup.sh

This alternative method redirects Android's "data/media" directory directly to SD-card, so it will not fall back to the original content of /data/media/ (or /home/.android/data/media/), if mounting it on the SD-card fails (i.e. specific SD-card not present or died, unable to unlock encrypted partition etc.): In this case a new, empty, "hidden" directory .android-sdcard is created in the (then not mounted) mount-directory /media/sdcard/XXXXXXXX/ on internal eMMC.
This is the preferable method, if you plan to delete the original content of /data/media/ (or /home/.android/data/media/) completely (see section 2.4.1) or to configure a freshly installed device.

Alter /opt/alien/system/script/platform_envsetup.sh by executing:

devel-su
cd /opt/alien/system/script/
cp -p platform_envsetup.sh platform_envsetup.sh.orig  # Backup

Edit line 16, so that diff platform_envsetup.sh.orig platform_envsetup.sh outputs (replace XXXXXXXX with the right UUID):

16c16
< MEDIA_STORAGE=$ANDROID_DATA/media
---
> MEDIA_STORAGE=/media/sdcard/XXXXXXXX/.android-sdcard

Backup your altered version with cp -p platform_envsetup.sh platform_envsetup.sh.patched
Check (user- and group-) ownership and access rights with ls -l platform_envsetup.sh* (compared to the backed up original) and adapt with chown and chmod, if necessary.

2.3 Finishing up

Reboot to see, if everything working as intended. If in doubt, issue as user nemo mount | grep mmcblk1pX as a first check.
If needed, debug with journalctl -r as root user, while having less installed (e.g. per pkcon install less): Search with "/" (e.g. for card bind or mmcblk1pX), use "n" to jump to the next hit and press PageUp / PageDown to look at adjacent messages.
When using method A, also check output of systemctl status data-media.mount # home-.android-data-media.mount for non-BTRFS devices

2.3.1 Removing data residing on internal eMMC

When working well, you may delete large files / directories (or everything) in the original /data/media/:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
umount /data/media  # /home/.android/data/media/ for non-BTRFS devices
cd /data/media/  # Ditto!
ls -la | more  # Check: You should *not* see ANDROID-SDCARD
# Delete large files, downloads etc., or empty completely with: rm -rf * .[^.]*

Before emptying /data/media/ (or /home/.android/data/media/) completely, mind that this is the fallback data for method A, when the partition on the SD-card is not available. So a deleting everything there is fine, as long you do not rely on Android apps (or yourself) to have this data available.

When done, reboot.

2.4 Updating externalised data to internal eMMC or vice versa

To update subdirectories or files in the original /data/media/ (or /home/.android/data/media/) with newer data from /media/sdcard/XXXXXXXX/.android-sdcard/ (or vice versa) you may use:

devel-su
systemctl stop alien-service-manager.service  # Stop AlienDalvik completely
umount /data/media  # /home/.android/data/media for non-BTRFS devices 
rsync -av /media/sdcard/XXXXXXXX/.android-sdcard/osmand /data/media/osmand  # Just an example; /home/.android/data/media/osmand for non-BTRFS devices.

When done, reboot.

edit retag flag offensive close delete

Comments

Richly instructive, thanks man, and as you pointed out it is a different approach to the one I have been using on my Jolla 1 since SFOS 1.1.7 which is upgrade proof but with the shortcoming of android storage on the SD Card not surviving a lipstick refresh or android support restart, a full restart is always required for the mount point to be effective once again. Is this shortcoming taken care of with this hack? Well done once again

DameCENO ( 2018-02-12 15:10:47 +0200 )edit

No such "shortcomings" was an achieved goal.

olf ( 2018-02-12 16:37:38 +0200 )edit

Good to know, I have another alternative to try. The work involved in starting afresh from my SDXC card scares me. Thanks once more for this guide I will definitely give it a go

DameCENO ( 2018-02-14 13:23:13 +0200 )edit

If you have android_storage and other data already on SD-card, why not reusing it for this solution (by adapting it)?

olf ( 2018-02-14 18:01:13 +0200 )edit
3

I'm not going to fiddle with these parts of the system. Jolla should finally listen to the user's complaints and make a fundamental and strategic change. With Android, when inserting the SD card, you are asked if you want to use it as extended memory. That's what Jolla should finally do.

Moo-Crumpus ( 2018-02-14 18:18:53 +0200 )edit