Guide: Externalising android_storage and other directories / files to SD-card
asked 2019-04-08 00:07:25 +0200
Although there are a couple of guides for extending the internal mass storage (eMMC) of devices running SailfishOS (in general; this is not addressing the "tiny system partition issue" of SailfishOS on non-BTRFS devices, i.e. all but the Jolla 1 phone) with an SD-card (on TMO and here on TJC), IMO none of them is completely satisfactory (technically, by functionality and usability), but they provided valuable ideas.
This is an enhanced, updated and streamlined guide created from the original "[How-to] Externalising android_storage and other directories / files to SD-card". It only covers SailfishOS 2.2.0 and newer.
For SailfishOS versions before 2.2.0 and the technical background of externalising android_storage, please consult the original how-to. The section topics and numbers are mostly the same for both guides.
- 1. Externalising regular files and directories in /home/nemo
- 2. Externalising /home/nemo/android_storage
This is part 2 "Externalising android_storage and other directories / files to SD-card", you may want to read part 1 "Creating partitions on SD-card, optionally encrypted" first.
All this was tested for years on Jolla 1 phones, Xperia Xs, XA2s and 10s, plus Gemini PDAs with SailfishOS versions from 2.2.0 to 3.2.1 and various SD-cards; if things should really turn out to look different on other devices (which is very unlikely, except for applying section 2 to the new, Android 8 based runtime environment, e.g. on an Xperia XA2 or 10), you may have to adapt the following sections (thus then some understanding of the commands issued is crucial); and last but not least this guide still may have flaws. Please report your experiences and adaptations in these cases, but do not blame anyone (e.g. me) for any mishaps.
Contributions and constructive suggestions are welcome.
1. Externalising regular files and directories in /home/nemo
For externalising android_storage, see section 2.
Mind using "hidden" directory names (starting with a ".") on the SD-card for externalised directories not to be indexed by the SailfishOS' Media Tracker service twice (there and in their original location, symlinked later in this guide).
Use regular directory names for all data, which solely resides on the SD-card (i.e. without being symlinked somewhere under /home/nemo/) and should be indexed by the Media Tracker.
As user nemo, execute the following commands:
cd /run/media/nemo/XXXXXXXX/
echo 'Data externalised from /home/nemo/ to SD-card resides in "hidden" directories starting with a ".".' > README-data.txt
mkdir .nemo-sdcard
cd
ln -s /run/media/nemo/XXXXXXXX/.nemo-sdcard SD-card
You may move existing, regular directories or subdirectories under /home/nemo/ (but not android_storage or anything under it, and one also shall not do that with "hidden" files or directories) onto SD-card by executing (you may want to back them up, first):
cd Foo
mkdir /run/media/nemo/XXXXXXXX/.foo
cp -av . /run/media/nemo/XXXXXXXX/.foo/ # Only continue, if copying has been successful
cd ..
rm -r Foo
ln -s /run/media/nemo/XXXXXXXX/.foo Foo
You can also externalise specific files (instead of whole directory trees) by executing (you may want to back them up, first):
cd Foo
mkdir /run/media/nemo/XXXXXXXX/.foo
cp -pv Bar /run/media/nemo/XXXXXXXX/.foo/ # Only continue, if copying has been successful
rm Bar
ln -s /run/media/nemo/XXXXXXXX/.foo/Bar Bar
Note that ".foo" and "Foo" are just a placeholders for specific directory names, "XXXXXXXX" is a placeholder for a specific mounted partition on the SD-card (rsp. technically: its UUID) and "Bar" is a placeholder for a specific file.
Arbitrary examples:
~nemo/Music --> /run/media/nemo/XXXXXXXX/.music
~nemo/Videos --> /run/media/nemo/XXXXXXXX/.videos
~nemo/Documents/Maps --> /run/media/nemo/XXXXXXXX/maps/osmscout/osmscout
~nemo/Maps/map.navit.bin --> /run/media/nemo/XXXXXXXX/maps/osm-maps/osm_bbox_X,Y,U,V.bin
2. Externalising /home/nemo/android_storage
Because AlienDalvik needs a filesystem providing classic UNIX access rights and ownerships, this section 2 does not work using a partition on SD-card formatted with (v)FAT.
You might utilise the "Guide: Creating partitions on SD-card, optionally encrypted" to employ a (at least one) proper, "native" filesystem on your SD-card before continuing.
- Devices with BTRFS as their native filesystem (e.g. Jolla 1 phones) use /data/media as AlienDalvik's media directory (i.e., the "virtual SD-card", which becomes bind-mounted at /home/nemo/android_storage) for their AOSP4.1.2 based Android runtime environment.
- "Second generation devices" employing EXT4 on LVM (e.g. Xperia X, Intex Aquafish, Jolla C, Jolla Tablet) with their AOSP4.4 based AlienDalvik all seem to use /home/nemo/android_storage directly (symlinked from /home/.android/data/media). Currently only tested with an Xperia X, please report differences on your "second generation device".
- "Third generation devices" with their AOSP8 based Android runtime environment (e.g. the Xperia 10 and XA2 families) are not analysed yet, currently lacking such a device.
Install mount-sdcard (e.g. per Storeman) to make automatic mounting on boot-up working reliably for AlienDalvik (i.e., to be race-free by a "Before=alien-service-manager.service" statement).
For automatic mounting of encrypted SD-cards, which have to be prepared as described in part 1 for externalising android_storage to them, have crypto-sdcard installed. Also running mount-sdcard is not really necessary in this case (when the content of android_storage resides on an encrypted partition), but it still provides some additional benefits.
2.1 Copying data to SD-card
Execute the following commands:
devel-su
systemctl stop alien-service-manager.service # Stop AlienDalvik completely
cd /home/nemo # cd /data for BTRFS-using devices
cp -av android_storage /run/media/nemo/XXXXXXXX/.android-sdcard # Pick the right UUID!
# cp -av media /run/media/nemo/XXXXXXXX/.android-sdcard # for BTRFS-using devices
touch /run/media/nemo/XXXXXXXX/.android-sdcard/ANDROID-SDCARD # Set a marker
chmod a= /run/media/nemo/XXXXXXXX/.android-sdcard/ANDROID-SDCARD # Make it persistent
Optionally check via checksums, that the data has been correctly written (needs rsync installed, per pkcon install rsync):
rsync -nc -avvhh /home/nemo/android_storage/ /run/media/nemo/XXXXXXXX/.android-sdcard/ | fgrep -v ' is uptodate' # Should detect no files not being up-to-date on "2nd gen" devices
rsync -nc -avvhh /data/media/ /run/media/nemo/XXXXXXXX/.android-sdcard/ | fgrep -v ' is uptodate' # Should detect no files not being up-to-date on BTRFS-using devices
2.2 Integrating an externalised android_storage when booting
Unfortunately there seems to be no proper way of integrating an externalised android_storage seamlessly into SailfishOS with AlienDalvik without altering one of its scripts.
Alter /opt/alien/system/script/platform_envsetup.sh by executing:
devel-su
cd /opt/alien/system/script/
cp -p platform_envsetup.sh platform_envsetup.sh.orig # Backup
Edit this file by adapting the MEDIA_STORAGE path variable, so that a diff platform_envsetup.sh.orig platform_envsetup.sh outputs (replace XXXXXXXX with the right UUID):
* On "2nd gen" devices:
20c20
< MEDIA_STORAGE=/home/nemo/android_storage
---
> MEDIA_STORAGE=/run/media/nemo/XXXXXXXX/.android-sdcard
* On BTRFS-using devices:
16c16
< MEDIA_STORAGE=$ANDROID_DATA/media
---
> MEDIA_STORAGE=/run/media/nemo/XXXXXXXX/.android-sdcard
Check (user- and group-) ownership and access rights with ls -l platform_envsetup.sh* (compared to the backed up original) and adapt them with chown and chmod, if necessary.
Finally backup your altered version with cp -p platform_envsetup.sh platform_envsetup.sh.patched!
Mind that the altered shell script may be overwritten by SailfishOS upgrades (as it happened when upgrading to SailfishOS 3.2.0, for the first time since 2.2.0), hence check for this then (at least when Android apps are failing to find their data after a SailfishOS upgrade) and reestablish the change (in the updated script file!), if overwritten (see timestamps with ls -l).
2.2.1 Additional step for devices with AOSP4.4 based AlienDalvik
On "2nd gen" devices (e.g. Xperia X, Jolla C, Intex Aquafish, Jolla Tablet) perform this additional step (i.e., not on BTRFS-using devices as the Jolla 1!), which can be done as user nemo (root user should also work):
cd /home/nemo
mv -v android_storage android_storage.bak
ln -s /run/media/nemo/XXXXXXXX/.android-sdcard android_storage
2.3 Finishing up
Reboot to see, if everything is working as intended: Check with your Android apps, if they access their data as before, as from their perspective everything should be the same as before externalising their data.
If in doubt, check if the relevant mounts succeeded:
Issue as user nemo a mount | fgrep mmcblk1 for unencrypted SD-cards respectively mount | fgrep crypto for encrypted SD-cards (employed per part 1): You should see each partition being mounted twice when AlienDalvik is running, once regularly at /run/media/nemo/XXXXXXXX as the mounted target (i.e., in the second field after the "on") and a second time at /opt/alien/run/media/nemo/XXXXXXXX for AlienDalvik.
- Additional check for BTRFS-using devices
Issue amount -t fuse: Again, you should see /dev/fuse being mounted twice when AlienDalvik is running, once regularly at /home/nemo/android_storage as the mounted target and at /opt/alien/home/nemo/android_storage for AlienDalvik.
If this does not yield the expected outcome, debug with journalctl -r as root user, while having less installed (per pkcon install less): Search with "/" for mmcblk1 rsp. crypto (if using an encrypted SD-card) (additionally on BTRFS-using devices: look for fuse), use "n" to jump to the next hit and press PageUp / PageDown to look at adjacent messages. Report your findings along with the SailfishOS version and the device you use in a comment here by describing your experience and pasting the output generated by above commands, which does not look as described.
2.3.1 Removing the old content of android_storage residing on internal eMMC
When working well, delete the old content of android_storage on internal eMMC, as this is the purpose of the whole procedure:
* On "2nd gen" devices:
devel-su
cd /home/nemo
rm -rf android_storage.bak
* On BTRFS-using devices:
devel-su
systemctl stop alien-service-manager.service # Stop AlienDalvik completely
cd /data/media
ls -la | more # Check: You should *not* see ANDROID-SDCARD
pwd # Check twice that you are really in the directory you cd'ed to two lines above!
rm -rf * .[^.]* # !!!
Done.
note that, like said, the AOSP 8.1 Oreo layer on the XA2, works in a completely different way:
- unlike X and J1, ~nemo/android_storage is actually empty.
- like them, the android data is in /home/.android
- on XA2, like on a real android, storage is mounted un /home/.android/data/media/0
- it's fuse mounted to ~nemo/android_storage while alien dalvik is running (and before running, any data in nemo/android_storage is imported in media/0 - for backward compatibility ?)
- i can presume that, once supported, SD in compatibility layer should appear in media/1 like on a real android ?
DrYak ( 2019-04-08 00:22:35 +0200 )editOh, that is half J1 and half "2nd gen" style:
- On a J1, ~nemo/android_storage is actually empty, too.
- But the J1 has its android data in /data
- /data/media is fuse mounted to ~nemo/android_storage while alien dalvik is running on a J1
- There already are artifacts of the Android SD compatibility layer on "2nd gen" devices
olf ( 2019-04-11 00:17:31 +0200 )edityou can just link the userdata storage dir from android emulator :
instead of moving android dir you copy the android storage dir to leave a backupdir on internal phone storage
if sudo is not installed use devel-su
speefak ( 2019-08-05 15:41:57 +0200 )edit@speefak, no, solely establishing a symlink does not do the job.
And why would one want to leave the (then stale) data on internal eMMC, when everything is working fine?
The primary reason for externalising data to SD-card is to free the internal eMMC of this data.
Side note: IMO backups should be carried out with a little thought, i.e. regularly and to external media (e.g. a USB stick per OTG, a PC, some cloud storage). Defining some left over, old data as a backup may lead to sorrow.
P.S.: Moving (per
olf ( 2020-06-17 17:18:25 +0200 )editmv) across filesystems performs exactly the same actions as copying (percp) and removing later (perrm), but is much more error prone (in case of a power loss, kernel panic, sudden reboot, inadvertently closing the Shell window etc.).