We have moved to a new Sailfish OS Forum. Please start new discussions there.
52

[How-to] Creating partitions on SD-card, optionally encrypted

asked 2018-02-11 23:20:08 +0300

olf gravatar image

updated 2019-09-20 16:29:19 +0300

Although there are a couple of guides for extending the internal mass storage (eMMC) of devices running SailfishOS (in general; this is not addressing the "tiny system partition issue" of SailfishOS on the Xperia X and Jolla C / Intex Aquafish) with a SD-card (on TMO and here on TJC), IMO none of them completely satisfactory (technically, by functionality and usability), but they provided valuable ideas.

Note: For SailfishOS 2.2.0 and newer, a reworked, updated and more streamlined version of this guide exists. This original how-to is kept for providing variations, details, technical background etc., and is still valid for SailfishOS 2.1.x.

This is part 1 "Creating partitions on SD-card, optionally encrypted", you may want to continue with part 2 "Externalising android_storage and other directories / files to SD-card" to utilise the space on SD-card thereafter.

0. Objectives and preface

  1. Primarily rely on SailfishOS provided tools, including Mer-Tools (but without using another device, e.g. a PC), so the whole configuration can be carried out directly on the SailfishOS device.
  2. Not altering SailfishOS' basic filesystem structure and avoid altering existing files. Thus no /home/nemo etc. on SD-card.
  3. But with the ability to move android_storage and other files and directories under /home/nemo to an (optionally encrypted) partition on the SD-card (see part 2 for this).
  4. A solution for all devices with SailfishOS, so a SD-card can be moved between different devices, while keeping data on it accessible. Hence the filesystems used are primarily EXT4 and vFAT32, because this is what Jolla officially supports, but you may alternatively use BTRFS or F2FS, if compiled into your SailfishOS kernel (see cat /proc/filesystems) and broad interoperability is not a goal.
  5. Protect partition(s) on SD-card by encryption, in order to keep the data on it safe, when the device is lost. I.e. the keys for DM-Crypt are based on a file stored at the internal mass store, which is accessible, when the SailfishOS device is unlocked. Additionally, with access to that file, one can copy it to other SailfishOS devices or Linux PCs to access the data on the SD-card there, too.
  6. And / or create unencrypted partition(s) for data exchange with any device and OS (by physically moving the SD-card), as "spare area" and for Jolla's Backup GUI-utility.
  7. Applicable to SDHC- and SDXC-cards of at least 4 GB (up to the maximum size the device hardware supports, as the technical limit of this scheme is 2 TBytes).
  8. Keep the reserved area of the SD-card intact.
  9. Align partition, DM-Crypt and filesystem data-structures to erase-block size / optimal alignment size.

Background on aligning data-structures on FLASH-based mass storage:
Write accesses to any NAND-FLASH based mass storage must write whole (complete) pages internally (typically 2^n multiples of 4 KBytes large), while for deleting data structures whole erase-blocks must be deleted. Note that the logical sector size such media exposes externally (usually 512 Bytes per logical sector) is much smaller. Recent NAND-FLASH memory chips (all FLASH-based mass storage is built with NAND-FLASH, plus a controller circuitry) have erase-block sizes of 1, 2, 4, 8, ... MBytes (while the numbers at Wikipedia refer to NAND-FLASH a couple of years ago, citing e.g. 512 KBytes; see also e.g. this explanation).
Not aligning data-structures to the erase-block size (or multiples of it) results in a much higher write amplification, leading to vastly degraded lifetime and lower performance of a reformatted SD-card.

README 1st:
Before you start, read the following sections completely and decide thereafter, if you (at least) roughly understand what these commands do (you may read their man pages, to be sure); if you don't, better leave it.
Mistyping (e.g. /dev/mmcblk0 instead of /dev/mmcblk1) may brick your SailfishOS device or quickly wear out your SD-card!
All this was tested on Jolla 1 phones and Xperia Xs under SailfishOS 2.1 and 2.2 (also should be applicable to SailfishOS 2.0) with multiple SD-cards, so if things turn out to look different with other devices or SD-cards, you may have to adapt the following sections (thus some understanding of the commands issued is crucial); and last but not least this description may be faulty. Please report your experiences and adaptations in these cases, but do not blame anyone (e.g. me) for any mishaps.
Contributions and constructive suggestions are welcome.

1. Preparation (installing needed utilities etc.)

Starting as user nemo in /home/nemo.
mkdir sd-card_layouts # Create a directory for backing up the original and altered partition and file system layouts of the SD-card
devel-su # Become root user
ssu ar mer-tools # Add Jolla's mer-tools repository to /etc/ssu/ssu.ini; Only needed for SailfishOS versions below 2.2.1
pkcon refresh # As usual, after adding a new repository
pkcon install util-linux # fdisk is needed for analysing and partitioning a SD-card; it has been moved from a standalone package to util-linux SailfishOS 2.1.4

2 Analysing a specific SD-card

2.1 Analysing a SD-card's partition layout

cd sd-card_layouts # Change to the backup directory created in section 1

First take a look at the partition layout with fdisk:
fdisk -l /dev/mmcblk1 | tee fdisk-l_mmcblk1-original.txt
Output of a 32 GB SDHC-card as an example:

Disk /dev/mmcblk1: 29 GiB, 31104958464 bytes, 60751872 sectors
Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device         Boot Start      End  Sectors Size Id Type
/dev/mmcblk1p1       8192 60751871 60743680  29G  c W95 FAT32 (LBA)

Note that there is (typically) a single FAT partition at /dev/mmcblk1p1 (some SD-cards may have a different partition layout, though) and the first LBA sector of this FAT partition is 8192 (which may vary depending upon the specific SD-card). One should retain this offset for the first partition (in this example of 8192 * 512 Bytes = 4 MBytes) for optimal alignment (see below).
Also note, that SD-cards usually "lie" about their internal page size for compatibility reasons and just report their externally provided logical sector size as physical sector size and optimal I/O size, even though the true page size is optimal for read and write accesses (while the erase-block size is optimal for deleting data).

2.2.a Analysing an existing FAT32 filesystem on a SDHC-card

To analyse a SDHC-card's pre-formatted FAT32 filesystem (advertised size typically "4 GB" to "32 GB"), use dosfsck (from the dosfstools):
pkcon install dosfstools
dosfsck -nv /dev/mmcblk1p1 | tee dosfsck-nv_mmcblk1p1-original.txt
Exemplary output of aforementioned 32 GB SD-card:

dosfsck 3.0.10 (12 Sep 2010)
dosfsck 3.0.10, 12 Sep 2010, FAT32, LFN
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "        "
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
     32768 bytes per cluster
     1556 reserved sectors
First FAT starts at byte 796672 (sector 1556)
         2 FATs, 32 bit entries
   3795968 bytes per FAT (= 7414 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 8388608 (sector 16384)
    948864 data clusters (31092375552 bytes)
63 sectors/track, 255 heads
      8192 hidden sectors
  60743680 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/mmcblk1p1: 3 files, 2507/948864 clusters

Note that the two File Allocation Tables (FATs) of 7414 sectors each plus the 1556 reserved sectors make up for the file system internal offset of 16384 sectors, at which the data area of this FAT partition starts.
To check that the whole calculation matches, multiply the size of the data area with the cluster size in sectors (i.e. 64 for the typical 32 KBytes cluster size and 512 Bytes logical sector size) and add the file system internal offset: 948864 * 64 + 16384 = 60743680, which is the number of total sectors reported by fdisk and dosfsck for this partition.
Note that the number of hidden sectors refers to the number of sectors preceding this FAT partition (i.e. outside and before the FAT partition proper) and hence shall equal its first sector number as reported by fdisk (i.e. its offset).

2.2.b Analysing an existing exFAT filesystem on a SDXC-card

To analyse a SDXC-card's pre-formatted exFAT filesystem (advertised size typically "64 GB" or larger), use dumpexfat (from the exfat-utils):
Install exfat-utils from OpenRepos (e.g. via Storeman), then execute:
dumpexfat /dev/mmcblk1p1 | tee dumpexfat_mmcblk1p1-original.txt
Exemplary output of a 128 GB SD-card:

Volume label                        
Volume serial number      0x64323432
FS version                       1.0
Sector size                      512
Cluster size                  131072
Sectors count              249704448
Free sectors               249670912
Clusters count                975280
Free clusters                 975277
First sector                   32768
FAT first sector               16384
FAT sectors count              16384
First cluster sector           32768
Root directory cluster             4
Volume state                  0x0000
FATs count                         1
Drive number                    0x80
Allocated space                   0%

The values correspond to the ones described in section 2.2.a (for dosfsck):

  • "First sector" (dumpexfat) equals "hidden sectors" (dosfsck), i.e. partition offset.
  • "FAT first sector" equals "First FAT starts at ... (sector X)", i.e. "reserved sectors".
  • "FAT sectors count" equals "Y bytes per FAT (= X sectors)".
  • "First cluster sector" equals "Data area starts at byte Y (sector X)".
  • The cluster size is provided by both (dumpexfat and dosfsck) in Bytes (not sectors!).

Note that typically just a single FAT is used for exFAT filesystems, as this is sufficient for FLASH-based media (the typically two FATs used in pre-formatted FAT32 filesystems on SD-cards are primarily a compatibility measure).

Check your understanding of the original filesystem layout as described in section 2.2.a.

2.3 Determining the optimal alignment size for a specific SD-card

A good estimate for the SD-card's optimal alignment size is the smaller of these two offsets, the partition offset (in this example: 8192 sectors = 4 MB) and the FAT file system internal offset for its data area (in this example: 16384 sectors = 8 MB, due to the FATs occupying more than 4 MB in this case). The data area offset should be a multiple (1, 2, 3, ...) of the partition offset, otherwise your SD-card is strangely formatted.
To determine the SD-card's erase-block size, look at the output of cat /sys/block/mmcblk1/queue/discard_granularity and especially cat /sys/block/mmcblk1/device/preferred_erase_size, if your SD-card provides these values (in Bytes).
Note that the SD-card's optimal alignment size is a 2^n multiple (i.e. times 1, 2, 4, 8, ...) of the erase-block size, so they are not necessarily the same.

All in all it should be plausible and fit together:

  • For the exemplary 32 GB SD-card, the partition offset is 4 MB, the data area offset 8 MB (due to the size of the FATs), discard_granularity is 4 MB and preferred_erase_size 4 MB: Hence the optimal alignment size is clearly 4 MB for this SD-card.
  • For the 128 GB SD-card mentioned above, the partition offset is 16 MB, the data area offset 16 MB too, but discard_granularity and preferred_erase_size are 4 MB: So, while the erase-block size is 4 MB, the manufacturer clearly prefers an alignment size of 16 MB for this SD-card.

Thus one goal is to align all significant data-structures to the determined optimal alignment size (e.g. for the exemplary 32 GB SD-card: 8192 sectors with 512 Bytes = 4 MBytes; for the aforementioned 128 GB SD-card: 32768 sectors with 512 Bytes = 16 MBytes) in the following sections.

Check, if your SD-card supports discards:
When cat /sys/block/mmcblk1/queue/discard_zeroes_data outputs "1" it definitely does, otherwise it likely does not.

Note, that all files on the SD-card will become inaccessible when carrying out the next steps, so now is the last chance to backup the data residing on your SD-card.

3. Re-partitioning of the SD-card

3.1 Preparation for re-partitioning

Note: For SailfishOS 2.2.0 (and newer) the path /media/sdcard/XXXXXXXX/ must be altered to /run/media/nemo/XXXXXXXX/ twice in section 3.1 (and in general).

Empty the existing filesystem(s) on SD-card completely in order to free the used blocks on the filesystem and the SD-card internally:
cd /media/sdcard/XXXXXXXX/ # Assure that you changed to this directory successfully, before proceeding:
pwd # O.K.?
rm -rf * .[^.]* # !
fstrim -v /media/sdcard/XXXXXXXX/ # For all filesystems other than FAT; fstrim does not work on FAT filesystems, but for FAT the mount option "discard" (used by SailfishOS) shall discard all empty blocks when executing the rm command.
Alternatively or additionally (preferably after having carried out the following paragraph: the unmounting), a blkdiscard -v /dev/mmcblk1pX can be used on the raw, logical partitions (but be careful with the whole device /dev/mmcblk1).

Check, which partition(s) is actually mounted and how often:
mount | grep mmcblk1 # Each partition is mounted twice (at different locations), if you have AlienDalvik running
Unmount the mounted partition(s) on the SD-card by issuing, e.g.:
umount /dev/mmcblk1p1 # If the output is "umount: /dev/mmcblkXXX: not mounted", you picked the wrong partition
mount | grep mmcblk1 # Check again, it should be mounted once less, now
Continue to issue umount /dev/mmcblk1p1 until the output is "umount: /dev/mmcblkXXX: not mounted" (and consequently mount | grep mmcblk1 outputs nothing).

Become familiar with the fdisk tool:
fdisk /dev/mmcblk1

  • p lists the partition layout as obtained in section 2
  • m provides the list of available commands
  • Do not set the "DOS compatibility flag" (with the command c), which is solely needed for computers running a real MS-DOS and will result in maximised write-amplification when activated (but it defaults to "off" in GNU fdisk since very long ago: more than 15 years).
  • If you have the impression anything went wrong, simply exit fdisk (without writing the altered partition table) via q and start anew.

3.2.a Setting up a single FAT32, Linux-native (e.g. EXT4) or encrypted partition

If you do not want to create two or three partitions, just leave the partition layout as it is and solely alter the partition type (even that is optional): Hit t, by default the only existing partition is selected and set its partition type to 83 ("Linux") or if you just want to reformat an SDXC-card (pre-formatted with exFAT) with FAT32 to c. Alternatively (e.g. for maximising plausible deniability when planning to set up encryption on this partition and facing the danger to be forced to reveal the encryption scheme / key-file, e.g. in the UK) you may leave it as it is (i.e. not changing anything with fdisk) or set it to 0 ("Unpartitioned"), which both should work, although other devices and operating systems may fail to recognise and to mount them automatically then (but e.g. SailfishOS has absolutely no issues automatically mounting a filesystem on a partition with a wrong type, as it probes them) or even try to reformat them.
Check by entering p that the partition table now looks as it should, and finally write it to the SD-card (and exit fdisk) via w (or q, if unaltered).

3.2.b Setting up two or three partitions: A FAT32 or native one plus a native or / and encrypted one

Prerequisites for FAT32 partitions:

  • When creating a FAT32 partition (type 0c) on a SD-card, it must become at least 16450560 sectors (with 512 Bytes each, i.e. 7,8 GBytes) large. Note that, although smaller size is theoretically possible, it does not make much sense to create such a small FAT partition IMO and requires a different partition type (0b) (to conform to the SD-card specification), which may provide additional hurdles, as that type usually indicates a non-LBA FAT32 partition (hence untested).
  • The maximum size for a FAT32 partition is 2 TB (no exFAT necessary for this, but the latter allows for file sizes larger than 2 GB).
  • A FAT partition should be the first primary partition on the SD-card, otherwise other operating systems may not recognise it.
  • Thus for a dual partition layout including a FAT partition, the SD-card should be at least a "16 GB" one.

Continue in the fdisk /dev/mmcblk1 and delete the existing partition with d.

Create a new, smaller first primary partition:

  • Create a new first primary partition by entering n, then p, then use the default of "1". Recreate this partition with the same starting LBA-sector the original partition had (which has been obtained and saved in section 2, in the example of the aforementioned 32 GB SD-card 8192).
  • As an easy optimisation (without any drawbacks), determine the last sector for the first partition (especially if it is going to be a FAT partition) by subtracting 1 from a desired size of n GBytes in sectors (hence the multiplier is 1024 * 1024 * 2 = 2097152 for 512 Byte sectors). In my example (still with that 32 GB SD-card with an offset of 8192 sectors for the first partition) and aiming at creating a first primary partition of slightly less than 8 GBytes, this results in 8 * 2097152 - 1 = 16777215 as the last sector for the new partition (and a partition size of 16777215 +1 - 8192 = 16769024 sectors = 7,996 GBytes).
    This measure also ensures that the second partition starts at an n GByte boundary.
  • Then enter this calculated value as the last sector for the new first partition.
  • As Linux fdisk always sets the partition type of newly created partitions to 83 ("Linux"), hit t afterwards and set the partition type to c ("W95 FAT32 (LBA)"), if you want a FAT32 partition. If you want a native, unencrypted partition as first primary partition, leave it as it is (on 83).

Create a new, second partition (which will be configured later to be used either natively or with DM-Crypt):

  • Start creating a new, second primary partition by entering n, then p for another primary partition (default, you may as well hit the Return key), then use the suggestion of "2".
  • Create this partition with the last LBA-sector of the first primary partition plus 1 as starting LBA-sector (e.g. with the exemplary 32 GB SD-card: 16777216). Note that fdisk will usually suggest a low sector number in the reserved ("hidden") area (e.g. 2048), which must not be used.
  • Use the value obtained for the last sector of the original partition (in section 2; this is usually, but not necessarily the last sector of the SD-card) as last sector (on my exemplary 32 GB SD-card it was the last available sector, so if that was the case for your original partition layout too, you may as well just hit the Return key to avoid mishaps by mistyping).
    If a third partition is going to be created, use +nG (e.g. +10G) to set size of the second partition in GBytes.
  • When going to create an encrypted partition, set its partition type (by hitting t) to e8 ("LUKS", not documented in fdisk, but elsewhere), for a Linux-native partition leave it on 83 (and for FAT32 set it to c).
    Optionally, only for maximising plausible deniability when creating a DM-Crypt "plain" partition, you may set its partition type to 0 (not tested).

If a third partition is wanted, repeat the steps for the second partition with fdisk for it: n, p, 3, <last sector of second partition + 1>, <noted last sector of original partition on sd-card>.

Finally check by entering p that the partition table now looks as it should, and write it to the SD-card (and exit fdisk) via w.

Save the altered partition layout for reference / as a backup and perform a final check:
fdisk -l /dev/mmcblk1 | tee fdisk-l_mmcblk1-altered.txt

3.3 Checks and balances, when calculating the partition layout

However you may deviate from this guide, you should follow these rules:

  • The first sector of any partition must be an even number, which must be evenly divisible by the erase-block size (or 0, if this indeed has been the starting sector of the original partition on the SD-card) and should be evenly divisible by the optimal alignment size.
  • The last sector of any partition must be an odd number.
  • The last sector of any partition plus 1 must be evenly divisible by the erase-block size (in sectors) and should be evenly divisible by the optimal alignment size.
  • Consequently any partition size will be evenly divisible by the erase-block / optimal alignment size, too (just to double-check).

4. Creating filesystems

4.1 vFAT32 / FAT32 LFN (unencrypted)

Mind that FAT filesystems do not support access rights, hence are unsuited for putting android_storage on them (and for many other use cases), plus its anachronistic design, which does not fit well to FLASH-based mass storage. Furthermore FAT32 only supports files up to 2 GBytes size (though up to 2 TB filesystem size).
Thus a FAT partition is primarily suited for data exchange by physically moving the SD-card to other devices. The same applies to exFAT, except for the file size limit.

Mimic the output of dosfsck from section 2 as much as possible:

  • Cluster size is 32, mostly 64 or 128 sectors, set with option -s; note that dosfsck and dumpexfat output the cluster size in Bytes, not sectors. If a preformatted exFAT partition had a cluster size of more than 128 sectors, use 128.
    When the partition is primarily used as an archive, i.e. not very often written to and not with many small files, as an optimisation for FLASH-media you may consider setting the cluster size to its maximum value of 128 sectors (regardless of its original value), but this may introduce interoperability issues with other operating systems / firmwares, especially digital cameras etc.
  • Logical sector size is usually 512 Bytes, set with option -S; needed for the file-based simulations, but do not use when creating a filesystem on a SD-card (as its sector size shall be implicitly used, then).
  • Hidden sectors count as obtained in section 2 is set with -h
  • Another optimisation for FLASH-based memory is to use a single FAT instead of two with -f 1, but this may also introduce interoperability issues with other operating systems / firmwares, especially digital cameras etc.

As the FATs are smaller now (due to the smaller capacity of the newly created filesystem and FATs), the number of reserved sectors (set with option -R) has to be adapted. As mkfs.vfat knows no "dry run" and I found no formula for calculating the exact size of a FAT according to the size of the complete file system, one simply starts a "trial run" within a sparse file on /tmp:
mkfs.vfat -S 512 -v -s 64 -h 8192 -C /tmp/fat-test.img <filesystem size in KBytes> # "Trial run 1", divide the intended number of 512 Bytes sectors by two for the filesystem size and set cluster size, hidden sectors count and sector size to the original values obtained in section 2 (if the cluster size of an pre-formatted exFAT partition on the SD-card was larger than 128 sectors, use 128 sectors)
dosfsck -nv /tmp/fat-test.img # Compare to the output generated in section 2 (sector and cluster size) and check, if the resulting filesystem size (e.g. in sectors) in the image file is the intended one for the freshly created (in section 3) real partition
Now calculate the number of reserved sectors, which are needed to add up with the FATs (one or two) to any multiple (e.g. times 1, 2, 3, ...) of the determined alignment size. Note that the number of reserved sectors must be at least 8 (for FAT32) and shall be at least 32 or the number of sectors per cluster (whichever is higher); in above example of an almost 8 GBytes large partition on my exemplary 32GB SD-card, each FAT occupies 2048 sectors, so it takes 4096 reserved sectors to sum up to the alignment size of 8192 sectors.
Now check, if this calculation is correct:
rm /tmp/fat-test.img # Delete the test file
mkfs.vfat -S 512 -v -s 64 -h 8192 -R 4096 -C /tmp/fat-test.img 8384512 # "Trial run 2" with 4096 reserved blocks as an example, for a filesystem slightly smaller than 8 GB ending on sector 16777215
(As a different example, also for slightly smaller than 8 GB filesystem ending on sector 16777215, but for the aforementioned 128 GB SD-card: mkfs.vfat -S 512 -v -f 1 -s 128 -h 32768 -R 31744 -C /tmp/fat-test.img 8372224 # Note that the offsets and consequently the filesystem size differ slightly)
dosfsck -nv /tmp/fat-test.img # Check if everything looks as intended
rm /tmp/fat-test.img # Delete the test file

Then do the real creation of the FAT filesystem on the SD-card:
mkfs.vfat -v -s 64 -h 8192 -R 4096 /dev/mmcblk1p1 # "Real run" with example values
dosfsck -nv /dev/mmcblk1p1 | tee dosfsck-nv_mmcblk1p1-altered.txt # Last check, if everything is as intended, and backup

Side note: Whatever you may consider to do, never use the mkdosfs / mkfs.vfat options -a or -c on FLASH-based media (e.g. SD-cards)!

4.2 Linux-native filesystems (unencrypted)


4.2.a EXT4

EXT4 is fast, widely used, trustworthy and "light" on the device (memory consumption, processing power and consequently energy consumption), plus supported by all SailfishOS and Linux devices in general. While it originally has not been designed with FLASH-based mass storage in mind, it is not "FLASH-unfriendly", even when a journal is used for metadata only (e.g. as in the default mount option data=ordered).

For creating an EXT4 filesystem, the defaults in /etc/mk2fs.conf are fine.
You may consider formatting your EXT4 filesystem without a journal per option "-O ^has_journal", although IMO the drawbacks (fsck time and corrupted files, when writing has been interrupted) outweigh the advantage of being slightly nicer to FLASH-based memory.

Hence simply enter:
mkfs.ext4 -v /dev/mmcblk1pX # Substitute "X" with 1 or 2

Side note: Whatever you may consider to do, never use the mkfs.extX option "-c -c" on FLASH-based media as a SD-card!

4.2.b BTRFS

While it took BTRFS some time to mature, since Linux kernel 3.4 it can be regarded as "production ready" (as the Jolla 1 phone shows). It has been designed with FLASH-based mass storage in mind, but has a couple of drawbacks, e.g. free space management (when over 85% full), performance on specific operations, being compute- and memory-intensive (especially when over 85% full), occasional "btrfs-balancer" runs are recommended for optimal space use and performance etc. OTOH BTRFS can provide transparent file compression, subvolumes etc.

If your kernel(s) support(s) BTRFS (see cat /proc/filesystems | grep btrfs), you may format a partition with it by issuing:
mkfs.btrfs -f -O ^extref /dev/mmcblk1pX # "-O ^extref" is only necessary on kernels before 3.7 (e.g. on Jolla 1 phones with their kernel 3.4.108) and for interoperability with them
(If mkfs.btrfs is not already on your device, install it per: pkcon install btrfs-progs)

Mind to check the mount options for BTRFS as described in section "6 Notes" after finishing the setup process (and before using the BTRFS filesystem).

4.2.c F2FS

F2FS has been designed specifically for FLASH-based mass storage and to overcome some of the drawbacks BTRFS has, but it is quite new (for a filesystem, for which reliability is crucial), not widely supported and IMO one should not put critical data on F2FS before kernel 3.18 (see uname -a on your device(s)).

If your kernel(s) support(s) F2FS (see cat /proc/filesystems | grep f2fs), you may consider installing mkfs.f2fs per f2fs-tools from OpenRepos (e.g. via Storeman) and format a partition with it by issuing:
mkfs.f2fs /dev/mmcblk1pX # The default options are fine

4.3 DM-Crypt (encrypted)

Encrypted volumes / partitions are a technically hard usage for NAND-FLASH based mass storage in some aspects. Nevertheless, this is one of the objectives, a desired general use case, relies on common components (DM-Crypt / cryptsetup and its volume format) and the hardness can be mostly alleviated by proper setup parameters.
It is easier for a FLASH-memory controller to handle this situation, the more "spare area" to "maneuver" (alike BTRFS) it has. Though this can be achieved by an unused (i.e. unformatted) partition with typically 12 - 20% of the SD-card's total capacity, a functional, unencrypted partition (of up to 20% of the SD-card's total capacity), which is only lightly used, is equally suited and hence preferable. This optimisation provides an additional incentive to use (at least) two partitions, e.g. an unencrypted and an encrypted one.

See what Cryptsetup versions are available and if you already have a Cryptsetup RPM installed with:
pkcon search name cryptsetup
Unfortunately Jolla's mer-tools repository for SailfishOS 2.1.4 and 2.2.0 provides a defunct cryptsetup-1.7.5+git1-1.2.1 (this is fixed in SailfishOS 2.2.1). If you have it installed, uninstall it with pkcon remove cryptsetup cryptsetup-libs.
Cryptsetup versions before 1.4.0 are not well suited to be used for FLASH-based mass storage, as they lack the option "--enable-discards" (2011-07-07) and the fix "If device is not rotational, do not use Gutmann wipe method." (2011-10-05). Thus, if you have Jolla's cryptsetup version 1.1.3 (2010-07-03) installed under SailfishOS 2.1.3.7 or earlier, uninstall it with pkcon remove cryptsetup-luks.

Consequently, on SailfishOS versions before 2.2.1, install (deviating from objective 1, but it is only used once, for setup) Cryptsetup 1.6.4 (compiled and packaged by NielDK) by issuing:
curl -O https://openrepos.net/sites/default/files/packages/500/cryptsetup-1.6.4-1.armv7hl.rpm
curl -O https://openrepos.net/sites/default/files/packages/500/libcryptsetup4-1.6.4-1.armv7hl.rpm
pkcon install-local libcryptsetup4-1.6.4-1.armv7hl.rpm cryptsetup-1.6.4-1.armv7hl.rpm # Enter "y" to confirm installing

If you had NielDK's Cryptsetup 1.6.4 installed when upgrading to SailfishOS 2.1.4 or 2.2.0, his cryptsetup RPM is updated to Jolla's defunct cryptsetup RPM, while keeping NielDK's libcryptsetup4 RPM; in this case, remove it completely per pkcon remove cryptsetup libcryptsetup4, then reinstall NielDK's Cryptsetup 1.6.4 as described above.
If you had NielDK's Cryptsetup 1.6.4 installed when upgrading to SailfishOS 2.2.1 or higher, only remove NielDK's libcryptsetup4 per pkcon remove libcryptsetup4in order to obtain a working installation of Jolla's fixed Cryptsetup 1.7.5+git3-1.3.1.jolla.

Create a "key file", e.g. by creating a random passphrase file on the root partition:
dd if=/dev/urandom of=/etc/mmcblk1pX.key bs=64 count=1 # Mind to adapt the partition number
For LUKS, it can be well shorter than 64 Bytes and manually created or half-automatically (and thus much easier to memorize) with pwgen (e.g. from OpenRepos), just as it fulfills your needs (i.e. it does not need to be cryptographically strong, but nevertheless should be a well chosen / "strong" passphrase); in general it can be any absolutely static file on the root partition up to a few KBytes (at most 8 KB) long.

Back up the "key file":
cp /etc/mmcblk1pX.key ./ # Better (for security) move the backed-up "key file" off-device later on, i.e. deleting it (and the backups of the DM-Crypt header generated later) in ~nemo/sdcard-layouts/ after copying them away
chmod ug=r,o= /etc/mmcblk1pX.key # Restrict access; the ownerships should be root:root

4.3.1 Cryptography options

  • The "password" / key file hashing algorithm does not need to be cryptographically strong, it just has to generate a cryptographic hash of 160 bits (more is discarded), see Cryptsetup FAQ section "5.20 LUKS [...] uses SHA-1!". It must be supported by DM-Crypt, plus (before SailfishOS 2.2.1) the kernel (cryptography support, see cat /proc/crypto # type: shash) rsp. kernels of all devices the encrypted partition is going to be used with, or OpenSSL (SailfishOS 2.2.1). It also ought to be "light" to the devices used on in order to avoid prolonging mount- / boot-time and energy consumption unnecessarily. Also note, when using LUKS, it hashes a random "key file" or a strong password many hundred times. (But on PCs / in the future with Cryptsetup 2, if a LUKS2 header is used, -h sha256 is the right choice.)
    Hence the obvious choice for the "password" hashing algorithm is SHA-1, set with: -h sha1
  • The block cipher algorithm also must be properly supported (by DM-Crypt, plus kernel crypto before SailfishOS 2.2.1 / OpenSSL), should be algorithmically "light" and efficiently implemented. Thus the obvious practical choice is AES with 128 bit key, as AES-256 is computationally heavier (resulting in higher energy consumption) and slower (resulting in slower filesystem performance: ~ 25% in practice) and other algorithms are even slower and / or not broadly used.
  • Chaining mode and initialisation vector ("IV") algorithm:
    • If all kernels involved support XTS, use it, as it provides much faster random accesses than CBC.
      Plus XTS needs no external IV-generation algorithm (as it implicitly uses AES for that), thus it should be used with "plain", as this is the computationally lightest IV-option ("plain64" is only needed for partitions larger than 2TB).
      So the resulting cryptography options are: -c aes-xts-plain
    • Note that the crypto kernel modules providing XTS support may not be loaded be default or on demand (see cat /proc/crypto | grep xts, cat /lib/modules/version/modules.builtin | fgrep crypto, ls -lR /lib/modules/version/kernel/crypto/, ls -lR /lib/modules/version/kernel/drivers/crypto/, lsmod etc.)
      E.g. on Jolla 1 phones (i.e. up to and including SailfishOS 2.2.0, altered to use OpenSSL's cryptography by SailfishOS 2.2.1), one has to issue a modprobe -v qcrypto to use XTS (do that once now and check again with cat /proc/crypto | grep xts, plus in the mount-sd.sh script edited in section 4.3.5 below for automatic start-up).
    • Otherwise use CBC, which needs an external IV-algorithm ("ESSIV") to protect against the watermarking attack. As DM-Crypt's ESSIV needs 256 bit input, SHA-256 is the obvious choice WRT "lightness", broad availability and use.
      The resulting cryptography options are then: -c aes-cbc-essiv:sha256
    • All other chaining modes available are outright broken (e.g. ECB), slower and / or not widely used.
    • Note that XTS cuts the key used in two halves, so AES is only fed with half of the key length (the other half is used in the XTS chaining algorithm)!
      Consequently, to set the AES key length to 128 bits, one has to specify -s 128 for CBC, but -s 256 for XTS.

For more information on available cryptography options, see cryptsetup --help, dmsetup targets, cat /proc/crypto and e.g. the Cryptsetup FAQ or ArchLinux wiki on Cryptsetup.

4.3.2 Experimenting with Cryptsetup / DM-Crypt

Create a image file for testing:
cd /media/sdcard/XXXXXXXX/# As fallocate does not work on tmpfs with old kernel and fallocate versions, use it on an already mounted partition on the SD-card (need to mount manually or reboot and become root again) or alternatively in rsp. under /home/nemo on internal eMMC
fallocate -v -l 64M cryptsetup-test.img # 64 MBytes are usually sufficient for experimenting

  • For Cryptsetup-LUKS, e.g.:
    cryptsetup -v -q -h sha1 -s 256 -c aes-xts-plain --align-payload=8192 luksFormat cryptsetup-test.img /etc/mmcblk1pX.key # Set "--align-payload" to the alignment size in 512 Byte sectors (here with the usual example of 4MB); alter options for experimenting
    cryptsetup -v --allow-discards -d /etc/mmcblk1pX.key luksOpen cryptsetup-test.img luks-test # Create DM-device file /dev/mapper/luks-test
    cryptsetup -v luksDump cryptsetup-test.img # Check, if everything is O.K. and as expected
    You may test the DM-device "luks-test", e.g. by creating, mounting and using a filesystem on it or dd'ing directly to / from it (e.g. with time dd if=/dev/zero of=/dev/mapper/luks-test bs=4M count=8 && time dd if=/dev/mapper/luks-test of=/dev/null bs=4M count=8).
    cryptsetup -v luksClose luks-test # Close DM-device
    rm cryptsetup-test.img # Remove image file, when finished with testing
  • For Cryptsetup "type: plain", e.g.:
    cat /etc/mmcblk1pX.key | cryptsetup -v -h sha1 -s 128 -c aes-cbc-essiv:sha256 --allow-discards --type plain open cryptsetup-test.img plain-test # You may want to adapt
    You may test the DM-device "plain-test" now (see bullet points above for details).
    cryptsetup -v close plain-test # Close dm-mapper device
    rm cryptsetup-test.img # Remove image file, when finished with testing

4.3.3.a Setting up Cryptsetup-LUKS (modern, recommended)

The LUKS header created with the following commands stores all relevant information to access this DM-Crypt partition, lateron only a valid "key" (corresponding to one of the keyslots) has to be provided in order to unlock and mount it.

The first command differs, depending on the chaining mode used:

  • For using XTS:
    cryptsetup -v -q -h sha1 -s 256 -c aes-xts-plain --align-payload=8192 luksFormat /dev/mmcblk1pX /etc/mmcblk1pX.key # Set "--align-payload" to the alignment size in 512 Byte sectors, pick the right partition number twice etc.
  • Alternatively, for using CBC:
    cryptsetup -v -q -h sha1 -s 128 -c aes-cbc-essiv:sha256 --align-payload=8192 luksFormat /dev/mmcblk1pX /etc/mmcblk1pX.key # Set "--align-payload" to the alignment size in 512 Byte sectors, pick the right partition number twice etc.

Continue with:
cryptsetup -v luksDump /dev/mmcblk1pX | tee ~nemo/sd-card_layouts/luksdump_mmcblk1pX.txt # Last check & backup
cryptsetup -v --header-backup-file ~nemo/sd-card_layouts/luks-header-backup_mmcblk1pX.bin luksHeaderBackup /dev/mmcblk1pX # Backup the Luks header, as without it being intact all encrypted data is inaccessible (e.g. when the Luks header on the SD-card is damaged)
cryptsetup -v --allow-discards -d /etc/mmcblk1pX.key luksOpen /dev/mmcblk1pX mmcblk1pX-crypt # Create DM-device file

4.3.3.b Setting up Cryptsetup "type: plain" (prone to administrative errors, but maximising plausible deniability, as without a LUKS header an encrypted partition appears to contain solely random data)

Note that the "key" file shall provide sufficient entropy for Cryptsetup "type: plain", i.e. 64 Bytes of saved random data or a static high-entropy file of a couple of hundred Bytes (at most 512 Bytes) are O.K., regular passphrases are not.
Also note, that Cryptsetup's "type: plain" does not allow to align data structures (as it has none).

  • For using XTS:
    cat /etc/mmcblk1pX.key | cryptsetup -v -h sha1 -s 256 -c aes-xts-plain --allow-discards --type plain open /dev/mmcblk1pX mmcblk1pX-crypt # Mind to pick the right partition number three times etc.
  • Alternatively, for using CBC:
    cat /etc/mmcblk1pX.key | cryptsetup -v -h sha1 -s 128 -c aes-cbc-essiv:sha256 --allow-discards --type plain open /dev/mmcblk1pX mmcblk1pX-crypt # Mind to pick the right partition number three times etc.

4.3.4 Creating a filesystem on encrypted partition

See section "4.2 Linux-native filesystems" for details of the mkfs.*-commands for various filesystems, but use /dev/mapper/mmcblk1pX-crypt as device (and insert the right partition number for "X")!

When successfully created a filesystem, do:
cryptsetup -v close mmcblk1pX-crypt # Finishing setup steps by closing the DM-crypt mapper-file

4.3.5 Let SailfishOS automatically mount the encrypted partition
This method described in section 4.3.5 only applies to SailfishOS versions below 2.2.0!
For SailfishOS 2.2.x, use addendum below
.

Up to (and including) SailfishOS 2.1.4 there seems to be no proper way for integrating filesystems on DM-Crypt partitions residing on SD-card seamlessly into SailfishOS without adapting one of its shell scripts.

Hence edit the shell script /usr/sbin/mount-sd.sh (backup first):
cd /usr/sbin
cp -p mount-sd.sh mount-sd.sh.orig

For use with DM-Crypt, add the following lines at the appropriate places ("diff -C 2" output; for cutting and pasting multiple lines, see complete script in section 8.1).

*** mount-sd.sh.orig
--- mount-sd.sh
***************
*** 28,31 ****
      eval "$(/sbin/blkid -c /dev/null -o export /dev/$2)"

+     if [ "${TYPE}" = "crypto_LUKS" ]; then
+#        /sbin/modprobe qcrypto  # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
+         if /usr/sbin/cryptsetup --allow-discards -d /etc/$(basename ${DEVNAME}).key luksOpen ${DEVNAME} $(basename ${DEVNAME})-crypt 
+         then
+             eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename ${DEVNAME})-crypt)"  # Reassigning DEVNAME, TYPE and UUID
+             if [ "${TYPE}" = "crypto_LUKS" ]  # Still?  Then blkid did not output anything!
+             then
+                 systemd-cat -t mount-sd /bin/echo "ERROR: Filesystem type missing for DM-crypt (LUKS) device ${DEVNAME}"
+                 /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
+                 exit 1
+             else systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (LUKS) device ${DEVNAME}"
+             fi
+         else 
+             systemd-cat -t mount-sd /bin/echo "ERROR: Failed to open ${DEVNAME} with cryptsetup (LUKS)"
+             exit 1
+         fi
+     fi
+     if [ -z "${TYPE}" ] && [ -z "${UUID}" ] && [ "${DEVNAME}" = "$2" ]; then  # trying cryptsetup "plain"
+#        /sbin/modprobe qcrypto  # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
+         for i in "sha1 256 aes-xts-plain" "sha1 128 aes-cbc-essiv:sha256" "ripemd160 256 aes-cbc-essiv:sha256"
+         do
+             cat /etc/$(basename $2).key | /usr/sbin/cryptsetup -h "$(echo "$i" | cut -f 1 -d ' ')" -s "$(echo "$i" | cut -f 2 -d ' ')" -c "$(echo "$i" | cut -f 3 -d ' ')" --allow-discards --type plain open /dev/$2 $(basename $2)-crypt  # Cryptsetup "plain" will always succeed, when the key-file is readable and the device is accessible
+             eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename $2)-crypt)"  # Reassigning DEVNAME, TYPE and UUID
+             if [ -n "${TYPE}" ]  # Success?
+             then 
+                 systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (\"plain\") device ${DEVNAME}"
+                 break
+             fi 
+             /usr/sbin/cryptsetup close "$(basename $2)-crypt"
+             systemd-cat -t mount-sd /bin/echo "NOTICE: Tried opening ${DEVNAME} as DM-crypt \"plain\" device with options \"$i\", but failed to obtain a filesystem type"
+             DEVNAME="$2" ; unset UUID  # Revert to state before trying cryptsetup "plain"
+         done
+     fi
+ 
      if [ -z "${TYPE}" ]; then

***************
*** 99,102 ****
       fi
       umount $DIR || umount -l $DIR
+         if [ -e "/dev/mapper/$(basename ${DEVNAME})-crypt" ]; then
+             if /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
+             then systemd-cat -t mount-sd /bin/echo "Cryptsetup closed ${DEVNAME} successfully"
+             else systemd-cat -t mount-sd /bin/echo "WARNING: Cryptsetup failed to close ${DEVNAME}"
+             fi
+         fi
       systemd-cat -t mount-sd /bin/echo "Finished ${ACTION}ing ${DEVNAME} at ${DIR}"
       fi

When finished editing the shell script, back it up:
cp -p mount-sd.sh mount-sd.sh.patched

Mind that the altered shell script may be overwritten by SailfishOS updates, hence check for this then and reestablish the changes (in the newly updated file!), if overwritten (see timestamps with ls -l).

Ultimately reboot to see, if everything working as intended: Issue as user nemo mount | grep mmcblk1
If not, debug with journalctl -r as root user, while having less installed (e.g. per pkcon install less): Search with "/" (e.g. for mount-sd or mmcblk1), use "n" to jump to the next hit and press PageUp / PageDown to look at adjacent messages.
Note that due to a flaw in older Linux kernels you may see journal messages as follows WRT mapped DM-crypt partitions, which are harmless and should be ignored:

systemd-udevd[ZZZZ]: conflicting device node '/dev/mapper/mmcblk1pX-crypt' found, link to '/dev/dm-0' will not be created
systemd-udevd[YYYY]: inotify_add_watch(9, /dev/dm-0, 10) failed: No such file or directory

5. Finishing up

Reboot (if not done in previous section 4.3.3) and finally "donate" all mountpoints and filesystems on the SD-card to user nemo, else you cannot use them as nemo:

devel-su

  • For SailfishOS 2.1 / 2.0:
    cd /media/sdcard/
    chown -v nemo:nemo * */.

  • For SailfishOS 2.2:
    cd /run/media/nemo/
    chown -v nemo:nemo */.

You can check that discards are correctly configured with lsblk -D /dev/mmcblk1 (crypto-devices do not return zeroes).

6. Notes

  • Multiple unencrypted partitions on a SD-card with filesystems supported by the SailfishOS kernel (e.g. EXT4, vFAT32) are all automatically mounted by SailfishOS (and with aforementioned changes to mount-sd.sh encrypted ones, too).
    Additionally a filesystem (both unencrypted or encrypted) using the whole device mmcblk1 (i.e without a partition table) is also supported (but not generally recommended).
  • Note that the SailfishOS backup tool (in the SailfishOS settings) stores backups only on the first and unencrypted (stating it cannot write to it, although untrue, if encrypted) partition of a SD-card: If you rely on backing up your contacts etc. with SailfishOS' backup tool onto SD-card, create an unencrypted partition (formatted with any supported filesystem) at /dev/mmcblk1p1 (a few GBytes is sufficient, if a large encrypted partition is wanted).
  • While deviating from objective 2, there are special use cases for which you may consider the mount option "discard" (see the following section 7 "Additional notes" for details) for specific non-FAT filesystems (which support discards), just as SailfishOS does for FAT by default. To achieve this add in /usr-sbin/mount-sd.sh (which has to be altered for seamlessly using encrypted partitions on the SD-card under SailfishOS, anyway; see section 8.1 for the complete patched file, but with this change disabled by comment marks "#" there), e.g. for EXT4:

    ext4)
    mount ${DEVNAME} $MNT/${UUID} -o "$MOUNT_OPTS",discard || /bin/rmdir $MNT/${UUID}
    ;;
    
  • If you choose to use BTRFS, check and ensure that the mount options "noacl", "space_cache" and "ssd" are active (they are on a Jolla 1 under SFOS 2.1.3.7 and later). You may also consider using the mount options "autodefrag" (as on Jolla 1's internal filesystems, see output of mount), "compress=lzo" (promising size and speed improvements) and / or "enospc_debug" (e.g. by setting them in /usr/sbin/mount-sd.sh as shown in section 8.1), but do read its wiki-page for the mount options first.
    If the mount option "ssd" is not automatically detected and enabled, consider using "ssd_spread".
  • The mount option "dirsync" for SD-cards in the "header" of /usr/sbin/mount-sd.sh is debatable (synchronous writes are "hard" to SSDs) and SailfishOS does not use it for internal eMMC, so I prefer to disable it (see script in section 8.1), consequently the "noatime" option also affects directories. If you really need directory access times to be recorded, use "diratime" in conjunction with "noatime" (or "relatime", if you need correct access times for files, too); both options ("diratime" and "relatime") are "hard" to SSDs (but less than "atime", "dirsync" or even "sync").
  • One reason for objective 1 is, that all third party (FUSE-based) filesystems appear to be somewhat unmaintained, hence one cannot rely on them being updated, if incompatibilities with newer SailfishOS versions arise: F2FS has been retracted on Openrepos and for encfs_sailfish a precompiled archive or a RPM and a page at OpenRepos are also missing. The same has been the case throughout 2016 & 2017 for exFAT, although it was newly compiled and published 2018-01-15.
    I wonder, if there are any hurdles to utilise the filesystem related kernel modules in /lib/modules/"latest"/kernel/fs/ (as provided by SailfishOS), which cover CIFS (SMB), NFS and eCryptFS. Answer: No hurdles, as FishNetMount does that for CIFS and NFS, so does cifs-utils for CIFS, but unfortunately I still have not found anything WRT eCryptFS.

7. Additional notes

(Originally planned as foreword, but it grew too large.)

  • Use quality SD-cards, e.g. from SanDisk (any series, the "Ultra" series is reasonably priced and reasonably fast, or alternatively one from Samsung). I personally regard all other brands as low quality (as e.g. they do not manufacture controller, flash memory and firmware by themselves, in contrast to SanDisk and Samsung) and "Noname" cards as rubbish.
  • If the SD-card has been heavily used before (especially if the partition layout was altered before), it may make sense to format it with the SD-card formatter from the SD Association (Windows and MacOS only, unfortunately), which is supposed to put the SD-card into a (more or less) juvenile state by issuing a "Secure erase" (likely addressing the "Security erase enhanced" command per ATA8-ACS3 specification, not the simple "Security erase") command to it and partition the SD-card anew with a layout conformant to the SD-card specification. The former can also be achieved with hdparm under e.g. Linux, but the options --security-erase-enhanced / --security-erase are still marked as "dangerous", while the option --trim-sectors (which sounds spot on) is still marked as "extremely dangerous" (all three for many years, already?!?), and I have not found a hdparm RPM for SailfishOS.
    Hence under SailfishOS and Linux in general blkdiscard -v /dev/mmcblk1pX does that well (discarding all sectors of a partition / logical block device), without altering the SD-card's partition layout (as long as not used on the whole, physical device, e.g. /dev/mmcblk1). If a filesystem is recreated anyway, blkdiscard on its partition is a complementary / additional measure to issuing a fstrim command to its mount point when the old (non-FAT) filesystem still exists (i.e. fstrim -v /media/sdcard/XXXXXXXX).
    Do not use any tool or tool-option (e.g. blkdiscard -z), which overwrites every sector on the device: These are meant for HDDs and are absolutely detrimental to lifetime and performance of FLASH-based media (e.g. SD-cards).
  • It is generally not recommended to use the mount option "discard" for non-FAT partitions (as the synchronous nature of implementations of the discard / trim command in many SD-card firmwares slows down file-overwrite and -remove operations significantly), rather rely on SailfishOS issuing fstrim occasionally (Does it? Check in section 8.2 unsuccessful, yet.) and / or manually issue fstrim -v /media/sdcard/XXXXXXXX (working fine with fstrim from util-linux 2.28.1 provided by Jolla's mer-tools, using the check in section 8.2) every now and then (thus making this an asynchronous operation, plus some other advantages). Note that a fstrim -av trims all mounted filesystems, which it supports (e.g. EXT4, BTRFS), including the ones on internal eMMC (which does no harm, AFAICS on a Jolla 1 phone, and makes sense to apply occasionally, after having thoroughly cleaned up and balanced BTRFS). FAT filesystems are not supported by fstrim, hence they should be mounted with "discard".
    You may still consider using "discard" for non-FAT filesystems on SD-card, if you do not put android_storage on SD-card and use it solely as an archive (e.g. for media files or backups), so slow erase and overwrite operations are irrelevant.
    For further details, see e.g.

8. Appendix:

8.1 /usr/sbin/mount-sd.sh.patched

#!/bin/bash

# The only case where this script would fail is:
# mkfs.vfat /dev/mmcblk1 then repartitioning to create an empty ext2 partition
# Plus when used on a base device with a partition table (e.g. /dev/mmcblk1 
# on every boot, when a regular SD-card is inserted) and (unpatched) 
# when partitions are encrypted with DM-Crypt.

DEF_UID=$(grep "^UID_MIN" /etc/login.defs |  tr -s " " | cut -d " " -f2)
DEF_GID=$(grep "^GID_MIN" /etc/login.defs |  tr -s " " | cut -d " " -f2)
DEVICEUSER=$(getent passwd $DEF_UID | sed 's/:.*//')
MNT=/media/sdcard
MOUNT_OPTS="noatime,users"  # "dirsync" removed
ACTION=$1
DEVNAME=$2

if [ -z "${ACTION}" ]; then
    systemd-cat -t mount-sd /bin/echo "ERROR: Action needs to be defined."
    exit 1
fi

if [ -z "${DEVNAME}" ]; then
    systemd-cat -t mount-sd /bin/echo "ERROR: Device name needs to be defined."
    exit 1
fi

systemd-cat -t mount-sd /bin/echo "Called to ${ACTION} ${DEVNAME}"

if [ "$ACTION" = "add" ]; then
    # Setting variables DEVNAME (anew!), UUID and TYPE (but all three will be not set for "plain" DM-crypt devices):
    eval "$(/sbin/blkid -c /dev/null -o export /dev/$2)"

    if [ "${TYPE}" = "crypto_LUKS" ]; then
#       /sbin/modprobe qcrypto  # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
        if /usr/sbin/cryptsetup --allow-discards -d /etc/$(basename ${DEVNAME}).key luksOpen ${DEVNAME} $(basename ${DEVNAME})-crypt 
        then
            eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename ${DEVNAME})-crypt)"  # Reassigning DEVNAME, TYPE and UUID
            if [ "${TYPE}" = "crypto_LUKS" ]  # Still?  Then blkid did not output anything!
            then
                systemd-cat -t mount-sd /bin/echo "ERROR: Filesystem type missing for DM-crypt (LUKS) device ${DEVNAME}"
                /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
                exit 1
            else systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (LUKS) device ${DEVNAME}"
            fi
        else 
            systemd-cat -t mount-sd /bin/echo "ERROR: Failed to open ${DEVNAME} with cryptsetup (LUKS)"
            exit 1
        fi
    fi
    if [ -z "${TYPE}" ] && [ -z "${UUID}" ] && [ "${DEVNAME}" = "$2" ]; then  # trying cryptsetup "plain"
#       /sbin/modprobe qcrypto  # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
        for i in "sha1 256 aes-xts-plain" "sha1 128 aes-cbc-essiv:sha256" "ripemd160 256 aes-cbc-essiv:sha256"
        do
            cat /etc/$(basename $2).key | /usr/sbin/cryptsetup -h "$(echo "$i" | cut -f 1 -d ' ')" -s "$(echo "$i" | cut -f 2 -d ' ')" -c "$(echo "$i" | cut -f 3 -d ' ')" --allow-discards --type plain open /dev/$2 $(basename $2)-crypt  # Cryptsetup "plain" will always succeed, when the key-file is readable and the device is accessible
        eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename $2)-crypt)"  # Reassigning DEVNAME, TYPE and UUID
            if [ -n "${TYPE}" ]  # Success?
            then 
                systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (\"plain\") device ${DEVNAME}"
                break
            fi 
            /usr/sbin/cryptsetup close "$(basename $2)-crypt"
            systemd-cat -t mount-sd /bin/echo "NOTICE: Tried opening ${DEVNAME} as DM-crypt \"plain\" device with options \"$i\", but failed to obtain a filesystem type"
            DEVNAME="$2" ; unset UUID  # Revert to state before trying cryptsetup "plain"
        done
    fi

    if [ -z "${TYPE}" ]; then

        # In case filesystem type is missing, try reading it.
        TYPE=$(lsblk -n -o FSTYPE ${DEVNAME} | tail -n 1)  # Fails grossly when trying to mount the basic device (e.g. mmcblk1) with multiple partitions: Picks the type of last partition (e.g. mmcblk1pX) then!  Still no harm done, as mounting fails.

        if [ -z "${TYPE}" ]; then
            systemd-cat -t mount-sd /bin/echo "ERROR: Filesystem type missing for ${DEVNAME}"
            exit 1
        fi
    fi

    if [ -z "${UUID}" ]; then
        # In case device does not have UUID lets create one for it based on
        # the card identification.
        PKNAME=$(lsblk -n -o PKNAME ${DEVNAME} | tail -n 1)

        # If there is no PKNAME try NAME instead.
        if [ -z "${PKNAME}" ]; then
            PKNAME=$(lsblk -n -o NAME ${DEVNAME} | head -n 1)
        fi

        if [ -e "/sys/block/${PKNAME}/device/cid" ]; then
            CID=$(cat /sys/block/${PKNAME}/device/cid)
            if [ -n "${CID}" ]; then
                IDNAME=$(lsblk -n -o NAME ${DEVNAME} | tail -1 | cut -d "-" -f2)
                UUID="${CID}-${IDNAME}"
            fi
        fi

        if [ -z "${UUID}" ]; then
            # Exit here as in the future there might be things like USB OTG disks or
            # sdcards attached via adapter that might behave differently and needs special case
            # in case such happens fail so we don't break anything.
            systemd-cat -t mount-sd /bin/echo "ERROR: Could not find or generate UUID for device ${DEVNAME}."
            exit 1
        fi
    fi

    DIR=$(grep -w ${DEVNAME} /proc/mounts | cut -d \  -f 2)
    if [ -n "$DIR" ]; then
        systemd-cat -t mount-sd /bin/echo "${DEVNAME} already mounted on ${DIR}, ignoring"
        exit 0
    fi

    test -d $MNT/${UUID} || mkdir -p $MNT/${UUID}
    chown $DEF_UID:$DEF_GID $MNT $MNT/${UUID}
    touch $MNT/${UUID}

    case "${TYPE}" in
    vfat|exfat)
        mount ${DEVNAME} $MNT/${UUID} -o uid=$DEF_UID,gid=$DEF_GID,$MOUNT_OPTS,utf8,flush,discard || /bin/rmdir $MNT/${UUID}
        ;;
    # NTFS support has not been tested but it's being left to please the ego of an engineer!
    ntfs)
        mount ${DEVNAME} $MNT/${UUID} -o uid=$DEF_UID,gid=$DEF_GID,$MOUNT_OPTS,utf8 || /bin/rmdir $MNT/${UUID}
        ;;
#   ext4)
#       mount ${DEVNAME} $MNT/${UUID} -o $MOUNT_OPTS,discard || /bin/rmdir $MNT/${UUID}
#       ;;
    btrfs)
        mount ${DEVNAME} $MNT/${UUID} -o $MOUNT_OPTS,enospc_debug,autodefrag,compress=lzo || /bin/rmdir $MNT/${UUID}  # Check if chosen options suit your use well
        ;;
    *)
        mount ${DEVNAME} $MNT/${UUID} -o $MOUNT_OPTS || /bin/rmdir $MNT/${UUID}
        ;;
    esac
    test -d $MNT/${UUID} && touch $MNT/${UUID}
    systemd-cat -t mount-sd /bin/echo "Finished ${ACTION}ing ${DEVNAME} of type ${TYPE} at $MNT/${UUID}"

else
    DIR=$(grep -w ${DEVNAME} /proc/mounts | cut -d \  -f 2)
    if [ -n "${DIR}" ] ; then
        if [ "${DIR##$MNT}" = "${DIR}" ]; then
            systemd-cat -t mount-sd /bin/echo "${DEVNAME} mountpoint ${DIR} is not under ${MNT}, ignoring"
            exit 0
        fi
        umount $DIR || umount -l $DIR
        if [ -e "/dev/mapper/$(basename ${DEVNAME})-crypt" ]; then
            if /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt" 
            then systemd-cat -t mount-sd /bin/echo "Cryptsetup closed ${DEVNAME} successfully" 
            else systemd-cat -t mount-sd /bin/echo "WARNING: Cryptsetup failed to close ${DEVNAME}" 
            fi
        fi
        systemd-cat -t mount-sd /bin/echo "Finished ${ACTION}ing ${DEVNAME} at ${DIR}"
    fi
fi

8.2 Practical trim / discard test

(adapted for SailfishOS, only usable without filesystem compression etc.)

devel-su
cat /sys/block/mmcblk1/queue/discard_zeroes_data # "1", if discard / trim is supported
lsblk -D /dev/mmcblk1 # See, if discards are correctly configured
cd /media/sdcard/XXXXXXXX/
yes | dd bs=4K count=1K of=trim.test # 4 MB; must consist of whole erase-blocks (at least one; if not, increase count)
sync ; echo 1 > /proc/sys/vm/drop_caches
hexdump trim.test # outputs only "0a79"
filefrag -s -v trim.test
df trim.test
dd bs=<blocksize_from_filefrag> skip=<starting-offset_from_filefrag> count=<number-of-blocks_from_filefrag> if=<underlying-device_from_df> | hexdump # outputs only "0a79"
rm trim.test
sync ; echo 1 > /proc/sys/vm/drop_caches

Then perform tests (fstrim, reboot, just wait etc.), issue
sync ; echo 1 > /proc/sys/vm/drop_caches
thereafter (if not rebooted) and ultimately repeat
dd bs=<blocksize_from_filefrag> skip=<starting-offset_from_filefrag> count=<number-of-blocks_from_filefrag> if=<underlying-device_from_df> | hexdump
If it still only outputs "0a79", no discard / trim occurred, if successful the output should be all zeroes (or random data for encrypted devices).

edit retag flag offensive close delete

Comments

WOW! Amazing how-to. Don't get me wrong, but I hope this get partially obsolete with the announced encryption for SD card and filesystem ;-)

bomo ( 2018-02-12 22:34:12 +0300 )edit
1

So do I, but no ETA announced, no exact feature set defined, in the past features took "Jolla time" to materialise (i.e. late or never) and I was finishing this last week after months of research (while most older guides have significant flaws and / or do not work anymore due to SailfishOS updates):
So for the time being, this seems to be the only practical solution.

olf ( 2018-02-13 01:42:29 +0300 )edit

Great work! Thumbs up! I know what I will do tomorrow with my empty 64GB-SD-card. :-)

ds1979 ( 2018-02-14 19:39:41 +0300 )edit

Hmm I notice that the kernel does supports f2fs. I feel fsf2-tools should be available in the repo so we can make f2fs partitions on the sdcard. f2fs is far preferable for solid state devices than ext4 is. From the wikipedia page

The motive for F2FS was to build a file system that, from the start, takes into account the characteristics of NAND flash memory-based storage devices (such as solid-state disks, eMMC, and SD cards), which are widely used in computer systems ranging from mobile devices to servers.

F2FS was designed on a basis of a log-structured file system approach, which is adapted to newer forms of storage. Jaegeuk Kim, the principal F2FS author, has stated that it remedies some known issues[2] of the older log-structured file systems, such as the snowball effect of wandering trees and high cleaning overhead. In addition, since a NAND-based storage device shows different characteristics according to its internal geometry or flash memory management scheme (such as the Flash Translation Layer or FTL), it supports various parameters not only for configuring on-disk layout, but also for selecting allocation and cleaning algorithms.

veritanuda ( 2018-02-19 23:02:48 +0300 )edit

Yes, F2FS is somewhat nicer to FLASH-based mass storage than EXT4 (although by "far" is debatable), but just like BTRFS it is not available on all SailfishOS devices. Specifically the Jolla 1 phones may never receive F2FS support in their Linux kernel. Hence it is not suitable for a general description, but sure you can create a F2FS filesystem with mkfs.f2fs (from the f2fs-tools / f2fs-utils e.g. in Warehouse, if not already on your device) and use it in my guide instead of EXT4, if your Linux kernel supports F2FS.

What is the output of [nemo@Sailfish ~]$ uname -soirvmp on your device and which device do you use?
On a Jolla 1 phone with SailfishOS 2.1.3.7 it is:

Linux 3.4.108.20171017.1 #1 SMP PREEMPT Thu Oct 19 18:48:59 UTC 2017 armv7l armv7l armv7l GNU/Linux

And [nemo@Sailfish ~]$ cat /proc/filesystems does not list F2FS.

olf ( 2018-02-20 00:28:08 +0300 )edit

5 Answers

Sort by » oldest newest most voted
2

answered 2018-03-05 16:41:15 +0300

olf gravatar image

updated 2018-03-06 00:10:31 +0300

Quick guide

While above guide evolved, it became quite lengthy, hence here brief descriptions ("walk-throughs") of typical use cases:

A. Reformatting a SDXC-card (pre-formatted with exFAT) to vFAT32

Walk through sections 1, 2.1, (omit 2.2.a), 2.2.b, 2.3, 3.1, 3.2.a (just set partition type to "c"), 4.1 and 5.: Done.

B. Reformatting a SD-card to a Linux-native filesystem

Walk through sections 1, 2.1, (you can omit 2.2.x), 3.1, 3.2.a (just set partition type to "83"), one of the subsections 4.2.x and finally 5.: Done.

C. Repartitioning a SD-card, deploying unencrypted and encrypted filesystems

As this is what above guide was originally developed for, there is no shortcut one can take (except for the obvious possible omissions: sections 3.2.a and 4.3.2), just a few choices to make.

Typical use cases / partitioning schemes are:

  • C.1 Create a small FAT32 partition (slightly below 8 GB, as described in section 3.2.b) and a large second, encrypted partition
  • C.2 Split the space on SD-card between an unencrypted and an encrypted partition
  • C.3 Create a small FAT32 partition (slightly below 8 GB, see section 3.2.b) and split the remaining space on SD-card between an unencrypted and an encrypted partition
edit flag offensive delete publish link more
1

answered 2018-09-19 01:56:21 +0300

olf gravatar image

updated 2018-10-16 19:27:29 +0300

These configuration files enable automated mounting of encrypted (partitions on) SD-card under SailfishOS 2.2.

After the design stabilised, a few weeks of testing, enhancing and optimising led to this set of configuration files, now being called "crypto-sdcard 0.1 (2018-10-07)".

Create these files as root (i.e. with the usual ownerships and access rights: root root -rw-r--r--), with their contents as depicted below.
Note, that archives (tar.gz / ZIP) are available of newer versions of these files, but they require to rename the "key"-files.
Also note, that installable RPMs containing these newer versions are available at OpenRepos.

Then reboot and check as described in section 2.3 of the second part, if working fine.
You may also check the status of single units per systemctl status -l "<unit-name>" (with e.g. crypto-sd-luks@mmcblk1p2.service as unit name).
The last paragraph of section 4.3.5 (quick troubleshooting guide and superfluous messages in journal) is also applicable.

Check, if a link in /media/sdcard/ has been created:
cd /data/media/
ls -l
If not, maybe a directory of the same name exists: Backup / (re)move it.

Please continue with section 5.


Features this set of files provides:

  • Start mounting encrypted (partitions on) SD-card via udisks at the earliest sensible time: Right after udisks2.service has started.
  • Unmount before udisks2 begins stopping, hence achieving a clean unmount.
  • Ensure, that AlienDalvik (alien-service-manager.service) begins starting after mounting succeeded, to allow for android_storage on SD-card. Even more importantly this also ensures, that unmounting occurs only after AlienDalvik is completely stopped.
    Nevertheless, these configuration files are also applicable to devices without AlienDalvik installed.
  • These configuration files do not alter, replace or delete any extant files.
  • Boot time is not significantly prolonged, as unlocking encrypted partitions per Cryptsetup occurs in parallel to starting udisks2; after both succeeded, all mount operations are also started concurrently.
  • Create a "compatibility symlink" to allow older apps seamlessly accessing encrypted (partitions on) SD-cards at their new (since SailfishOS 2.2.0) mount point.


/etc/udev/rules.d/82-crypto-sd.rules

# For DM-Crypt LUKS, match sda0 to mmcblk1 to both SUBSYSTEM=="block" and ENV{ID_FS_TYPE}=="crypto_LUKS"
KERNEL=="mmcblk1*|sd[a-z][0-9]", SUBSYSTEM=="block", ENV{ID_FS_TYPE}=="crypto_LUKS", ACTION=="add", MODE="0660", TAG+="systemd", ENV{SYSTEMD_WANTS}="crypto-sd-luks@%k.service"

# For DM-Crypt "plain", also match sda0 to mmcblk1 to SUBSYSTEM=="block", but ensure (by ENV{ID_*}!= statements) that it appears to be unused space
KERNEL=="mmcblk1*|sd[a-z][0-9]", SUBSYSTEM=="block", ENV{ID_FS_USAGE}!="filesystem", ENV{ID_FS_TYPE}!="crypto_LUKS", ENV{ID_PART_TABLE_TYPE}!="dos", ENV{ID_PART_TABLE_TYPE}!="gpt", ACTION=="add", MODE="0660", ENV{DM_CRYPT_PLAIN_SDCARD}="1", TAG+="systemd", ENV{SYSTEMD_WANTS}="crypto-sd-plain@%k.service"

# Carefully match resulting virtual node dm-* to trigger mounting it; see /lib/udev/rules.d/10-dm.rules for details
KERNEL=="dm-[0-9]*", SUBSYSTEM=="block", SYMLINK=="mapper/mmcblk1*|mapper/sd[a-z][0-9]", ENV{ID_FS_USAGE}=="filesystem", ENV{DM_UDEV_RULES_VSN}=="[1-9]*", ACTION=="change", ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}=="1", ENV{DM_SUSPENDED}=="0", MODE="0660", GROUP="disk", TAG+="systemd", ENV{SYSTEMD_WANTS}="crypto-sd-luks-udisks@%E{DM_NAME}.service"

# Ditto for DM-Crypt "plain":
KERNEL=="dm-[0-9]*", SUBSYSTEM=="block", SYMLINK=="mapper/mmcblk1*|mapper/sd[a-z][0-9]", ENV{ID_FS_USAGE}=="filesystem", ENV{DM_UDEV_RULES_VSN}=="[1-9]*", ACTION=="change", ENV{DM_UDEV_PRIMARY_SOURCE_FLAG}=="1", ENV{DM_ACTIVATION}=="1", ENV{DM_SUSPENDED}=="0", ENV{DM_CRYPT_PLAIN_SDCARD}=="1", MODE="0660", GROUP="disk", TAG+="systemd", ENV{SYSTEMD_WANTS}="crypto-sd-plain-udisks@%E{DM_NAME}.service"

/etc/polkit-1/localauthority/50-local.d/69-crypto-sd-udisks.pkla

[Allow nemo mounting encrypted SD-card]
Identity=unix-group:system
Action=org.freedesktop.udisks2.filesystem-mount-system
ResultAny=yes
ResultInactive=yes
ResultActive=yes

/etc/systemd/system/crypto-sd-luks@.service

[Unit]
Description=Open DM-Crypt LUKS on SD-card %i
After=systemd-udevd.service systemd-udev-settle.service dev-%i.device
BindsTo=dev-%i.device
Conflicts=rescue.target actdead.target factory-test.target
AssertFileNotEmpty=/etc/%i.key

[Service]
Type=oneshot
RemainAfterExit=yes
# Only needed on Jolla 1 phones running SailfishOS 2.2.0 for using XTS:
# ExecStartPre=/sbin/modprobe qcrypto
# For various reasons (dependency on udisks2, allow discards etc.), do not use "udisksctl unlock --key-file", call cryptsetup directly:
ExecStart=/usr/sbin/cryptsetup --allow-discards -d /etc/%i.key luksOpen /dev/%i %i
ExecStop=/usr/sbin/cryptsetup close %i

/etc/systemd/system/crypto-sd-luks-udisks@.service

[Unit]
Description=Mount LUKS encrypted SD-card %i with udisks
After=udisks2.service crypto-sd-luks@%i.service dev-mapper-%i.device
BindsTo=udisks2.service crypto-sd-luks@%i.service dev-mapper-%i.device
Wants=crypto-sd-symlink@%i.service
Before=alien-service-manager.service

[Service]
User=nemo
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/udisksctl mount -b /dev/mapper/%i
ExecStop=/usr/bin/udisksctl unmount -b /dev/mapper/%i

/etc/systemd/system/crypto-sd-plain@.service

[Unit]
Description=Open DM-Crypt "plain" on SD-card %i
After=systemd-udevd.service systemd-udev-settle.service dev-%i.device
BindsTo=dev-%i.device
Conflicts=rescue.target actdead.target factory-test.target
AssertFileNotEmpty=/etc/%i.key

[Service]
Type=oneshot
RemainAfterExit=yes
# Only needed on Jolla 1 phones running SailfishOS 2.2.0 for using XTS:
# ExecStartPre=/sbin/modprobe qcrypto
ExecStart=/bin/sh -c 'cat /etc/%i.key | /usr/sbin/cryptsetup -h sha1 -s 256 -c aes-xts-plain --allow-discards --type plain open /dev/%i %i'
ExecStop=/usr/sbin/cryptsetup close %i

/etc/systemd/system/crypto-sd-plain-udisks@.service

[Unit]
Description=Mount "plain" encrypted SD-card %i with udisks
After=udisks2.service crypto-sd-plain@%i.service dev-mapper-%i.device
BindsTo=udisks2.service crypto-sd-plain@%i.service dev-mapper-%i.device
Wants=crypto-sd-symlink@%i.service
Before=alien-service-manager.service

[Service]
User=nemo
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/udisksctl mount -b /dev/mapper/%i
ExecStop=/usr/bin/udisksctl unmount -b /dev/mapper/%i

/etc/systemd/system/crypto-sd-symlink@.service

[Unit]
Description=Create media symlink to encrypted SD-card %i
Documentation=https://github.com/Olf0/crypto-sdcard
After=dev-mapper-%i.device
Requisite=dev-mapper-%i.device

[Service]
# The symlink is meant to stay, so it cannot be accidentially overwritten.
# OTOH, ln does not overwrite anything; it just fails, if a file, directory or link of that name already exists.
ExecStart=-/bin/sh -c 'UUID="$(/sbin/blkid -c /dev/null -s UUID -o value /dev/mapper/%i)" && ln -s /run/media/nemo/$UUID /media/sdcard/$UUID'
edit flag offensive delete publish link more

Comments

1

Note that for Cryptsetup "plain" setups, crypto-sd-plain-udisks@.service is not automatically triggered with above v0.1. It works fine, if manually started per systemctl.
Edit: This is fixed in v0.2.

While a proper solution is working well, it also alter alters the name of the "key"-files. While at it, a new UUID-based "key"-file naming scheme will be introduced in another version.
Edit: "Key"-files are based on UUIDs since v0.3.

olf ( 2018-10-07 18:09:29 +0300 )edit
0

answered 2018-02-18 00:21:01 +0300

Holger gravatar image

Hi, I have successfully created an ext4 formatted sdcard (unencrypted) with this guide. Thanks! However, I found one glitch: in step 5 "Finishing up" the chown command is not complete: the option "-R" is missing (recursive), so that all files and all directories in all subdirectories are changed to nemo:nemo as well. Bye

edit flag offensive delete publish link more

Comments

The chown nemo:nemo * */. in section 5 adapts the (user- and group-) ownership of each mountpoint (i.e. the directory named UUID) in /media/sdcard/ and each root directory of mounted filesystem (/media/sdcard/XXXXXXXX/.): Nothing more is needed for nemo to have full access, other existing directories (e.g. the "lost+found" directory) should stay untouched.
Any copying of data should be done as user nemo.

@Holger: What was your issue, which you resolved by changing the ownership recursively?

olf ( 2018-02-18 01:44:33 +0300 )edit

I used my pc for copying the files to the card, and on the pc I had no nemo user. So file ownership had to be changed.

Holger ( 2018-02-18 08:28:26 +0300 )edit
0

answered 2018-04-28 18:51:22 +0300

pcfe gravatar image

updated 2018-04-29 11:54:16 +0300

Thank you olf for the comprehensive guide.

On my Sailfish X | SailfishOS 2.1.4.14 (Lapuanjoki) I have successfully set up both a 16GB FAT partition for backing up and a f2fs on luks … -c aes-xts-plain … with the rest of my 128GB SD card.

I'm currently successfully downloading maps for OSM Scout Server. :-)

I added the needed bits to /usr/sbin/mount-sd.sh and I hope @Jolla will add an easy way for end users to do just this (small FAT for backing up plus rest luks+f2fs or supporting backup/restore on/from luks+f2fs). /usr/sbin/mount-sd.sh on the SailfishX being slightly different, I have attached my mount-sd.sh.patched.

Both partitions

Disk /dev/mmcblk1: 119.1 GiB, 127864930304 bytes, 249736192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x920700d3

Device         Boot    Start       End   Sectors   Size Id Type
/dev/mmcblk1p1         32768  33554431  33521664    16G  c W95 FAT32 (LBA)
/dev/mmcblk1p2      33554432 249735167 216180736 103.1G e8 unknown

mount fine on boot.

[nemo@Sailfish ~]$ lsblk /dev/mmcblk1
NAME                MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
mmcblk1             179:64   0 119.1G  0 disk  
|-mmcblk1p1         179:65   0    16G  0 part  /opt/alien/media/sdcard/6B[…]
`-mmcblk1p2         179:66   0 103.1G  0 part  
  `-mmcblk1p2-crypt 253:2    0 103.1G  0 crypt /opt/alien/media/sdcard/33f[…]
[nemo@Sailfish ~]$ df -h /media/sdcard/33[…]
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/mmcblk1p2-crypt
                      104G  5.3G   98G   6% /media/sdcard/33[…]
[nemo@Sailfish ~]$ df -h /opt/alien/media/sdcard/6B[…]
Filesystem            Size  Used Avail Use% Mounted on
/dev/mmcblk1p1         16G   64K   16G   1% /opt/alien/media/sdcard/6B[…]

One minor annoyance on the UI side; the _Storage_ screen under _Settings_ shows crypto_LUKS as unsupported and offers a Format to vfat button if selected. (see screenshots below).

EDIT: also the Camera application does not allow selecting the encrypted storage.

Storage screen

Format button

Camera app storage settings

edit flag offensive delete publish link more

Comments

Thanks for your feedback, @pcfe.

Nice to hear that above guide works flawlessly on the Xperia X and to see that the original mount-sd.sh script there shows only negligible differences (compared to the original one on a Jolla 1 phone), so the adaptions apply unaltered and at obvious places.

WRT the annoyances in SailfishOS' Settings --> Storage and its Camera app:

  • IMO it constitutes a usability bug, that Settings --> Storage so strongly suggests to format partitions on which it successfully detected a file system: It should rather rely on the exit code of the blkid command or if the returned variable TYPE contains anything, than looking up the returned TYPE in a list of known types to determine when to strongly suggest formatting.
    But OTOH Jolla's logic and reasoning is clear: Only ext4, vfat and btrfs are officially supported by SailfishOS, as they have stated multiple times.
  • SailfishOS' camera app stating "Memory card is unmounted" is likely a typical result of not considering encrypted partitions, as it seems to look for all partitions on the SD-card, which are mounted under /media/sdcard/. But as an encrypted partition is mounted under /dev/mapper/ (and that in turn is mounted under /media/sdcard/), the camera apps' current logic fails.
    BTW, the excellent "Open Camera" Android app (available at F-Droid) nicely writes to an encrypted partitions on SD-card, after executing part 2 "Externalising android_storage and other directories / files to SD-card" of this guide. Alternatively you may try the SailfishOS-native camera apps "MiracuCam" (Jolla Store) or CameraPlus (rsp. compiled for x86 (e.g. Jolla Tablet) or its alternative packages for ARMv7).

But ultimately both issues constitute rather minor annoyances, IMHO.

olf ( 2018-05-12 02:58:07 +0300 )edit
0

answered 2018-06-09 16:10:41 +0300

pcfe gravatar image

updated 2018-06-09 18:11:39 +0300

I just updated my Xperia X to Sailfish OS 2.2.0.29 (Mouhijoki) as follows

  1. Settings / Patch Manager: disable all patches
  2. reboot
  3. verify all patches are disabled
  4. upgrade Sailfish OS including doing a backup when offered by the UI
  5. verify that /etc/mmcblk1p2.key is still present. (it was, so no need to pull out the key backup)
  6. check for /usr/sbin/mount-sd.sh

At this point I noticed that there no longer is a /usr/sbin/mount-sd.sh to patch.

I'll go dig what mounts the µSD card in 2.2.0.29 and update this post with my findings.

===

EDIT: trying first to get it to mount manually is already failing, this will take more debugging time...

Seems 2.2.0.29 needs TLC on the luks side but both 1.7.5 and 1.6.4 did not work for me.

[root@Sailfish nemo]# /usr/sbin/cryptsetup --allow-discards --key-file /etc/mmcblk1p2.key luksOpen /dev/disk/by-uuid/47a7f0d8-cf18-4ea3-bb5d-742de2afec59 47a7f0d8-cf18-4ea3-bb5d-742de2afec59-crypt
Cannot initialize crypto backend.

Additionally I might be running into https://github.com/storaged-project/udisks/issues/481 with udisksctl

[root@Sailfish nemo]# lsblk
[...]
mmcblk1           179:64   0 119.1G  0 disk 
|-mmcblk1p1       179:65   0    16G  0 part /opt/alien/run/media/nemo/6B7C-0E63
`-mmcblk1p2       179:66   0 103.1G  0 part 
[...]
[root@Sailfish nemo]# blkid /dev/mmcblk1p2
/dev/mmcblk1p2: UUID="47a7f0d8-cf18-4ea3-bb5d-742de2afec59" TYPE="crypto_LUKS" PARTUUID="920700d3-02"
[root@Sailfish nemo]# udisksctl unlock --block-device /dev/disk/by-uuid/47a7f0d8-cf18-4ea3-bb5d-742de2afec59 --key-file /etc/mmcblk1p2.key 
Error unlocking /dev/mmcblk1p2: GDBus.Error:org.freedesktop.UDisks2.Error.Failed: Error unlocking /dev/mmcblk1p2: Failed to load device's parameters: Invalid argument

For now I'll do without encryption again, I need the phone working which includes providing me maps and music. None of these really warrant encryption.

Sadly this constant luks SNAFU on Sailfish means that as of now I can not use the phone for anything serious like work data :-(

I might have another poke at this if I one again have a spare card lying around for playing with. But not today, it's a sunny day and I already wasted too much time on something Jolla has been asked to provide out of the box for ages :-/

@olf: you may want to put off upgrading to 2.2.0.29 until I or someone else found time to debug this further.

pcfe

edit flag offensive delete publish link more

Comments

@pcfe, thanks for the heads up.

It was already apparent from the SFOS 2.2.0 release notes and changelog (and the comments there) during the quite short "EA" (early access) phase, that the mount-sd.sh shell script (on which the above solution relies) has been eliminated in favor of using udisks.
As I do not participate in the EA program and due to these changes hesitate to update my Jolla 1 yet, I just installed SFOS 2.2.0 on an Xperia X and will try to document how to mount existing, encrypted partitions on SD-card in the next couple of weeks.

The final aim is to reestablish above functionality (automatically mounting encrypted partitions) under SFOS 2.2.

olf ( 2018-06-11 00:30:52 +0300 )edit

Mounting encrypted partitions manually works just the way it did before SailfishOS 2.2.0 (if Cryptsetup is properly installed).
Example for an encrypted partition using Cryptsetup-LUKS:

  1. devel-su
  2. modprobe -v qcrypto # Only needed on Jolla 1 phones for using XTS
  3. cryptsetup -v --allow-discards -d /etc/mmcblk1p2.key luksOpen /dev/mmcblk1p2 mmcblk1p2-crypt # Might have to adapt partition number
  4. mount -v /dev/mapper/mmcblk1p2-crypt /media/sdcard/XXXX-XXXX/ -o noatime,users # Adapt UUID, eventually partition number and may add more mount options

But when updating to SailfishOS 2.2.0 or 2.1.4 with Cryptsetup installed and Jolla's mer-tools repository enabled, Jolla's broken cryptsetup-1.7.5+git1-1.2.1 is installed in the SailfishOS update process. You seem to have run into this issue.
In order to obtain a working Cryptsetup again, one has to uninstall all components of cryptsetup-1.7.5+git1-1.2.1 and install NielDK's cryptsetup-1.6.4, as described at the beginning of section 4.3.
HTH

olf ( 2018-07-08 14:15:53 +0300 )edit

Upgrading to SailfishOS 2.2.1 or higher without having removed NielDK's cryptsetup and libcryptsetup4 RPMs before that, results in a non-functional Cryptsetup (again), due to Jolla's cryptsetup and cryptsetup-libs RPMs being installed (working since SFOS 2.2.1) while NielDK's libcryptsetup4 RPM remains installed. The resulting "mixed" Cryptsetup installation does not work.
When trying to resolve this following above guide in this situation (i.e., under SFOS 2.1.1 and higher, in order to obtain a working installation of Jolla's Cryptsetup) a whole lot of dependent RPMs are going to be uninstalled, too.
Hence remove solely NielDK's libcryptsetup4 (with pkcon remove libcryptsetup4), if you ran into this situation (which seems to be O.K., i.e resulting in a properly working installation). Alternatively one may let pkcon uninstall all dependent RPMs along Cryptsetup (with pkcon remove cryptsetup cryptsetup-libs libcryptsetup4) and reinstall them right after that together with Jolla's Cryptsetup: This is the tedious, proper way for nitpickers (considering the possibility, that the uninstallation of the libcryptsetup4 RPM removes files, which were installed by Jolla's cryptsetup-libs RPM during the preceding SFOS upgrade), which bears its own dangers (a non-functional SailfishOS between the two steps) and thus shall be carried out booted into recovery mode (thus untested).

olf ( 2018-11-08 23:17:32 +0300 )edit
Login/Signup to Answer

Question tools

Follow
37 followers

Stats

Asked: 2018-02-11 23:20:08 +0300

Seen: 7,745 times

Last updated: Sep 20 '19