[How-to] Creating partitions on SD-card, optionally encrypted
Although there are a couple of guides for extending the internal mass storage (eMMC) of devices running SailfishOS (in general; this is not addressing the "tiny system partition issue" of SailfishOS on the Xperia X and Jolla C / Intex Aquafish) with a SD-card (on TMO and here on TJC), IMO none of them completely satisfactory (technically, by functionality and usability), but they provided valuable ideas.
Note: For SailfishOS 2.2.0 and newer, a reworked, updated and more streamlined version of this guide exists. This original how-to is kept for providing variations, details, technical background etc., and is still valid for SailfishOS 2.1.x.
- 0. Objectives and preface
- 1. Preparation (installing needed utilities etc.)
- 2 Analysing a specific SD-card
- 3. Re-partitioning of the SD-card
- 4. Creating filesystems
- 5. Finishing up
- 6. Notes
- 7. Additional notes
- 8. Appendix:
This is part 1 "Creating partitions on SD-card, optionally encrypted", you may want to continue with part 2 "Externalising android_storage and other directories / files to SD-card" to utilise the space on SD-card thereafter.
0. Objectives and preface
- Primarily rely on SailfishOS provided tools, including Mer-Tools (but without using another device, e.g. a PC), so the whole configuration can be carried out directly on the SailfishOS device.
- Not altering SailfishOS' basic filesystem structure and avoid altering existing files. Thus no /home/nemo etc. on SD-card.
- But with the ability to move android_storage and other files and directories under /home/nemo to an (optionally encrypted) partition on the SD-card (see part 2 for this).
- A solution for all devices with SailfishOS, so a SD-card can be moved between different devices, while keeping data on it accessible. Hence the filesystems used are primarily EXT4 and vFAT32, because this is what Jolla officially supports, but you may alternatively use BTRFS or F2FS, if compiled into your SailfishOS kernel (see
cat /proc/filesystems
) and broad interoperability is not a goal. - Protect partition(s) on SD-card by encryption, in order to keep the data on it safe, when the device is lost. I.e. the keys for DM-Crypt are based on a file stored at the internal mass store, which is accessible, when the SailfishOS device is unlocked. Additionally, with access to that file, one can copy it to other SailfishOS devices or Linux PCs to access the data on the SD-card there, too.
- And / or create unencrypted partition(s) for data exchange with any device and OS (by physically moving the SD-card), as "spare area" and for Jolla's Backup GUI-utility.
- Applicable to SDHC- and SDXC-cards of at least 4 GB (up to the maximum size the device hardware supports, as the technical limit of this scheme is 2 TBytes).
- Keep the reserved area of the SD-card intact.
- Align partition, DM-Crypt and filesystem data-structures to erase-block size / optimal alignment size.
Background on aligning data-structures on FLASH-based mass storage:
Write accesses to any NAND-FLASH based mass storage must write whole (complete) pages internally (typically 2^n multiples of 4 KBytes large), while for deleting data structures whole erase-blocks must be deleted. Note that the logical sector size such media exposes externally (usually 512 Bytes per logical sector) is much smaller. Recent NAND-FLASH memory chips (all FLASH-based mass storage is built with NAND-FLASH, plus a controller circuitry) have erase-block sizes of 1, 2, 4, 8, ... MBytes (while the numbers at Wikipedia refer to NAND-FLASH a couple of years ago, citing e.g. 512 KBytes; see also e.g. this explanation).
Not aligning data-structures to the erase-block size (or multiples of it) results in a much higher write amplification, leading to vastly degraded lifetime and lower performance of a reformatted SD-card.
README 1st:
Before you start, read the following sections completely and decide thereafter, if you (at least) roughly understand what these commands do (you may read their man pages, to be sure); if you don't, better leave it.
Mistyping (e.g. /dev/mmcblk0 instead of /dev/mmcblk1) may brick your SailfishOS device or quickly wear out your SD-card!
All this was tested on Jolla 1 phones and Xperia Xs under SailfishOS 2.1 and 2.2 (also should be applicable to SailfishOS 2.0) with multiple SD-cards, so if things turn out to look different with other devices or SD-cards, you may have to adapt the following sections (thus some understanding of the commands issued is crucial); and last but not least this description may be faulty. Please report your experiences and adaptations in these cases, but do not blame anyone (e.g. me) for any mishaps.
Contributions and constructive suggestions are welcome.
1. Preparation (installing needed utilities etc.)
Starting as user nemo in /home/nemo.
mkdir sd-card_layouts
# Create a directory for backing up the original and altered partition and file system layouts of the SD-card
devel-su
# Become root user
ssu ar mer-tools
# Add Jolla's mer-tools repository to /etc/ssu/ssu.ini; Only needed for SailfishOS versions below 2.2.1
pkcon refresh
# As usual, after adding a new repository
pkcon install util-linux
# fdisk is needed for analysing and partitioning a SD-card; it has been moved from a standalone package to util-linux SailfishOS 2.1.4
2 Analysing a specific SD-card
2.1 Analysing a SD-card's partition layout
cd sd-card_layouts
# Change to the backup directory created in section 1
First take a look at the partition layout with fdisk:
fdisk -l /dev/mmcblk1 | tee fdisk-l_mmcblk1-original.txt
Output of a 32 GB SDHC-card as an example:
Disk /dev/mmcblk1: 29 GiB, 31104958464 bytes, 60751872 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Device Boot Start End Sectors Size Id Type
/dev/mmcblk1p1 8192 60751871 60743680 29G c W95 FAT32 (LBA)
Note that there is (typically) a single FAT partition at /dev/mmcblk1p1 (some SD-cards may have a different partition layout, though) and the first LBA sector of this FAT partition is 8192 (which may vary depending upon the specific SD-card). One should retain this offset for the first partition (in this example of 8192 * 512 Bytes = 4 MBytes) for optimal alignment (see below).
Also note, that SD-cards usually "lie" about their internal page size for compatibility reasons and just report their externally provided logical sector size as physical sector size and optimal I/O size, even though the true page size is optimal for read and write accesses (while the erase-block size is optimal for deleting data).
2.2.a Analysing an existing FAT32 filesystem on a SDHC-card
To analyse a SDHC-card's pre-formatted FAT32 filesystem (advertised size typically "4 GB" to "32 GB"), use dosfsck (from the dosfstools):
pkcon install dosfstools
dosfsck -nv /dev/mmcblk1p1 | tee dosfsck-nv_mmcblk1p1-original.txt
Exemplary output of aforementioned 32 GB SD-card:
dosfsck 3.0.10 (12 Sep 2010)
dosfsck 3.0.10, 12 Sep 2010, FAT32, LFN
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID " "
Media byte 0xf8 (hard disk)
512 bytes per logical sector
32768 bytes per cluster
1556 reserved sectors
First FAT starts at byte 796672 (sector 1556)
2 FATs, 32 bit entries
3795968 bytes per FAT (= 7414 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 8388608 (sector 16384)
948864 data clusters (31092375552 bytes)
63 sectors/track, 255 heads
8192 hidden sectors
60743680 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/mmcblk1p1: 3 files, 2507/948864 clusters
Note that the two File Allocation Tables (FATs) of 7414 sectors each plus the 1556 reserved sectors make up for the file system internal offset of 16384 sectors, at which the data area of this FAT partition starts.
To check that the whole calculation matches, multiply the size of the data area with the cluster size in sectors (i.e. 64 for the typical 32 KBytes cluster size and 512 Bytes logical sector size) and add the file system internal offset: 948864 * 64 + 16384 = 60743680, which is the number of total sectors reported by fdisk and dosfsck for this partition.
Note that the number of hidden sectors refers to the number of sectors preceding this FAT partition (i.e. outside and before the FAT partition proper) and hence shall equal its first sector number as reported by fdisk (i.e. its offset).
2.2.b Analysing an existing exFAT filesystem on a SDXC-card
To analyse a SDXC-card's pre-formatted exFAT filesystem (advertised size typically "64 GB" or larger), use dumpexfat (from the exfat-utils):
Install exfat-utils from OpenRepos (e.g. via Storeman), then execute:
dumpexfat /dev/mmcblk1p1 | tee dumpexfat_mmcblk1p1-original.txt
Exemplary output of a 128 GB SD-card:
Volume label
Volume serial number 0x64323432
FS version 1.0
Sector size 512
Cluster size 131072
Sectors count 249704448
Free sectors 249670912
Clusters count 975280
Free clusters 975277
First sector 32768
FAT first sector 16384
FAT sectors count 16384
First cluster sector 32768
Root directory cluster 4
Volume state 0x0000
FATs count 1
Drive number 0x80
Allocated space 0%
The values correspond to the ones described in section 2.2.a (for dosfsck):
- "First sector" (dumpexfat) equals "hidden sectors" (dosfsck), i.e. partition offset.
- "FAT first sector" equals "First FAT starts at ... (sector X)", i.e. "reserved sectors".
- "FAT sectors count" equals "Y bytes per FAT (= X sectors)".
- "First cluster sector" equals "Data area starts at byte Y (sector X)".
- The cluster size is provided by both (dumpexfat and dosfsck) in Bytes (not sectors!).
Note that typically just a single FAT is used for exFAT filesystems, as this is sufficient for FLASH-based media (the typically two FATs used in pre-formatted FAT32 filesystems on SD-cards are primarily a compatibility measure).
Check your understanding of the original filesystem layout as described in section 2.2.a.
2.3 Determining the optimal alignment size for a specific SD-card
A good estimate for the SD-card's optimal alignment size is the smaller of these two offsets, the partition offset (in this example: 8192 sectors = 4 MB) and the FAT file system internal offset for its data area (in this example: 16384 sectors = 8 MB, due to the FATs occupying more than 4 MB in this case). The data area offset should be a multiple (1, 2, 3, ...) of the partition offset, otherwise your SD-card is strangely formatted.
To determine the SD-card's erase-block size, look at the output of cat /sys/block/mmcblk1/queue/discard_granularity
and especially cat /sys/block/mmcblk1/device/preferred_erase_size
, if your SD-card provides these values (in Bytes).
Note that the SD-card's optimal alignment size is a 2^n multiple (i.e. times 1, 2, 4, 8, ...) of the erase-block size, so they are not necessarily the same.
All in all it should be plausible and fit together:
- For the exemplary 32 GB SD-card, the partition offset is 4 MB, the data area offset 8 MB (due to the size of the FATs), discard_granularity is 4 MB and preferred_erase_size 4 MB: Hence the optimal alignment size is clearly 4 MB for this SD-card.
- For the 128 GB SD-card mentioned above, the partition offset is 16 MB, the data area offset 16 MB too, but discard_granularity and preferred_erase_size are 4 MB: So, while the erase-block size is 4 MB, the manufacturer clearly prefers an alignment size of 16 MB for this SD-card.
Thus one goal is to align all significant data-structures to the determined optimal alignment size (e.g. for the exemplary 32 GB SD-card: 8192 sectors with 512 Bytes = 4 MBytes; for the aforementioned 128 GB SD-card: 32768 sectors with 512 Bytes = 16 MBytes) in the following sections.
Check, if your SD-card supports discards:
When cat /sys/block/mmcblk1/queue/discard_zeroes_data
outputs "1" it definitely does, otherwise it likely does not.
Note, that all files on the SD-card will become inaccessible when carrying out the next steps, so now is the last chance to backup the data residing on your SD-card.
3. Re-partitioning of the SD-card
3.1 Preparation for re-partitioning
Note: For SailfishOS 2.2.0 (and newer) the path /media/sdcard/XXXXXXXX/ must be altered to /run/media/nemo/XXXXXXXX/ twice in section 3.1 (and in general).
Empty the existing filesystem(s) on SD-card completely in order to free the used blocks on the filesystem and the SD-card internally:
cd /media/sdcard/XXXXXXXX/
# Assure that you changed to this directory successfully, before proceeding:
pwd
# O.K.?
rm -rf * .[^.]*
# !
fstrim -v /media/sdcard/XXXXXXXX/
# For all filesystems other than FAT; fstrim does not work on FAT filesystems, but for FAT the mount option "discard" (used by SailfishOS) shall discard all empty blocks when executing the rm command.
Alternatively or additionally (preferably after having carried out the following paragraph: the unmounting), a blkdiscard -v /dev/mmcblk1pX
can be used on the raw, logical partitions (but be careful with the whole device /dev/mmcblk1).
Check, which partition(s) is actually mounted and how often:
mount | grep mmcblk1
# Each partition is mounted twice (at different locations), if you have AlienDalvik running
Unmount the mounted partition(s) on the SD-card by issuing, e.g.:
umount /dev/mmcblk1p1
# If the output is "umount: /dev/mmcblkXXX: not mounted", you picked the wrong partition
mount | grep mmcblk1
# Check again, it should be mounted once less, now
Continue to issue umount /dev/mmcblk1p1
until the output is "umount: /dev/mmcblkXXX: not mounted" (and consequently mount | grep mmcblk1
outputs nothing).
Become familiar with the fdisk tool:
fdisk /dev/mmcblk1
p
lists the partition layout as obtained in section 2m
provides the list of available commands- Do not set the "DOS compatibility flag" (with the command c), which is solely needed for computers running a real MS-DOS and will result in maximised write-amplification when activated (but it defaults to "off" in GNU fdisk since very long ago: more than 15 years).
- If you have the impression anything went wrong, simply exit fdisk (without writing the altered partition table) via
q
and start anew.
3.2.a Setting up a single FAT32, Linux-native (e.g. EXT4) or encrypted partition
If you do not want to create two or three partitions, just leave the partition layout as it is and solely alter the partition type (even that is optional): Hit t
, by default the only existing partition is selected and set its partition type to 83
("Linux") or if you just want to reformat an SDXC-card (pre-formatted with exFAT) with FAT32 to c
.
Alternatively (e.g. for maximising plausible deniability when planning to set up encryption on this partition and facing the danger to be forced to reveal the encryption scheme / key-file, e.g. in the UK) you may leave it as it is (i.e. not changing anything with fdisk) or set it to 0
("Unpartitioned"), which both should work, although other devices and operating systems may fail to recognise and to mount them automatically then (but e.g. SailfishOS has absolutely no issues automatically mounting a filesystem on a partition with a wrong type, as it probes them) or even try to reformat them.
Check by entering p
that the partition table now looks as it should, and finally write it to the SD-card (and exit fdisk) via w
(or q
, if unaltered).
3.2.b Setting up two or three partitions: A FAT32 or native one plus a native or / and encrypted one
Prerequisites for FAT32 partitions:
- When creating a FAT32 partition (type 0c) on a SD-card, it must become at least 16450560 sectors (with 512 Bytes each, i.e. 7,8 GBytes) large. Note that, although smaller size is theoretically possible, it does not make much sense to create such a small FAT partition IMO and requires a different partition type (0b) (to conform to the SD-card specification), which may provide additional hurdles, as that type usually indicates a non-LBA FAT32 partition (hence untested).
- The maximum size for a FAT32 partition is 2 TB (no exFAT necessary for this, but the latter allows for file sizes larger than 2 GB).
- A FAT partition should be the first primary partition on the SD-card, otherwise other operating systems may not recognise it.
- Thus for a dual partition layout including a FAT partition, the SD-card should be at least a "16 GB" one.
Continue in the fdisk /dev/mmcblk1
and delete the existing partition with d
.
Create a new, smaller first primary partition:
- Create a new first primary partition by entering
n
, thenp
, then use the default of "1". Recreate this partition with the same starting LBA-sector the original partition had (which has been obtained and saved in section 2, in the example of the aforementioned 32 GB SD-card 8192). - As an easy optimisation (without any drawbacks), determine the last sector for the first partition (especially if it is going to be a FAT partition) by subtracting 1 from a desired size of n GBytes in sectors (hence the multiplier is 1024 * 1024 * 2 = 2097152 for 512 Byte sectors). In my example (still with that 32 GB SD-card with an offset of 8192 sectors for the first partition) and aiming at creating a first primary partition of slightly less than 8 GBytes, this results in 8 * 2097152 - 1 = 16777215 as the last sector for the new partition (and a partition size of 16777215 +1 - 8192 = 16769024 sectors = 7,996 GBytes).
This measure also ensures that the second partition starts at an n GByte boundary. - Then enter this calculated value as the last sector for the new first partition.
- As Linux fdisk always sets the partition type of newly created partitions to 83 ("Linux"), hit
t
afterwards and set the partition type toc
("W95 FAT32 (LBA)"), if you want a FAT32 partition. If you want a native, unencrypted partition as first primary partition, leave it as it is (on 83).
Create a new, second partition (which will be configured later to be used either natively or with DM-Crypt):
- Start creating a new, second primary partition by entering
n
, thenp
for another primary partition (default, you may as well hit the Return key), then use the suggestion of "2". - Create this partition with the last LBA-sector of the first primary partition plus 1 as starting LBA-sector (e.g. with the exemplary 32 GB SD-card: 16777216). Note that fdisk will usually suggest a low sector number in the reserved ("hidden") area (e.g. 2048), which must not be used.
- Use the value obtained for the last sector of the original partition (in section 2; this is usually, but not necessarily the last sector of the SD-card) as last sector (on my exemplary 32 GB SD-card it was the last available sector, so if that was the case for your original partition layout too, you may as well just hit the Return key to avoid mishaps by mistyping).
If a third partition is going to be created, use+nG
(e.g. +10G) to set size of the second partition in GBytes. - When going to create an encrypted partition, set its partition type (by hitting
t
) toe8
("LUKS", not documented in fdisk, but elsewhere), for a Linux-native partition leave it on83
(and for FAT32 set it toc
).
Optionally, only for maximising plausible deniability when creating a DM-Crypt "plain" partition, you may set its partition type to0
(not tested).
If a third partition is wanted, repeat the steps for the second partition with fdisk for it: n
, p
, 3
, <last sector of second partition + 1>
, <noted last sector of original partition on sd-card>
.
Finally check by entering p
that the partition table now looks as it should, and write it to the SD-card (and exit fdisk) via w
.
Save the altered partition layout for reference / as a backup and perform a final check:
fdisk -l /dev/mmcblk1 | tee fdisk-l_mmcblk1-altered.txt
3.3 Checks and balances, when calculating the partition layout
However you may deviate from this guide, you should follow these rules:
- The first sector of any partition must be an even number, which must be evenly divisible by the erase-block size (or 0, if this indeed has been the starting sector of the original partition on the SD-card) and should be evenly divisible by the optimal alignment size.
- The last sector of any partition must be an odd number.
- The last sector of any partition plus 1 must be evenly divisible by the erase-block size (in sectors) and should be evenly divisible by the optimal alignment size.
- Consequently any partition size will be evenly divisible by the erase-block / optimal alignment size, too (just to double-check).
4. Creating filesystems
4.1 vFAT32 / FAT32 LFN (unencrypted)
Mind that FAT filesystems do not support access rights, hence are unsuited for putting android_storage on them (and for many other use cases), plus its anachronistic design, which does not fit well to FLASH-based mass storage. Furthermore FAT32 only supports files up to 2 GBytes size (though up to 2 TB filesystem size).
Thus a FAT partition is primarily suited for data exchange by physically moving the SD-card to other devices. The same applies to exFAT, except for the file size limit.
Mimic the output of dosfsck from section 2 as much as possible:
- Cluster size is 32, mostly 64 or 128 sectors, set with option -s; note that dosfsck and dumpexfat output the cluster size in Bytes, not sectors. If a preformatted exFAT partition had a cluster size of more than 128 sectors, use 128.
When the partition is primarily used as an archive, i.e. not very often written to and not with many small files, as an optimisation for FLASH-media you may consider setting the cluster size to its maximum value of 128 sectors (regardless of its original value), but this may introduce interoperability issues with other operating systems / firmwares, especially digital cameras etc. - Logical sector size is usually 512 Bytes, set with option -S; needed for the file-based simulations, but do not use when creating a filesystem on a SD-card (as its sector size shall be implicitly used, then).
- Hidden sectors count as obtained in section 2 is set with -h
- Another optimisation for FLASH-based memory is to use a single FAT instead of two with -f 1, but this may also introduce interoperability issues with other operating systems / firmwares, especially digital cameras etc.
As the FATs are smaller now (due to the smaller capacity of the newly created filesystem and FATs), the number of reserved sectors (set with option -R) has to be adapted. As mkfs.vfat knows no "dry run" and I found no formula for calculating the exact size of a FAT according to the size of the complete file system, one simply starts a "trial run" within a sparse file on /tmp:
mkfs.vfat -S 512 -v -s 64 -h 8192 -C /tmp/fat-test.img <filesystem size in KBytes>
# "Trial run 1", divide the intended number of 512 Bytes sectors by two for the filesystem size and set cluster size, hidden sectors count and sector size to the original values obtained in section 2 (if the cluster size of an pre-formatted exFAT partition on the SD-card was larger than 128 sectors, use 128 sectors)
dosfsck -nv /tmp/fat-test.img
# Compare to the output generated in section 2 (sector and cluster size) and check, if the resulting filesystem size (e.g. in sectors) in the image file is the intended one for the freshly created (in section 3) real partition
Now calculate the number of reserved sectors, which are needed to add up with the FATs (one or two) to any multiple (e.g. times 1, 2, 3, ...) of the determined alignment size. Note that the number of reserved sectors must be at least 8 (for FAT32) and shall be at least 32 or the number of sectors per cluster (whichever is higher); in above example of an almost 8 GBytes large partition on my exemplary 32GB SD-card, each FAT occupies 2048 sectors, so it takes 4096 reserved sectors to sum up to the alignment size of 8192 sectors.
Now check, if this calculation is correct:
rm /tmp/fat-test.img
# Delete the test file
mkfs.vfat -S 512 -v -s 64 -h 8192 -R 4096 -C /tmp/fat-test.img 8384512
# "Trial run 2" with 4096 reserved blocks as an example, for a filesystem slightly smaller than 8 GB ending on sector 16777215
(As a different example, also for slightly smaller than 8 GB filesystem ending on sector 16777215, but for the aforementioned 128 GB SD-card: mkfs.vfat -S 512 -v -f 1 -s 128 -h 32768 -R 31744 -C /tmp/fat-test.img 8372224
# Note that the offsets and consequently the filesystem size differ slightly)
dosfsck -nv /tmp/fat-test.img
# Check if everything looks as intended
rm /tmp/fat-test.img
# Delete the test file
Then do the real creation of the FAT filesystem on the SD-card:
mkfs.vfat -v -s 64 -h 8192 -R 4096 /dev/mmcblk1p1
# "Real run" with example values
dosfsck -nv /dev/mmcblk1p1 | tee dosfsck-nv_mmcblk1p1-altered.txt
# Last check, if everything is as intended, and backup
Side note: Whatever you may consider to do, never use the mkdosfs / mkfs.vfat options -a or -c on FLASH-based media (e.g. SD-cards)!
4.2 Linux-native filesystems (unencrypted)
4.2.a EXT4
EXT4 is fast, widely used, trustworthy and "light" on the device (memory consumption, processing power and consequently energy consumption), plus supported by all SailfishOS and Linux devices in general. While it originally has not been designed with FLASH-based mass storage in mind, it is not "FLASH-unfriendly", even when a journal is used for metadata only (e.g. as in the default mount option data=ordered).
For creating an EXT4 filesystem, the defaults in /etc/mk2fs.conf are fine.
You may consider formatting your EXT4 filesystem without a journal per option "-O ^has_journal", although IMO the drawbacks (fsck time and corrupted files, when writing has been interrupted) outweigh the advantage of being slightly nicer to FLASH-based memory.
Hence simply enter:
mkfs.ext4 -v /dev/mmcblk1pX
# Substitute "X" with 1 or 2
Side note: Whatever you may consider to do, never use the mkfs.extX option "-c -c" on FLASH-based media as a SD-card!
4.2.b BTRFS
While it took BTRFS some time to mature, since Linux kernel 3.4 it can be regarded as "production ready" (as the Jolla 1 phone shows). It has been designed with FLASH-based mass storage in mind, but has a couple of drawbacks, e.g. free space management (when over 85% full), performance on specific operations, being compute- and memory-intensive (especially when over 85% full), occasional "btrfs-balancer" runs are recommended for optimal space use and performance etc. OTOH BTRFS can provide transparent file compression, subvolumes etc.
If your kernel(s) support(s) BTRFS (see cat /proc/filesystems | grep btrfs
), you may format a partition with it by issuing:
mkfs.btrfs -f -O ^extref /dev/mmcblk1pX
# "-O ^extref" is only necessary on kernels before 3.7 (e.g. on Jolla 1 phones with their kernel 3.4.108) and for interoperability with them
(If mkfs.btrfs is not already on your device, install it per: pkcon install btrfs-progs
)
Mind to check the mount options for BTRFS as described in section "6 Notes" after finishing the setup process (and before using the BTRFS filesystem).
4.2.c F2FS
F2FS has been designed specifically for FLASH-based mass storage and to overcome some of the drawbacks BTRFS has, but it is quite new (for a filesystem, for which reliability is crucial), not widely supported and IMO one should not put critical data on F2FS before kernel 3.18 (see uname -a
on your device(s)).
If your kernel(s) support(s) F2FS (see cat /proc/filesystems | grep f2fs
), you may consider installing mkfs.f2fs per f2fs-tools from OpenRepos (e.g. via Storeman) and format a partition with it by issuing:
mkfs.f2fs /dev/mmcblk1pX
# The default options are fine
4.3 DM-Crypt (encrypted)
Encrypted volumes / partitions are a technically hard usage for NAND-FLASH based mass storage in some aspects. Nevertheless, this is one of the objectives, a desired general use case, relies on common components (DM-Crypt / cryptsetup and its volume format) and the hardness can be mostly alleviated by proper setup parameters.
It is easier for a FLASH-memory controller to handle this situation, the more "spare area" to "maneuver" (alike BTRFS) it has. Though this can be achieved by an unused (i.e. unformatted) partition with typically 12 - 20% of the SD-card's total capacity, a functional, unencrypted partition (of up to 20% of the SD-card's total capacity), which is only lightly used, is equally suited and hence preferable. This optimisation provides an additional incentive to use (at least) two partitions, e.g. an unencrypted and an encrypted one.
See what Cryptsetup versions are available and if you already have a Cryptsetup RPM installed with:
pkcon search name cryptsetup
Unfortunately Jolla's mer-tools repository for SailfishOS 2.1.4 and 2.2.0 provides a defunct cryptsetup-1.7.5+git1-1.2.1 (this is fixed in SailfishOS 2.2.1). If you have it installed, uninstall it with pkcon remove cryptsetup cryptsetup-libs
.
Cryptsetup versions before 1.4.0 are not well suited to be used for FLASH-based mass storage, as they lack the option "--enable-discards" (2011-07-07) and the fix "If device is not rotational, do not use Gutmann wipe method." (2011-10-05). Thus, if you have Jolla's cryptsetup version 1.1.3 (2010-07-03) installed under SailfishOS 2.1.3.7 or earlier, uninstall it with pkcon remove cryptsetup-luks
.
Consequently, on SailfishOS versions before 2.2.1, install (deviating from objective 1, but it is only used once, for setup) Cryptsetup 1.6.4 (compiled and packaged by NielDK) by issuing:
curl -O https://openrepos.net/sites/default/files/packages/500/cryptsetup-1.6.4-1.armv7hl.rpm
curl -O https://openrepos.net/sites/default/files/packages/500/libcryptsetup4-1.6.4-1.armv7hl.rpm
pkcon install-local libcryptsetup4-1.6.4-1.armv7hl.rpm cryptsetup-1.6.4-1.armv7hl.rpm
# Enter "y" to confirm installing
If you had NielDK's Cryptsetup 1.6.4 installed when upgrading to SailfishOS 2.1.4 or 2.2.0, his cryptsetup RPM is updated to Jolla's defunct cryptsetup RPM, while keeping NielDK's libcryptsetup4 RPM; in this case, remove it completely per pkcon remove cryptsetup libcryptsetup4
, then reinstall NielDK's Cryptsetup 1.6.4 as described above.
If you had NielDK's Cryptsetup 1.6.4 installed when upgrading to SailfishOS 2.2.1 or higher, only remove NielDK's libcryptsetup4 per pkcon remove libcryptsetup4
in order to obtain a working installation of Jolla's fixed Cryptsetup 1.7.5+git3-1.3.1.jolla.
Create a "key file", e.g. by creating a random passphrase file on the root partition:
dd if=/dev/urandom of=/etc/mmcblk1pX.key bs=64 count=1
# Mind to adapt the partition number
For LUKS, it can be well shorter than 64 Bytes and manually created or half-automatically (and thus much easier to memorize) with pwgen (e.g. from OpenRepos), just as it fulfills your needs (i.e. it does not need to be cryptographically strong, but nevertheless should be a well chosen / "strong" passphrase); in general it can be any absolutely static file on the root partition up to a few KBytes (at most 8 KB) long.
Back up the "key file":
cp /etc/mmcblk1pX.key ./
# Better (for security) move the backed-up "key file" off-device later on, i.e. deleting it (and the backups of the DM-Crypt header generated later) in ~nemo/sdcard-layouts/ after copying them away
chmod ug=r,o= /etc/mmcblk1pX.key
# Restrict access; the ownerships should be root:root
4.3.1 Cryptography options
- The "password" / key file hashing algorithm does not need to be cryptographically strong, it just has to generate a cryptographic hash of 160 bits (more is discarded), see Cryptsetup FAQ section "5.20 LUKS [...] uses SHA-1!". It must be supported by DM-Crypt, plus (before SailfishOS 2.2.1) the kernel (cryptography support, see cat /proc/crypto # type: shash) rsp. kernels of all devices the encrypted partition is going to be used with, or OpenSSL (SailfishOS 2.2.1). It also ought to be "light" to the devices used on in order to avoid prolonging mount- / boot-time and energy consumption unnecessarily. Also note, when using LUKS, it hashes a random "key file" or a strong password many hundred times. (But on PCs / in the future with Cryptsetup 2, if a LUKS2 header is used, -h sha256 is the right choice.)
Hence the obvious choice for the "password" hashing algorithm is SHA-1, set with:-h sha1
- The block cipher algorithm also must be properly supported (by DM-Crypt, plus kernel crypto before SailfishOS 2.2.1 / OpenSSL), should be algorithmically "light" and efficiently implemented. Thus the obvious practical choice is AES with 128 bit key, as AES-256 is computationally heavier (resulting in higher energy consumption) and slower (resulting in slower filesystem performance: ~ 25% in practice) and other algorithms are even slower and / or not broadly used.
- Chaining mode and initialisation vector ("IV") algorithm:
- If all kernels involved support XTS, use it, as it provides much faster random accesses than CBC.
Plus XTS needs no external IV-generation algorithm (as it implicitly uses AES for that), thus it should be used with "plain", as this is the computationally lightest IV-option ("plain64" is only needed for partitions larger than 2TB).
So the resulting cryptography options are:-c aes-xts-plain
- Note that the crypto kernel modules providing XTS support may not be loaded be default or on demand (see cat /proc/crypto | grep xts, cat /lib/modules/version/modules.builtin | fgrep crypto, ls -lR /lib/modules/version/kernel/crypto/, ls -lR /lib/modules/version/kernel/drivers/crypto/, lsmod etc.)
E.g. on Jolla 1 phones (i.e. up to and including SailfishOS 2.2.0, altered to use OpenSSL's cryptography by SailfishOS 2.2.1), one has to issue amodprobe -v qcrypto
to use XTS (do that once now and check again with cat /proc/crypto | grep xts, plus in the mount-sd.sh script edited in section 4.3.5 below for automatic start-up). - Otherwise use CBC, which needs an external IV-algorithm ("ESSIV") to protect against the watermarking attack. As DM-Crypt's ESSIV needs 256 bit input, SHA-256 is the obvious choice WRT "lightness", broad availability and use.
The resulting cryptography options are then:-c aes-cbc-essiv:sha256
- All other chaining modes available are outright broken (e.g. ECB), slower and / or not widely used.
- Note that XTS cuts the key used in two halves, so AES is only fed with half of the key length (the other half is used in the XTS chaining algorithm)!
Consequently, to set the AES key length to 128 bits, one has to specify-s 128
for CBC, but-s 256
for XTS.
- If all kernels involved support XTS, use it, as it provides much faster random accesses than CBC.
For more information on available cryptography options, see cryptsetup --help
, dmsetup targets
, cat /proc/crypto
and e.g. the Cryptsetup FAQ or ArchLinux wiki on Cryptsetup.
4.3.2 Experimenting with Cryptsetup / DM-Crypt
Create a image file for testing:
cd /media/sdcard/XXXXXXXX/
# As fallocate does not work on tmpfs with old kernel and fallocate versions, use it on an already mounted partition on the SD-card (need to mount manually or reboot and become root again) or alternatively in rsp. under /home/nemo on internal eMMC
fallocate -v -l 64M cryptsetup-test.img
# 64 MBytes are usually sufficient for experimenting
- For Cryptsetup-LUKS, e.g.:
cryptsetup -v -q -h sha1 -s 256 -c aes-xts-plain --align-payload=8192 luksFormat cryptsetup-test.img /etc/mmcblk1pX.key
# Set "--align-payload" to the alignment size in 512 Byte sectors (here with the usual example of 4MB); alter options for experimentingcryptsetup -v --allow-discards -d /etc/mmcblk1pX.key luksOpen cryptsetup-test.img luks-test
# Create DM-device file /dev/mapper/luks-testcryptsetup -v luksDump cryptsetup-test.img
# Check, if everything is O.K. and as expected
You may test the DM-device "luks-test", e.g. by creating, mounting and using a filesystem on it or dd'ing directly to / from it (e.g. withtime dd if=/dev/zero of=/dev/mapper/luks-test bs=4M count=8 && time dd if=/dev/mapper/luks-test of=/dev/null bs=4M count=8
).cryptsetup -v luksClose luks-test
# Close DM-devicerm cryptsetup-test.img
# Remove image file, when finished with testing - For Cryptsetup "type: plain", e.g.:
cat /etc/mmcblk1pX.key | cryptsetup -v -h sha1 -s 128 -c aes-cbc-essiv:sha256 --allow-discards --type plain open cryptsetup-test.img plain-test
# You may want to adapt
You may test the DM-device "plain-test" now (see bullet points above for details).cryptsetup -v close plain-test
# Close dm-mapper devicerm cryptsetup-test.img
# Remove image file, when finished with testing
4.3.3.a Setting up Cryptsetup-LUKS (modern, recommended)
The LUKS header created with the following commands stores all relevant information to access this DM-Crypt partition, lateron only a valid "key" (corresponding to one of the keyslots) has to be provided in order to unlock and mount it.
The first command differs, depending on the chaining mode used:
- For using XTS:
cryptsetup -v -q -h sha1 -s 256 -c aes-xts-plain --align-payload=8192 luksFormat /dev/mmcblk1pX /etc/mmcblk1pX.key
# Set "--align-payload" to the alignment size in 512 Byte sectors, pick the right partition number twice etc. - Alternatively, for using CBC:
cryptsetup -v -q -h sha1 -s 128 -c aes-cbc-essiv:sha256 --align-payload=8192 luksFormat /dev/mmcblk1pX /etc/mmcblk1pX.key
# Set "--align-payload" to the alignment size in 512 Byte sectors, pick the right partition number twice etc.
Continue with:
cryptsetup -v luksDump /dev/mmcblk1pX | tee ~nemo/sd-card_layouts/luksdump_mmcblk1pX.txt
# Last check & backup
cryptsetup -v --header-backup-file ~nemo/sd-card_layouts/luks-header-backup_mmcblk1pX.bin luksHeaderBackup /dev/mmcblk1pX
# Backup the Luks header, as without it being intact all encrypted data is inaccessible (e.g. when the Luks header on the SD-card is damaged)
cryptsetup -v --allow-discards -d /etc/mmcblk1pX.key luksOpen /dev/mmcblk1pX mmcblk1pX-crypt
# Create DM-device file
4.3.3.b Setting up Cryptsetup "type: plain" (prone to administrative errors, but maximising plausible deniability, as without a LUKS header an encrypted partition appears to contain solely random data)
Note that the "key" file shall provide sufficient entropy for Cryptsetup "type: plain", i.e. 64 Bytes of saved random data or a static high-entropy file of a couple of hundred Bytes (at most 512 Bytes) are O.K., regular passphrases are not.
Also note, that Cryptsetup's "type: plain" does not allow to align data structures (as it has none).
- For using XTS:
cat /etc/mmcblk1pX.key | cryptsetup -v -h sha1 -s 256 -c aes-xts-plain --allow-discards --type plain open /dev/mmcblk1pX mmcblk1pX-crypt
# Mind to pick the right partition number three times etc. - Alternatively, for using CBC:
cat /etc/mmcblk1pX.key | cryptsetup -v -h sha1 -s 128 -c aes-cbc-essiv:sha256 --allow-discards --type plain open /dev/mmcblk1pX mmcblk1pX-crypt
# Mind to pick the right partition number three times etc.
4.3.4 Creating a filesystem on encrypted partition
See section "4.2 Linux-native filesystems" for details of the mkfs.*
-commands for various filesystems, but use /dev/mapper/mmcblk1pX-crypt
as device (and insert the right partition number for "X")!
When successfully created a filesystem, do:
cryptsetup -v close mmcblk1pX-crypt
# Finishing setup steps by closing the DM-crypt mapper-file
4.3.5 Let SailfishOS automatically mount the encrypted partition
This method described in section 4.3.5 only applies to SailfishOS versions below 2.2.0!
For SailfishOS 2.2.x, use addendum below.
Up to (and including) SailfishOS 2.1.4 there seems to be no proper way for integrating filesystems on DM-Crypt partitions residing on SD-card seamlessly into SailfishOS without adapting one of its shell scripts.
Hence edit the shell script /usr/sbin/mount-sd.sh (backup first):
cd /usr/sbin
cp -p mount-sd.sh mount-sd.sh.orig
For use with DM-Crypt, add the following lines at the appropriate places ("diff -C 2" output; for cutting and pasting multiple lines, see complete script in section 8.1).
*** mount-sd.sh.orig
--- mount-sd.sh
***************
*** 28,31 ****
eval "$(/sbin/blkid -c /dev/null -o export /dev/$2)"
+ if [ "${TYPE}" = "crypto_LUKS" ]; then
+# /sbin/modprobe qcrypto # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
+ if /usr/sbin/cryptsetup --allow-discards -d /etc/$(basename ${DEVNAME}).key luksOpen ${DEVNAME} $(basename ${DEVNAME})-crypt
+ then
+ eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename ${DEVNAME})-crypt)" # Reassigning DEVNAME, TYPE and UUID
+ if [ "${TYPE}" = "crypto_LUKS" ] # Still? Then blkid did not output anything!
+ then
+ systemd-cat -t mount-sd /bin/echo "ERROR: Filesystem type missing for DM-crypt (LUKS) device ${DEVNAME}"
+ /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
+ exit 1
+ else systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (LUKS) device ${DEVNAME}"
+ fi
+ else
+ systemd-cat -t mount-sd /bin/echo "ERROR: Failed to open ${DEVNAME} with cryptsetup (LUKS)"
+ exit 1
+ fi
+ fi
+ if [ -z "${TYPE}" ] && [ -z "${UUID}" ] && [ "${DEVNAME}" = "$2" ]; then # trying cryptsetup "plain"
+# /sbin/modprobe qcrypto # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
+ for i in "sha1 256 aes-xts-plain" "sha1 128 aes-cbc-essiv:sha256" "ripemd160 256 aes-cbc-essiv:sha256"
+ do
+ cat /etc/$(basename $2).key | /usr/sbin/cryptsetup -h "$(echo "$i" | cut -f 1 -d ' ')" -s "$(echo "$i" | cut -f 2 -d ' ')" -c "$(echo "$i" | cut -f 3 -d ' ')" --allow-discards --type plain open /dev/$2 $(basename $2)-crypt # Cryptsetup "plain" will always succeed, when the key-file is readable and the device is accessible
+ eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename $2)-crypt)" # Reassigning DEVNAME, TYPE and UUID
+ if [ -n "${TYPE}" ] # Success?
+ then
+ systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (\"plain\") device ${DEVNAME}"
+ break
+ fi
+ /usr/sbin/cryptsetup close "$(basename $2)-crypt"
+ systemd-cat -t mount-sd /bin/echo "NOTICE: Tried opening ${DEVNAME} as DM-crypt \"plain\" device with options \"$i\", but failed to obtain a filesystem type"
+ DEVNAME="$2" ; unset UUID # Revert to state before trying cryptsetup "plain"
+ done
+ fi
+
if [ -z "${TYPE}" ]; then
***************
*** 99,102 ****
fi
umount $DIR || umount -l $DIR
+ if [ -e "/dev/mapper/$(basename ${DEVNAME})-crypt" ]; then
+ if /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
+ then systemd-cat -t mount-sd /bin/echo "Cryptsetup closed ${DEVNAME} successfully"
+ else systemd-cat -t mount-sd /bin/echo "WARNING: Cryptsetup failed to close ${DEVNAME}"
+ fi
+ fi
systemd-cat -t mount-sd /bin/echo "Finished ${ACTION}ing ${DEVNAME} at ${DIR}"
fi
When finished editing the shell script, back it up:
cp -p mount-sd.sh mount-sd.sh.patched
Mind that the altered shell script may be overwritten by SailfishOS updates, hence check for this then and reestablish the changes (in the newly updated file!), if overwritten (see timestamps with ls -l).
Ultimately reboot to see, if everything working as intended: Issue as user nemo mount | grep mmcblk1
If not, debug with journalctl -r
as root user, while having less installed (e.g. per pkcon install less
): Search with "/" (e.g. for mount-sd or mmcblk1), use "n" to jump to the next hit and press PageUp / PageDown to look at adjacent messages.
Note that due to a flaw in older Linux kernels you may see journal messages as follows WRT mapped DM-crypt partitions, which are harmless and should be ignored:
systemd-udevd[ZZZZ]: conflicting device node '/dev/mapper/mmcblk1pX-crypt' found, link to '/dev/dm-0' will not be created
systemd-udevd[YYYY]: inotify_add_watch(9, /dev/dm-0, 10) failed: No such file or directory
5. Finishing up
Reboot (if not done in previous section 4.3.3) and finally "donate" all mountpoints and filesystems on the SD-card to user nemo, else you cannot use them as nemo:
devel-su
For SailfishOS 2.1 / 2.0:
cd /media/sdcard/
chown -v nemo:nemo * */.
For SailfishOS 2.2:
cd /run/media/nemo/
chown -v nemo:nemo */.
You can check that discards are correctly configured with lsblk -D /dev/mmcblk1
(crypto-devices do not return zeroes).
6. Notes
- Multiple unencrypted partitions on a SD-card with filesystems supported by the SailfishOS kernel (e.g. EXT4, vFAT32) are all automatically mounted by SailfishOS (and with aforementioned changes to mount-sd.sh encrypted ones, too).
Additionally a filesystem (both unencrypted or encrypted) using the whole device mmcblk1 (i.e without a partition table) is also supported (but not generally recommended). - Note that the SailfishOS backup tool (in the SailfishOS settings) stores backups only on the first and unencrypted (stating it cannot write to it, although untrue, if encrypted) partition of a SD-card: If you rely on backing up your contacts etc. with SailfishOS' backup tool onto SD-card, create an unencrypted partition (formatted with any supported filesystem) at /dev/mmcblk1p1 (a few GBytes is sufficient, if a large encrypted partition is wanted).
While deviating from objective 2, there are special use cases for which you may consider the mount option "discard" (see the following section 7 "Additional notes" for details) for specific non-FAT filesystems (which support discards), just as SailfishOS does for FAT by default. To achieve this add in /usr-sbin/mount-sd.sh (which has to be altered for seamlessly using encrypted partitions on the SD-card under SailfishOS, anyway; see section 8.1 for the complete patched file, but with this change disabled by comment marks "#" there), e.g. for EXT4:
ext4) mount ${DEVNAME} $MNT/${UUID} -o "$MOUNT_OPTS",discard || /bin/rmdir $MNT/${UUID} ;;
- If you choose to use BTRFS, check and ensure that the mount options "noacl", "space_cache" and "ssd" are active (they are on a Jolla 1 under SFOS 2.1.3.7 and later). You may also consider using the mount options "autodefrag" (as on Jolla 1's internal filesystems, see output of
mount
), "compress=lzo" (promising size and speed improvements) and / or "enospc_debug" (e.g. by setting them in /usr/sbin/mount-sd.sh as shown in section 8.1), but do read its wiki-page for the mount options first.
If the mount option "ssd" is not automatically detected and enabled, consider using "ssd_spread". - The mount option "dirsync" for SD-cards in the "header" of /usr/sbin/mount-sd.sh is debatable (synchronous writes are "hard" to SSDs) and SailfishOS does not use it for internal eMMC, so I prefer to disable it (see script in section 8.1), consequently the "noatime" option also affects directories. If you really need directory access times to be recorded, use "diratime" in conjunction with "noatime" (or "relatime", if you need correct access times for files, too); both options ("diratime" and "relatime") are "hard" to SSDs (but less than "atime", "dirsync" or even "sync").
- One reason for objective 1 is, that all third party (FUSE-based) filesystems appear to be somewhat unmaintained, hence one cannot rely on them being updated, if incompatibilities with newer SailfishOS versions arise: F2FS has been retracted on Openrepos and for encfs_sailfish a precompiled archive or a RPM and a page at OpenRepos are also missing. The same has been the case throughout 2016 & 2017 for exFAT, although it was newly compiled and published 2018-01-15.
I wonder, if there are any hurdles to utilise the filesystem related kernel modules in /lib/modules/"latest"/kernel/fs/ (as provided by SailfishOS), which cover CIFS (SMB), NFS and eCryptFS. Answer: No hurdles, as FishNetMount does that for CIFS and NFS, so does cifs-utils for CIFS, but unfortunately I still have not found anything WRT eCryptFS.
7. Additional notes
(Originally planned as foreword, but it grew too large.)
- Use quality SD-cards, e.g. from SanDisk (any series, the "Ultra" series is reasonably priced and reasonably fast, or alternatively one from Samsung). I personally regard all other brands as low quality (as e.g. they do not manufacture controller, flash memory and firmware by themselves, in contrast to SanDisk and Samsung) and "Noname" cards as rubbish.
- If the SD-card has been heavily used before (especially if the partition layout was altered before), it may make sense to format it with the SD-card formatter from the SD Association (Windows and MacOS only, unfortunately), which is supposed to put the SD-card into a (more or less) juvenile state by issuing a "Secure erase" (likely addressing the "Security erase enhanced" command per ATA8-ACS3 specification, not the simple "Security erase") command to it and partition the SD-card anew with a layout conformant to the SD-card specification. The former can also be achieved with hdparm under e.g. Linux, but the options --security-erase-enhanced / --security-erase are still marked as "dangerous", while the option --trim-sectors (which sounds spot on) is still marked as "extremely dangerous" (all three for many years, already?!?), and I have not found a hdparm RPM for SailfishOS.
Hence under SailfishOS and Linux in generalblkdiscard -v /dev/mmcblk1pX
does that well (discarding all sectors of a partition / logical block device), without altering the SD-card's partition layout (as long as not used on the whole, physical device, e.g. /dev/mmcblk1). If a filesystem is recreated anyway, blkdiscard on its partition is a complementary / additional measure to issuing a fstrim command to its mount point when the old (non-FAT) filesystem still exists (i.e.fstrim -v /media/sdcard/XXXXXXXX
).
Do not use any tool or tool-option (e.g. blkdiscard -z), which overwrites every sector on the device: These are meant for HDDs and are absolutely detrimental to lifetime and performance of FLASH-based media (e.g. SD-cards). - It is generally not recommended to use the mount option "discard" for non-FAT partitions (as the synchronous nature of implementations of the discard / trim command in many SD-card firmwares slows down file-overwrite and -remove operations significantly), rather rely on SailfishOS issuing fstrim occasionally (Does it? Check in section 8.2 unsuccessful, yet.) and / or manually issue
fstrim -v /media/sdcard/XXXXXXXX
(working fine with fstrim from util-linux 2.28.1 provided by Jolla's mer-tools, using the check in section 8.2) every now and then (thus making this an asynchronous operation, plus some other advantages). Note that afstrim -av
trims all mounted filesystems, which it supports (e.g. EXT4, BTRFS), including the ones on internal eMMC (which does no harm, AFAICS on a Jolla 1 phone, and makes sense to apply occasionally, after having thoroughly cleaned up and balanced BTRFS). FAT filesystems are not supported by fstrim, hence they should be mounted with "discard".
You may still consider using "discard" for non-FAT filesystems on SD-card, if you do not put android_storage on SD-card and use it solely as an archive (e.g. for media files or backups), so slow erase and overwrite operations are irrelevant.
For further details, see e.g.
8. Appendix:
8.1 /usr/sbin/mount-sd.sh.patched
#!/bin/bash
# The only case where this script would fail is:
# mkfs.vfat /dev/mmcblk1 then repartitioning to create an empty ext2 partition
# Plus when used on a base device with a partition table (e.g. /dev/mmcblk1
# on every boot, when a regular SD-card is inserted) and (unpatched)
# when partitions are encrypted with DM-Crypt.
DEF_UID=$(grep "^UID_MIN" /etc/login.defs | tr -s " " | cut -d " " -f2)
DEF_GID=$(grep "^GID_MIN" /etc/login.defs | tr -s " " | cut -d " " -f2)
DEVICEUSER=$(getent passwd $DEF_UID | sed 's/:.*//')
MNT=/media/sdcard
MOUNT_OPTS="noatime,users" # "dirsync" removed
ACTION=$1
DEVNAME=$2
if [ -z "${ACTION}" ]; then
systemd-cat -t mount-sd /bin/echo "ERROR: Action needs to be defined."
exit 1
fi
if [ -z "${DEVNAME}" ]; then
systemd-cat -t mount-sd /bin/echo "ERROR: Device name needs to be defined."
exit 1
fi
systemd-cat -t mount-sd /bin/echo "Called to ${ACTION} ${DEVNAME}"
if [ "$ACTION" = "add" ]; then
# Setting variables DEVNAME (anew!), UUID and TYPE (but all three will be not set for "plain" DM-crypt devices):
eval "$(/sbin/blkid -c /dev/null -o export /dev/$2)"
if [ "${TYPE}" = "crypto_LUKS" ]; then
# /sbin/modprobe qcrypto # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
if /usr/sbin/cryptsetup --allow-discards -d /etc/$(basename ${DEVNAME}).key luksOpen ${DEVNAME} $(basename ${DEVNAME})-crypt
then
eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename ${DEVNAME})-crypt)" # Reassigning DEVNAME, TYPE and UUID
if [ "${TYPE}" = "crypto_LUKS" ] # Still? Then blkid did not output anything!
then
systemd-cat -t mount-sd /bin/echo "ERROR: Filesystem type missing for DM-crypt (LUKS) device ${DEVNAME}"
/usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
exit 1
else systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (LUKS) device ${DEVNAME}"
fi
else
systemd-cat -t mount-sd /bin/echo "ERROR: Failed to open ${DEVNAME} with cryptsetup (LUKS)"
exit 1
fi
fi
if [ -z "${TYPE}" ] && [ -z "${UUID}" ] && [ "${DEVNAME}" = "$2" ]; then # trying cryptsetup "plain"
# /sbin/modprobe qcrypto # Needed on SbJ1 under SFOS up to & incl. 2.1.4 for using XTS
for i in "sha1 256 aes-xts-plain" "sha1 128 aes-cbc-essiv:sha256" "ripemd160 256 aes-cbc-essiv:sha256"
do
cat /etc/$(basename $2).key | /usr/sbin/cryptsetup -h "$(echo "$i" | cut -f 1 -d ' ')" -s "$(echo "$i" | cut -f 2 -d ' ')" -c "$(echo "$i" | cut -f 3 -d ' ')" --allow-discards --type plain open /dev/$2 $(basename $2)-crypt # Cryptsetup "plain" will always succeed, when the key-file is readable and the device is accessible
eval "$(/sbin/blkid -c /dev/null -o export /dev/mapper/$(basename $2)-crypt)" # Reassigning DEVNAME, TYPE and UUID
if [ -n "${TYPE}" ] # Success?
then
systemd-cat -t mount-sd /bin/echo "Successfully opened DM-crypt (\"plain\") device ${DEVNAME}"
break
fi
/usr/sbin/cryptsetup close "$(basename $2)-crypt"
systemd-cat -t mount-sd /bin/echo "NOTICE: Tried opening ${DEVNAME} as DM-crypt \"plain\" device with options \"$i\", but failed to obtain a filesystem type"
DEVNAME="$2" ; unset UUID # Revert to state before trying cryptsetup "plain"
done
fi
if [ -z "${TYPE}" ]; then
# In case filesystem type is missing, try reading it.
TYPE=$(lsblk -n -o FSTYPE ${DEVNAME} | tail -n 1) # Fails grossly when trying to mount the basic device (e.g. mmcblk1) with multiple partitions: Picks the type of last partition (e.g. mmcblk1pX) then! Still no harm done, as mounting fails.
if [ -z "${TYPE}" ]; then
systemd-cat -t mount-sd /bin/echo "ERROR: Filesystem type missing for ${DEVNAME}"
exit 1
fi
fi
if [ -z "${UUID}" ]; then
# In case device does not have UUID lets create one for it based on
# the card identification.
PKNAME=$(lsblk -n -o PKNAME ${DEVNAME} | tail -n 1)
# If there is no PKNAME try NAME instead.
if [ -z "${PKNAME}" ]; then
PKNAME=$(lsblk -n -o NAME ${DEVNAME} | head -n 1)
fi
if [ -e "/sys/block/${PKNAME}/device/cid" ]; then
CID=$(cat /sys/block/${PKNAME}/device/cid)
if [ -n "${CID}" ]; then
IDNAME=$(lsblk -n -o NAME ${DEVNAME} | tail -1 | cut -d "-" -f2)
UUID="${CID}-${IDNAME}"
fi
fi
if [ -z "${UUID}" ]; then
# Exit here as in the future there might be things like USB OTG disks or
# sdcards attached via adapter that might behave differently and needs special case
# in case such happens fail so we don't break anything.
systemd-cat -t mount-sd /bin/echo "ERROR: Could not find or generate UUID for device ${DEVNAME}."
exit 1
fi
fi
DIR=$(grep -w ${DEVNAME} /proc/mounts | cut -d \ -f 2)
if [ -n "$DIR" ]; then
systemd-cat -t mount-sd /bin/echo "${DEVNAME} already mounted on ${DIR}, ignoring"
exit 0
fi
test -d $MNT/${UUID} || mkdir -p $MNT/${UUID}
chown $DEF_UID:$DEF_GID $MNT $MNT/${UUID}
touch $MNT/${UUID}
case "${TYPE}" in
vfat|exfat)
mount ${DEVNAME} $MNT/${UUID} -o uid=$DEF_UID,gid=$DEF_GID,$MOUNT_OPTS,utf8,flush,discard || /bin/rmdir $MNT/${UUID}
;;
# NTFS support has not been tested but it's being left to please the ego of an engineer!
ntfs)
mount ${DEVNAME} $MNT/${UUID} -o uid=$DEF_UID,gid=$DEF_GID,$MOUNT_OPTS,utf8 || /bin/rmdir $MNT/${UUID}
;;
# ext4)
# mount ${DEVNAME} $MNT/${UUID} -o $MOUNT_OPTS,discard || /bin/rmdir $MNT/${UUID}
# ;;
btrfs)
mount ${DEVNAME} $MNT/${UUID} -o $MOUNT_OPTS,enospc_debug,autodefrag,compress=lzo || /bin/rmdir $MNT/${UUID} # Check if chosen options suit your use well
;;
*)
mount ${DEVNAME} $MNT/${UUID} -o $MOUNT_OPTS || /bin/rmdir $MNT/${UUID}
;;
esac
test -d $MNT/${UUID} && touch $MNT/${UUID}
systemd-cat -t mount-sd /bin/echo "Finished ${ACTION}ing ${DEVNAME} of type ${TYPE} at $MNT/${UUID}"
else
DIR=$(grep -w ${DEVNAME} /proc/mounts | cut -d \ -f 2)
if [ -n "${DIR}" ] ; then
if [ "${DIR##$MNT}" = "${DIR}" ]; then
systemd-cat -t mount-sd /bin/echo "${DEVNAME} mountpoint ${DIR} is not under ${MNT}, ignoring"
exit 0
fi
umount $DIR || umount -l $DIR
if [ -e "/dev/mapper/$(basename ${DEVNAME})-crypt" ]; then
if /usr/sbin/cryptsetup close "$(basename ${DEVNAME})-crypt"
then systemd-cat -t mount-sd /bin/echo "Cryptsetup closed ${DEVNAME} successfully"
else systemd-cat -t mount-sd /bin/echo "WARNING: Cryptsetup failed to close ${DEVNAME}"
fi
fi
systemd-cat -t mount-sd /bin/echo "Finished ${ACTION}ing ${DEVNAME} at ${DIR}"
fi
fi
8.2 Practical trim / discard test
(adapted for SailfishOS, only usable without filesystem compression etc.)
devel-su
cat /sys/block/mmcblk1/queue/discard_zeroes_data
# "1", if discard / trim is supported
lsblk -D /dev/mmcblk1
# See, if discards are correctly configured
cd /media/sdcard/XXXXXXXX/
yes | dd bs=4K count=1K of=trim.test
# 4 MB; must consist of whole erase-blocks (at least one; if not, increase count)
sync ; echo 1 > /proc/sys/vm/drop_caches
hexdump trim.test
# outputs only "0a79"
filefrag -s -v trim.test
df trim.test
dd bs=<blocksize_from_filefrag> skip=<starting-offset_from_filefrag> count=<number-of-blocks_from_filefrag> if=<underlying-device_from_df> | hexdump
# outputs only "0a79"
rm trim.test
sync ; echo 1 > /proc/sys/vm/drop_caches
Then perform tests (fstrim, reboot, just wait etc.), issue
sync ; echo 1 > /proc/sys/vm/drop_caches
thereafter (if not rebooted) and ultimately repeat
dd bs=<blocksize_from_filefrag> skip=<starting-offset_from_filefrag> count=<number-of-blocks_from_filefrag> if=<underlying-device_from_df> | hexdump
If it still only outputs "0a79", no discard / trim occurred, if successful the output should be all zeroes (or random data for encrypted devices).
WOW! Amazing how-to. Don't get me wrong, but I hope this get partially obsolete with the announced encryption for SD card and filesystem ;-)
bomo ( 2018-02-12 22:34:12 +0200 )editSo do I, but no ETA announced, no exact feature set defined, in the past features took "Jolla time" to materialise (i.e. late or never) and I was finishing this last week after months of research (while most older guides have significant flaws and / or do not work anymore due to SailfishOS updates):
olf ( 2018-02-13 01:42:29 +0200 )editSo for the time being, this seems to be the only practical solution.
Great work! Thumbs up! I know what I will do tomorrow with my empty 64GB-SD-card. :-)
ds1979 ( 2018-02-14 19:39:41 +0200 )editHmm I notice that the kernel does supports f2fs. I feel fsf2-tools should be available in the repo so we can make f2fs partitions on the sdcard. f2fs is far preferable for solid state devices than ext4 is. From the wikipedia page
veritanuda ( 2018-02-19 23:02:48 +0200 )editYes, F2FS is somewhat nicer to FLASH-based mass storage than EXT4 (although by "far" is debatable), but just like BTRFS it is not available on all SailfishOS devices. Specifically the Jolla 1 phones may never receive F2FS support in their Linux kernel. Hence it is not suitable for a general description, but sure you can create a F2FS filesystem with mkfs.f2fs (from the f2fs-tools / f2fs-utils e.g. in Warehouse, if not already on your device) and use it in my guide instead of EXT4, if your Linux kernel supports F2FS.
What is the output of
[nemo@Sailfish ~]$ uname -soirvmp
on your device and which device do you use?On a Jolla 1 phone with SailfishOS 2.1.3.7 it is:
And
olf ( 2018-02-20 00:28:08 +0200 )edit[nemo@Sailfish ~]$ cat /proc/filesystems
does not list F2FS.