BTRFS forced readonly [answered]
asked 2018-07-01 23:24:22 +0200
So yesterday I wanted to quickly balance the BTRFS volume on my Jolla 1 using btrfs-balancer balance. It started to balance normally but at 75% my Jolla suddenly shut down.
After that I am not able to boot it up again. It boots up past the first Jolla logo and then the white LED stays on and it never boots.
I then went into the recovery mode and tried to recover the BTRFS filesystem with option 5 several times without success.
Using the shell with option 4 of the recovery mode I am able to mount the home subvolume as described here and browse my data.
I also tried to repair the btrfs volume several times with the following command in the shell:
/ # btrfs check --repair /dev/mmcblk0p28
enabling repair mode
Checking filesystem on /dev/mmcblk0p28
UUID: 0f8a2490-53ed-xxxx-xxxx-xxxxx
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
checking quota groups
Counts for qgroup id: 264 are different
our: referenced 817819648 referenced compressed 817819648
disk: referenced 817819648 referenced compressed 817819648
our: exclusive 813629440 exclusive compressed 813629440
disk: exclusive 4096 exclusive compressed 4096
diff: exclusive 813625344 exclusive compressed 813625344
Counts for qgroup id: 265 are different
our: referenced 38080512 referenced compressed 38080512
disk: referenced 38080512 referenced compressed 38080512
our: exclusive 3330048 exclusive compressed 3330048
disk: exclusive 4096 exclusive compressed 4096
diff: exclusive 3325952 exclusive compressed 3325952
found 5871114520 bytes used err is 0
total csum bytes: 10431468
total tree bytes: 427667456
total fs tree bytes: 387694592
total extent tree bytes: 24133632
btree space waste bytes: 140464600
file data blocks allocated: 11176611840
referenced 10299916288
Btrfs v3.16
But this does also not solve it either and I am still unable to boot my Jolla.
So executing dmesg in the shell prints me the following btrfs related output:
/ # dmesg
[...]
[ 925.904104] btrfs: disk space caching is enabled
[ 925.907462] btrfs: bdev /dev/mmcblk0p28 errs: wr 0, rd 0, flush 0, corrupt 0, gen 0
[ 925.923821] Btrfs detected SSD devices, enabling SSD mode
[ 926.360018] btrfs: unlinked 1 orphans
[ 926.361025] btrfs: continuing balance
[ 927.616664] btrfs: relocating block group 2108773105664 flags 1
[ 928.129467] ------------[ cut here ]------------
[ 928.129497] WARNING: at fs/btrfs/super.c:221 __btrfs_abort_transaction+0x38/0x98()
[ 928.129497] btrfs: Transaction aborted
[ 928.129528] Modules linked in:
[ 928.129558] [<c010b73c>] (unwind_backtrace+0x0/0x118) from [<c0173780>] (warn_slowpath_common+0x4c/0x64)
[ 928.129589] [<c0173780>] (warn_slowpath_common+0x4c/0x64) from [<c01737c4>] (warn_slowpath_fmt+0x2c/0x3c)
[ 928.129589] [<c01737c4>] (warn_slowpath_fmt+0x2c/0x3c) from [<c032566c>] (__btrfs_abort_transaction+0x38/0x98)
[ 928.129619] [<c032566c>] (__btrfs_abort_transaction+0x38/0x98) from [<c0335874>] (find_free_extent+0x9d4/0xc08)
[ 928.129650] [<c0335874>] (find_free_extent+0x9d4/0xc08) from [<c03396cc>] (btrfs_reserve_extent+0xe0/0x294)
[ 928.129680] [<c03396cc>] (btrfs_reserve_extent+0xe0/0x294) from [<c034ff90>] (__btrfs_prealloc_file_range+0xa8/0x2ec)
[ 928.129711] [<c034ff90>] (__btrfs_prealloc_file_range+0xa8/0x2ec) from [<c035557c>] (btrfs_prealloc_file_range+0x34/0x3c)
[ 928.129741] [<c035557c>] (btrfs_prealloc_file_range+0x34/0x3c) from [<c038a010>] (prealloc_file_extent_cluster+0x13c/0x19c)
[ 928.129741] [<c038a010>] (prealloc_file_extent_cluster+0x13c/0x19c) from [<c038a1e4>] (relocate_file_extent_cluster+0x64/0x368)
[ 928.129772] [<c038a1e4>] (relocate_file_extent_cluster+0x64/0x368) from [<c038a52c>] (relocate_data_extent+0x44/0xe0)
[ 928.129803] [<c038a52c>] (relocate_data_extent+0x44/0xe0) from [<c038f398>] (relocate_block_group+0x3a8/0x5a4)
[ 928.129833] [<c038f398>] (relocate_block_group+0x3a8/0x5a4) from [<c038f6dc>] (btrfs_relocate_block_group+0x148/0x2a4)
[ 928.129864] [<c038f6dc>] (btrfs_relocate_block_group+0x148/0x2a4) from [<c036efec>] (btrfs_relocate_chunk.isra.12+0x3c/0x688)
[ 928.129894] [<c036efec>] (btrfs_relocate_chunk.isra.12+0x3c/0x688) from [<c0372ee4>] (btrfs_balance+0xbb4/0xd6c)
[ 928.129894] [<c0372ee4>] (btrfs_balance+0xbb4/0xd6c) from [<c03730e8>] (balance_kthread+0x4c/0x6c)
[ 928.129925] [<c03730e8>] (balance_kthread+0x4c/0x6c) from [<c0192000>] (kthread+0x84/0x90)
[ 928.129955] [<c0192000>] (kthread+0x84/0x90) from [<c0106778>] (kernel_thread_exit+0x0/0x8)
[ 928.129986] ---[ end trace da227214a82491b9 ]---
[ 928.129986] BTRFS error (device mmcblk0p28) in find_free_extent:5803: error 28
[ 928.130016] btrfs is forced readonly
[ 928.211628] btrfs-balance (269): undefined instruction: pc=c038efc0
[ 928.211658] Code: e3a03001 ebfeabf4 e3500000 aa000000 (e7f001f2)
[ 928.211689] ------------[ cut here ]------------
[ 928.211719] kernel BUG at fs/btrfs/relocation.c:2278!
[ 928.211719] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[ 928.211750] Modules linked in:
[ 928.211750] CPU: 1 Tainted: G W (3.4.108.20171107.1 #1)
[ 928.211780] PC is at merge_reloc_roots+0xf4/0x124
[ 928.211811] LR is at btrfs_drop_snapshot+0x540/0x564
[ 928.211811] pc : [<c038efc0>] lr : [<c033a4cc>] psr: a0000013
[ 928.211811] sp : d2d31d98 ip : 00380039 fp : ee763508
[ 928.211841] r10: ee763000 r9 : ffffffe2 r8 : ee8f4c00
[ 928.211841] r7 : fffffe5c r6 : ee7634f0 r5 : d2d31d98 r4 : ee763000
[ 928.211872] r3 : 271ae970 r2 : 271ae970 r1 : c092cdb8 r0 : ffffffe2
[ 928.211872] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 928.211902] Control: 10c5787d Table: ae9ec06a DAC: 00000015
[... ]
Some googling brought me to the following link: https://www.suse.com/support/kb/doc/?id=7018233 where it says the following:
Background: This issue can be triggered _only_ with a filesystem where almost no free space is available anymore. Under rare conditions balancing the filesystem tries to relocate an extend which fails due to missing space. The error is unconditionally overwritten and success returned. The presumably relocated extent is missing later because it wasn't actually relocated. This results in remounting the filesystem readonly.
It should be noted that none of the file systems affected by this are suffering corruption. This is entirely a runtime issue and no repair is required. User action as noted above is required to avoid hitting the issue immediately on remount once the issue is encountered, though.
and it recommends to execute the following command to stop the automatic btrfs balancing opperation:
mount -o remount,rw,skip_balance <mountpoint>
So I executed mount -o remount,rw,skip_balance /dev/mmcblk0p28 and rebooted my Jolla but it still does not work.
Could someone please help me to recover my Jolla without factory resetting?