Unrecoverable Btrfs errors
asked 2014-08-11 23:42:25 +0300
My phone stopped booting and there seem to be unrecoverable errors in the btrfs root filesystem.
Recovery menu entries for factory reset and btrfs recovery just hang when they try to mount the btrfs, which causes the kernel errors in dmesg below. (Thus, it's different from https://together.jolla.com/question/53061/btrfs-recovery-segfaults-when-trying-to-repair-root-partition/ )
[ 346.453044] device label sailfish devid 1 transid 2100490 /dev/mmcblk0p28
[ 346.455119] btrfs: disk space caching is enabled
[ 346.458782] btrfs: bdev /dev/mmcblk0p28 errs: wr 0, rd 0, flush 0, corrupt 0, gen 0
[ 346.466839] Btrfs detected SSD devices, enabling SSD mode
[ 346.471173] btrfs: free space inode generation (0) did not match free space cache generation (1992516)
[ 346.483656] btrfs: corrupt leaf, bad key order: block=121073664,root=1, slot=18
[ 346.483717] ------------[ cut here ]------------
[ 346.483747] WARNING: at fs/btrfs/super.c:221 __btrfs_abort_transaction+0x38/0x98()
[ 346.483778] btrfs: Transaction aborted
[ 346.483808] Modules linked in:
[ 346.483961] [<c0014c5c>] (unwind_backtrace+0x0/0x11c) from [<c007de78>] (warn_slowpath_common+0x4c/0x64)
[ 346.484205] [<c007de78>] (warn_slowpath_common+0x4c/0x64) from [<c007df10>] (warn_slowpath_fmt+0x2c/0x3c)
[ 346.484266] [<c007df10>] (warn_slowpath_fmt+0x2c/0x3c) from [<c0221f48>] (__btrfs_abort_transaction+0x38/0x98)
[ 346.484297] [<c0221f48>] (__btrfs_abort_transaction+0x38/0x98) from [<c0230858>] (__btrfs_free_extent+0x6a8/0x6cc)
[ 346.484358] [<c0230858>] (__btrfs_free_extent+0x6a8/0x6cc) from [<c0235430>] (run_clustered_refs+0x850/0x964)
[ 346.484419] [<c0235430>] (run_clustered_refs+0x850/0x964) from [<c023579c>] (btrfs_run_delayed_refs+0x258/0x3d8)
[ 346.484632] [<c023579c>] (btrfs_run_delayed_refs+0x258/0x3d8) from [<c0244a70>] (btrfs_commit_transaction+0x9c/0x940)
[ 346.484663] [<c0244a70>] (btrfs_commit_transaction+0x9c/0x940) from [<c027f308>] (btrfs_recover_log_trees+0x364/0x3b8)
[ 346.484724] [<c027f308>] (btrfs_recover_log_trees+0x364/0x3b8) from [<c0243420>] (open_ctree+0xfa8/0x13fc)
[ 346.484785] [<c0243420>] (open_ctree+0xfa8/0x13fc) from [<c021dbc0>] (btrfs_mount+0x488/0x888)
[ 346.484999] [<c021dbc0>] (btrfs_mount+0x488/0x888) from [<c013cb74>] (mount_fs+0x10/0xb0)
[ 346.485060] [<c013cb74>] (mount_fs+0x10/0xb0) from [<c0151f20>] (vfs_kern_mount+0x4c/0xb4)
[ 346.485121] [<c0151f20>] (vfs_kern_mount+0x4c/0xb4) from [<c015289c>] (do_kern_mount+0x34/0xd0)
[ 346.485151] [<c015289c>] (do_kern_mount+0x34/0xd0) from [<c0154094>] (do_mount+0x614/0x690)
[ 346.485212] [<c0154094>] (do_mount+0x614/0x690) from [<c0154278>] (sys_mount+0x84/0xc4)
[ 346.485243] [<c0154278>] (sys_mount+0x84/0xc4) from [<c000df80>] (ret_fast_syscall+0x0/0x30)
[ 346.485304] ---[ end trace da227214a82491b9 ]---
[ 346.485334] BTRFS error (device mmcblk0p28) in __btrfs_free_extent:5173: IO failure
[ 346.485365] btrfs: run_one_delayed_ref returned -5
[ 346.485395] BTRFS error (device mmcblk0p28) in btrfs_run_delayed_refs:2455: IO failure
[ 346.485426] BTRFS warning (device mmcblk0p28): Skipping commit of aborted transaction.
[ 346.485487] BTRFS error (device mmcblk0p28) in cleanup_transaction:1226: IO failure
[ 346.591301] btrfs: corrupt leaf, bad key order: block=121073664,root=1, slot=18
[ 346.591637] btrfs: corrupt leaf, bad key order: block=121073664,root=1, slot=18
Latest btrfsck from upstream git is not able to fix the errors either, and just aborts in the middle of it:
checking extents
bad key ordering 18 19
bad key ordering 18 19
bad key ordering 18 19
bad key ordering 18 19
bad block 121073664
Errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots
bad key ordering 18 19
bad key ordering 18 19
root 268 inode 335 errors 200, dir isize wrong
root 268 inode 390 errors 2001, no inode item, link count wrong
unresolved ref dir 334 index 3 namelen 7 name America filetype 2 errors 4, no inode ref
root 268 inode 559 errors 2001, no inode item, link count wrong
unresolved ref dir 334 index 4 namelen 10 name Antarctica filetype 2 errors 4, no inode ref
...
root 268 inode 822468 errors 2001, no inode item, link count wrong
unresolved ref dir 268 index 3226 namelen 18 name harbour-citybikeme filetype 1 errors 4, no inode ref
root 268 inode 822471 errors 2001, no inode item, link count wrong
unresolved ref dir 321 index 300 namelen 4 name solv filetype 1 errors 4, no inode ref
root 268 inode 939477 errors 2001, no inode item, link count wrong
unresolved ref dir 261 index 3581 namelen 9 name boardname filetype 1 errors 4, no inode ref
parent transid verify failed on 190312448 wanted 216588 found 2098833
parent transid verify failed on 190312448 wanted 216588 found 2098833
parent transid verify failed on 190312448 wanted 216588 found 2098833
parent transid verify failed on 190312448 wanted 216588 found 2098833
Ignoring transid failure
btrfsck: cmds-check.c:1342: walk_down_tree: Assertion `!(1)' failed.
Would it be ok to create a new btrfs filesystem and re-use the filesystem recovery snapshots factory-@ and factory-@home from another phone? (Or are there device-specific parameters/keys stored there?)
Update: I went ahead with the repair attempt with some support from users tbr and Stskeeps on the #sailfishos chat channel and with the filesystem recovery snapshots from another phone. In the end, I have a working phone again!
For reference, here's the steps I took - zero guarantees that they won't break your phone though: https://gist.github.com/tuukka/7a81112ef161ee77eccb
Please, build something like this (or the much simpler full btrfs filesystem backup and restore) into the recovery menu.
TuukkaH ( 2014-08-13 13:59:28 +0300 )editUpdate: Entering comment via my N9, I failed.
One of the first things I do when I finally get a Jolla is to build that recovery image, using ideas from TuukkaH's link. Since technically no one would be redistributing it, would it be legal if Jolla supply a script that at first start, can build a recovery image, and store it on an SD card or computer?
GD ( 2014-08-27 13:35:56 +0300 )edit