Ask / Submit
3

Last call for help: Restore btrfs or bricked?

asked 2017-05-15 23:04:32 +0200

NGC-Ollie gravatar image

Hello community, I already contacted the care and support, but they need my Jolla 1 with an insecure outcome which means: not an option.

History of what happened:

Undefined behavioral on 2.1.0.11, sth. like file system full, but wasn't the case, I always use my SD card. You'll see that below:

  1. After reboot apps didn't start anymore,
  2. 3 times hooray for my backups, tried factory reset over preference app, got stuck.
  3. Saw, read and applied the recovery mode article
  4. Boot loop
  5. Recovery mode console: Mount, analyse and repair:

    / # mount -t btrfs -o skip_balance /dev/mmcblk0p28 /mnt/
    / # btrfs fi show
    ERROR: unable get label Inappropriate ioctl for device
    Label: 'sailfish'  uuid: …
            Total devices 1 FS bytes used 2.73GiB
            devid    1 size 13.75GiB used 5.75GiB path /dev/mmcblk0p28
    
    Btrfs v3.16
    ----- Reboot ----
    / # btrfs rescue chunk-recover /dev/mmcblk0p28
    Check chunks successfully with no orphans
    Recover the chunk tree successfully.
    / # btrfs check --repair /dev/mmcblk0p28
    enabling repair mode
    Checking filesystem on /dev/mmcblk0p28
    UUID: …
    checking extents
    bad item end 3999 wanted 3995
    […]
    Incorrect local backref count on 4032372736 root 269 owner 4957 offset 655360 found 0 wanted 1 back 0x2229818
    Backref disk bytenr does not match extent record, bytenr=4032372736, ref bytenr=0
    backpointer mismatch on [4032372736 528384]
    owner ref check failed [4032372736 528384]
    ref mismatch on [4032921600 720896] extent item 1, found 0
    attempting to repair backref discrepency for bytenr 4032921600
    Failed to find [158896128, 168, 4096]
    btrfs unable to find ref byte nr 158896128 parent 0 root 269  owner 2 offset 0
    repaired damaged extent references
    Failed to find [31232000, 168, 4096]
    btrfs unable to find ref byte nr 31232000 parent 0 root 1  owner 1 offset 0
    btrfs: extent-tree.c:1752: write_one_cache_group: Assertion `!(ret)' failed.
    Aborted
    / # btrfs fi show
    Label: 'sailfish'  uuid: …
            Total devices 1 FS bytes used 2.73GiB
            devid    1 size 13.75GiB used 5.75GiB path /dev/mmcblk0p28
    
    Btrfs v3.16
    
  6. Nothing changed, first support mail. Their solution: an over-full file system --> balancing. I was wondering, because my home is empty and the file system only uses 5GB, but why not:

    /mnt/rootfs # ls -la … drwxr-xr-x 1 0 0 2164 Aug 19 2016 etc -rw-r----- 1 0 2000 980 Oct 2 2015 fstab.qcom drwxr-xr-x 1 0 0 0 Aug 19 2016 home …

    fresh restart and mount everything:

    -----------------------------
     Jolla Recovery v0.3.1
    -----------------------------
    Welcome to the recovery tool!
    The available options are:
    1) Reset device to factory state
    2) Reboot device
    3) Bootloader unlock [Current state: locked]
    4) Shell
    5) Try btrfs recovery if your device is in bootloop
    6) Exit
    Type the number of the desired action and press [Enter]:
    4
    
    If you continue, this may void your warranty. Are you really SURE? [y/N] y
    [OK] No lockcode has been set. Proceeding...
    / # mount /dev/mmcblk0p28 /mnt
    / # mount
    rootfs on / type rootfs (rw)
    none on /dev type devtmpfs 
    (rw,relatime,size=412500k,nr_inodes=103125,mode=755)
    proc on /proc type proc (rw,relatime)
    sysfs on /sys type sysfs (rw,relatime)
    none on /tmp type tmpfs (rw,relatime,size=10240k)
    /proc/sys/debug on /proc/sys/debug type debugfs (rw,relatime)
    none on /var/run type tmpfs (rw,nosuid,noatime,size=256k,mode=755)
    devpts on /dev/pts type devpts (rw,relatime,mode=600)
    /dev/mmcblk0p28 on /mnt type btrfs (rw,relatime,ssd,noacl,space_cache)
    / # btrfs balance start -v -dusage=50 /mnt
    Dumping filters: flags 0x1, state 0x0, force is off
     DATA (flags 0x2): balancing, usage=50
    ERROR: error during balancing '/mnt' - Read-only file system
    There may be more info in syslog - try dmesg | tail
    / # dmesg | tail
    [ 29.293087] [<c038f13c>] (relocate_block_group+0x528/0x5a0) from 
    [<c038f2fc>] (btrfs_relocate_block_group+0x148/0x2a4)
    [ 29.293117] [<c038f2fc>] (btrfs_relocate_block_group+0x148/0x2a4) 
    from [<c036eba4>] (btrfs_relocate_chunk.isra.12+0x3c/0x688)
    [ 29.293148] [<c036eba4>] (btrfs_relocate_chunk.isra.12+0x3c/0x688) 
    from [<c0372aa8>] (btrfs_balance+0xbbc/0xd74)
    [ 29.293148] [<c0372aa8>] (btrfs_balance+0xbbc/0xd74) from 
    [<c0372cac>] (balance_kthread+0x4c/0x6c)
    [ 29.293178] [<c0372cac>] (balance_kthread+0x4c/0x6c) from 
    [<c0191ecc>] (kthread+0x84/0x90)
    [ 29.293209] [<c0191ecc>] (kthread+0x84/0x90) from [<c0106758>] 
    (kernel_thread_exit+0x0/0x8)
    [ 29.293239] Code: e3a03001 ebfeabe0 e3500000 aa000000 (e7f001f2)
    [ 29.293270] ---[ end trace da227214a82491ba ]---
    [ 29.293270] kevent_trigger: queue event work event id is 0
    [ 29.293331] kevent_work: send kobject uevent, EVENT=Oops
    / # mount
    rootfs on / type rootfs (rw)
    none on /dev type devtmpfs 
    (rw,relatime,size=412500k,nr_inodes=103125,mode=755)
    proc on /proc type proc (rw,relatime)
    sysfs on /sys type sysfs (rw,relatime)
    none on /tmp type tmpfs (rw,relatime,size=10240k)
    /proc/sys/debug on /proc/sys/debug type debugfs (rw,relatime)
    none on /var/run type tmpfs (rw,nosuid,noatime,size=256k,mode=755)
    devpts on /dev/pts type devpts (rw,relatime,mode=600)
    /dev/mmcblk0p28 on /mnt type btrfs (ro,relatime,ssd,noacl,space_cache)
    / # btrfs fi show
    ERROR: unable get label Inappropriate ioctl for device
    Label: 'sailfish' uuid: …
     Total devices 1 FS bytes used 2.73GiB
     devid 1 size 13.75GiB used 5.75GiB path /dev/mmcblk0p28
    
    Btrfs v3.16
    / #
    

    The file system turns suddenly into a read only system

  7. Found this post: https://together.jolla.com/question/160081/phone-bricked-during-last-update/?answer=160350#post-id-160350 used dd in this readme' http://www.htheb.com/list/jolla/factoryimage/Readme.txt

  8. Boot loop [end of ideas]

TL;DR: Brick?

edit retag flag offensive close delete

Comments

1

Can you enable persistent logging while in recovery mode and post the result of journalctl here?

louisbob ( 2017-05-16 00:04:10 +0200 )edit
1

Btrfs is odd, but don't you do a check on an unmounted volume?

chemist ( 2017-05-16 15:37:58 +0200 )edit

When I've had btrfs issues in the past (non-Jolla) I've had success using newer versions of the btrfs tools (eg, you might be able to get a newer version off OBS, or from a different distro and then copy them on?) -- or maybe try exposing the partition via the telnet interface (eg, I'm pretty sure I've used sshfs (to the device) when in rescue mode) to allow interacting with the partition directly (eg to allow dd-ing it to make an image to hack at more aggressively)

bruce_one ( 2017-05-17 12:32:15 +0200 )edit

@luisbob: which recovery do you mean? btrfs balance, btrfs rescue chunk-recover, btrfs rescue super-recover, factory reset or the dd?

@chemist: Nope, I did a reboot in between

@bruce_one: can you give me hint which version you used? I don't want to run into unresolved dependencies or sth.

Ah and sorry for my delay, you know, work and stuff to do, but i appreciate your help, thanks a lot!

NGC-Ollie ( 2017-05-21 16:44:37 +0200 )edit

1.) Go into recovery mode: https://jolla.zendesk.com/hc/en-us/articles/204709607 (watch out, for jolla C you need to use the UP volume key, not the down one)

2.) In the menu, select Shell

3.) When the shell prompt appears, and edit the following file: /rootfs/etc/systemd/journald.conf

4.) Change the line "Storage=volatile" to "Storage=persistent"

5.) Reboot, wait a bit so your log will be populated with error hopefully.

6.) Go back to the recovery mode -> Shell, and use journalctl command to display the log. Note that you will probably need to do a chroot before (command is "chroot /rootfs/")

Resources: https://together.jolla.com/question/4842/how-to-enable-more-detailed-and-persistent-logs-on-jolla-device/

louisbob ( 2017-05-21 21:27:28 +0200 )edit

2 Answers

Sort by » oldest newest most voted
1

answered 2017-05-21 18:05:12 +0200

NGC-Ollie gravatar image

updated 2017-05-21 18:07:22 +0200

Dear all, thank you, I did it, but I don't know how. I did the same things in the same order and now the factory reset did his job. Yes btrfs is odd, here my log:

/ # btrfs rescue chunk-recover -v /dev/mmcblk0p28
All Devices:
        Device: id = 1, name = /dev/mmcblk0p28

DEVICE SCAN RESULT:
Filesystem Information:
        sectorsize: 4096
        leafsize: 4096
        tree root generation: 20
        chunk root generation: 8

All Devices:
        Device: id = 1, name = /dev/mmcblk0p28

All Block Groups:
        Block Group: start = 0, len = 4194304, flag = 2
        Block Group: start = 4194304, len = 8388608, flag = 4
        Block Group: start = 12582912, len = 8388608, flag = 1
        Block Group: start = 20971520, len = 8388608, flag = 22
        Block Group: start = 29360128, len = 1073741824, flag = 24
        Block Group: start = 1103101952, len = 1073741824, flag = 1

All Chunks:
        Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 1, offset = 0
        Chunk: start = 4194304, len = 8388608, type = 4, num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 1, offset = 4194304
        Chunk: start = 12582912, len = 8388608, type = 1, num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 1, offset = 12582912
        Chunk: start = 20971520, len = 8388608, type = 22, num_stripes = 2
            Stripes list:
            [ 0] Stripe: devid = 1, offset = 20971520
            [ 1] Stripe: devid = 1, offset = 29360128
        Chunk: start = 29360128, len = 1073741824, type = 24, num_stripes = 2
            Stripes list:
            [ 0] Stripe: devid = 1, offset = 37748736
            [ 1] Stripe: devid = 1, offset = 1111490560
        Chunk: start = 1103101952, len = 1073741824, type = 1, num_stripes = 1
            Stripes list:
            [ 0] Stripe: devid = 1, offset = 2185232384

All Device Extents:
        Device extent: devid = 1, start = 0, len = 4194304, chunk offset = 0
        Device extent: devid = 1, start = 4194304, len = 8388608, chunk offset = 4194304
        Device extent: devid = 1, start = 12582912, len = 8388608, chunk offset = 12582912
        Device extent: devid = 1, start = 20971520, len = 8388608, chunk offset = 20971520
        Device extent: devid = 1, start = 29360128, len = 8388608, chunk offset = 20971520
        Device extent: devid = 1, start = 37748736, len = 1073741824, chunk offset = 29360128
        Device extent: devid = 1, start = 1111490560, len = 1073741824, chunk offset = 29360128
        Device extent: devid = 1, start = 2185232384, len = 1073741824, chunk offset = 1103101952

CHECK RESULT:
Healthy Chunks:
  Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 1, offset = 0
      Block Group: start = 0, len = 4194304, flag = 2
      Device extent list:
          [ 0]Device extent: devid = 1, start = 0, len = 4194304, chunk offset = 0
  Chunk: start = 4194304, len = 8388608, type = 4, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 1, offset = 4194304
      Block Group: start = 4194304, len = 8388608, flag = 4
      Device extent list:
          [ 0]Device extent: devid = 1, start = 4194304, len = 8388608, chunk offset = 4194304
  Chunk: start = 12582912, len = 8388608, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 1, offset = 12582912
      Block Group: start = 12582912, len = 8388608, flag = 1
      Device extent list:
          [ 0]Device extent: devid = 1, start = 12582912, len = 8388608, chunk offset = 12582912
  Chunk: start = 20971520, len = 8388608, type = 22, num_stripes = 2
      Stripes list:
      [ 0] Stripe: devid = 1, offset = 20971520
      [ 1] Stripe: devid = 1, offset = 29360128
      Block Group: start = 20971520, len = 8388608, flag = 22
      Device extent list:
          [ 0]Device extent: devid = 1, start = 29360128, len = 8388608, chunk offset = 20971520
          [ 1]Device extent: devid = 1, start = 20971520, len = 8388608, chunk offset = 20971520
  Chunk: start = 29360128, len = 1073741824, type = 24, num_stripes = 2
      Stripes list:
      [ 0] Stripe: devid = 1, offset = 37748736
      [ 1] Stripe: devid = 1, offset = 1111490560
      Block Group: start = 29360128, len = 1073741824, flag = 24
      Device extent list:
          [ 0]Device extent: devid = 1, start = 1111490560, len = 1073741824, chunk offset = 29360128
          [ 1]Device extent: devid = 1, start = 37748736, len = 1073741824, chunk offset = 29360128
  Chunk: start = 1103101952, len = 1073741824, type = 1, num_stripes = 1
      Stripes list:
      [ 0] Stripe: devid = 1, offset = 2185232384
      Block Group: start = 1103101952, len = 1073741824, flag = 1
      Device extent list:
          [ 0]Device extent: devid = 1, start = 2185232384, len = 1073741824, chunk offset = 1103101952
Bad Chunks:

Total Chunks:   6
  Heathy:       6
  Bad:  0

Orphan Block Groups:

Orphan Device Extents:
Check chunks successfully with no orphans
Recover the chunk tree successfully.

/ # btrfs check --repair  /dev/mmcblk0p28
enabling repair mode
Checking filesystem on /dev/mmcblk0p28
UUID: …
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 104367220 bytes used err is 0
total csum bytes: 862624
total tree bytes: 29380608
total fs tree bytes: 27267072
total extent tree bytes: 1069056
btree space waste bytes: 8385932
file data blocks allocated: 884224000
 referenced 884224000
Btrfs v3.16
/ # 



-----------------------------
     Jolla Recovery v0.3.1      
-----------------------------
Welcome to the recovery tool!
The available options are:
1) Reset device to factory state
2) Reboot device
3) Bootloader unlock [Current state: locked]
4) Shell
5) Try btrfs recovery if your device is in bootloop
6) Exit
Type the number of the desired action and press [Enter]: 
1

ALL DATA WILL BE ERASED! Clears everything from the device and reverts the 
software back to factory state. This means LOSING EVERYTHING you have added to 
the device (e.g. updates, apps, accounts, contacts, photos and other media). 
Are you really SURE? [y/N] y[CLEANUP] Starting cleanup!
[CLEANUP] Umounting top volume...
[CLEANUP] Deleting /mnt
[CLEANUP] Cleanup done.
Mounting /dev/mmcblk0p28 on /mnt
sh: unlock: unknown operand
[OK] No lockcode has been set. Proceeding...
sh: unlock: unknown operand
sh: shell: unknown operand
[OK] Factory snapshots found.
Resetting procedure started!
Backing up current root and home subvolumes. If the backup fails, the old data will be deleted to let the recovery process continue.
[Done]
Restoring factory subvolumes...
Create a snapshot of '/mnt/factory-@' in '/mnt/@'
Create a snapshot of '/mnt/factory-@home' in '/mnt/@home'
[Done]
FSSync '/mnt'
Checking swap subvolume
[OK] Swap subvolume found, no need to recreate it.
[DONE] Swap is ok.
Running flashing scripts from recovered snapshot ...
Running /var/lib/platform-updates/flash-bootimg-sbj.sh ... 
1760+1 records in
1760+1 records out
7211008 bytes (7.2 MB) copied, 0.373997 s, 19.3 MB/s
[OK]
Running /var/lib/platform-updates/flash-persist.sh ... 
2048+0 records in
2048+0 records out
8388608 bytes (8.4 MB) copied, 0.385839 s, 21.7 MB/s
[OK]
Running /var/lib/platform-updates/flash-platform-images-sbj.sh ... 
23+1 records in
23+1 records out
96464 bytes (96 kB) copied, 0.00640928 s, 15.1 MB/s
30+1 records in
30+1 records out
124588 bytes (125 kB) copied, 0.00860674 s, 14.5 MB/s
88+1 records in
88+1 records out
364360 bytes (364 kB) copied, 0.0223714 s, 16.3 MB/s
47+1 records in
47+1 records out
194360 bytes (194 kB) copied, 0.0146803 s, 13.2 MB/s
31+1 records in
31+1 records out
129180 bytes (129 kB) copied, 0.00738593 s, 17.5 MB/s
483+1 records in
483+1 records out
1980704 bytes (2.0 MB) copied, 0.103983 s, 19.0 MB/s
11360+1 records in
11360+1 records out
46532096 bytes (47 MB) copied, 2.05164 s, 22.7 MB/s
2048+0 records in
2048+0 records out
8388608 bytes (8.4 MB) copied, 0.355043 s, 23.6 MB/s
[OK]
Running /var/lib/platform-updates/flash-recoveryimg-sbj.sh ... 
2366+1 records in
2366+1 records out
9693184 bytes (9.7 MB) copied, 0.482558 s, 20.1 MB/s
[OK]
[DONE] flashing script succeeded.
Deleting old subvolumes
Transaction commit: none (default)
Delete subvolume '/mnt/rec-19700101_004024_@home'
Transaction commit: none (default)
Delete subvolume '/mnt/rec-19700101_004024_@'
Recovery procedure terminated SUCCESSFULLY! Now cleaning up...
[CLEANUP] Starting cleanup!
[CLEANUP] Umounting top volume...
[CLEANUP] Umounting /mnt
[CLEANUP] Deleting /mnt
[CLEANUP] Cleanup done.


[DONE] DEVICE RECOVERED!

[NOTE]: please note that since the phone will now reboot, after you press 
[Enter], this connection will be interrupted and you won't be able to interact 
with this screen anymore. If you wish to use the recovery tool again, switch 
off the phone and boot it to recovery mode again (VolDown + Power keys).
Press [Enter] to reboot the phone...
edit flag offensive delete publish link more

Comments

Phew, That was educational!! I just love TJC I know I don't have the issue as of now, but this will save anyone a lot of heartache when needed. Thanks for the thread, and maybe the procedure failed the first time but wasn't altogether meaningless, maybe it had it's use in aiding the fixing of it in the second attempt which tells me a lot, always try the same approach more than once before deeming it a failure.

DameCENO ( 2017-05-22 19:42:44 +0200 )edit
0

answered 2017-06-05 22:28:47 +0200

NGC-Ollie gravatar image

Last friday the btrfs died again. So the same game again:

  1. dd'ing, update, update, update, btrfs died
  2. dd'ing, update, update, btrfs died
  3. dd'ing, update, update, update, update, btrfs died (update to 2.1.0.11)
  4. dd'ing, update, update, update, update, btrfs died (update to 2.1.0.11)
  5. Finally: dd'ing, update, complete btrfs check, update, complete btrfs check, update, complete btrfs check, update, complete btrfs check, update to 2.1.0.11

And now it's stable, but the last check always returns this:

btrfs check --repair  /dev/mmcblk0p28
enabling repair mode
Checking filesystem on /dev/mmcblk0p28
UUID: 7436e663-696e-4d5d-b78d-644b698edf6c
checking extents
incorrect offsets 1279 1283
bad block 373514240
Errors found in extent allocation tree or chunk allocation
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
root 268 inode 100260 errors 0
        unresolved ref dir 92250 index 962 namelen 255 name  filetype 0 errors 103, no dir item, no dir index, name too long
found 247268260 bytes used err is 1
total csum bytes: 1128788
total tree bytes: 49717248
total fs tree bytes: 46170112
total extent tree bytes: 1970176
btree space waste bytes: 14676148
file data blocks allocated: 1685921792
 referenced 1405644800
Btrfs v3.16

Can anyone explain these lines, I don't get it and the search "google" deosn't give me an answer. Thank you.

edit flag offensive delete publish link more

Comments

It happened again, twice. It took me me more than 16 hours to accept, it's bricked. I guess the memory die got something when it once hit the floor. Shorty after all these things happened. I used it only 5 month, now I•m back at my old half broken Jolla from 2014. A friend of a friend will send me a third one with a brocken display, if someone got a manual for a display repair, please send me a hint here or jolla-display -at- misc.ngc-ollie.de

NGC-Ollie ( 2017-06-22 22:34:40 +0200 )edit
Login/Signup to Answer

Question tools

Follow
1 follower

Stats

Asked: 2017-05-15 23:04:32 +0200

Seen: 563 times

Last updated: Jun 05