Revision history [back]

This is a bug report.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space.

Df -h showed 50% free disk space. This was quite exactly a half of the size of /dev/mmcblk0p28.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After a reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have been got automatically fixed just by rebooting the device if btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla.

This is a bug report.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space.

Df -h showed 50% free disk space. This was quite exactly a half of the size of /dev/mmcblk0p28.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After a reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have been got automatically fixed just by rebooting the device if btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla.

This is a report.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space.

Df -h showed 50% free disk space. This was quite exactly a half of the size of /dev/mmcblk0p28.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After a the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have been got automatically fixed just by rebooting the device if btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla.

This is a report.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space.

Df -h showed 50% free disk space. This was quite exactly a half of the size of /dev/mmcblk0p28.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have been got automatically fixed just by rebooting the device if btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla.

This is a report.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space.

Df -h showed 50% free disk space. space for both root and home. This was quite exactly a half of the size of /dev/mmcblk0p28./dev/mmcblk0p28 partition.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have been got automatically fixed just by rebooting the device if btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' are before a reboot when the problem was still on, and the ones named 'after' are after the reboot.

This is a report.report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition.partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have been got automatically fixed just by rebooting the device if _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla.Jolla by methods instructed elsewhere.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' are were taken before a the reboot when while the problem was still on, and the ones named 'after' are after the reboot.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

btrfs fi show
btrfs fi df /home
btrfs fi df /root

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before a doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

Should have checkedchecked these first. Afterwards they showed all ok, and now I have to wait the third time for this to happen.

btrfs fi show
btrfs fi df /home
btrfs fi df /root

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

Should have checked these first. Afterwards they showed all ok, and now I have to wait the third time for this to happen.

btrfs fi show
btrfs fi df /home
btrfs fi df /root

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /root
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.48MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.48MB
Metadata: total=8.00MB, used=0.00

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

Should have checked these first. Afterwards they showed all ok, and now I have to wait the third time for this to happen.

btrfs fi show
btrfs fi df /home
btrfs fi df /root

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /root
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.48MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.48MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood had very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. More comparisons with this device possibly tomorrow.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

Should have checked these first. Afterwards they showed all ok, and now I have to wait the third time for this to happen.

btrfs fi show
btrfs fi df /home
btrfs fi df /root

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /root
/
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.48MB
used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.48MB
used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood had very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. More comparisons with this device possibly tomorrow.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

Should have checked these first. Afterwards they showed all ok, and now I have to wait the third time for this to happen.

btrfs fi show
btrfs fi df /home
btrfs fi df /root
/

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood had has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. More comparisons The output of differences is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
**Data: total=5.08GB, used=2.24GB**
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with this btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

Some random but possibly notable things:

• I haven't done factory reset at any point. The device possibly tomorrow.
  is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

Should have checked these first. Afterwards they showed all ok, and now I have to wait the third time for this to happen.

btrfs fi show
btrfs fi df /home
btrfs fi df /

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state.state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of device.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. The output of differences second device is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
**Data: total=5.08GB, used=2.24GB**
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

Some random but possibly notable things:

• I haven't done factory reset at any point. The device is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of device.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. The output of second device is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
**Data: total=5.08GB, used=2.24GB**
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

Some random but possibly notable things:

• I haven't done factory reset at any point. The device is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.
• The second device has had shorter uptimes than my device. It has been restarted usually once per day to counter Ingress problems with GPS. My device has usually 7 days or longer uptime.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of device.the devices.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. The output of second device is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
**Data: Data: total=5.08GB, used=2.24GB**
used=2.24GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

Some random but possibly notable things:

• I haven't done factory reset at any point. The device is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.
• The second device has had shorter uptimes than my device. It has been restarted usually once per day to counter Ingress problems with GPS. My device has usually 7 days or longer uptime.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of the devices.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. The output of second device is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
Data: total=5.08GB, used=2.24GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

I tested more with other Jolla device that had lots of unused free space. It looks like btrfs allocates space to filesystem in 1 GB chunks. These chunks get added to 'btrfs fi df /' total. Eventually all unallocated space is used.

Deleting files, rebooting or doing fsck does not bring allocated space back as unallocated.

It was notable that this space reservation happens a lot earlier than what one would expect. It chooses to do this even when there is several gigs of free allocated space left on mountpoints. Maybe the false error of running out of space originates from this? The actual reason for allocating more space too early could be the wear leveling or trim on sdcard, and I would not be surprised if this proved to be cause of the problem.

It would be nice to know if people experiencing symptoms mentioned earlier have all space allocated in their /dev/mmcblk0p28 device:

btrfs fi show

Some random but possibly notable things:things about my device:

• I haven't done factory reset at any point. The device is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.
• The second device has had shorter uptimes than my device. It has been restarted usually once per day to counter Ingress problems with GPS. My device has usually 7 days or longer uptime.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of the devices.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. The output of second device is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
Data: total=5.08GB, used=2.24GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

I tested more with other Jolla device that had lots of unused free space. It looks like btrfs allocates space to filesystem in 1 GB chunks. These chunks ("used" on Devid 1-line) get added to 'btrfs fi df /' total. Eventually all unallocated space is used.

Deleting files, rebooting or doing fsck does not bring allocated space back as unallocated.

It was notable that this space reservation happens a lot earlier than what one would expect. It chooses to do this even when there is several gigs of free allocated space left on mountpoints. Maybe the false error of running out of space originates from this? The actual reason for allocating more space too early could be the wear leveling or trim on sdcard, and I would not be surprised if this proved to be cause of the problem.

It would be nice to know if people experiencing symptoms mentioned earlier have all space allocated in their /dev/mmcblk0p28 device:

btrfs fi show

Some random but possibly notable things about my device:

• I haven't done factory reset at any point. The device is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.
• The second device has had shorter uptimes than my device. It has been restarted usually once per day to counter Ingress problems with GPS. My device has usually 7 days or longer uptime.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of the devices.

EDIT7: Added results about filesystem behavior with second Jolla device.

This is a report about btrfs filesystem issue.

I had a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At least following symptoms were observed. They happened separately of each other:

1. Sailfish browser could not load pages. Also back button did not work (nothing happened when pressed).
2. Messages application crashed, until restarted manually again.
3. Email application crashed.
4. Sailfish user interface crashed and restarted (green blinking led, blank screen).
5. Flight-mode did not engage (possibly connman failed to bring interfaces down).
6. Connecting Jolla to charger caused GUI crash and blinking green led.

I would suspect that anything that would have tried to save any preferences etc. would have crashed, such as creating or editing ambience.

Journalctl showed that several systems had problems writing on the disk. Manually trying to copy files under /home/nemo or anywhere under the btrfs filesystem gave an error about lack of disk space. Regular methods showed that there was no shortage of space.

Df -h showed 50% free disk space for both root and home. This was quite exactly half of the size of /dev/mmcblk0p28 partition used for root, home and couple of other mountpoints.

At this point I saved journalctl, dmesg, mount, df -h and df -i logs under '/run/user/100000/media/sdcard'.

Not taking any risks with a reboot with a full root, I first tried to run 'btrfsck /dev/mmcblk0p28' without success. Also it was not possible to unmount /dev/mmcblk0p28 or remount it as read only (far too many processes using /).

After clearing up couple of megabytes manually from /home/nemo, I was able to write forcefsck to root.

touch /forcefsck
sync

Then I cleared up some 100 MB to make sure Sailfish would not run out of space in next reboot in case that fsck failed. After that:

reboot

After the reboot the problem was gone. I'll add dmesg, journalctl, df and mount logs after I get extracted them from the device and online.

I suppose this problem might have got automatically fixed just by rebooting the device _if_ btrfs does automatic recovery every time the device boots (seems very fast so it could be possible). Alternatively one could run btrfsck in recovery mode of Jolla by methods instructed elsewhere.

For a future reference output of various btrfs commands after the reboot were:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs fi df /
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs fi df /home
Data: total=13.08GB, used=6.94GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=119.50MB
Metadata: total=8.00MB, used=0.00

[root@localhost nemo]# btrfs sub list /
ID 259 gen 109906 top level 5 path @swap
ID 264 gen 55 top level 5 path factory-@
ID 265 gen 56 top level 5 path factory-@home
ID 266 gen 119488 top level 5 path @
ID 267 gen 119488 top level 5 path @home

At least one known Jolla in my neighbourhood has very different values on the 'devid 1' line of btrfs fi show. The values were indicating that only half of the space was used eg, 13.75 and 5.75, when on my device they are 13.75 and 13.75. There seemed to be differences in full capacity of /home too. The output of second device is below:

[root@localhost nemo]# btrfs filesystem show 
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 2.32GB
    devid    1 size 13.75GB used 5.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
Data: total=5.08GB, used=2.24GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

Currently I am not experiencing problems in writing on neither / or /home. I'll have to wait for the problem to reappear. It is possible that this difference is business as usual for btrfs, eg. if the device has ever been full, the volume status stay recorded and won't change unless something is done actively with btrfs-commands (balancing?). I don't have experience on btrfs at all, so I'm not going to do anything advanced yet, unless instructed.

I tested more with other Jolla device that had lots of unused free space. It looks like btrfs allocates space to filesystem in 1 GB chunks. These chunks ("used" on Devid 1-line) get added to 'btrfs fi df /' total. Eventually all unallocated space is used.

Deleting files, rebooting or doing fsck does not bring allocated space back as unallocated.

It was notable that this space reservation happens a lot earlier than what one would expect. It chooses to do this even when there is several gigs of free allocated space left on mountpoints. Maybe the false error of running out of space originates from this? The actual reason for allocating more space too early could be the wear leveling or trim on sdcard, and I would not be surprised if this proved to be cause of the problem.

It would be nice to know if people experiencing symptoms mentioned earlier have all space allocated in their /dev/mmcblk0p28 device:

btrfs fi show

Good info:

https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21

" If your device is large (>16GiB)

btrfs fi df /mountpoint

will probably report available space in both metadata and data. The problem here is that one particular 256MiB or 1GiB block is full, and wants to allocate another whole block. The easy fix is to run, as root:

btrfs fi balance /mountpoint -dusage=5

This may take a while (although the system is otherwise usable during this time), but when completed, you should be able to use most of the remaining space. We know this isn't ideal, and there are plans to improve the behavior. Running close to empty is rarely the ideal case, but we can get far closer to full than we do. "

Some random but possibly notable things about my device:

• I haven't done factory reset at any point. The device is how I got in the end of November 2013 in Jolla launch event. I simply haven't had any serious problems before.
• The device internal sdcard has been (almost) filled up with music once (during the very first week I got the device, so it is long way off).
• Before the first problem appeared I had an external sdcard linked to /home/nemo/sdcard for two months. The card was formatted as btrfs couple of times in the beginning, but since the automatic mounting script didn't support it, I reverted to FAT32. No btrfs-commands were run, eg. I didn't add it as a raid volume to the system or anything like that. Unless the device registers all btrfs volumes by itself during the boot. After the first problem appearance, I removed the link 'touch /forcefsck' and rebooted the device, so it wasn't there any more the second time.
• The device still has over 16 GB of files on /run/usr/100000/media/sdcard.
• The second device has had shorter uptimes than my device. It has been restarted usually once per day to counter Ingress problems with GPS. My device has usually 7 days or longer uptime.

EDIT: The logs can be found here:

http://perhonen.keilarannanpalvelin.fi/public.php?service=files&t=1febcefdde9c804159b94ba2c31b9fff

The ones named 'full' were taken before the reboot while the problem was still on, and the ones named 'after' are after the reboot.

EDIT2: Added symptoms to description.

EDIT3: A useful article. Too bad I missed it before doing fsck the second time:

http://askubuntu.com/questions/278811/btrfs-not-enough-free-disk-space-but-device-not-fully-used

EDIT4: Added GUI crash when connecting Jolla to charger as one observed symptom.

EDIT5: Added outputs of various btrfs commands when not in the disk full state. Changed these a bit later (root was / not /root).

EDIT6: Added output of 'btrfs fi show' and 'btrfs fi df /home' from other Jolla device. Added various details about the use of the devices.

EDIT7: Added results about filesystem behavior with second Jolla device.

This is a report about btrfs filesystem issue.

I issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a sudden problem with networking with builtin browser. Cure all, eg. flight mode didn't stay on and the phone sprang back online immediately after the flight mode triggered. Uptime was 6 days.

At reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms were observed. They happened separately of each other:may occur:

1. Sailfish browser could may crash or does not load pages. Also back Back button did not work (nothing happened when pressed).may stop working.
2. Messages application crashed, until restarted manually again.crashes
3. Email application crashed.crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashed and restarted crashes and restarts (green blinking led, blank screen).screen)
6. Flight-mode did does not engage (possibly connman failed fails to bring interfaces down).down). Could affect any connection changes.
7. Connecting Jolla to charger caused causes GUI crash and blinking green led.led

touch /forcefsck
sync
btrfs fi show
btrfs fi df /home
btrfs sub list /

reboot
journalctl
dmesg | less

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 7.06GB
6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# 

[root@localhost nemo]# btrfs filesystem show 
fi balance /home
Done, had to relocate 13 out of 13 chunks

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
 Total devices 1 FS bytes used 2.32GB
    6.42GB
devid    1 size 13.75GB used 5.75GB 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

[root@localhost nemo]# btrfs filesystem df /home/
Data: total=5.08GB, used=2.24GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=330.00MB, used=82.62MB
Metadata: total=8.00MB, used=0.00

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing block is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ̃9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable block where it could write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For regular user this is visible as crashing applications, including the interface of Jolla itself.

Note that both data and metadata can run out of space, but so far it has been observed only only on the data parts of the filesystem. Chunk allocation size for metadata is smaller, 256 MB.

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence it might be best to keep Jolla's internal sdcard as empty as possible and use only external sdcard.

Notes:

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

• It is not known whether the delete operations should be done as adviced in btrfs FAQ, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This would make sense as it may free up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. Personally I was successful (enough) with just delete.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing block is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ̃9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable block where it could write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For regular user this is visible as crashing applications, including the interface of Jolla itself.

Note that both data and metadata can run out of space, but so far it has been observed only only on the data parts of the filesystem. Chunk allocation size for metadata is smaller, 256 MB.

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence it might be best to keep Jolla's internal sdcard as empty as possible and use only external sdcard.

Notes:

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

• It is not known whether the delete operations should be done as adviced in btrfs FAQ, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This would make sense as it may free up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. Personally I was successful (enough) with just delete.This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing block is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ̃9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable block where it could write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For regular user this is visible as crashing applications, including the interface of Jolla itself.

Note that both data and metadata can run out of space, but so far it has been observed only only on the data parts of the filesystem. Chunk allocation size for metadata is smaller, 256 MB.

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence it might be best to keep Jolla's internal sdcard as empty as possible and use only external sdcard.

Notes:

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

• It is not known whether the delete operations should be done as adviced in btrfs FAQ, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This would make sense as it may free up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing block is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ̃9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable block where it could write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For regular user this is visible as crashing applications, including the interface of Jolla itself.

Note that both data and metadata can run out of space, but so far it has been observed only only on the data parts of the filesystem. space. It is possible that fully allocated device does not have enough space to increase metadata allocation level. Chunk allocation size for metadata is smaller, 256 MB.

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence it might be best to keep Jolla's internal sdcard as empty as possible and use only external sdcard.

Notes:

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

• It is not known whether the delete operations should be done as adviced in btrfs FAQ, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This would make sense as it may free up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing block blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ̃9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable block blockgroup where it could to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of Jolla the phone itself.

Note that both data and metadata can run out of space. It is possible that fully space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to increase allocate a new metadata allocation level. blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence it might be best to keep Jolla's internal sdcard as empty as possible and use only external sdcard.

Notes:

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

• It is not known whether the delete operations should be done as adviced in btrfs FAQ, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This would make sense as it may free up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ̃9 ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence it might be best to keep Jolla's internal sdcard as empty as possible and use only external sdcard.

Notes:

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

• It is not known whether the delete operations should be done as adviced in btrfs FAQ, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This would make sense as it may free up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the balancing operation your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space. In essence space.

Notes:

• To be absolutely certain of the health of the filesystem, it might would be best to keep Jolla's internal sdcard as empty as possible possible, and use only external sdcard.

  ---

  Notes:

  sdcard for storing lots of space consuming data.

  • Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk.

  • It is not known whether the delete operations should be done as adviced in disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

  • When clearing up space before balance, btrfs FAQ, FAQ recommends clobbering files instead of deleting them by, eg. either:

    true > /path/to/file

    echo > /path/to/file

    This would make sense as it may free clears up space by freeing up space in chunks, as the file does not get deleted, only updated as 0 sized file object. without causing need for new metadata allocation. This is useful if you are running out of metadata space.

  • Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

  • 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. During the The balancing operation your is heavy operation for the filesystem, and you phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes.

   minutes and keep checking the situation.
   • If the screen goes blank and you can't wake it up by double tap or short press power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots are needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed with a device with lots of unallocated space left, while copying a large file on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. The balancing operation is heavy operation for the filesystem, and you phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking the situation.

   • If the screen goes blank and you can't wake it up by double tap or short press power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots are needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space balancing can clear up only 3,5 GB of space.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size noticed was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

3. The balancing operation is heavy operation for the filesystem, and you phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking the situation.

   • If the screen goes blank and you can't wake it up by double tap or short press power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots are needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of space data balancing can clear up only 3,5 GB of space.raw space for future allocations.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

3. The balancing operation is heavy operation for the filesystem, and you your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking the situation.checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots are should be needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3,5 3.5 GB of raw space for future allocations.allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line.line. For further information about how to achieve devel-su (eg. root), see this wiki article.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If the user has filled up his/her /home mountpoint with data, all free chunks are already allocated. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

According to answer below from November 2013 a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

3. The balancing operation is heavy for the filesystem, and your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work.work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. In case that not a When no single existing blockgroup is not suitable writing a specific change, new 1 GB chunk gets allocated.

If When the user has filled fills up his/her /home mountpoint with data, all free chunks are already allocated. get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Note that both data and metadata can run out of space separately (but they do allocate blockgroups from the same unallocated pool at least). Fully allocated device does not have enough space to allocate a new metadata blockgroup. Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below from below, posted in November 2013 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful or not.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

3. The balancing operation is heavy for the filesystem, and your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, it is successful write is succeeds or not.doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

3. The balancing operation is heavy for the filesystem, and your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, write is either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

3. The balancing operation is heavy for the filesystem, and your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.

4. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.

5. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balancing operation is heavy for the filesystem, and your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.

2. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation.operation. If you have developer mode enabled and can connect to your phone through SSH, that will do too (actually might be required if your device is in bad shape enough already).

3. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace during the upgrade could cause difficult situations to resolve.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files you may have, preferably couple of gigabytes

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balancing operation is heavy for the filesystem, and your phone may become unresponsive. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.

2. Open two terminal windows. In a second one you can run dmesg command to see any possible errors during the operation. If you have developer mode enabled and can connect to your phone through SSH, that will do too (actually might be required if your device is in bad shape enough already).

3. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of diskspace space during the upgrade could cause difficult situations to resolve.a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, at least / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

The btrfs balance operation introduced in this article can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, and even reboots. Reboots during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed. It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB (see below), new chunks cannot be allocated anymore. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this btrfs blockgroup allocation problem.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files from the device you may have, have copied on it earlier. Try to free up preferably couple of gigabytesseveral gigabytes of space in filesystem.

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balancing balance operation is heavy for the filesystem, cpu and mmc and your phone may become unresponsive. unresponsive during the process. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking.checking. Preferably start the process directly through SSH so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed.needed. Absolutely do not remove the battery!

2. Open two terminal windows. windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation. If you have developer mode enabled and can connect to your phone through SSH, that will do too (actually might be required if your device is in bad shape enough already).operation.

3. Run as devel-su

   btrfs fi balance /home

Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume you can use just plain 'btrfs fi balance /' too.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare.spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata superblock.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

The btrfs balance operation introduced in this article can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, and even reboots. Reboots during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed. It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore. anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered this a situation where btrfs blockgroup allocation problem.thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balance operation is heavy for the cpu and mmc and your phone may become unresponsive during the process. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking. Preferably start the process directly through SSH so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

2. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

3. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume you can use just plain 'btrfs fi balance /' too.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata superblock.blockgroup.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, and or you have problems very seldom, please ignore this article for now. For instance it is not known how this affects internal sdcard's wear and tear.

The If you just plain want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation introduced in this article is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even reboots. real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed. accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally.locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balance operation is heavy for the cpu and mmc and your phone may become unresponsive during the process. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking. Preferably start the process directly through SSH so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

2. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

3. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume volume, you can use just plain 'btrfs fi balance /' too.too. The result is same.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you have don't experience problems very seldom, often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just plain want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • If you are unable to delete files, your problem may be that you are running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balance operation is heavy for the cpu and mmc and your phone may become unresponsive during the process. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to 10 minutes and keep checking. Preferably start the process directly through SSH so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on screen or short press of power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

2. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

3. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may crash
5. Sailfish user interface crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcard's have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balance operation is heavy for on the cpu and mmc and your phone may become unresponsive during the process. Since it can take 10 to 30 minutes, it might be good to set the screen timeout to from 10 minutes and keep checking. Preferably start to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

2. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

3. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcard's have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

2. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

3. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcard's sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

1. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

2. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

3. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Some ten minutes later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

• If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

  https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to confirm that balance operation has finished through ssh or telnet if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Some ten minutes later Later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs too long you can pause or cancel it.

You can cancel, pause or resume running balance operation with these commands. Use another SSH session to run them if you did not start the balance operation as shell background process:

btrfs fi balance cancel /home
btrfs fi balance pause /home
btrfs fi balance resume /home

It has also been observed that in the case that your device reboots during the balance operation, it is triggered to run automatically when the system comes around again.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that balance the operation has finished through finished. Use ssh or telnet to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs too long you can pause or cancel it.

You can cancel, pause or resume running balance operation with these commands. Use another SSH session to run them if you did not start the balance operation as shell background process:

btrfs fi balance cancel /home
btrfs fi balance pause /home
btrfs fi balance resume /home

It has also been observed that in the case that your device reboots during the balance operation, it is triggered to run automatically when the system comes around again.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs fi balance /home

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' too. The result is same.

Later you get the result:

[root@localhost nemo]# btrfs fi balance /home
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs too long you can pause or cancel it.

You can cancel, pause or resume running balance operation with these commands. Use another SSH session to run them if you did not start the balance operation as shell background process:

btrfs fi balance cancel /home
btrfs fi balance pause /home
btrfs fi balance resume /home

It has also been observed that in the case that your device reboots during the balance operation, it is triggered to run automatically when the system comes around again.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process safer, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps through telnet using telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs fi balance /homestart /

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs fi balance /' start /home' too. The result is same.

Later you get the result:

[root@localhost nemo]# btrfs fi balance /home
start /
Done, had to relocate 13 out of 13 chunks

After the balancing operation, run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'; 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs too long you can pause or cancel it.

You can cancel, pause or resume running balance operation with these commands. Use another SSH session to run them if you did not start the balance operation as shell background process:

btrfs fi balance cancel /home
/
btrfs fi balance pause /home
/
btrfs fi balance resume /home
/

It has also been observed that in the case that your device reboots during the balance operation, it is triggered to run automatically when the system comes around again.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process safer, less prone to loose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /home
/
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs balance start /home' too. The result is same.

Later When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

After the balancing operation, Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. This is not destructive. You can clear up more files from the disk and try again.

If the balance operation runs too long you can pause or cancel it.

You can cancel, Cancel, pause or resume running balance operation with these commands. Use another SSH session to run them if you did not start the balance operation as a shell background process:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Please note that you get the result message from these commands in the original shell window where you started the balance, unless the cancel/pause/resume command itself fails. It may take a minute or two until the balance gets cancelled.

It has also been observed that in the case that if your device reboots during the balance operation, balance, it is triggered to run automatically when the system comes around again.afterwards.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to loose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Home is the mountpoint users have all their data and therefore most of the salvageable allocation space. Since both / and /home reside on the save btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. This In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it.

Cancel, pause or resume running balance operation it with these commands. Use another SSH session to run them if you did not start the balance operation as a shell background process:commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get the successful result message from these commands in the original shell window where you started the balance, unless the cancel/pause/resume command itself fails. balance. It may take a minute or two until the balance gets cancelled.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically afterwards.cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to loose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which contains /, swap and home mountpoints. Due to the design of btrfs, / and home mountpoints can run out of disk space even when there is several gigabytes of free space left on the device.

Please note that this article can be disputed, and hopefully will be. It is still too much of guess work, but the details found so far are alarming. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Since both / and /home reside on the save same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to loose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28, which /dev/mmcblk0p28. The device contains /, swap and home mountpoints. Due to the design of btrfs, / This issue manifests itself as inability to write on root and home mountpoints mounts and their subdirectories. This can run out of disk space happen even when there is are several gigabytes of free space left on the device.device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Many of the symptoms are not unique to just this issue, which makes it mandatory to enable developer mode and have a look at the filesystem status and logs.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article can may be disputed, and hopefully will be. disputed. It is still too much of guess work, but the work. The details found so far are alarming. alarming; Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jollacare.Jolla care.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstances should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to loose lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Many of the symptoms are not unique to just this issue, which makes it mandatory to enable developer mode and have a look at the filesystem status and logs.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. It is still too much of guess work. The details found so far are alarming; Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is not safe. NOT SAFE. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and and lead to a bricked device, recoverable only by Jolla care.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstances circumstance should you remove the battery during the balance process. process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
6. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
7. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Many of the symptoms are not unique to just this issue, which makes it mandatory to enable developer mode and have a look at the filesystem status and logs.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. It is still too much of guess work. The details found so far are alarming; Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, the actual btrfs balance operation is NOT SAFE. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, something that look like reboots but aren't, and even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Many of the symptoms are not unique to just this issue, which makes it mandatory to enable developer mode and have a look at the filesystem status and logs.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. It is still too much of guess work. The details found so far are alarming; Internal internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you just suspect you are having this issue, it is better to try to fix it before any operations requiring serious rewrite of system files are committed. Trying upgrade or factory reset may complicate your situation considerably. Latter does not fix the ongoing issue on your device. If you want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

However, Unfortunately the actual btrfs balance operation is NOT SAFE. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, or something that look like reboots but aren't, and aren't. Or even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care.care with a firmware flasher.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (because of (in case of high CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led
9. Factory reset fails
10. After factory reset you may have serious issues when trying to update the phone back to current level

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if you do not free up enough space manually, you may run out of space during the manual balancing operation depicted below.

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. Run as devel-su

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything in the FAQ is up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Many of the symptoms are not unique to just this issue, which makes it mandatory to enable developer mode and have a look at the filesystem status and logs.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. It is still too much of guess work. The details found so far are alarming; internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you suspect you are having this issue, it is better to try to fix it before any operations requiring serious rewrite of system files are committed. Trying upgrade or factory reset may complicate your situation considerably. Latter does not fix the ongoing issue on your device. If you want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

Unfortunately the actual btrfs balance operation is NOT SAFE. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, or something that look like reboots but aren't. Or even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care with a firmware flasher.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (in case of high CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led
9. Factory reset fails
10. After factory reset you may have serious issues when trying to update the phone back to current level

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

More so, if The method to free up space reserved by unused or sparsely used blockgroups is called 'balancing'. It resembles filesystem 'defrag' operation, but instead of just speeding up the filesystem by regrouping splintered files, balance also frees up space. Classic filesystems do not suffer from space usage problems to the extent of btrfs.

Balance operation can consume more space than what you have available. If you do not free up enough space in existing blockgroups manually, you may run out of space during the manual balancing operation depicted below.

Btrfs balance command manual can be found here:

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. As devel-su, at first try to balance any block groups that have under 5% of space in use. This should be a lot faster operation than full balance.

   btrfs balance start -dusage=5 /

7. Run as devel-sufull balance if previous did not free up enough block group chunks

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• Not everything -dusage parameter after start command is useful and a lot faster in the FAQ is situations where you have freed up to date; for instance many balancing command parameters have become obsolete (-dusage=5 for instance).lots of space by deleting files, and just want to free up unused block groups fast.

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Many Because most of the symptoms are not unique to just this issue, which makes issue only, it is mandatory to enable developer mode and have a look at the filesystem status and logs.logs to be sure.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. It is still too much of guess work. The details found so far are alarming; internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you suspect you are having this issue, it is better to try to fix it before any operations requiring serious rewrite of system files are committed. Trying upgrade or factory reset may complicate your situation considerably. Latter does not fix the ongoing issue on your device. If you want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

Unfortunately the actual btrfs balance operation is NOT SAFE. It can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, or something that look like reboots but aren't. are only GUI related. Or even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care with a firmware flasher.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (in case of high CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led
9. Factory reset fails
10. After factory reset you may have serious issues when trying to update the phone back to current level

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

The method to free up space reserved by unused or sparsely used blockgroups is called 'balancing'. It resembles filesystem 'defrag' operation, but instead of just speeding up the filesystem by regrouping splintered files, balance also frees up space. Classic filesystems do not suffer from space usage problems to the extent of btrfs.

Balance operation can consume more space than what you have available. If you do not free up enough space in existing blockgroups manually, you may run out of space during the balancing operation depicted below.

Btrfs balance command manual can be found here:

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. As devel-su, at first try to balance any block groups that have under 5% of space in use. This should be a lot faster operation than full balance.

   btrfs balance start -dusage=5 /

7. Run full balance if previous did not free up enough block group chunks

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• -dusage parameter after start command is useful and a lot faster in situations where you have freed up lots of space by deleting files, and just want to free up unused block groups fast.

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Because most of the symptoms are not unique to this issue only, it is mandatory to enable developer mode and have a look at the filesystem status and logs to be sure.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. The details found so far are alarming; internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you suspect you are having this issue, it is better to try to fix it before any operations requiring serious rewrite of system files are committed. Trying upgrade or factory reset may complicate your situation considerably. Latter does not fix the ongoing issue on your device. If you want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

Unfortunately the actual btrfs balance operation is NOT SAFE. It When run without balance filter (-dusage=5 or less) it can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, or something that look like reboots but are only GUI related. Or even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care with a firmware flasher.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (in case of high CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led
9. Factory reset fails
10. After factory reset you may have serious issues when trying to update the phone back to current level

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

The method to free up space reserved by unused or sparsely used blockgroups is called 'balancing'. It resembles filesystem 'defrag' operation, but instead of just speeding up the filesystem by regrouping splintered files, balance also frees up space. Classic filesystems do not suffer from space usage problems to the extent of btrfs.

Balance operation can consume more space than what you have available. If you do not free up enough space in existing blockgroups manually, you may run out of space during the balancing operation depicted below.

Btrfs balance command manual can be found here:

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. As devel-su, at first try to balance any block groups that have 0% or under 5% of space in use. This should be a lot faster operation than full balance.balance. (Check the result with 'btrfs fi show'.)

   btrfs balance start -dusage=0 /

   btrfs balance start -dusage=5 /

7. Run full balance if previous did not free up enough block group chunks

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• -dusage parameter after start command is useful and a lot faster in situations where you have freed up lots of space by deleting files, and just want to free up unused block groups fast.

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Because most of the symptoms are not unique to this issue only, it is mandatory to enable developer mode and have a look at the filesystem status and logs to be sure.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. The details found so far are alarming; internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you suspect you are having this issue, it is better to try to fix it before any operations requiring serious rewrite of system files are committed. Trying upgrade or factory reset may complicate your situation considerably. Latter does not fix the ongoing issue on your device. If you want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

Unfortunately the actual btrfs balance operation is NOT SAFE. When run Running full balance without balance filter (-dusage=5 or less) it filters can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, or something that look like reboots but are only GUI related. Or even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care with a firmware flasher.

Running btrfs balance with balance filter is recommended; the load and risks are significantly lower than with full balance operation.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (in case of high CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led
9. Factory reset fails
10. After factory reset you may have serious issues when trying to update the phone back to current level

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

The method to free up space reserved by unused or sparsely used blockgroups is called 'balancing'. It resembles filesystem 'defrag' operation, but instead of just speeding up the filesystem by regrouping splintered files, balance also frees up space. Classic filesystems do not suffer from space usage problems to the extent of btrfs.

Balance operation can consume more space than what you have available. If you do not free up enough space in existing blockgroups manually, you may run out of space during the balancing operation depicted below.

Btrfs balance command manual can be found here:

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. As devel-su, at first try to balance any block groups that have 0% or under 5% of space in use. This should be a lot faster operation than full balance. (Check the result with 'btrfs fi show'.)

   btrfs balance start -dusage=0 /

   btrfs balance start -dusage=5 /

7. Run full balance if previous did not free up enough block group chunks

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.

• -dusage parameter after start command is useful and a lot faster in situations where you have freed up lots of space by deleting files, and just want to free up unused block groups fast.

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/

This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen even when there are several gigabytes of free space left on the device and it does not match with the actual file space usage. You may only want to write 10 kilobytes change and that too fails.

There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Because most of the symptoms are not unique to this issue only, it is mandatory to enable developer mode and have a look at the filesystem status and logs to be sure.

It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel.

Please note that this article may be disputed. The details found so far are alarming; internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.

If you haven't had problems with your Jolla device you could not fix with a reboot, or you don't experience problems often, please ignore this article for now. For instance it is not known how much the balance procedure mentioned in this article affects internal sdcard's wear and tear.

If you suspect you are having this issue, it is better to try to fix it before any operations requiring serious rewrite of system files are committed. Trying upgrade or factory reset may complicate your situation considerably. Latter does not fix the ongoing issue on your device. If you want to overrule the possibility of btrfs allocation problems causing trouble on your device, the 'btrfs fi show' command is completely safe to use. See 'How to evaluate the situation' down below.

Unfortunately the actual btrfs balance operation is NOT SAFE. Running full balance without filters can cause high CPU and IO loads that may lead to restarts to sailfish services during the procedure, or something that look like reboots but are only GUI related. Or even real reboots if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care with a firmware flasher.

Running btrfs balance with balance filter is recommended; the load and risks are significantly lower than with full balance operation.

Btrfs balance operation requires enabling developer mode on Jolla phone, and some basic experience of working with linux command line. For further information about how to achieve devel-su (eg. root), see this wiki article.

It is vital that you have SSH connection enabled, as you cannot enable it anymore if the user interface cannot be accessed at all (in case of high CPU and IO load or crashed services). It would be also best that you are familiar with recovery steps using telnet in case that things do not go smoothly.

In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Use ssh to do this if you cannot access terminal window locally from the Jolla phone.

General symptoms

When the user runs out of disk space, at least following symptoms may occur:

1. Sailfish browser may crash or does not load pages. Back button may stop working.
2. Messages application crashes
3. Email application crashes
4. Any program that wants to write on disk and fails may freeze and crash
5. Any application database may corrupt while the writes fail
6. Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
7. Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
8. Connecting Jolla to charger causes GUI crash and blinking green led
9. Factory reset fails
10. After factory reset you may have serious issues when trying to update the phone back to current level

What happens under the bonnet

In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.

When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space.

When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.

Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.

Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.

According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066

(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)

For more information about btrfs, including space problems, see:

https://btrfs.wiki.kernel.org/index.php/FAQ

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F

How to evaluate the situation

Useful commands to evaluate the situation are (run as devel-su):

btrfs fi show
btrfs fi df /
btrfs sub list /

Check also these for errors:

journalctl
dmesg | less

When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
    Total devices 1 FS bytes used 6.42GB
    devid    1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.

Measures to free up raw space

Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".

The method to free up space reserved by unused or sparsely used blockgroups is called 'balancing'. It resembles filesystem 'defrag' operation, but instead of just speeding up the filesystem by regrouping splintered files, balance also frees up space. Classic filesystems do not suffer from space usage problems to the extent of btrfs.

Balance operation can consume more space than what you have available. If you do not free up enough space in existing blockgroups manually, you may run out of space during the balancing operation depicted below.

Btrfs balance command manual can be found here:

https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance

Before running the balance operation

1. Take backups of everything you can't afford to lose.

2. Close all unnecessary applications. Ensure that network connections work.

3. Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.

   • Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.

   • If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.

     echo > /path/to/yourfile

     After this you can try normal deleting again.

   • If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:

     https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/

4. The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.

   • If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!

5. Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.

6. As devel-su, at first try to balance any block groups that have 0% or under 5% of space in use. This should be a lot faster operation than full balance. (Check the result with 'btrfs fi show'.)

   btrfs balance start -dusage=0 /

   btrfs balance start -dusage=5 /

7. Run full balance if previous did not free up enough block group chunks

   btrfs balance start /

   Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start /home' too. The result is the same.

When the balance operation is finished (normally in under an hour), you get the result:

[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks

Now run 'btrfs fi show' again:

[root@localhost nemo]# btrfs fi show
Label: 'sailfish'  uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid    1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1

As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.

Ending balance operation prematurily

If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.

It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.

If the balance operation runs too long you can pause or cancel it with these commands:

btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /

Use another SSH session to run them if you did not start the balance operation as a shell background process.

Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.

Notes:Notes and tips:

• To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.

• Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.

• When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:

  true > /path/to/file

  echo > /path/to/file

  This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space.space. Try '-dusage=0' parameter in balance command first, as it deletes all unused block groups.

• -dusage parameter after start command is useful and a lot faster in situations where you have freed up lots of space by deleting files, and just want to free up unused block groups fast.

• If you you fail at clearing up enough space for balance via previous methods you can try Juho's method. It may be a bit risky if the phone happens to reboot during the operation:

In case you get "no space left on device" when you are running the balance, you can use the external SD card to get the extra space:

dd if=/dev/zero of=/media/sdcard/0000-BDCB/btrfs

bs=100M count=5 losetup -v -f /media/sdcard/0000-BDCB/btrfs btrfs device add /dev/loop0 /

That way the balance can get the needed extra. You can remove the device like this:

btrfs device delete /dev/loop0 /

• 1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.

• It has been observed with btrfs fi df / that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.

• Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/