[Request] Automatic/background btrfs balance (defrag type cleaning of filesystem)

Question

54

[Request] Automatic/background btrfs balance (defrag type cleaning of filesystem)

asked 2014-11-27 08:42:12 +0300

pata

3661 ●22 ●30 ●26

updated 2015-02-06 18:01:29 +0300

Btrfs, the file system of Jolla, is copy-on-write type fs. This means that every time somdthing is written to the flash, btrfs needs to find new unallocated space to it. There are some internal structures of btrfs that handle that, but one key issue with btrfs and Jolla is btrfs use of 1 GB allocation chunks. It's obvious that 16 GB flash do not have much breathing room of that kind of allocation scheme. This only mattters if you have once filled your device almost full and all available chunks are already allocated. If you then try to write something big to the flash it may fail even fs reports that you have more space left.

This phenonenon may show up like this bug report.

There are excellent thread about al this here.

This btrfs issue can be addressed simply to run defragment type of fs operation from cli (devel-su):

btrfs bal start /

The problem is that this take a long time, ~ hour or so, and the system load is high meanwhile. It can akso trigger reboot, if system thinks that there are endless loop going on because high system load of the command if not run by lowered priority.

There are also less resource heavy version of that balance operation like:

btrfs bal start -dusage=5 /

which cures many issues we sailors are experiencing and don't take long to accomplish. The background task should favor this if the logic detects no need to do full balance as nnth ooints out.

Jolla/Sailfish should have scheduled background job that automatically runs if there are reasons to do that (it can be check out by using other btrfs commands). The system job should run with low priority like with nice settings and only when the phone has been unused sometime with charger connected. The balance operation is quite write heavy so the system should avoid to run the task if not necessarily to prevent excessive wear of flash memory.

EDIT 21-01-2015 upcoming updates will run a btrfs-balance upon system update as mentioned here

EDIT 06-02-2015 there are new systemd btrfs balance service coming hopefully soon, please see below!

edit retag flag offensive close delete

Comments

6

With setting option in system settings to start it manually by user as well.

With visual indication what is been done.

Perhaps it would be good to have also the possible to see the current status of the btrfs allocations before actual start of manual balance?

Kari ( 2014-11-27 08:46:27 +0300 )edit

3

I agree, good idea. The new Utilities? (Apuohjelmat) system area is good place for this.

pata ( 2014-11-27 08:58:24 +0300 )edit

2

It's better to use the 'btrfs balance start -dusage=5 /' command, with that -dusage parameter added. It fixes most issues people will have and only takes a few seconds instead of an hour.

nthn ( 2014-11-27 09:41:04 +0300 )edit

1

Having faced the symptoms of BTRFS storage space congestion myself (in my case I was required to factory-reset the device via the Bootloader) I wholeheartedly agree with your idea of a background job. I would argue that, for the same reason, the Sailfish OS "free space" indicator (Settings -> System -> About product) should accommodate for the BTRFS headroom and show less free space accordingly.

t-lo ( 2014-11-27 11:34:09 +0300 )edit

2

Wont the filesystem get hosed when the device reboots/shuts down during the balance ? That could be quite dangerous considering all the spontaneous reboots & shutdowns happening on the Jolla.

And periodically running the balance could only make that more likely/certain that sooner or later there would be a reboot or shutdown during the balance, breaking the filesystem.

MartinK ( 2014-11-27 12:12:14 +0300 )edit

With setting option in system settings to start it manually by user as well.
With visual indication what is been done.
Perhaps it would be good to have also the possible to see the current status of the btrfs allocations before actual start of manual balance?
Kari ( 2014-11-27 08:46:27 +0300 )edit
I agree, good idea. The new Utilities? (Apuohjelmat) system area is good place for this.
pata ( 2014-11-27 08:58:24 +0300 )edit
It's better to use the 'btrfs balance start -dusage=5 /' command, with that -dusage parameter added. It fixes most issues people will have and only takes a few seconds instead of an hour.
nthn ( 2014-11-27 09:41:04 +0300 )edit
Having faced the symptoms of BTRFS storage space congestion myself (in my case I was required to factory-reset the device via the Bootloader) I wholeheartedly agree with your idea of a background job. I would argue that, for the same reason, the Sailfish OS "free space" indicator (Settings -> System -> About product) should accommodate for the BTRFS headroom and show less free space accordingly.
t-lo ( 2014-11-27 11:34:09 +0300 )edit
Wont the filesystem get hosed when the device reboots/shuts down during the balance ? That could be quite dangerous considering all the spontaneous reboots & shutdowns happening on the Jolla.
And periodically running the balance could only make that more likely/certain that sooner or later there would be a reboot or shutdown during the balance, breaking the filesystem.
MartinK ( 2014-11-27 12:12:14 +0300 )edit

Answer 1

0

answered 2015-10-27 11:01:36 +0300

evo3de

13567 ●164 ●227 ●231

http://www.tomatenmark.ne...

Is there now implemented or planed to implement a auto balance task or something by default?

edit flag offensive delete publish link

Comments

Yes, it's been there since 1.1.4, see for example the release notes in Zendesk:

- An automatic balancing of the file system is executed in the background on Tuesday mornings starting at 3 am, on condition that the phone is connected to a battery charger.

raketti ( 2015-10-27 11:53:47 +0300 )edit

Ok, then i must say it doesn´t work probably. I´ll open a new question on this.. Thanks @raketti

evo3de ( 2015-10-27 11:57:08 +0300 )edit

Sure, no problem. I haven't checked what my status is, but haven't had any issues with the phone and it's not connected to the charger on every Tuesday morning. Do you have a lot of allocated space or some other problems?

raketti ( 2015-10-27 12:04:17 +0300 )edit

Yes of course, it runs full if i don´t balance manually. And this problem persist on all 3 devices

evo3de ( 2015-10-27 12:16:28 +0300 )edit

Then there's something wrong indeed, sad to hear.

raketti ( 2015-10-27 12:56:13 +0300 )edit

Answer 2

7

answered 2015-02-06 11:38:06 +0300

Nekron
3162 ●47 ●60 ●62

updated 2015-02-06 11:39:13 +0300

As seen on the Sailfish repository at https://github.com/sailfishos/btrfs-balancer/blob/master/service/btrfs-balancer.timer there is the new btrfs systemd service that will regulary cleanup the filesystem running a btrfs-balance script.

I looked at the timer settings for the new service. There it is defined:

OnCalendar=Tue, 3:00

This means that the balance service will run every Tuesday morning at 3 am. However I guess that many regular Jolla owners will not run their smartphone as a Linux server 24h/7d like @kimmoli did (I think he had a 130+ days uptime record), so what will happen if you turn off your phone at night? You will never have that script running.

My suggestion would be to add the Persistent=1 option to the .timer configuration file so that even if a scheduled btrfs-balance run is not possible because of phone not running it will catchup as soon as the phone has been booted.

edit flag offensive delete publish link

Comments

1

Great find, thanks Nekron!

Manatus ( 2015-02-06 17:25:24 +0300 )edit

Answer 3

5

answered 2014-12-18 14:05:52 +0300

dez

7104 ●78 ●94 ●94

moderator

Copying the answer to the later question.

Btrfs balancing procedure is not yet proven to be absolutely safe on the version of the kernel used on the Jolla phone. So, until we will be sure it is safe enough we can't enable it to be executed automatically, as also add it e.g. to the Sailfish Utilities package. Back-porting of btrfs fixes from recent kernels is the hard task (if it is still possible) - a lot of things was changed.

But Jolla engineers are working on getting it enabled for non-power users with minimal risk.

edit flag offensive delete publish link

Comments

4

We are here to test such things. Smart use of this community can solve may issues and speed up many things. Jolla be smart and use us.

tvicol ( 2014-12-18 14:46:34 +0300 )edit

@tvicol Smart is the good word for it :-) Yes, we should use volunteering more.

dez ( 2014-12-18 16:54:41 +0300 )edit

1

I suppose updating the kernel is no option as it'd break Android drivers?

schmittlauch ( 2015-02-06 21:56:05 +0300 )edit

Jolla failed

Trizt ( 2015-05-21 18:51:23 +0300 )edit

Answer 4

2

answered 2014-11-27 12:13:57 +0300

MartinK
18500 ●154 ●178 ●174

Wont the filesystem get hosed when the device reboots/shuts down during the balance ? That could be quite dangerous considering all the spontaneous reboots & shutdowns happening on the Jolla - or just even plain old running out of battery or user initiated shutdown.

And periodically running the balance could only make that more likely/certain that sooner or later there would be a reboot or shutdown during the balance, breaking the filesystem.

edit flag offensive delete publish link

Comments

It's always possible that something bad happens for the fs if there are activities going during shutdown (unexpected, like running out juice - thats why I think that charger is one obvious requirement of this) or crash.

Still, balance operation should be "safe" by desing and fs should be able recover after reboot and even automacally continue balance operation if set up so.

The logic should also prefer dusage setting as nnth suggests and only go with with full balance if really needed.

pata ( 2014-11-27 13:04:03 +0300 )edit

If you want to avoid the btrfs balance lagging down your phone if you have to run it in the day time, look here on how to reduce disk latency during heavy disk operations: https://together.jolla.com/question/92552/howto-make-disk-io-more-responsive-for-example-during-btrfs-balance/)

00prometheus ( 2015-05-15 16:39:52 +0300 )edit

From my own experience, my Dalvik is quite messed up, can't install applications anymore, some applications just don't start like google play, all this thanks to balance and the only way to stop it is by restarting the phone (cancel balance do not work as expected, I have sometimes tried to leave the phone on for 48hours without the balance been cancelled). Why I had to reboot the phone, just for the balance made it totally unresponsive.

I recommend you to disable the balance service made by Jolla, it's just not worth to have. They should at least go up to kernel 3.18, which has a built in balance for btrfs.

Trizt ( 2015-05-21 18:50:55 +0300 )edit

[Request] Automatic/background btrfs balance (defrag type cleaning of filesystem)

Comments

4 Answers

Comments

Comments

Comments

Comments

Question tools

Stats

Related questions

[Request] Automatic/background btrfs balance (defrag type cleaning of filesystem)

Comments

4 Answers

Comments

Comments

Comments

Comments

Question tools

Public thread

Stats

Related questions