Root and home disks full and causing various problems
asked 2014-02-27 13:37:14 +0300
SailfishOS 1.1.4.28 (Äijänpäivänjärvi) and onwards: Jolla has implemented a workaround
Jolla has implemented filesystem checks to their upgrade process. Balance will be run when you are updating your phone to the latest versions of SailfishOS.
This does not fix any issues you may get in between the update releases. It is still a good idea to avoid filling up your phone's own filesystem more than 7 GB, and use sdcard for storing big amounts of data instead. In any situation where you've gone past the 7 GB limit, even temporarily, you should check the btrfs allocation level on your phone and run balance if necessary.
Btrfs-balancer command requires developer mode.
devel-su
btrfs-balancer allocation
Should the allocation (used) show higher value than 13153337344 bytes, you should run balance operation. Before doing this, make sure you can connect to your phone via SSH.
btrfs-balancer balance
If you've successfully upgraded to SailfishOS 1.1.7.24 or later, this operation is unlikely to take very long time, as your filesystem has been balanced recently during the upgrade.
Problems and notes:
- Btrfs-balancer's prerequisites seem to be in order now. (Vitaminj found and opened up a bug report regarding btrfs-balance command here: https://together.jolla.com/question/90031/new-btrfs-balance-service-in-114-seems-to-ignore-battery-percentage-gate/
- Btrfs-balance command is scheduled to run in Tuesdays 3:00am as a service. Unfortunately it does not seem to work. It did in 1.1.4.28 but not in later versions, regardless of whether the phone is in sleep state or not.
Below are files introduced with 'btrfs-balancer' command. See also the rest of this article for plain 'btrfs balance' command.
/lib/systemd/system/btrfs-balancer.timer
/lib/systemd/system/btrfs-balance.service
/usr/sbin/btrfs-balancer
/usr/share/btrfs-balancer/btrfs-balancer.conf
Applicable: SailfishOS 1.1.2.16 (Yliaavanlampi) or earlier
This is a report about btrfs filesystem issue on Jolla device's internal sdcard partition /dev/mmcblk0p28. The device contains /, swap and home mountpoints. This issue manifests itself as inability to write on root and home mounts and their subdirectories. This can happen when there are several gigabytes of free space left on the device. You may only want to write 10 kilobytes change and that too fails. With btrfs usual methods of checking free space on the device do not apply, such as Sailfish OS built in report or commands du or df.
There is no clear indication to the user that btrfs is incapable of writing requested changes on the volume, but many usability problems it causes are very clearly noticeable. Because most of the symptoms are not unique to this issue only, it is mandatory to enable developer mode and have a look at the filesystem status and logs.
It is not known whether this is an unfortunate design feature of btrfs or a bug in btrfs implementation of current kernel. Internal mass memory as small as 16 GB may not simply be big enough for a filesystem as complicated as btrfs. Or at least not if the user saves his/hers data on the same volume in any reasonable amount.
If you have constant stability problems with your phone and suspect you are having this issue, it is best to try and fix it before the situation gets worse. Please note that trying to upgrade or factory reset the device instead may complicate your situation considerably. So far updates or factory reset do not fix the ongoing issue on your device! There is a high probability that this problem is going to be fixed or worked around in coming Jolla updates. See the notes in the end of this article.
If you want to overrule the possibility of btrfs allocation problems on your device, the btrfs fi show command is completely safe to use. See 'How to evaluate the situation' down below.
Unfortunately the actual btrfs balance operation is NOT SAFE. According to Dez's answer in this article, running balance is not safe with current kernel version of Sailfish OS.
Running full balance without filters can cause high CPU and IO loads. This may lead to restarts to sailfish services or something that look like reboots but are only GUI related (green led, black screen). Reboots may happen too if you are very unlucky. Reboots or power cuts during the filesystem operations can cause loss of data and lead to a bricked device, recoverable only by Jolla care with a firmware flasher. However reboot during balance is not "autobrick"; btrfs recovery and balance operations may start and continue after the reboot without problems. This seems to be most often the case.
To keep the balance operation as light as possible running it with balance filter (parameter '-dusage') is highly recommended. With filter CPU and IO load and risks are significantly lower compared to a full balance.
Btrfs balance operation requires enabling developer mode on Jolla phone and some basic experience of working with linux command line. For how to achieve devel-su (eg. root), see this wiki article.
It is vital that you have SSH connection enabled. You cannot do this afterwards if the graphical interface freezes, or your screen turns and stays black just because of high CPU and IO load or crashing services. In addition it would be best that you are familiar with recovery steps using telnet in case that things do not go smoothly.
In no circumstance should you remove the battery during the balance process. You have to cancel the operation or confirm that the operation has finished. Previously enabled and established SSH connection is a way to do it should you not be able to access the terminal window locally on the Jolla phone anymore.
General symptoms
When the user runs out of disk space, at least following symptoms may occur:
- Sailfish browser may crash or does not load pages. Back button may stop working.
- Messages application crashes
- Email application crashes
- Any program that wants to write on disk and fails may freeze and crash
- Any application database may corrupt while the writes fail
- Sailfish user interface freezes, crashes and restarts (green blinking led, blank screen)
- Flight-mode does not engage (possibly connman fails to bring interfaces down). Could affect any connection changes.
- Connecting Jolla to charger causes GUI crash and blinking green led
- Factory reset fails
- Serious issues after the factory reset when trying to update the phone back to current level
What happens under the bonnet
In short the thesis is that the shortage of disk space is caused by how btrfs allocates 1 GB chunks of raw space for filesystem use. When no single existing blockgroup is suitable writing a specific change, new 1 GB chunk gets allocated.
When the user fills up his/her /home mountpoint with data, all free chunks get allocated very easily. Of available 14 GB space on Jolla device, it is enough to write only ~9 GB (estimate) data on the device to allocate whole 14 GB of free space. Generally speaking the longer the device has been in use, more likely it is that all space has already been allocated.
When btrfs filesystem does not find suitable blockgroup where to write the data in, and it cannot allocate a new 1 GB chunk, it gives an error that there is no space left. For a regular Jolla phone user this is visible as crashing applications, including the graphical interface of the phone itself.
Chunk allocation size for filesystem metadata is smaller, 256 MB, but on Jolla device this is duplicated and requires 512 MB.
Note that both data and metadata can run out of space separately, yet the error and symptoms are the same. They also allocate blockgroups from the same unallocated pool on internal sdcard of Jolla device. Fully allocated device does not have enough space to allocate a new metadata blockgroup. This is more difficult situation than just with data blockgroups: You may not be able to delete files, as there is no space to mark this change to the metadata, and the filesystem refuses the operation.
According to answer below, posted in November 2013, a minimum unallocated space for the btrfs filesystem should be at least 1.5 GB. That would be 1 GB for data and 256+256 MB for metadata.
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066
(Original question http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047)
For more information about btrfs, including space problems, see:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
https://btrfs.wiki.kernel.org/index.php/FAQ
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29
https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F
How to evaluate the situation
Useful commands to evaluate the situation are (run as devel-su):
btrfs fi show
btrfs fi df /
btrfs sub list /
Check also these for errors:
journalctl
dmesg | less
When 'btrfs fi show' command shows 13.75GB used of 13.75GB for 'devid 1' (see example below), new chunks cannot be allocated anymore and the random problems begin. Depending on what, when and where is written, write either succeeds or doesn't.
[root@localhost nemo]# btrfs fi show
Label: 'sailfish' uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid 1 size 13.75GB used 13.75GB path /dev/mmcblk0p28
Btrfs v0.20-rc1
At this point you should also try copying ~500 MB of files under your home directory mountpoint with a 'cp' command. If the copying fails with disk space errors, you have encountered a situation where btrfs thinks it absolutely cannot use any existing blockgroups, and tries to allocate new ones, no matter what.
Measures to free up raw space
Deleting big files helps freeing up space both in under visible filesystem and in the area of allocated space. However it does not shrink the allocated space automatically. When you run out of space it is recommended to delete several gigabytes of data to make sure that the filesystem has enough space to "breathe".
The method to free up space reserved by unused or sparsely used blockgroups is called 'balancing'. It resembles filesystem 'defrag' operation, but instead of just speeding up the filesystem by regrouping splintered files, balance also frees up space. Classic filesystems do not suffer from space usage problems to the extent of btrfs.
Balance operation can consume more space than what you have available. If you do not free up enough space in existing blockgroups manually, you may run out of space during the balancing operation depicted below.
Btrfs balance command manual can be found here:
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance
Before running the balance operation
Take backups of everything you can't afford to lose.
Close all unnecessary applications. Ensure that network connections work.
Remove any excess files from the device you may have copied on it earlier. Try to free up preferably several gigabytes of space in filesystem.
Especially look out for directories with very high amounts of files. It has been observed that these can prolong and cause balance operation to run out of disk space (to stop, not harmful). For instance Android's Tripadvisor can cache tens of thousands of files under its data directory, and btrfs balance operation has hard time going through those with IO-rates as bad as sdcards have. Plain uninstalling Android programs does not help; you have to find and delete files manually.
If you are unable to delete files, your problem may be that you are in a chicken-egg situation of running out of metadata space. You can free up metadata space in already used block group by overwriting the data in files you do not need.
echo > /path/to/yourfileAfter this you can try normal deleting again.
If you have previously done a factory reset on your device, you may have old snapshots of the filesystem taking up lots of space. To get rid of these, follow instructions in this case:
https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/
The balance operation is heavy on the cpu and mmc and your phone may become unresponsive during the process. Since it can take from 10 minutes to hours, and even days(!), it is preferred to have the phone connected to a computer or directly in mains. Start the process directly through SSH session, so you can see what is happening all the time.
- If the screen goes blank and you can't wake it up by double tap on the screen or short press of a power button, do not longpress power button or remove the battery from the phone! You would risk losing crucial data or cause bootloop. The phone should work normally when the process is finished. No forced reboots should be needed. Absolutely do not remove the battery!
Open two terminal windows or SSH connections. In a second one you can run dmesg command to see any possible errors during the operation.
As devel-su, at first try to balance any block groups that have 0% or under 5% of space in use. This should be a lot faster operation than full balance. The command will tell you how many chunks were relocated but you can check the result with 'btrfs fi show' too.
btrfs balance start -dusage=0 /
btrfs balance start -dusage=5 /
Keep going to bigger -dusage parameter values in increments of 5 or 10 % until you see difference. After each balance operation, check the situation with btrfs fi show. Really it shouldn't be necessary to go higher than 25 %. Run the full balance without -dusage parameter only if previous did not free up enough block group chunks (atleast 1.5 GB).
Since both / and /home reside on the same btrfs volume, you can use 'btrfs balance start -dusage=xx /home' too. The result is the same.
When the balance operation is finished (normally in under an hour), you get the result:
[root@localhost nemo]# btrfs balance start /
Done, had to relocate 13 out of 13 chunks
Now run 'btrfs fi show' again:
[root@localhost nemo]# btrfs fi show
Label: 'sailfish' uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387
Total devices 1 FS bytes used 6.42GB
devid 1 size 13.75GB used 10.13GB path /dev/mmcblk0p28
Btrfs v0.20-rc1
As we can see here, with 6.42 GB of actual data balancing can clear up 3.5 GB of raw space for future allocations. You should always have at least 1.5 GB of unallocated disk to spare on the volume 'devid 1'. 1 GB for data and 0.5 GB for next metadata blockgroup.
Cancelling balance operation
If the balance operation runs out of space it ends with an error. In experience this is not destructive. You can clear up more files from the disk and try again.
It has also been observed that if your device reboots during the balance, it is triggered to run automatically after the reboot again.
If the balance operation runs too long you can pause or cancel it with these commands:
btrfs balance cancel /
btrfs balance pause /
btrfs balance resume /
Use another SSH session to run them if you did not start the balance operation as a shell background process.
Please note that you get successful result message from these commands in the original shell window where you started the balance. It may take a minute or two until the balance gets cancelled. If the command is not successful, you get an error message on the same window where tried to do cancel/pause/resume.
Notes and tips
According to Nekron's find here and here, Jolla is planning to add a systemd service and scheduler script for btrfs balance in coming updates.
To be absolutely certain of the health of the filesystem, it would be best to keep Jolla's internal sdcard as empty as possible, and use only external sdcard for storing lots of space consuming data.
Before installing any big Jolla upgrades, it might be wise to clear up lots of free space on disk. Running out of space during the upgrade could cause a reboot loop curable only through the recovery mode and factory reset.
When clearing up space before balance, btrfs FAQ recommends clobbering files instead of deleting them by, eg. either:
true > /path/to/file
echo > /path/to/file
This clears up space without causing need for new metadata allocation. This is useful if you are running out of metadata space. Try '-dusage=0' parameter in balance command first, as it deletes all unused block groups.
-dusage parameter after start command is useful and a lot faster in situations where you have freed up lots of space by deleting files, and just want to free up unused block groups fast.
If you you fail at clearing up enough space for balance via previous methods just keep trying. If nothing else works, you can try Juho's method. It is risky if the phone happens to reboot during the operation.
In case you get "no space left on device" when you are running the balance, you can use the external SD card to get the extra space. Please note that if balance fails when loop device is in use, and you cannot delete it properly, you will end up with devided system partition and your device may not boot anymore. At this stage you cannot factory reset either because factory reset on Jolla depends on btrfs snapshots on now broken filesystem.
dd if=/dev/zero of=/media/sdcard/0000-BDCB/btrfs bs=100M count=5 losetup -v -f /media/sdcard/0000-BDCB/btrfs btrfs device add /dev/loop0 /That way the balance can get the needed extra. You can remove the device like this:
btrfs device delete /dev/loop0 /It might be a good idea to use one liner, as suggested by lpr, to make the process a bit less prone to fail:
btrfs device add /dev/loop0 / && btrfs-balancer balance && btrfs device delete /dev/loop0 /
1 GB chunk allocation size was noted with a device with lots of unallocated space left, while copying a large file (4.6 GB) on the device through ssh.
It has been observed with
btrfs fi df /that during the balance operation filesystem disk usage on the disk slowly but constantly rises to a 1 GB higher figure than what was the original disk usage. When it reaches 1 GB, the usage drops to original level, only to start rising again. It is assumed that filesystem is moving files between block groups, and the process does not discard old block group until the new block group has been completely written. This makes balance as a process less prone to lose data if the system shuts down, but it is ever more important to have enough free space before starting it.Jolla device does have a recovery mode of its own where you can run Btrfs recovery (option 5 in recovery mode). Do this if you cannot access command line through the phone or SSH: https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/
Ok, now this problem happened second time. It started again with a crashing browser, then GUI going down couple of times with green led. I was able to make one copy of 75 MB file I've used for testing, but after that the filesystem claims that /home is full. Yet df-command clains over 6 gigs of free space. I think many of the problems reported here at together.jolla.com about the phone crashing all the time, and led blinking green, may be related to this same filesystem or mmc device problem.
Manatus ( 2014-03-02 01:27:32 +0300 )editIn the linked article the problem was resolved by upgrading kernel from 3.2.x to.3.11.x. As Jolla is in 1.0.3.8 running with #Linux kernel 3.4.0 does this suggest that the kernel is maybe too old for reliable btrfs (which is still in heavy development)?
foss4ever ( 2014-03-02 04:35:42 +0300 )editPossibly yes. But of course there is a possibility that Jolla could have backported things from newer kernels.
Being just an enthusiast and not knowing kernel development is hard to say what is the current status of btrfs. Discussions two years ago mentioned that autorecovery that seems to be enabled (according to dmesg) could make things worse. But that was then and I expect Jolla devs knew its current status when they chose btrfs for the release version.
Manatus ( 2014-03-02 13:25:52 +0300 )editStrange. I got UI reset twice. And my disk space is same strange ..
And remove rec* snapshots not helped with this ..
Kaacz ( 2014-03-03 19:52:00 +0300 )editThanks Kaacz! What does
say?
Have you ever filled up your internal storage knowingly on Jolla?
I'm suspecting the cause for this problem could be that btrfs tries to increase the btrfs volume size instead of writing on already existing free space. The reasons could vary, starting from trimming to something else. If you haven't ever filled up your device, then that figure has kept rising for other reasons than pure file size.
Manatus ( 2014-03-03 20:14:38 +0300 )editbtrfs fi df /
FS is normaly filled only to 4.1G with 9.3G free, but why - in btrfs point of view - is full? I was do factory reset one month back ..
Kaacz ( 2014-03-04 02:18:01 +0300 )editThanks again. :)
What you see there is probably what I'm going to see the next time I get problems, unfortunately. I was hoping to see metadata bulging or something.
AFAIK Jolla's factory reset is based on btrfs snapshots. It just reverts to factory snapshot of the same filesystem. It does not overwrite the whole volume or partitions like commonly used image based factory reset methods do... So btrfs status may stay, unless something else is done to revert the space usage/reservation.
Manatus ( 2014-03-04 14:57:42 +0300 )editthis started happening to me too since last week. Maybe because I'm keeping 5-6 apps open?
c.la ( 2014-03-05 13:34:21 +0300 )editI too have 5-6 six apps open so that does match.
I have uptime of ~4 days and still I haven't had any problems. Apps use swap that is on the same btrfs device, but it has 512 MB size limit. I'm more and more getting inclined that the problem is caused by btrfs trim or sdcard wear leveling process. For instance on Windows some Intel SSD's trim function works so that it writes the whole disk full of dummy files once per week (I know it sound weird). We could be witnessing same kind of logic here.
Manatus ( 2014-03-05 17:15:48 +0300 )editi shutdown the phone every night. Tidings is also one of the apps that frequently freezes
c.la ( 2014-03-05 17:49:43 +0300 )editOk, thanks for that info. I would not have expected problems on a phone that is regularly rebooted.
I'm currently running btrfs balance. Looks like it may be a long wait and I had to turn on 10 minutes screen timer. The phone responds very badly to screen wake during that so I don't want to take any more risks. We'll see what happens.
Manatus ( 2014-03-05 18:02:16 +0300 )editI was able to salvage space. It may have some risks though. I'll update and clean up the report...
Manatus ( 2014-03-05 18:33:26 +0300 )editWow, I am using my sdcard with subvolumes for "Desktop Documents Downloads Music Pictures Playlists Public Templates Videos" and have only 2GB on /home.
Label: 'sailfish' uuid: 0f8a2490-53ed-4ff6-ba34-b81df3430387 Total devices 1 FS bytes used 2.18GB devid 1 size 13.75GB used 13.75GB path /dev/mmcblk0p28 btrfs fi balance /home Total devices 1 FS bytes used 2.18GB devid 1 size 13.75GB used 4.07GB path /dev/mmcblk0p28I will have an eye on that -- especially next week.
jolladiho ( 2014-03-05 22:09:15 +0300 )edit@jolladiho That was an afwul lot compared to actual use. :)
Had you ever added much data on the internal sdcard before, even for a brief time? Did you have any actual problems like mentioned in symptoms?
Manatus ( 2014-03-05 22:17:17 +0300 )edit@Manatus I was up to 6GB until 6th Januar (http://talk.maemo.org/showpost.php?p=1404380). Until then I started testing mount-sd.sh to use optimal my sdcard and moved the date from the home folders to the sd subvolumes. Never had very much data in home. And I did a factory reset after the last sailfishOS release. But I don't remember why. After factory reset I reformated the sdcard, created the subvolumes and moved the data again to the subvolumes. In /home are only the (hidden) config data from apps (a bit more then nothing). df -h shows 2.3GB for / /home and /swap.
I have had sometimes a hanging browser and thought it was a connection problem.
jolladiho ( 2014-03-05 23:06:42 +0300 )editI also forgot to mention (sorry about that) that I have yet to insert a sdcard in jolla. I always used only jolla disk.
yesterday evening the issues appeared again but on system information it showed about 2,5 GB of disk space used (don't remember the exact figures now), the space that usually is shown on the device as used
c.la ( 2014-03-06 10:09:19 +0300 )edit@jolladiho Thanks! :)
Manatus ( 2014-03-06 11:56:11 +0300 )edit@c.la Can you check what the both used figures show currently for you?
And also metadata total and used with:
Thanks! :)
Manatus ( 2014-03-06 12:00:41 +0300 )editOk, there is my story :)
OK, next Jolla update, I am ready for you.. :)
Kaacz ( 2014-03-06 18:30:12 +0300 )edit@Kaacz My metadata total has raised to 512 MB from original 330 MB too. It does not fit into 256 MB chunks for metadata, but balance operation does not seem restricted for same allocation space limits. Maybe it has gone up to 582 MB in the beginning, and then down again to 512.
I wouldn't do update before doing balancing, not anymore after problems. But all of the updates I've run to date have worked flawlessly, and I suppose unallocated space on my device has been 0 from the mid December.
Manatus ( 2014-03-06 21:35:27 +0300 )editAh, looks like the blockgroup size for metadata is indeed 256, the reason it is now 512 is that it is duplicated (DUP). I've absolutely no idea why the original value was 330 MB. Hopefully some Jolla dev could enlighten us about it.
This was a very good read:
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066
Original question was:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/30047
I'll add these to report.
Manatus ( 2014-03-06 22:08:59 +0300 )editUnfortunately (for me) I know nothing about btrfs. But as linux geek, I am ready for next steps .. :) PS: Jolla use many new interesting features (btrfs,wayland,.. ). Some of this things need more time .. :) But in this point of view, Jolla is true innovative leader in this time. :)
Kaacz ( 2014-03-07 03:26:22 +0300 )edit@Manatus I will but I need some days to do it as I currently have quite a bit of backlog
c.la ( 2014-03-07 10:41:17 +0300 )edit@Kaacz Yup, I was also surprised when I noticed that Jolla uses btrfs. And I was not surprised at all that something like this could happen. Somebody has to lead. ;D
Of course btrfs brings in goodies too, so I am all for it.
I'm speculating that maybe they should have used mixed block group mode mentioned in the btrfs wiki, or plain bigger mmc to start with. In the long run it shouldn't matter; one should be able to increase the size of root or home with external mmc. :)
Manatus ( 2014-03-07 10:56:50 +0300 )edit@c.la No hurry. This case has been a learning experience already. Useful in the future too...
Manatus ( 2014-03-07 11:09:06 +0300 )editWhat about run balance on / volume ? After several app updates there are also some changes .. ?snapshots?
Kaacz ( 2014-03-07 20:18:10 +0300 )editI've run balance on / after cleaning up android data from /sdcard just in case, but I don't remember what the result was. My impression this far is that balance happens always on whole volume, as
all seem to give output for all subvolumes, regardless of path used.
Manatus ( 2014-03-08 12:20:25 +0300 )editrebalance of / ..
Kaacz ( 2014-03-09 22:14:05 +0300 )editWhat do you think about adding a btrfs formated partition from SDcard to '/ and or /home'? It is simple possible with a the btrfs:
btrfs device add /dev/mmcblk1p1 /
and/or
btrfs device add /dev/mmcblk1p2 /home
Remember: I am thinking about it, I HAVE NOT DONE IT. What do you think? Can we try it?
I read it there: http://www.zdnet.com/btrfs-hands-on-an-extremely-cool-file-system-7000023734/
jolladiho ( 2014-03-12 11:27:55 +0300 )edit@jolladiho I expect this in the future, but currently I would not dare to do it.
We are not talking here about touching the partition structure, but adding more disk to the volume could cause problematic situations with Jolla recovery, depending on how it works (I have not found the script that btrfs recovery uses). And if the recovery does not work... :\
Also see the state of affairs with Jolla flasher:
https://together.jolla.com/question/32050/have-gdisk-available-in-mer-tools-repository/
Manatus ( 2014-03-12 14:29:23 +0300 )editI probably got into the situation due to nested symlinks but to be honest - I have no idea.
Now I try to follow the routine described above but run into trouble - namely
both:
true > /path/to/file
echo > /path/to/file
return an more space left on device error message. How do I free any space that way? Or do I have to do the rebalance step already?
This is what I get:
Maybe unmounting and remounting with cache clear command could help?
http://www.novell.com/support/kb/doc.php?id=7011860
If only I had the slightest clue on what the partitions and mountpoints were that'd be needed for this.
I also found an interesting POV on BTRFS
http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/70
I guess it's too late for Jolla to introduce a change now, is it? I don't like where this is going :-/
MoritzJT ( 2014-03-19 00:43:33 +0300 )edit@MoritzJT One might be able to unmount root for repairs in recovery mode (telnetting in you device). I have not checked. For me there was no need to go that far as I was able to clear up enough space.
Note that my first attempt at balance operation failed with 'no space left on device' after some time. It was still able to free up some space and it did not seem to break anything. Looking at the output of yours I would guess that you are running out of metadata space.
Manatus ( 2014-03-19 10:31:55 +0300 )edit@MoritzJT If the updates could start in the fastboot mode and one could unmount the regular root device there, Jolla might be able to do a script that would arrange filesystem to different partitions or even start using different filesystem.
But it is not likely as the phone mmc could already be filled with stuff by user, and rearranging past that would be time consuming process.
Alternatively they may wait for btrfs developers to sort out the problems and just update the kernel.
Manatus ( 2014-03-19 10:41:57 +0300 )editI found a few files that could be deleted after a long tiring search. Rebalance did work, but my developer mode is broken. It's active but I can't access its settings applet. Jolla store doesn't load my apps and shows i can reinstall all, which then fails. Android apps no longer start and half of my native apps fail too :-(
MoritzJT ( 2014-03-19 10:44:47 +0300 )editGenerally the whole situation is bad as btrfs induced common disk based problems get mixed up with the actual real bugs in SailfishOS. The first thing a user should do when having any problems with his/her phone, would be checking the amount of free raw space.
Manatus ( 2014-03-19 10:46:36 +0300 )editIt's on my radar now for sure.
Do you by any chance know how to enable/disable install/uninstall dev mode with pkcon/terminal?
MoritzJT ( 2014-03-19 10:49:38 +0300 )edit@MoritzJT Sorry to hear that. :(
Manatus ( 2014-03-19 10:54:14 +0300 )editIf you are still able to access devel-su, I would keep trying to clear up even more space and running balance. Not sure if that is the best method.
Recovery through telnet and use fix btrfs option from there. Before doing this you should try a regular reboot once, as autorecovery has been able to fix the situation for short time periods.
Plain factory reset might not fix raw space problem but could buy you enough free space to do a successful balance.
I achieved that luckily already. Deleted some gigabytes of Android app Game files in sdcard that also still were in my Downloads folder. Rebalance worked fine, but the issues are still present after a reboot.Though BTRFS reports enough space again.
MoritzJT ( 2014-03-19 10:57:00 +0300 )edit@MoritzJT What does 'btrfs fi show' currently show to you? Answer to previous question; I don't know how to enable dev mode on the console, sorry.
Your experiences are valuable. Actual problems with the software may be caused by corrupted databases due to lack of space at the time. Many of the Jolla functions and programs seem to write their statuses to their respective databases or gconf. Now they might have problems reading those settings.
Manatus ( 2014-03-19 11:41:07 +0300 )editSounds plausible. I guess that's what happened. I think a factory reset will now be the most clean solution. Or do you think I should wait and contribute with further investigation?
I don't believe fixing it all manually is an achievable task for me :D
MoritzJT ( 2014-03-19 11:47:39 +0300 )edit@MoritzJT Fixing all possibly broken places might be as simple as deleting the original databases and restarting the associated services or plain reboot, but with gconf it could eventually lead to factory reset anyway. And it is definitely faster, of course. :)
After the factory reset you may want to check the btrfs for traces of previous state and do some cleanups. See:
https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/
After that maybe do balance again.
Manatus ( 2014-03-19 13:29:21 +0300 )editSo to recap it all.
I ran into the no space issue and couldn't even 'clobber' some files with echo/true commands. I stumbled upon https://together.jolla.com/question/14633/bug-factory-reset-no-storage-left/ and followed the routine.
It was easy for me as I still had developer mode enabled and could thereby access the shell via Putty SSH. For all those who can't, a look at https://together.jolla.com/question/22079/howto-all-pc-users-recover-or-reset-a-device-that-is-stuck-in-boot-loop/ and get access to a shell that way.
Once you have it sorted, you might try a rebalance and if that still fails, try looking for small files to delete first. I managed to free enough space that way until I could again delete the big files.
A rebalance later I could at leaste go for the factory reset option. Now I'm back to 1.0.4.2 and looking for any leftovers to clean up.
Thanks for your help
@Manatus
I'm wondering now, with all back to normal, wouldn't it be wise to create a snapshow of each clean update?
MoritzJT ( 2014-03-19 16:04:11 +0300 )edit@MoritzJT Thanks for the report. Out of curiosity, had you done a factory reset before, meaning you had those subvolumes named rec* in existence? (Haven't seen them on my device yet.)
I do think that advanced features such as snapshots and subvolumes got us into this situation in the first place. Deduplication information for snapshots and duplicated metadata has to be stored somewhere, and as a result filesystem itself doesn't seem to be very well aware of the amount of the total data used. :(
Manatus ( 2014-03-19 20:40:25 +0300 )editI managed to give rebalance a successful run prior to doing a factory reset. It didn't show any rec* before I did that. But afterwards, I updated to 1.0.2.5 and then to 1.0.4.2 and that left me with two large rec*s that I then deleted before finally running rebalance again.
MoritzJT ( 2014-03-19 20:42:27 +0300 )editOk, thanks for clarification. :)
Manatus ( 2014-03-19 21:16:02 +0300 )editReboot while running "btrfs fi balance /home".
I didn't experience crashing applications or anything like that, but ran "btrfs fi balance /home" as "btrfs fi show" command gave "13.75GB used of 13.75GB" output. Before the balance finished my Jolla suddenly rebooted. Maybe this is because a watchdog is not getting response from the OS in a timely fashion or something like that. Before the reboot the phone had a high load average and felt a bit unresponsive. The result was that it rebooted to a Jolla logo screen, not to the normal UI, then rebooted again and after longer than normal boot it came up to the normal UI.
Using "top" from a ssh session I could see that the balance was still running.
As a result I'm not sure if it's wise to recommend users to run the balance operation if it can result in a bricked device in case of a sudden reboot.
Jonni ( 2014-06-10 15:01:17 +0300 )edit@Jonni If the balance was still running after reboot, I expect that it was a restart to several services because of watchdog, as you mentioned, and not a real power cycle of the device itself. You are correct that this operation may cause such problems, especially with a file system that hasn't been balanced before. Mine was noticeably heavy on CPU first time I run it, although all later ones have been a lot faster and lighter.
You are also correct that this may cause bricked device, especially if it happens at the same time with other bugs.
I'll add a warning in the first paragraph.
Manatus ( 2014-06-10 15:17:29 +0300 )edit@Manataus the uptime of my device went to zero during the reboot I experienced while the rebalancing was running, so I think it was a real reboot.
Also /var/log/systemboot.log shows:
20140610_143709 Startup: pwr_on_status=pwr_on_by_HW_reboot
20140610_143739 Received: dsme internal state USER
20140610_143748 Startup: pwr_on_status=pwr_on_by_HW_reboot
20140610_143752 Received: dsme internal state USER
20140610_143853 Startup: pwr_on_status=pwr_on_by_HW_reboot
20140610_143859 Received: dsme internal state USER
I started the balance command around 14:35, don't remember when exactly.
Jonni ( 2014-06-10 17:21:29 +0300 )edit@Jonni Ok, thanks. So what we could expect happened was that the phone rebooted, as you said. Then it came around, or didn't properly, which then triggered instantly some kind of recovery. And after that the second reboot.
Looks like btrfs has skip_balance function (since kernel version 3.3). This implies that if it is not set, balance operation will be rerun when Jolla comes around again after unclean shutdown. That's why it was running then...
https://btrfs.wiki.kernel.org/index.php/Mount_options
Manatus ( 2014-06-10 19:46:31 +0300 )editAny ideas, still doing sync after 19h?
Bysmyyr ( 2014-07-11 08:10:57 +0300 )editIt was a happy accident that I came across this... My Jolla had it like this:
Now it's this:
And yes, I did encounter occationally green led, reboots, and even email and browser crashes. Hope this helps. For the record, I did
btrfs balance start /homeandbtrfs balance start /separately, don't really know if it was a good or bad thing, but hey, I didn't brick it!I knew it that btrfs must have some backsides too, and I guess I now know one... Then again, now that I know it, I can live with it!
Direc ( 2014-07-11 14:34:36 +0300 )edit@Direc, Nice to know, it looks like that balance syntax without fi or filesystem in the middle is accepted.
I think that both btrfs commands did not run at the same time as they are targeting essentially the same volume. Possibly the second balance started after the first had finished. It probably was a lot faster too.
If you still have similar crashes and freezes in coming months without hitting devid 1 used higher than 12.25 GB (1,5 GB margin), please inform us here. Thanks! :)
My own problems with Jolla have ended and stayed away since I started keeping an eye on free raw space on the volume.
Manatus ( 2014-07-11 15:37:50 +0300 )edit@Manatus, running just
Direc ( 2014-07-11 15:52:05 +0300 )editbtrfs filesystem balancefor more help actually said "btrfs filesystem balanceis deprecated, please usebtrfs balance startcommand instead." I first ran the /home balancing, then for / and actually the later was significantly faster, and it didn't free up any further space. So far so good; I'll keep my eye on free raw space, too, and will let you know it my hiccups continue.@Direc, Thanks. I changed the balance syntax throughout the whole article.
Manatus ( 2014-07-12 19:13:36 +0300 )editI had also this problem without really noticing any very bad consequences before. But by chance, I tested 'btrfs fi show' and it was full! I have not done anything on my phone apart from activating 'dev' side. So I really wonder why I had it full and how easy it seems to have it full! Probably something for Jolla dev to consider?
pat_o ( 2014-07-23 14:34:39 +0300 )editThis far it looks to me that the reserved group block space is often almost double the size of the actual data.
I would reckon that one gets into real trouble when it is the metadata that tries to expand and cannot do that. Btrfs would be a very bad filesystem indeed if it couldn't just rearrange file locations between the 1 GB data block groups, without assigning a new one every time it has to do housekeeping.
We have to remember that the devs of btrfs (or Jolla) don't necessarily consider that a fully allocated disk is a problem to begin with. We do know that we get around the problems by trying to avoid it, but in the end it could be just a simple bug in kernel for btrfs that is causing all the woes.
Manatus ( 2014-07-23 18:37:41 +0300 )editHmm, my jolla has leds blinking and various slow downs and jitter after days of uptime. There should be space free but:
Copying a 500mb file to /home/nemo works without errors. I wonder if I should do the balancing thingy?
mcfrisk ( 2014-08-20 16:41:56 +0300 )edit@mcfrisk Since you already have problems with the device, I would give it a go. Just be prepared that it may take a long time if you have had your device for a longer period and it is the very first time you run the balance.
Manatus ( 2014-08-20 20:04:59 +0300 )editafter this incident email is not synced any more and I cannot set the interval either
ortylp ( 2014-09-03 22:45:09 +0300 )edit@ortylp It is possible that a database or gconfig has corrupted when the client hasn't been able to write the changes properly. It would be best to delete the account and create it a new.
Manatus ( 2014-09-08 10:52:04 +0300 )edit@Manatus I did exactly that and it works again
ortylp ( 2014-09-08 21:29:40 +0300 )editJust commenting to tell Jolla that I (and many others, possibly) still have that problem. Since I'm more of an average user and I guess that Jolla is targeting people like me doing the solutions mentioned above is not an option, period. I won't risk bricking my device to fix something that shouldn't have happened in the first place. Mine is close to unusable at the moment despite around 7gb of space available. Can I expect a fix in the next one or two updates?
dom1n1kus ( 2014-09-17 18:39:57 +0300 )editSo, I was getting weird freezeups etc. that I couldn't explain with CPU usage or similar (generally on app startups or swiping back to the homescreen, maybe memory paging related), so I started a rebalance at about 9pm on Sunday. The UI stayed responsive until 11pm where it had the green flashing LED and restarted the homescreen. UI says it's not connected to WLAN, various things won't start but my SSH sessions are still working so it's just not telling me the truth. I requested a cancel because I wanted to take my phone with me the next day, but it didn't immediately respond so I left it overnight.
Anyway, it's Tuesday morning and it's still going (I took the SIM out on Monday morning and put it in another phone, left the Jolla on my home wifi). Status is (and has been since Sun 11pm):
topshowsbtrfsandbtrfs-transactitaking a consistent 9 to 11% CPU.Any suggestions? Might get cavalier and reboot it if it's still going with no progress when I get back from work tonight.
vitaminj ( 2014-10-14 12:04:45 +0300 )editCheck the journal (journalctl -f as root), there were a lot of btrfs messages there when I did a rebalance.
BTW, I did rebalance with about 6 GB of free space and it took about an hour.
MartinK ( 2014-10-14 20:42:45 +0300 )editAs above, and if it is stuck because there is no space to handle btrfs operations at all, try deleting lots of unnecessary files where possible to give it room to play with.
Manatus ( 2014-10-15 08:54:30 +0300 )editSame problem, luckily found out it early while upgrading Debian in chroot. Also had browser crashes, green LED, etc., but reason was not clear then. Fixed with rebalance, everything seems fine for now.
I have a correction: flags -d<filter> to the balance command are not deprecated, at least they are present in FAQ, official man page [1], Debian man page. And options like -dusage=5 are rather useful. As far as I understand, full rebalance needs more free space to run, while partial rebalance may still work in more restricted conditions. You can decide what to rebalance, -m for metadata, -d for data, and then increase percentage in the option from 5 to 100 until enough space will be free. Also partial balance runs much faster.</filter>
[1] https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance
butler ( 2014-10-31 13:13:04 +0300 )editTo follow up on my woes, the process had indeed hung waiting for free space even though I think it had more than 1gb free before I started (thanks @Manatus, though why it takes 20% cpu just waiting is beyond me). Nothing suspicious in the journal except the wifi driver spitting out a stack trace on some btrfs call (which explains the general UI death).
I rebooted for fun, but obviously the rebalance tried to resume on next startup. So then I tried to delete some big files, but the btrfs metadata store must have been full during the rebalance as it wouldn't allow it. And the "echo > file" zeroing trick only worked on a few files before complaining about no free space left again. Emptying more files seemed to work after a few minutes (I assume the background rebalance was stealing the free space). So with a bit of strategic xargs'ing to recursively discard files, running every 5 mins, eventually I got rid of enough that there was enough room for the rebalance to work properly.
A reboot and all was good again. Back to normal, and predictable lockups gone.
How any "normal" user is supposed to go through this ridiculous hand-cranking process is beyond me. And indeed the symptoms for me weren't too bad (before my filesystem was totally full), just freezeups which I could have attributed to (and I bet many customers do) inherent dodginess of the phone, which is probably really tarnishing Jolla's image with a subset of users.
How would Jolla fix this in a firmware update though?
"Please free up 2gb for this update and then wait while we rebalance for you on next boot, hope you didn't need your phone for the next hour"? Or maybe rebalance in the background on next boot - just don't turn the phone off or run out of battery until we say you can!
Oh actually you can't free up 2gb because your metadata partition is full, please get out an SSH session and xargs, or send your phone to Jolla Care with a list of files you don't care for any more.
OK, I'm rambling now. Enough.
vitaminj ( 2014-11-05 21:17:35 +0300 )edit@butler, Good point about -dusage option. It might be actually best to use that in the beginning instead of full balance as you said. Just tried it and it did run very fast.
I don't remember why I thought it was deprecated, but I have a recollection that it did not run when I tried the option last time. It may have been because the original syntax I used was already deprecated and it didn't work then, or I misinterpreted output of the command.
I'll change the article a bit. I was planning to make a shorter wiki article about balance command for future reference; no point for everybody to go through the whole thing and hunt for changes. I haven't dared to open up this one as wiki as I wanted to keep it structured to my liking.
Manatus ( 2014-11-05 23:42:36 +0300 )edit@vitaminj, Thanks for the follow up. You pretty much nailed the problem there. :)
Manatus ( 2014-11-05 23:44:11 +0300 )editHi, thanks for this.. I had "desktop folder opening" issues, and helpful #sailfishos people directed me here.
Initial state:
I ran:
I took maybe 10 seconds to run and now I have 8 Gigs free again :)
After running the command my state is:
Juuba ( 2014-11-10 15:39:19 +0300 )edit@Juuba, Thanks for the report, this confirms that it would be best to use usage filter for balance. It will also make balance much more safer to run for everybody. All one should do is clear up couple of gigs of files and run balance with -dusage=5 or preferably -dusage=0 for the starters. Unless the whole device is already borked OS-wise, which is a different matter altogether.
I could swear that that btrfs balance manpage did not exist in February 2014. May be it did or didn't but things don't look so grim regarding this problem anymore. I wonder if balance with -dusage=0 could be automated on device. It deletes the unused block groups and is basically without risk.
People should still be warned about not to fill up their devices to the brim, though.
Manatus ( 2014-11-11 08:58:41 +0300 )editHi,
you could add following information to the post:
In case you get "no space left on device" when you are running the balance you can use the external SD card to get the extra space:
That way the balance can get the needed extra. You can remove the device like this:
I found this tip from: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
Juho ( 2014-11-24 09:13:28 +0300 )edit@Juho, Sorry I did notice your comment only now... Thanks for the tip and the link, it is a nice read. :)
I don't dare to put it up in the article, though. The reason being; if the phone crashes and reboots during the balance, while the root system is still spread on several devices, there might be no way to get the phone back in booting condition (I'm not pro enough for that at least). The recovery might completely fail loading. It is much safer to try and find out the files you can delete or "clobber", even if it takes a while to do that.
EDIT: Your trick is too nice to go unnoticed. I added it to the notes... Thanks! :)
Manatus ( 2014-12-07 18:06:49 +0300 )editThanks for the reply @Manatus. Good you mentioned it is risky. When I did it the phone rebooted and the btrfs didn't like it at all. I had to overwrite the filesystem mainly because my btrfs skills were not good enough. Maybe it is a last resort thing.
Juho ( 2014-12-07 20:16:13 +0300 )editI would very much like to see a mechanism showing the user a big red dialog window when the space btrfs needs is about to run out. (I.e. when less than 1.5 G are unallocated if I understand http://permalink.gmane.org/gmane.comp.file-systems.btrfs/30066 correctly.) A daily check on the system side might be enough.
I'm not close to facing this problem, but I don't like the idea of having this kind of time-bomb built into my phone. IMHO the worst part of this is that the user only sees weird random failures and does not get any hint about what could be actually wrong.
Jolly-Jo ( 2014-12-08 12:27:08 +0300 )edit