Original link: https://blog.lilydjwg.me/posts/216670.html
This article comes from Evian’s Blog , please indicate when reprinting.
The previous article “Btrfs Overturned” described the accident of btrfs on our server, which seemed to scare some users QAQ. In fact, that situation was rather special. Generally speaking, even if the metadata is full, it is not necessary to change the kernel code to save it. However, the problem of full metadata has indeed troubled many users. It happens that many people in the group have encountered it these days. This article will record how to deal with it when the metadata is full.
Problem and Disposal
The phenomenon of the problem is that some file operations report an error “No space left on device”, but tools such as df clearly report that there is still space. The output of btrfs filesystem usage is this:
We can see that there is still 373G of free space (Free). However, “Device unallocated” is less than 1G. On a sufficiently large file system, btrfs will allocate block groups (block groups, bg for short) in units of 1G. So in this situation, it is no longer possible to allocate new bg. Then let’s look down, the “Metadata” part has a total of 4.5G, and 4G has been used. There are 512M left, which is just the size of the Global reserve. In other words, if the reserved space is not counted, there is no space available for metadata, so an error will be reported.
So what do we do now? If there are unnecessary large files on the file system, and there is no snapshot, if the space can be released immediately after deletion, you can delete it and see if you can just empty a bg. Otherwise, try running balance, like this:
This command is to sort out the data bg with a usage rate of 0%. From the output “had to relocate 0 out of …”, it can be seen that without such a bg, the operation has no effect. You can try increasing the value of -dusage to see if it works. Unfortunately, it was unsuccessful in this case:
Then only add some space to move the data. If there is an idle partition (or a swap partition that is temporarily unused), it can be used. Otherwise, you can also insert a U disk. It doesn’t need to be too big, just a few G will do. It can be removed after moving the data, and it will not be used for a long time. Another very risky approach is to use memory to temporarily store data, but once the system crashes or the power is cut off, the entire file system will be ruined, so it is not recommended.
After preparing the free partition, you can add it with btrfs device add . The -f parameter is usually added here to erase the original data in the partition. Be careful not to add the wrong device.
It can be seen that after the device is added, data can be written to the file system again. Next, run commands like btrfs balance -duage=10 to make some data bg out. Here, we should pay attention to only balance data bg, and do not move metadata bg, because the more centralized the metadata is stored, the more likely it is necessary to allocate new bg in the future, and the more likely it is that there will be no bg to allocate.
The btrfs-heatmap tool can view the distribution and usage of bg. The following is the state after the balance is good (using the --curve linear parameter):
In the figure, the white ones are data bg, and the blue ones are metadata bg. The brighter the color, the higher the usage rate. Solid black is unallocated space. As you can see, there are a lot of data bg that is not used much. The balance operation is to merge some of them, leaving a lot of black areas (the bottom black part is the unallocated space on the newly added device). Here is a screenshot of usage:
After doing the math, excluding the space of the newly added device, the original storage device can also have 46G of unallocated space (it seems that the newly added device has not started to be used, but just makes btrfs believe that it has enough space). Next, just delete the previously added device: btrfs device del /dev/sdb1 / . Wait until it finishes running to unplug the device (if it’s removable media).
prevention
The essence of this problem is that the fragmentation of bg leads to obviously having space, but the metadata cannot be used, so an error is reported and manual processing is required. It is also very simple to identify the file system that is about to cause problems: Take a look at btrfs filesystem usage . If the “unallocated” is very small (less than 1G), you should quickly balance it (of course, the premise is that there is a lot of fragmented free space). Note, do not delete the snapshot at this time! Deleting snapshots may quickly consume the reserved metadata space, resulting in the situation that adding devices cannot be added, and an error is reported as read-only.
btrfs recently added automatic block group reclaim (automatic block group reclaim), but it is not enabled by default. Because it is a new function, there may be bugs, and you don’t know when it will run, so I don’t recommend it for the time being. It is also good to write a timing script by yourself and run it when the system is idle.
The image materials in this article are provided by friends who have encountered problems.
This article is transferred from: https://blog.lilydjwg.me/posts/216670.html
This site is only for collection, and the copyright belongs to the original author.