How does your boss react if the database stopped working because of a full filesystem? Depending on what application you have on your server the filesystem can fill up rapidly and a monitoring software like Nagios, can warn you before it happens. But will there be enough time after the warning to avoid the worst-case scenario? we have a database running on our clustered system. The database filesystems are all monitored and there is space for them to grow. One of the ,so called, temporary databases is stored on a 50GB LUN. Due to errors in queries it is possible that this database grows by 10GB in minutes. That means that the time between a warning of 80% utilization and a ‘no-space-left-incident’ is very short. Whenever there is no space left, the whole database will stop working. …. welcome to the magic world of databases 🙁
The only way to keep the system working is to stop the task that generates the high amount of rubbish in the temporary database. This usually costs time.
After having that issue on a Sunday morning, I decided to get me more time to react in creating empty files on the filesystem. So whenever I get a warning I can delete one of my dummyfiles and gain ~5 minutes reaction time.
one could say that decreasing the threshold of the warning percentage would also help. Good point, but we are monitoring ~50 servers with ~200 filesystems. I don’t want to have a separate Nagios configuration for different filesystems.
I wrote a very small script just to automate the process of creating dummyfiles on the system.
#!/bin/bash ########################################################### # crt_dummy.sh # creates dummyfiles # number = number of dummyfiles # size = size in bytes per dummyfile ########################################################### NUM=$1 SIZE=$2 if [ "$#" -ne "2" ] then echo Usage: "$0 number size" echo number = number of dummyfiles echo size = size of dummyfiles in Bytes exit 1 fi for ((i=1;i<=$NUM;i++)) do dd if=/dev/zero of=delete_me${i}.dummy bs=$SIZE count=1 done
A [root@testbox ~]# crt_dummy.sh 5 10485760
creates 5 dummyfiles with a size of 10MB.
Some useful sizes are:
Size | Bytes |
---|---|
10MB | 10485760 |
50MB | 52428800 |
100MB | 104857600 |
200MB | 209715200 |
1GB | 1073741824 |
5GB | 5368709120 |