Filesystem full. Howto avoid the worst case

How does your boss react if the database stopped working because of a full filesystem? Depending on what application you have on your server the filesystem can fill up rapidly and a monitoring software like Nagios, can warn you before it happens. But will there be enough time after the warning to avoid the worst-case scenario? we have a database running on our clustered system. The database filesystems are all monitored and there is space for them to grow. One of the ,so called, temporary databases is stored on a 50GB LUN. Due to errors in queries it is possible that this database grows by 10GB in minutes. That means that the time between a warning of 80% utilization and a ‘no-space-left-incident’ is very short. Whenever there is no space left, the whole database will stop working. …. welcome to the magic world of databases 🙁
The only way to keep the system working is to stop the task that generates the high amount of rubbish in the temporary database. This usually costs time.
After having that issue on a Sunday morning, I decided to get me more time to react in creating empty files on the filesystem. So whenever I get a warning I can delete one of my dummyfiles and gain ~5 minutes reaction time.
one could say that decreasing the threshold of the warning percentage would also help. Good point, but we are monitoring ~50 servers with ~200 filesystems. I don’t want to have a separate Nagios configuration for different filesystems.
I wrote a very small script just to automate the process of creating dummyfiles on the system.

#!/bin/bash
###########################################################
# crt_dummy.sh
# creates dummyfiles
# number = number of dummyfiles
# size = size in bytes per dummyfile
###########################################################
NUM=$1
SIZE=$2

if [ "$#" -ne "2" ]
then
        echo Usage: "$0 number size"
        echo number = number of dummyfiles
        echo size = size of dummyfiles in Bytes
        exit 1
fi

for ((i=1;i<=$NUM;i++))
do
        dd if=/dev/zero of=delete_me${i}.dummy bs=$SIZE count=1
done

A [root@testbox ~]# crt_dummy.sh 5 10485760 creates 5 dummyfiles with a size of 10MB.
Some useful sizes are:

Size Bytes
10MB 10485760
50MB 52428800
100MB 104857600
200MB 209715200
1GB 1073741824
5GB 5368709120

About Juergen Caris

I am 54yo, MSc(Dist) and BSc in Computer Science, German and working as a Senior Server Engineer for the NHS Lothian. I am responsible for the patient management system, called TrakCare. I am a UNIX/Linux guy, working in this sector for more than 20 years now. I am also interested in robotics, microprocessors, system monitoring, Home automation and programming.
This entry was posted in Bash, Linux. Bookmark the permalink.

Leave a Reply

Your email address will not be published.