Using Monit to Monitor and Restart a Python Script

This is the first follow-up to my post about my Raspberry Pi Fish Tank sensor, I have a python script for reading the water temperature from the sensor and recording it to a Postgres DB hosted on another server. The script works pretty well and seems to handle any errors ok, but I wanted the script to be restarted after a series of errors (which might just be due to the other server being patched and rebooted). At first I thought about just wrapping the script in a bash block to keep retrying, but then I remembered that I learned of a simple tool called Monit a while back, but just hadn’t got around to trying it out. “Trying things out” is exactly why I put together this sensor and system, so the following is my quick and dirty guide to using Monit to manage a python sensor script on a Raspberry Pi. 

First off, what is Monit? Well it’s billed as “a free, open-source process supervision tool for Unix and Linux.” It watches a process and restarts it if it fails, along with some other bells and whistles. Sounds great, exactly what I need for my temperature sensor.

 First install monit, which is simple enough…

apt-get install monit

Then edit the configuration file, which on the Raspberry Pi is located at /etc/monit/monitrc, (for reference, the general documentation is at https://mmonit.com/monit/documentation/monit.html)

First I’ll change the timing (in seconds) for how often the daemon is going to check to make sure my script is running. I like once per minute, it’s a balance between often but not too often to overload the Pi zero (though I’m sure I could set that lower without issue).

set daemon 60    

Monit also comes with its own simple web interface, which sounds useful so I want  that to be running and I will make it available on my network (since it’s all behind a firewall and not public anyway, if your not on a trusted network or to just want to be more secure you might want it listening on https and also protected by a login since the web interface allows you to start/stop jobs/commands).

set httpd port 2812 and
use address 192.168.0.14
allow 0.0.0.0/0.0.0.0 
# allow admin:monit # how to enable a login

Finally, I also add my local mail relay so that when Monit takes actions I will get a notification about it. 

set mailserver 192.168.0.10              # primary mailserver

That’s it for the configuration file, though you might want to read through it and adjust other options for your environment or setup. I can then use monit with the -t flag to verify the syntax of the configuration file is correct:

$ sudo monit -t
Control file syntax OK

Now, I want Monit to also start at boot so I need to edit the service configuration file at /etc/default/monit and ensure “START” is set to “yes”, which I believe is already the default on the Raspberry pi. 

# Set START to yes to start the monit
START=yes

Now to add in the check for my python script. Monit has a config directory where I can drop in my process definition file. So I make the file /etc/monit/conf.d/1-wire and the contents are:

check process 1-wire with pidfile /tmp/1-wire.pid
   start = "/bin/su - pi -c 'cd /home/pi/Code/pi-fishtank-sensors; /usr/bin/python3 1-wire.py  >> /tmp/fishtank.log 2>&1 &'"
   stop = "/usr/bin/killall python3"

This should hopefully be straight forward, Monit is going to monitor a process called 1-wire by checking a pid file in the /tmp directory (which, with a little code modification to my python code, I now have it make the PID file when it starts up). If that process is not running monit will attempt to restart the process by first switching to the Pi user, changing to the correct directory, then executing the python script and redirecting STDIN & STDERR to a log file also in the /tmp directory. And if Monit needs to stop the script it does so by executing a killall on python3 (which can be a little dangerous, since it would kill any running python, but in this case the only thing the Pi is doing is running this script). Once the changes are saved in the file and Monit restarted, I can check the status by passing the “status” option to monit which also outputs the same information as what you can see on the web interface. 

sudo monit status

And the web UI outputs:

Overall I’m pretty pleased. A nifty little tool that does the job and was really easy to set up. Definitely will be keeping this in my tool chest and using it for little one offs.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.