Use Superlance to recieve error mail notification when a Supervisor process fails

  •  
  •  

Superlance is a plugin written in Python and it has a multiple Event Handlers that can trigger SMS or Email notifications or even call an http endpoint to setup a hartbeat signal.

Even though the setup process is very easy, there are a few things that need to be in place for Superlance to function. We'll also tell you how to setup the latest Supervisor and a service (assuming you're running a Linux distribution with systemd), though you can skip this step if you already have an instance of Supervisor running.

Install latest Supervisor & Superlance using pip

Skip this step if your supervisor is already running, though check your supervisord.conf to make sure supervisor's HTTP server is working.

Instead of installing the Supervisor shipped with your distribution, using apt-get install supervisor for example, we'll get the latest version using pip :

sudo apt-get install pip
sudo pip install supervisor superlance

# Required files and folders (if not exist)
mkdir -p /etc/supervisor/conf.d
touch /etc/supervisor/supervisord.conf
touch /etc/supervisor/conf.d/00superlance.conf

Minimal supervisord.conf

To get supervisor running, we'll need a minimal configuration. Especially the [unix-http-server] and [supervisorctl] sections are needed to make Superlance able to talk to Supervisor. Put in the contents below into your /etc/supervisor/supervisord.conf :

/etc/supervisor/supervisord.conf
[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[include]
files = conf.d/*.conf

Supervisor Daemon: Systemd supervisord service

You can skip this step if you've installed supervisor trough your package-manager (the supervisord service will already exist). Otherwise, we'll manually have to make a service file, to allow supervisor run as a daemon or background process. Put the following contents in your /etc/systemd/system/supervisor.service :

/etc/systemd/system/supervisor.service
[Unit]
Description=Supervisor daemon
Documentation=http://supervisord.org
After=network.target

[Service]
ExecStart=/usr/local/bin/supervisord -n -c /etc/supervisor/supervisord.conf
ExecStop=/usr/local/bin/supervisorctl $OPTIONS shutdown
ExecReload=/usr/local/bin/supervisorctl $OPTIONS reload
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target
Alias=supervisor.service supervisord.service

And then enable the service and start supervisor, to see if you did right:

systemctl enable supervisor
systemctl start supervisor

Credits go to CloudWaferHQ for providing the outline for the systemd-file, though we've renamed the service to supervisor for it to be consistent with standard installation (trough package managers). The aliases are there to ensure both supervisor and supervisord can be used as service name.

Setup Superlance

Assuming you've already Supervisor and Superlance (see the Install steps above), we'll give an example how to set up an listener to receive an email when a process exists.
To set up other triggers, the configuration steps are the same, though the trigger and command you call will change.

Crashmail event listener: 00crashmail.conf

Even though Superlance documentation states the event-listener should be added to supervisord.conf , we like it in a separate file.
Put the contents below into your /etc/supervisor/conf.d/00crashmail.conf ;

/etc/supervisor/conf.d/00crashmail.conf
[eventlistener:crashmail]
command=crashmail -a -m root
events=PROCESS_STATE_EXITED

A few notes on the 00crashmail.conf file:

  • The -m option defines the target mailaddress, it must be set for crashmail to work.
    We've set it to root and use an alias in /etc/aliases to set the actual final address. You can use your own mailaddress here as well.
  • The events option has been set to PROCESS_STATE_EXITED, to only receive a message if a process exists.
    Use PROCESS_STATE to receive a message on every state change a process makes, or have a look at other Event Types to set as your trigger.
    Separate multiple events with a comma, e.g.: PROCESS_STATE_EXITED,PROCESS_STATE_FATAL .
  • The filename 00superlance.conf starting with 00 ensures the event listener is loaded before other processes are.
    Optionally you could add priority=1 instead, though we haven't tested this.
    Or just put the lines in your supervisord.conf , might there be issues.

 When you're done, simply restart supervisor to apply your changes:

systemctl restart supervisor

Errors, debugging and troubleshooting Superlance

First and foremost, it's important to understand the chain for an email to be sent: Supervisor → Superlance → Crashmail → Sendmail.
For debugging it's good to follow the chain as it should work, to make sure every chain link works properly.

For all errors on Linux, the troubleshooting is often a matter of reading logs and output messages until you understand where it goes wrong. Mostly…
Use the following commands to get some insights:

# If supervisord service doesn't start
systemctl status supervisor

# Supervisor log output, CTRL+C to exit
tail -f /var/log/supervisor/supervisord.log
 
# Mail log ouytput, CTRL+C to exit
tail -f /var/log/mail.info
tail -f /var/log/mail.err
tail -f /var/log/mail.log
 
# Global system log (to see both supervisor and mail activity)
tail -f /var/log/syslog

Test outgoing mail

Even though the /var/log/mail.… -examples from abouve should show if an email is sent, sometimes it's easier to just manually invoke sending an email:

echo "Test message body" | mail -s "Test subject1" your@email.addr.com

Test event trigger: Manually kill process

To see if the event gets triggered at all, you can manually kill the process manually. This should trigger the event listener.

# Find your process
ps aux | grep process.php

# Then kill the found processess
ps aux | grep process.php | grep -v grep | awk '{print $2}' | xargs kill

Sources