upstart subshells

Ran into an interesting problem with an upstart service config recently.  So an engineering team was running a third-party daemon using an upstart script, but they needed to capture the stdout and stderr of the daemon for debugging purposes.  After thinking for a bit, I just appended ‘ | logger -t <somelabel>’ to the exec line in the config, and everything seemed to work.

Of course what I didn’t realize is that if any shell metacharacters are included in the exec statement (eg. the pipe), upstart will automatically open a subshell and run the daemon in it.  The pid of the subshell is then tracked by upstart as the pid of the service.  This is fine in theory as when upstart stops the job and sends a SIGTERM to the parent shell, the child process receives a SIGHUP and should shut down.

In this case, the daemon apparently handled SIGHUP and continued running.  This means that upstart thinks the daemon is stopped when it is actually still running, which isn’t good news for anyone as now multiple daemon instances are possible.

A possible solution is to simply use file redirection (eg. something like > /var/log/somelog), but a new logfile means a logrotate update, splunk forwarder updates, and possible other headaches.  It would be really nice to use logger and let syslog handle the messages if possible.

Thankfully, I stumbled across this very clever approach.  This solution creates a temporary fifo to negotiate a shared file descriptor that the given daemon can write to and logger can read from.  Very elegant IMHO.

http://serverfault.com/questions/114052/logging-a-daemons-output-with-upstart

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s