Anton Lindstrom (about, @twitter, @github)

Adventures with Puppet and Sensu

Published:

I took some time to play around with the Sensu Puppet module this weekend. It's isn't the most mature module but it's working terrific. Once I did some diving into the module it was pretty easy to get around to using. This will be somewhat a description of what I did to get Sensu up and running in my own infrastructure.

The readme will probably get you up and running, the thing I missed however was that the default handler isn't included (I really should make a pull request).

Note that the module might change a lot so this will probably be some faults in this config in a few days/weeks/months.

Current README example:

node 'sensu-server.foo.com' {
  class { 'sensu':
    rabbitmq_password => 'secret',
    server            => true,
    plugins           => [
      'puppet:///data/sensu/plugins/ntp.rb',
      'puppet:///data/sensu/plugins/postfix.rb'
    ]
  }

  sensu::check { 'check_ntp':
    command     => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H pool.ntp.org -w 30 -c 60',
    handlers    => 'default',
    subscribers => 'sensu-test'
  }

  sensu::check { '...':
    ...
  }
}

What we'll have to add is the following:

sensu::handler { 'default':
  type      => 'set',
  command   => 'true',
  handlers  => [ 'mailer' ],
}

sensu::handler { 'mailer':
  type        => 'pipe',
  source      => 'puppet:///modules/data/sensu/handlers/notification/mailer.rb',
  config      => {
    mail_from     => '[email protected]',
    mail_to       => '[email protected]',
    smtp_address  => 'localhost',
    smtp_port     => 25,
    smtp_domain   => 'example.com',
  }
}

The example above will create a default handler that is a set, which means that it's passing information to other handlers, in this case mailer. I have a separate module called data where I put the handlers and plugins, so that's where mailer.rb is. The config is a hash which is generated into json as the config for the handler. It'll be called mailer.json.

Let's add a new check to better explain how it works. You can either add it in the class { 'sensu': } block or by defining it as a plugin.

sensu::plugin { 'puppet:///modules/data/sensu/plugins/http/check-http.rb': }

sensu::check { 'http_antonlindstrom_com':
  command     => '/etc/sensu/plugins/check-http.rb -u "http://antonlindstrom.com/"',
  handlers    => 'mailer',
  subscribers => 'sensu-test',
}

By using a little hiera magic we can make a pretty clean node definition and it's pretty awesome.

I've been running sensu for a few hours now and can say that it's the easiest thing to get running when you have a dynamic and configuration management controlled environment. The adventure has been great as it alerts pretty smooth and works faster than Nagios. There aren't that much of reports and GUI fanciness but it does really what it should do. Sensu alerts when it should. The only problem I found was that it didn't alert that fast when nodes became unresponsive (for instance when shutting down a machine).

The problem could be solved by running checks that alerts on the services on the node and do real service metrics instead of host metrics. By running checks on your service and checking load and metrics for how it is performing, a host should not really be important if the service still responds well.

Conclusion: I really found Sensu great as it pinpointed the most troublesome parts of Nagios. Configuration. In Sensu it really works great with the supported configuration management modules.