Model instance verification

[ianunruh]

During instance deployment, it should be possible to verify that an instance was correctly configured.

Deployment workflow w/ Sensu

1. Add client definition, check definitions and metric check definitions to Sensu
2. Wait for configuration to be pushed to instance
3. Wait for notifications of OK check events

Termination workflow w/ Sensu

1. Remove client definition, check definitions and metric check definitions from Sensu

Check mechanism

For this process to work, we have to be able to model what service checks we want to define in Sensu. There are two parts to a check:

1. The check definition on the Sensu monitoring server (this is in JSON)
2. The check itself on Sensu clients (arbitrary executable, probably Ruby)

We have to distribute the checks via Puppet. The check definitions can be managed via Puppet as well. Results of the check can be distributed in a number of ways, including over RabbitMQ via AMQP. After instance deployment is finished, check failures can be used to notify the adaptation engine of issues.

Metric mechanism

Metrics work in nearly the same way. Metric check definitions are setup on the monitoring server and metric checks are deployed to the clients.

Unlike service checks though, we don't need to wait on metric check results. These checks are just used as a feedback mechanism for the adaptation engine.

Resources

http://docs.sensuapp.org/0.12/index.html
https://github.com/sensu/sensu-community-plugins