Since it looks like EA2D's GitHub account has been deleted (the group doesn't exist anymore), I've moved the (kinda) maintained versions of several open source projects to my personal account.
The includes: - pingdom-python - Python library for Pingdom's REST API - loggly-python - Python library for Loggly's REST API - loggly-cookbook - Chef coookbook for Loggly
(originally written for EA2D's engineering blog)
As mentioned in a previous post, we aggregate and store our logs using a service called Loggly.
Since we wrote a library for programmatically managing Loggly inputs and devices, it was only natural for us to integrate it with our Chef deployment.
We've written and open sourced a Chef cookbook for Loggly.
From the README:
Installs the loggly-python library and provides a definition for the configuration of Loggly logging.
More specifically, the logglyconf definition will configure rsyslog to watch a log file and send its lines to a Loggly input. When first run, logglyconf will create the input and authorize the server to publish events to that input.
Developed for and tested on Ubuntu 10.10 LTS
You can grab it from the Opscode site:
$ knife cookbook site vendor loggly
Or from our GitHub repository:
A core tenant of DevOps is the notion of "Infrastructure as Code." Provisioning and deployment should be done programmatically, potentially with a configuration management (CM) system such as Chef or Puppet.
Jesse Robbins describes the goal well:
“Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources”
While CM tools facilitate this reconstruction, there are some tricks for getting the most of your implementation. Below are the key points from a lightning talk I gave on this at the last ArchCamp.
Using the configuration management server as the system of record. A Chef example is creating and modifying roles directly on the Chef server:
$ knife node create myrole
$ knife node edit myrole
This decreases visibility and introduces unnecessary risk. Losing the Chef server is now a major event. Yes, you can mitigate this risk with regular backups, but you still lack visibility into changes.
Place your CM data into source control and treat that as your system of record. Changes are committed to source control, then deployed to the CM server. Under Jesse's description, this buckets CM configs as source code rather than application data.
Going back to the Chef example, your workflow would instead look like:
$ git commit -am "added newfeature to myrole"
$ knife role from file roles/myrole.json
This gives you an auditable history of every operational config change.
But what if instead of deploying Chef changes via knife, you did so using git? Changes to multiple roles or cookbooks could be pushed simultaneously, and with one command.
In other words, this:
$ git commit -am "added newfeature to mycookbook, integrated it with myrole, and added supporting data to mydatabag"
$ knife cookbook upload mycookbook
$ knife role from file roles/myrole.json
$ knife data bag from file mydatabag newitem
Would become:
$ git commit -am "added newfeature to mycookbook, integrated it with myrole, and added supporting data to mydatabag"
$ git push origin master
This type of deployment can be implemented via a process that monitors the git repo for changes and deploys any changes to the Chef server.
A simplified version of the upload code is:
#!bash
for cookbook in $(git_diff("cookbooks")); do
knife cookbook upload $cookbook
done
for role in $(git_diff("roles")); do
knife role from file $role
done
for bag in $(git_diff("databags")); do
for item in $(git_diff("items", $bag)); do
knife data bag from file $bag $item
done
done
By putting this on a continuous integration server (Jenkins, BuildBot, etc.) and detecting repository changes through polling or post-commit hooks, you implement continuous deployment of your operational configs.
That wasn't so bad, was it?
Don't forget to check out the slides.
Pingdom is one of several monitoring tools we use at EA2D. Besides alerting us when things go down, we query Pingdom's API to include check status in our dashboards.
The old Pingdom SOAP API was unwieldy and slow. Fortunately, Pingdom released a new, JSON-ified REST API that remedied the problems of its predecessor.
I've written a Python library for this new API and released it as open source. For now, it supports only a subset of available resources, but the framework is there for others to be added easily.
https://github.com/mbabineau/pingdom-python
Set up a Pingdom connection:
>>> import pingdom
>>> c = pingdom.PingdomConnection(PINGDOM_USERNAME, PINGDOM_PASSWORD) # Same credentials you use for the Pingdom website
Create a new Pingdom check:
>>> c.create_check('EA2D Website', 'ea2d.com', 'http')
Check:EA2D Website
Get basic information about a Pingdom check:
>>> check = c.get_all_checks(['EA2D Website'])[0] # Expects a list, returns a list
>>> check.id
302632
>>> check.status
u'up'
Get more detailed information about a Pingdom check:
>>> check = c.get_check(210702) # Look up by check ID
>>> check.lasterrortime
1289482981
Delete a Pingdom check:
>>> c.delete_check(302632)
{u'message': u'Deletion of check was successful!'}
Go check it out on GitHub, or install it directly from PyPI:
sudo easy_install pingdom
If you have any questions, drop me a line: michael.babineau@gmail.com.
(originally written for EA2D's engineering blog)
For centralized logging, we use a service called Loggly. We forward our logs to Loggly and aggregate them by application and environment. This gives us a handy web interface for viewing logs across all servers within an application group, and provides some great tools for search, comparison, and alerting.
We send events to Loggly using syslog over TCP, and these events are bucketed based on destination port. For security, Loggly locks down each port to a list of authorized IP addresses.
Since we make heavy use of EC2 Auto Scaling Groups, we simply can not maintain this authorized IP list manually. Additionally, we're constantly launching new applications and environments, so our list of buckets (Loggly "inputs") is in constant flux.
Fortunately, Loggly has exposed a set of administration APIs for managing inputs and authorized devices. Since no library was available, we ended up writing one ourselves (in Python) and releasing it as open source.
You can find it on GitHub:
https://github.com/mbabineau/loggly-python
Or install it from PyPI:
sudo easy_install loggly
This package includes scripts for managing inputs and devices. To use them, simply set up your credentials:
export LOGGLY_USERNAME='someuser'
export LOGGLY_PASSWORD='somepassword'
export LOGGLY_DOMAIN='somesubdomain.loggly.com'
Create an input:
$ loggly-create-input -i testinput -s syslogtcp
Creating input "testinput" of type "syslogtcp"
Input:testinput2
Add a device to an input:
$ loggly-add-device -i testinput -d 192.168.1.1
Adding device "192.168.1.1" to input "testinput"
Device:192.168.1.1
Delete a device:
$ loggly-remove-device -d 192.168.1.1
Removing device "192.168.1.1" from all inputs
Delete an input:
$ loggly-delete-input -i testinput
Deleting input testinput
Enjoy!