Mike's head

occasional ramblings on technology, travel, food, and frivolity
About Me

Cloud wrangler

http://mikebabineau.com

My Other Sites
TwitterFacebookLinkedIn
Tags
Archive
2011 (7)
2010 (7)
2009 (11)
 

Chef Cookbook for Loggly - EA2D Engineering

Released another bit of code, this time a Chef cookbook for Loggly. Go check it out.

As mentioned in a previous post, we aggregate and store our logs using a service called Loggly.

Since we wrote a library for programmatically managing Loggly inputs and devices, it was only natural for us to integrate it with our Chef deployment.

We’ve written and open sourced a Chef cookbook for Loggly.

From the README:

Installs the loggly-python library and provides a definition for the configuration of Loggly logging.

More specifically, the loggly_conf definition will configure rsyslog to watch a log file and send its lines to a Loggly input. When first run, loggly_conf will create the input and authorize the server to publish events to that input.

Developed for and tested on Ubuntu 10.10 LTS

You can grab it from the Opscode site:

$ knife cookbook site vendor loggly

Or from our GitHub repository:

https://github.com/EA2D/loggly-cookbook

via eng.ea2d.com

 

Posted
 

Continuous Deployment of Operational Configs - EA2D Engineering

Gave a talk the other night on continuously deploying Chef configs. Slides below, but I wrote a full post over on the EA2D engineering blog.

via eng.ea2d.com

 

Posted
 

pingdom-python, an Open Source Library - EA2D Engineering

Woo, I got to open source another library:

Pingdom is one of several monitoring tools we use here at EA2D. Besides alerting us when things go down, we query Pingdom’s API to include check status in our dashboards.

The old Pingdom SOAP API was unwieldy and slow. Fortunately, Pingdom released a new, JSON-ified REST API that remedied the problems of its predecessor.

We’ve written a Python library for this new API and released it as open source. For now, it supports only a subset of available resources, but the framework is there for others to be added easily.

https://github.com/EA2D/pingdom-python

pingdom-python in action

Set up a Pingdom connection:

>>> import pingdom
>>> c = pingdom.PingdomConnection(PINGDOM_USERNAME, PINGDOM_PASSWORD)  # Same credentials you use for the Pingdom website

Create a new Pingdom check:

>>> c.create_check('EA2D Website', 'ea2d.com', 'http')
Check:EA2D Website

Get basic information about a Pingdom check:

>>> check = c.get_all_checks(['EA2D Website'])[0]   # Expects a list, returns a list
>>> check.id
302632
>>> check.status
u'up'

Get more detailed information about a Pingdom check:

>>> check = c.get_check(210702)  # Look up by check ID
>>> check.lasterrortime
1289482981

Delete a Pingdom check:

>>> c.delete_check(302632)
{u'message': u'Deletion of check was successful!'}

Go check it out on GitHub, or install it directly from PyPI:

sudo easy_install pingdom

If you have any questions, drop me a line: mikeb@ea2d.com.

via eng.ea2d.com

 

Posted
 

Loggly-Python, an Open Source Library - EA2D Engineering

I started an engineering blog over at EA2D. First post is on the open source Python lib I wrote for Loggly:

For centralized logging, we use a service called Loggly. We forward our logs to Loggly and aggregate them by application and environment. This gives us a handy web interface for viewing logs across all servers within an application group, and provides some great tools for search, comparison, and alerting.

We send events to Loggly using syslog over TCP, and these events are bucketed based on destination port. For security, Loggly locks down each port to a list of authorized IP addresses.

Since we make heavy use of EC2 Auto Scaling Groups, we simply can not maintain this authorized IP list manually. Additionally, we’re constantly launching new applications and environments, so our list of buckets (Loggly “inputs”) is in constant flux.

Fortunately, Loggly has exposed a set of administration APIs for managing inputs and authorized devices. Since no library was available, we ended up writing one ourselves (in Python) and releasing it as open source.

Getting the library

You can find it on GitHub:

https://github.com/EA2D/loggly-python

Or install it from PyPI:

sudo easy_install loggly

Batteries-included

This package includes scripts for managing inputs and devices. To use them, simply set up your credentials:

export LOGGLY_USERNAME='someuser'
export LOGGLY_PASSWORD='somepassword'
export LOGGLY_DOMAIN='somesubdomain.loggly.com'

Create an input:

$ loggly-create-input -i testinput -s syslogtcp
Creating input "testinput" of type "syslogtcp"
Input:testinput2

Add a device to an input:

$ loggly-add-device -i testinput -d 192.168.1.1
Adding device "192.168.1.1" to input "testinput"
Device:192.168.1.1

Delete a device:

$ loggly-remove-device -d 192.168.1.1
Removing device "192.168.1.1" from all inputs

Delete an input:

$ loggly-delete-input -i testinput
Deleting input testinput

Enjoy!

via eng.ea2d.com

Posted
 

Going global with EC2 and GSLB

Media_http3bpblogspot_nkcak
via dev.bizo.com

I've written a post on the use of Global Server Load Balancing (GSLB) to drastically improve EC2 application performance. Check it out on Bizo's dev blog.

Filed under  //  aws   bizo   ec2   gslb  
Posted
 

Cloudgrapher, a free dashboard for Amazon CloudWatch, launched

I've launched a new service for graphing Amazon CloudWatch metrics, Cloudgrapher.  It's free, it's based off of cloudviz, and I think EC2 users, especially those with highly dynamic environments, will find it pretty handy.

For more info, check out the announcement on the Cloudgrapher blog.  Or just head on over to http://www.cloudgrapher.com/ and give it a try!  It's free and requires zero setup - what's not to like?

Filed under  //  aws   cloudgrapher   cloudviz   cloudwatch   ec2  
Posted
 

Cloudviz now has region support

I've added region support to Cloudviz, so you can now graph CloudWatch data for entities in us-west-1 and eu-west-1.  Download the latest version here.

Filed under  //  aws   cloudviz   cloudwatch  
Posted
 

Introducing Cloudviz

Media_httpmbabineaugi_tevso
via dev.bizo.com

I've written a tool to graph CloudWatch data. Read the announcement on the Bizo dev blog or check out the project on GitHub.

Filed under  //  aws   bizo   cloudviz   cloudwatch  
Posted
 

SSH to EC2 instance ID

I often find myself looking up EC2 nodes by instance ID so I can grab the external DNS name and SSH in. Fed up with the extra “ec2-describe-instance , copy, paste” layer, I threw together a function (basically a fancy alias) to SSH into an EC2 instance referenced by ID.

Assuming you’re on Mac OS X / Linux, just put this somewhere in ~/.profile, reload your terminal, and you’re good to go.  Alternatively, you can use the shell script version.

Update (3/5/10): Added region support

Filed under  //  aws   ec2   terminal  
Posted
 

Joining Bizo

I’m excited to announce that starting today, I have joined San Francisco-based startup Bizo.

Bizo was spun out of ZoomInfo a year and a half ago to provide a platform for targeted B2B advertising.  They’ve had tremendous success, reaching more than 50 million business people and counting among their customers seven of the top 10 B2B advertisers and marquee brands such as JPMorgan Chase, Aston Martin, Microsoft, and Verizon.  They recently closed a $6 million round of funding from the venerable Bessemer Venture Partners.

Bizo is hosted entirely on Amazon Web Services, and was recently named runner-up in the 2009 AWS Start-Up Challenge; their presentation at the finals is available online.

I’m joining Bizo’s engineering team as their first Operations hire.  Initially, I’ll be focusing on monitoring, security, and optimization of cloud infrastructure.  As Bizo is embracing of open source and the sharing of information, expect to see me posting results (including, hopefully, code) to their development blog.

This also means I’ll be relocating from Cincinnati to San Francisco (“le pouvre homme!”).  The move won’t be immediate -I’ll be telecommuting until my wife finishes grad school in June- but I will be making frequent trips to the Bay Area in the interim.

I’m thrilled at the opportunity to be a part of Bizo, to work with their excellent team, to remain focused on cloud computing, to continue living the startup life, and to make a new home in sunny California!

Filed under  //  bizo   career  
Posted