Matt Whiteley

IAM Policy For Kitchen EC2

2014-10-25T00:00:00+00:00

IAM policies can be tricky but that is certainly not a reason to use your AWS root account credentials all over the place. In the test-kitchen-berks-api example I created a new user and granted it permissions for the kitchen-ec2 tests. This requires being much more explicit than authorizing ec2:RunInstances. The user needs to be able to tag the running instances and to prevent complications we limit the instance termination permissions.

To create a working policy, you'll want to use the iam-policy-generator and the iam-policy-simulator. The permissions required for this test are very well explained on the AWS Security Blog.

iam-usercreate -u test-kitchen-berks-api -k
iam-useruploadpolicy -f iam.json -p test-kitchen-berks-api-instances -u test-kitchen-berks-api

It should be noted that the restrictions in this policy are not secure in that they allow the tagging of all resources in the account and do not limit tags to instances created by this user. To force a tag onto a newly booted instance it would need to be applied to the AMI.

Test Kitchen on Travis CI

2014-10-23T00:00:00+00:00

The world of Chef cookbook development and testing has improved a ton in the last year or two. We can be far more confident in our configuration quality since we run them via Test Kitchen using our internal custom AMIs.

At this point, Berkshelf can't directly index cookbooks hosted in chef-server and a Berkshelf API server is required. We had need for multiple configurations during development and I wanted to save every engineer on the team from running their own API server locally.

We use Travis CI to test our cookbooks and why not have the server running right where the information is needed. To simplify this example, I replaced hosted chef in the config.json with the Supermarket configuration but this would work with any of the supported endpoints.

ENV['BERKSHELF_API_PATH'] is set to the root of the repository to override the default of ~/.berkshelf/api-server
Travis encrypted environment variables are provided for aws_access_key_id and aws_secret_access_key
The SSH private key is added via travis encrypt-file
Cache for APT and Bundler are both enabled to speed up test runs
Package containing libarchive.so installed for berks-api dependency
AWS resource tags are applied to the EC2 instances to help identify them

After figuring out the Bundler cache hiccups, I ran into an issue that one of my coworkers had reported earlier this year. As pointed out there, uploading compressed tarballs or switching to sftp may be the best course of action but for now removing the empty directories results in a massive performance improvement.

The trickiest part of this test was getting the IAM configuration correct, so I'll explain that in another post.

Travis Makes a Better Launchpad

2014-08-24T00:00:00+00:00

At work we have a love/hate relationship with Launchpad ... we can turn Debian source packages into APT repositories but only if we jump through the hoops just right. Discussing the many limitations of Launchpad is a black hole, so I just want to share an experiment I completed recently. Most Debian packaging tutorials start with the GNU Hello program, so I used that for consistency.

I first started with a list of requirements:

Store packaging files in git
Build packages for multiple Ubuntu releases
Create APT repositories

We use (and love) Travis CI a ton and it seemed like the perfect place to iterate on packages. With the typical build environment already available, the only extra tools I used were:

deb-s3 for creating and managing APT repositories in the cloud without rsyncing local copies of everything
pbuilder to isolate the package building environment

The contents of the hello/debian directory will be straightforward to anyone familiar with Debian packaging and all of the interesting bits are contained in the .travis.yml file.

The resulting repository is hosted on S3 and can be added to your system with the following APT source:

deb https://s3.amazonaws.com/s3pa-mwhiteley-hello precise main

Next steps:

Use sbuild or similar to solve the multiple distributions issue
Inject the APT repository into the build sources so that dependent packages can be created
Sign the packages and repository catalogs
Create a community index for GitHub repositories and S3 buckets

Docker on i386

2013-08-31T00:00:00+00:00

I have had a much outdated but heroically stable Dell PowerEdge 2550 for many years now. It's great to have a place to work on personal development projects without dealing with hourly costs etc, but sometimes the lack of performance can really be an issue. A lot of the work I've done in the past few years has been centered around virtualization and so the machine has collected more dust than it once did. I've been doing some work with Docker recently and got excited that I could deploy some of my projects on this machine again. Somehow, I hadn't ever noticed that Docker is only supported on amd64 at this point.

"focusing on amd64 for now"
"it should work fine on 32-bit"
"you just need new images and a 32-bit docker binary"

After some more searching I found that it wasn't incompatible but only unsupported. With comments like these, I had a new yak to shave.

The server started with a fully updated Ubuntu 13.04 install. I originally tried to use the available Go deb packages but they were too old (1.0.2) for Docker which requires at least 1.1. Next, I tried the binary distribution but that was compiled with SSE2 optimizations enabled. I would have to build Go from source.

Building Go doesn't require a lot of prerequisites, but we need a compiler and mercurial to clone the sources.

$ sudo apt-get -y install build-essential mercurial

Not only is this server 32-bit, but it's old enough to lack support for SSE2.

$ export GO386=387

Clone the Go sources.

$ hg clone -u release https://code.google.com/p/go /home/mwhiteley/p/go
...

Build Go and add it to our path.

$ pushd /home/mwhiteley/p/go/src
$ ./all.bash
...
$ popd
$ export GOPATH=/home/mwhiteley/go
$ export PATH=$GOPATH/bin:$PATH:/home/mwhiteley/p/go/bin
$ go version
go version go1.1.2 linux/386

For Docker, we need a few more prerequisites such as the Linux Containers userspace tools and a kernel module for the Advanced Multi Layered Unification Filesystem.

$ sudo apt-get -y install git linux-image-extra-$(uname -r) lxc xz-utils

Clone the Docker sources.

$ git clone dotcloud/docker $GOPATH/src/github.com/dotcloud/docker
...

Enable and build a 32-bit Docker.

$ pushd $GOPATH/src/github.com/dotcloud/docker
$ curl https://gist.github.com/whiteley/6400552/raw/server.go.diff | git apply -
$ go get -v github.com/dotcloud/docker/...
$ go install -v github.com/dotcloud/docker/...
$ popd
$ docker version
Go version (client): go1.1.2
Go version (server): go1.1.2
Last stable version: 0.6.1

A base image is just the contents of a minimal file system and we can create one with debootstrap. The /etc/apt/sources.list file will not contain anything except the single install repository.

$ sudo debootstrap raring /tmp/rootfs
$ for d in raring raring-security raring-updates raring-backports
for> do echo "deb http://archive.ubuntu.com/ubuntu ${d} main universe multiverse"
for> done | sudo tee /tmp/rootfs/etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu raring main universe multiverse
deb http://archive.ubuntu.com/ubuntu raring-security main universe multiverse
deb http://archive.ubuntu.com/ubuntu raring-updates main universe multiverse
deb http://archive.ubuntu.com/ubuntu raring-backports main universe multiverse
$ sudo tar czf /home/mwhiteley/p/raring_base32_rootfs.tgz -C /tmp/rootfs .

We can now create a docker group and start the daemon. The group members will have access to connect directly to the daemon running as root and potentially gain root access on the host. Make sure to read the instructions carefully and only do this on a development machine.

$ sudo addgroup docker
Adding group `docker' (GID 1001) ...
Done.
$ sudo addgroup mwhiteley docker
Adding user `mwhiteley' to group `docker' ...
Adding user mwhiteley to group docker
Done.
$ sudo docker -d
Loading containers: done.
2013/08/31 17:25:41 WARNING: Your kernel does not support cgroup swap limit.
2013/08/31 17:25:41 Listening for HTTP on /var/run/docker.sock (unix)

Now, in another terminal (a new login shell so your group membership takes effect), we can import the base image.

$ cat /home/mwhiteley/raring_base32_rootfs.tgz | docker import - mwhiteley/base32
c755b018548a
$ docker images
REPOSITORY          TAG                 ID                  CREATED             SIZE
mwhiteley/base32    latest              c755b018548a        21 minutes ago      187.4 MB (virtual 187.4 MB)

Run a simple test to make sure that everything seems operational.

$ docker run c755b018548a env
HOME=/
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=lxc
HOSTNAME=d60803fb36fe

Create a Dockerfile and build an image

$ cat <<'EOP' >/home/mwhiteley/p/docker/chef/Dockerfile
heredoc> FROM c755b018548a
heredoc> MAINTAINER Matt Whiteley <mattwhiteley@gmail.com>
heredoc> RUN apt-get update
heredoc> RUN apt-get -y install curl git ruby1.9.3
heredoc> RUN curl -L https://www.opscode.com/chef/install.sh | bash
heredoc> EOP
$ docker build /home/mwhiteley/p/docker/chef
 ...
 ---> b8d729360f11
Successfully built b8d729360f11
$ docker images
REPOSITORY          TAG                 ID                  CREATED             SIZE
<none>              <none>              b8d729360f11        19 seconds ago      88.5 MB (virtual 710.8 MB)
mwhiteley/base32    latest              c755b018548a        35 minutes ago      187.4 MB (virtual 187.4 MB)

Wow, look at the size of that thing! Evidently the ruby1.9.3 package in Ubuntu pulls in a ton of dependencies that we won't be needing and a different route for Ruby will save us a ton of space. For now, let's just make sure it all worked.

$ docker run b8d729360f11 chef-solo --version
Chef: 11.6.0

Awesome, Docker running on my 866MHz Pentium III! Now, on to make some better images for test environments.

Packer Definition for Ubuntu Raring on VMware Fusion

2013-07-26T00:00:00+00:00

I needed a virtual machine on my laptop to use with Oz for offline testing. Since Oz uses kvm to install the guest image, I would need nested hypervisor support. Virtualbox doesn't provide this, but VMware Fusion does.

Another requirement for any development testing cycle using virtual machines is Vagrant and so the first thing I needed was the Vagrant VMware Provider. Step 2 was setting up a base box with the required environment.

Hashicorp, the people behind vagrant, have a new product out called Packer that I used to create the base box. Packer could actually be used to replace Oz in a lot of situations, but this is for a larger project that relies on a number of Oz specifics. You can grab my packer-raring-fusion definition which sets up a small image with:

Installs VMware tools with some needed patches for 3.8 kernels
Installs Vagrant insecure public key
Downloads Packer and unzips in bin/
Enables VHV for nested hypervisors on the guest
Sets Vagrant User password to 'packer'

or the packaged base box (650MB).

My Adventures with Jekyll

2013-07-23T13:45:32+00:00

I've decided to give Jekyll a try for posting some of my snippets of code, thoughts, and other work. This post is just here to get me started.

A random trick I pulled off today, that I'm sure to forget:

$ brew list emacs | xargs \
    pax -w -z -s ,$(brew --prefix),, \
    -f ~/Downloads/emacs-$(stat -f '%Sc' -t '%Y%m%d')-brewed.tgz

This created an archive of the Homebrew installed emacs package and I could then try to install it fresh from the git repo but restore the working version if it failed.

I'm sure that I will need this link to the Jekyll docs in the near future and maybe at some point, I'll even have a commit in Jekyll's GitHub repo.

Crawl EC2 meta-data and output JSON

2013-02-23T00:00:00+00:00

Last week, while debugging an issue with some EC2 instances at work, I wanted to quickly analyze and compare some of the instance meta-data. Our typical interaction with this is in Chef as learned by Ohai, but I wanted something standalone. I keep wanting to play with Go more, and this seemed like a fun Sunday morning hack (translation: homework avoidance).

# setup Go
wget -P /tmp http://go.googlecode.com/files/go1.0.3.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf /tmp/go1.0.3.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

# grab the source file
wget https://gist.github.com/whiteley/4751059/raw/md.go

# print meta-data
go run md.go
{"ami-id":"ami-aa941e9a","ami-launch-index":"0","ami-manifest-path":"ubuntu-us-west-2/images/ubuntu-precise-12.04-amd64-server-20130204.manifest.xml","block-device-mapping":{"ami":"sda1","ephemeral0":"sda2","root":"/dev/sda1","swap":"sda3"}, …

# pretty print meta-data
go run md.go --prettyprint
{
  "ami-id": "ami-aa941e9a",
  "ami-launch-index": "0",
  "ami-manifest-path": "ubuntu-us-west-2/images/ubuntu-precise-12.04-amd64-server-20130204.manifest.xml",
  "block-device-mapping": {
    "ami": "sda1",
    "ephemeral0": "sda2",
    "root": "/dev/sda1",
    "swap": "sda3"
  },
  …
}

Go has a number of exciting features, but the most interesting in this experiment is possibly the compilation to an executable binary. I can now quickly grab this file on a running instance and inspect the meta-data easily without installing Go or anything else.

# grab the binary
wget http://go-mwhiteley.s3.amazonaws.com/md && chmod 0755 md
# run
./md

Next up, I plan to investigate Rust.

EC2 Launching

2012-12-13T00:00:00+00:00

Everyday I launch a lot of EC2 instances as part of a controlled process:

deploying an application in the Engine Yard Cloud Dashboard
using the EC2 plugin for the knife cli tool for Opscode Chef

I also wind up running a lot of ad-hoc instances for quick experiments. This required a frustrating multi-step dance:

ec2-run-instances
highlight and copy the Instance ID
wait long enough that networking is provisioned
ec2-describe-instances with the pasted id
highlight and copy the Public DNS attribute
wait long enough that sshd will be available before the connection times out
ssh -oForwardAgent=yes -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -i ~/.ec2/keypair.pem with the pasted hostname

Last week I got to the breaking point during one of these cycles and hacked together a script to launch and log into these one-off instances. The helpful tcp_test_ssh and wait_for_sshd methods are grabbed mostly verbatim from the previously mentioned knife-ec2 plugin.

Checking Upstart configuration syntax

2012-12-11T00:00:00+00:00

There is a handy tool init-checkconf you can use to validate your freshly written Upstart configs. If you're trying to use this on an Ubuntu server install or similar environment, you might have run into this issue:

The problem is that init-checkconf is assuming you have a dbus-daemon running and the related environment variables set to find it. A graphical login session generally will start a dbus session for the user, but on a server install you are most likely running without one.

I wrote a quick script for a coworker that wraps init-checkconf with a temporary dbus session. It might come in handy for someone else as well.

Ubuntu Cloud AMI ID Finder

2012-12-07T00:00:00+00:00

Recently, I've found myself loading up the Ubuntu Cloud Images site way too frequently just to copy and paste an up to date AMI id.

I hacked together a script to grab the json source, filter it on some provided attributes and output a table in your terminal. My buddy JD helped out with some refactoring and it's a much more pleasant way to find that AMI id you need.

$ ubuntu-cloud-images suite:precise region:us-west-2 type:ebs
+-----------+---------+-------+------+------------+--------------+---------------------------------------------+
| region    | suite   | arch  | type | date       | ami          | launch                                      |
+-----------+---------+-------+------+------------+--------------+---------------------------------------------+
| us-west-2 | precise | amd64 | ebs  | 20121026.1 | ami-7eab224e | ec2-run-instances -k mwhiteley ami-7eab224e |
| us-west-2 | precise | i386  | ebs  | 20121026.1 | ami-7cab224c | ec2-run-instances -k mwhiteley ami-7cab224c |
+-----------+---------+-------+------+------------+--------------+---------------------------------------------+

Next up, I'll post my script that I use to run instances that waits for ssh access and drops you into a shell.