Connecting multiple VPC’s using OpenSwan IPSec VPN

Getting a VPN working to connect multiple regions is actually pretty simple. I’ll lay out the step’s I used to get this working and hopefully it helps someone else.

  • Setup security groups (Note: You can, and probably should, set the cidr / source address to be the other instances elastic IP only. This would probably be the more secure thing to do, but for demonstrative purposes I didn’t)
1
2
3
4
5
6
ec2-create-group ipsec_east -d "Ipsec East" -c <vpc_id> --region us-east-1
ec2-create-group ipsec_west -d "Ipsec West" -c <vpc_id> --region us-west-2
ec2-authorize ipsec_east --protocol udp -p 4500 --region us-east-1 --cidr 0.0.0.0/0
ec2-authorize ipsec_east --protocol udp -p 500 --region us-east-1 --cidr 0.0.0.0/0
ec2-authorize ipsec_west --protocol udp -p 4500 --region us-west-2 --cidr 0.0.0.0/0
ec2-authorize ipsec_west --protocol udp -p 500 --region us-west-2 --cidr 0.0.0.0/0
  • Deploy 2 small instances. I used two CentOS 6.4 AMI’s. Use the security groups from step 1, and in my example I am deploying one in east and one in west-2:
1
2
ec2-run-instances ami-bf5021d6 -t t1.small -s subnet-xxxxxxxx -k my-key-pair -g sg-xxxxxxxx --region us-east-1
ec2-run-instances ami-b3bf2f83 -t t1.small -s subnet-xxxxxxxx -k my-key-pair -g sg-xxxxxxxx --region us-west-1
  • Allocate two Elastic IP’s, one in east and one in west
1
2
ec2-allocate-address -d vpc --region region us-east-1
ec2-allocate-address -d vpc --region region us-west-2
  • Assign elastic IP’s from step 3 to the instances created in step 1.
1
2
ec2-associate-address -i <instance_id> --region us-east-1
ec2-associate-address -i <instance_id> --region us-west-2
  • Turn off source destination checks for each instance
1
2
ec2-modify-instance-attribute <instance_id> --source-dest-check false --region us-east-1
ec2-modify-instance-attribute <instance_id> --source-dest-check false --region us-west-2
  • Edit /etc/sysctl.conf and comment out, or add, the following lines:
1
2
3
4
5
6
7
net.ipv4.ip_forward=1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.eth0.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
  • Enable sysctl changes from the previous step sudo sysctl -p /etc/sysctl.conf

  • install openswan yum -y install openswan

  • Uncoment ipsec.d setting inside /etc/ipsec.conf sed -i -e "s_#include /etc/_include /etc/_" /etc/ipsec.conf

So for the remainder of this example we are going to esablish some “facts” to help.

1
2
3
4
5
6
7
8
9
# Instance West:
west_elastic_ip: 11.22.33.44
west_private_ip: 10.10.0.10
west_vpc_subnet_cidr: 10.10.0.0/16

# Instance East:
east_elastic_ip: 55.66.77.88
east_private_ip: 10.1.0.10
east_vpc_subnet_cidr: 10.1.0.0/16
  • So using these values we’ll go ahead and create the tunnel configs on each instance:

west-east.conf

/etc/ipsec.d/west-east.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
conn west-east
   authby=secret
   auto=start
   type=tunnel
   left=10.10.0.10
   leftid=11.22.33.44
   leftsubnet=10.10.0.0/16
   right=55.66.77.88
   rightsubnet=10.1.0.0/16
   ike=aes256-sha1;modp2048
   dpddelay=30
   dpdtimeout=120
   dpdaction=restart

east-west.conf

/etc/ipsec.d/east-west.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
conn east-west
   authby=secret
   auto=start
   type=tunnel
   left=10.1.0.10
   leftid=55.66.77.88
   leftsubnet=10.1.0.0/16
   right=11.22.33.44
   rightsubnet=10.10.0.0/16
   ike=aes256-sha1;modp2048
   dpddelay=30
   dpdtimeout=120
   dpdaction=restart
  • Next we’ll want to go ahead and setup the shared secret between the two instances.

    • West

      echo '11.22.33.44 44.55.66.77: PSK "super_secure_key"' > /etc/ipsec.secrets

    • East

      echo '44.55.66.77 11.22.33.44: PSK "super_secure_key"' > /etc/ipsec.secrets

  • Verify config, and make sure there are no FAILED items

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@vpn1 ~]# sudo ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                               [OK]
Linux Openswan U2.6.32/K2.6.32-358.el6.x86_64 (netkey)
Checking for IPsec support in kernel                          [OK]
 SAref kernel support                                         [N/A]
 NETKEY:  Testing for disabled ICMP send_redirects            [OK]
NETKEY detected, testing for disabled ICMP accept_redirects   [OK]
Checking that pluto is running                                [OK]
 Pluto listening for IKE on udp 500                           [OK]
 Pluto listening for NAT-T on udp 4500                        [OK]
Two or more interfaces found, checking IP forwarding          [OK]
Checking NAT and MASQUERADEing                                [OK]
Checking for 'ip' command                                     [OK]
Checking /bin/sh is not /bin/dash                             [OK]
Checking for 'iptables' command                               [OK]
Opportunistic Encryption Support                              [DISABLED]
[root@vpn1 ~]#
  • chkconfig and start services on both servers
1
2
3
4
sudo service ipsec status
sudo chkconfig ipsec on
sudo service ipsec stop
sudo service ipsec start
  • Setup routes in each VPC to route traffic over the VPN instance. Some illustrations to help with this part can be found here
1
2
ec2-create-route <route_table_id> -r 10.10.0.0/16  -i <west_instance_id> --region us-west-2
ec2-create-route <route_table_id> -r 10.1.0.0/16 -i <east_instance_id> --region us-east-1
  • Profit

At this point you should be able to connect to an instance in west and ssh to an instance in east. Make sure your security groups in east have SSH open for the new origin IP, which is 10.10.0.10/32 in this example.

Additional References:


Showing restarts in Graphite using uptime

Currently we use CollectD’s uptime plugin to track system uptime. We then use Curl-Json along with Jolokia to get key metrics from our Java application server. One of these metrics is the JVM uptime, which, which we’ll discuss in this post. So here are the facts.

  • The JVM uptime is reported in milliseconds
  • System uptime is in seconds.
  • Collectd’s reporting interval is set to 10 seconds

So in this contrived example I am going to plot memory used over a 7 day period on one of our test systems. I then wanted to show a vertical bar for any time the JVM or system is restarted. This way I can correlate these with any sudden changes I might see in memory utilization.

So lets build the request we would use to do this via Graphite’s Render API

Basics

1
https://graphite.mydomain.com/render/?width=500&height=600&from=-7days&until=now&areaMode=all&areaAlpha=0.7

First we want to set the width, height, and window dimensions and areaMode for our graph. All of these are well documented and shouldn’t require much explanation.

Java process uptime

1
2
3
4
5
6
7
8
9
10
11
alias(
  drawAsInfinite(
    transformNull(
      removeAboveValue(
        test100.*.collectd.curl_json-javaVM.frequency-java_lang:type=Runtime-Uptime,
        100000
      )
    )
  ),
  'jvm%20restart'
)

So here we need to count any time the RMI returns a value less then 100 seconds as a restart, since I feel that this can never actually happen unless the counter were reset due to a restart of the JVM. We used a value of 100000 since the JVM uptime is reported in milliseconds.

System uptime

1
2
3
4
5
6
7
8
9
10
11
alias(
  drawAsInfinite(
    transformNull(
      removeAboveValue(
        test100*.collectd.uptime.uptime,
        100
      )
    )
  ),
  'system%20restart'
)

The uptime plug-in returns the system uptime in seconds, and because this starts accumulating collectd starts I found a value of 100 seconds to work great. Be warned your millage may vary, so you might want to try a higher value if you are not seeing anything at first.

System used memory

1
2
3
4
5
6
7
color(
  alias(
    test100*.collectd.memory.memory-used,
    'used'
  ),
  'orange'
)

Here we are just using CollectD’s memory plug-in to track used memory. We then change the color because I personally found the orange graph a little easier to read.

End result

Hope someone found this helpful :)


Ohai, Vagrant and eth1… Making them work in harmony

Unfortunately Vagrant deployments always have the NAT adapter on eth0, which can cause headaches at times, especially when doing a multiVM deployment! Most of the time you end up having to modify the recipe to make things work. In my humble opinion these changes kind of defeat the purpose of testing in Vagrant to begin with. So what to do ? We’ll you can use an Ohai plugin and override the node.ipaddress value ! So what we’ll do is make eth1 pretend to be eth0 !

Now since I want to make sure my cookbooks are the same in development as production I will want to make sure we don’t use this plugin in production. That being the case we’ll go ahead and add some guards to ensure this is only loaded when we are on a VirtualBox VM. Just for good measure we’ll also check for the vagrant user, and make sure that there is in fact an eth1 adapter. I feel this is enough of a safe guard to ensure we only apply this plugin if running under Vagrant, and only when needed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
provide "ipaddress"
require_plugin "#{os}::network"
require_plugin "#{os}::virtualization"
require_plugin "passwd"

if virtualization["system"] == "vbox"
  if etc["passwd"].any? { |k,v| k == "vagrant"}
    if network["interfaces"]["eth1"]
      network["interfaces"]["eth1"]["addresses"].each do |ip, params|
        if params['family'] == ('inet')
          ipaddress ip
        end
      end
    end
  end
end

We can now distribute this w/ the Ohai Plugin Cookbook and not have to make any changes regardless of the environment ( Production / Development ).


Creating a release RPM

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
#
# spec file for VoxeoLabs Yum Repo
#
Summary: Voxeolabs repository configuration
Name: voxeolabs
Version:        0.0.1
Release:        1%{?dist}
License:        GPLv3
Group: System Environment/Base

URL:            http://yum.voxeolabs.net/
Source0:        http://yum.voxeolabs.net.s3.amazonaws.com/pub/voxeolabs-0.0.1.el.tgz

BuildRoot: %{_tmpdir}/%{name}-%{version}-%{release}

Vendor: Voxeo Labs
Packager: John Dyer <jdyer@voxeo.com>

%description
This package contains the Voxeo Labs yum repo.  It includes the  GPG key as well as configuration for yum.

%prep
%setup -q

%build

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d/
mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/pki/rpm-gpg/
cp -p voxeo-labs.repo $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d/
cp -p RPM-GPG-KEY-voxeo-labs $RPM_BUILD_ROOT%{_sysconfdir}/pki/rpm-gpg/

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
/etc/yum.repos.d/voxeo-labs.repo
/etc/pki/rpm-gpg/RPM-GPG-KEY-voxeo-labs

%post

# Retrieve Platform and Platform Version

if [ -f "/etc/redhat-release" ];
then
  platform=$(sed 's/^\(.\+\) release.*/\1/' /etc/redhat-release | tr '[A-Z]' '[a-z]')
  platform_version=$(sed 's/^.\+ release \([.0-9]\+\).*/\1/' /etc/redhat-release)

  # If /etc/redhat-release exists, we act like RHEL by default
  if [ "$platform" = "fedora" ];
  then
    # Change platform version for use below.
    platform_version="6.0"
  fi
  platform="el"
elif [ -f "/etc/system-release" ];
then
  platform=$(sed 's/^\(.\+\) release.\+/\1/' /etc/system-release | tr '[A-Z]' '[a-z]')
  platform_version=$(sed 's/^.\+ release \([.0-9]\+\).*/\1/' /etc/system-release | tr '[A-Z]' '[a-z]')
  # amazon is built off of fedora, so act like RHEL
  if [ "$platform" = "amazon linux ami" ];
  then
    platform="el"
    platform_version="6.0"
  fi
fi

sed -i -e "s/\$releasever/`echo $platform_version | cut -d. -f1`/g" /etc/yum.repos.d/voxeo-labs.repo
yum clean all

%changelog
* Sun Aug 27 2012 John Dyer <jdyer@voxeo.com> 0.0.1
- Initial release

Using SOX to batch convert audio files

The Plain old telephone service (POTS) we are all used to is almost a hundred years old so when it comes to audio codecs you get pretty much the bottom of the barrel. When doing IVR you will want to make sure you convert all your media to the lowest common denominator, which is generally 8bit, 8Khz Mono. Doing this w/ SOX is pretty simple

1
2
3
4
for i in *.wav
do
 sox $i -r 8000 -U -b 8 new-files/$(basename $i)
done

Thats pretty much it ! If you are using OSX you can get SOX by installing Homebrew and running:

1
brew install sox

Hope that helps someone

-John


Using Nagios to monitor Tropo

So I will start by saying that Tropo maintains a very redundant, and fault tollerent network, however we do understand that people still need to do their due dilligance and implement monitoring just to be safe. So yesterday we had a developer that wanted do just this, they wanted to monitor Tropo’s SBC’s to ensure they were responding to requests as expected. The only problem here is that Tropo’s SBC’s don’t implelment OPTIONS and all the plugins for Nagios seem to use OPTIONS in their tests, however one seems to allow you to specify the response code for a passing test. This plugin is the NagiosSIP plugin, and via the -c ( SIP_CODE) flag we can set a 501 ( SIP/501 Not Implemented) as a passing test, since a 501 would signify a responding SBC. Check out below for an example of just what I mean:

1
2
10:51:28 jdyer@aleph.local ./nagios_sip_plugin.rb -s sip.tropo.com -c 501
OK:status code = 501

We can also see the exit code is 0, which would be a pass

1
2
10:51:33 jdyer@aleph.local echo $?
0

Sure hope someone finds this helpful!

-John


Installing Asterisk 10.3 on OSX Lion

So I had a need to install Asterisk the other day on my laptop and doing the atypical:

1
2
3
./configure
make
make install

Gave me a whole bunch of suck

1
2
3
4
5
6
7
8
snmp/agent.c: In function ‘init_asterisk_mib’:
snmp/agent.c:835: error: ‘RONLY’ undeclared (first use in this function)
snmp/agent.c:835: error: (Each undeclared identifier is reported only once
snmp/agent.c:835: error: for each function it appears in.)
make[1]: *** [snmp/agent.o] Error 1
make[1]: *** Waiting for unfinished jobs....
   [LD] chan_unistim.o -> chan_unistim.so
make: *** [res] Error 2

Little bit of digging led to a bunch of solution thats didn’t seem to work, but after a little bit of trail and error I found a quick, and working solution, which I figured I would share

1
2
3
4
5
./configure --host=x86_64-darwin
make menuselect ( remove res_snmp under Resource Modules )
sudo make -j 4
sudo make install
sudo make samples

Worked for me, and I hope it helps someone else!

-John


API for AWS pricing

Looking for an easy way to get AWS pricing programitcaially? Just use the following three URI’s to get a JSON feed of transfer costs, as well as Reserved and On-Demand instance pricing:

1
2
3
https://aws.amazon.com/ec2/pricing/pricing-on-demand-instances.json
https://aws.amazon.com/ec2/pricing/pricing-reserved-instances.json
https://aws.amazon.com/ec2/pricing/pricing-data-transfer.json

Enjoy

-John



Installing ELB command line tools on OSX / Linux

The ELB command line tools are pretty nice, and they are actually required to get a lot of important information which is unfortunately missing from the AWS ELB UI. Installation is actually quite simple, first go ahead and download the zip from here, and then just go ahead and do the following:

1
2
3
4
wget http://ec2-downloads.s3.amazonaws.com/ElasticLoadBalancing.zip
unzip ElasticLoadBalancing.zip -d ~/bin
echo "EXPORT ~/bin/`ls ~/bin | grep Elastic`" >> ~/.bash_profile
source ~/.bash_profile

Then boom, magic

1
2
3
4
5
6
7
8
10:50:24 jdyer@aleph.local Downloads elb-describe-lbs -h                                                                                                                                                                    127 ↵

SYNOPSIS
   elb-describe-lbs
       [LoadBalancerNames [LoadBalancerNames ...] ]  [General Options]

DESCRIPTION
       Describes the state and properties of LoadBalancers

Hope that helps!


Print letters up to n in Ruby

Had a case the other day where I needed to create users accounts in a specific fashion. If I had less then 25 users I would use letters for the test accounts (eg: usera, userb, ect), if there were more then 25 I would use numbered accounts (eg: user1, user2, ect ) The problem was there could be any number of user accounts and I wanted to keep to this schema.

Unfortunately Enumerable wasn’t going to make this as easy as most things are in ruby, like this wasn’t going to work:

1
"a".upto(5)

However I figured I could just switch to their ascii equivalent and then this would be easy!

1
2
3
4
5
6
num_of_users = 25
if num_of_users <= 25
  (97..97+num_of_users).each{|e| puts e.chr}
else
  num_of_users.times{|e| puts e}
end

Worked like a charm!


Upgrade Chef server to 0.10.8

Upgrading to Chef Server ( 0.10.8 ) is dead simple, takes probably 30 seconds.

1
2
3
for s in server server-webui solr expander; do sudo /etc/init.d/chef-${s} stop; done
sudo gem update chef chef-server --no-ri --no-rdoc
for s in server server-webui solr expander; do sudo /etc/init.d/chef-${s} start; done

If you want you can backup the server first using Seth’s backup script ( here ). Installing, and running it, is dead simple:

1
2
curl -O https://raw.github.com/jtimberman/knife-scripts/master/chef_server_backup.rb
knife exec chef_server_backup.rb

This will create a backup fo the server in your .chef dir. Enjoy


Dynamic templates based on search results

So here is my use case, I have a two nodes which have to be configured to know about each other, but I can’t use multicast, and I want to be elastic, in that I can add a node and the others configs get updated. So to start we are going to have two nodes, gateway1 and gateway2, and each has a config file called sipmethod.xml which contains a node of XML data that defines the cluster.

1
2
3
4
5
 <Cluster>
   <Node name="gateway2" local="10.72.111.116:47520">
      <Peer remote="10.110.234.150:47520"/>
   </Node>
 </Cluster>

and here is my template that creates this section (sorta):

1
2
3
4
5
6
7
8
9
<% if node[:prism][:sipmethod][:cluster] %>
 <Cluster>
   <Node name="<%=node[:hostname]%>" local="<%=node[:ipaddress]%>:<%= node[:prism][:sipenv][:rmi_port] %>">
     <% search(:node, "role:rayo_gateway AND chef_environment:#{node.chef_environment}").each do |peer| %>
      <Peer remote="<%= (peer.ec2 ? peer.ec2.local_ipv4 : peer.ipaddress) %>:<%= node[:prism][:sipenv][:rmi_port] %>"/>
     <%end%>
   </Node>
 </Cluster>
 <%end%>

Now there is one problem with this approach, if you look carefully the peers are going to include itself, which is not going to work.

1
2
3
4
5
6
 <Cluster>
   <Node name="gateway2" local="10.72.111.116:47520">
      <Peer remote="10.110.234.150:47520"/>
      <Peer remote="10.72.111.116:47520"/>  <!-- PROBLEM -->
   </Node>
 </Cluster>

I need to not include the node running the search in the results, and this is actually pretty simple:

We are going to take this search and simply add a NOT clause:

1
 search(:node, "role:rayo_gateway AND chef_environment:#{node.chef_environment} NOT name:#{node.name}")

This will result in returning every gateway in my environment except the one running the search, and the end result is the dynamically configured cluster config:

1
2
3
4
5
 <Cluster>
   <Node name="gateway2" local="10.72.111.116:47520">
      <Peer remote="10.110.234.150:47520"/>
   </Node>
 </Cluster>

Query Chef’s RestAPI using Spice

Dan Ryan has an awesome gem on Github called Spice , which is a nice wrapper to the Chef Severs RestAPI. The one thing I didn’t find clearly documented was how to do a search based on things such as environment, roles, and run lists. However after a bit of digging I was able to get this working and I felt obliged to share w/ the class. The first thing you need to do is setup the Spice connection. This part of pretty straight forward and of course well documented:

1
2
3
4
5
  Spice.setup do |s|
    s.server_url  = "http://chef.server.net"
    s.client_name = "client-key"
    s.key_file    = "#{File.dirname(__FILE__)}/keys/client-key.pem"
  end

Then you perform the query using Spice.connection.get

1
  results = Spice.connection.get('/search/node', :params => {:q => "role:base AND role:rayo_gateway AND chef_environment:ec2_rayo_functional_tests"})["rows"]

This will return a collection that you can easily iterate over to get the ‘goods’

1
2
3
4
5
6
7
8
9
10
11
12
13
14
require 'spice'

Spice.setup do |s|
  s.server_url  = "http://chef.server.net"
  s.client_name = "client-key"
  s.key_file    = "#{File.dirname(__FILE__)}/keys/client-key.pem"
end


results = Spice.connection.get('/search/node', :params => {:q => "role:base AND role:rayo_gateway AND chef_environment:ec2_rayo_functional_tests"})["rows"]

results.each do |node|
  puts node["automatic"]["ec2"]["public_ipv4"]
end

` The great thing is this not only contains the node’s attributes and run lists but it also contains all the Ohai data for each node. This is some powerful stuff, hope you enjoy!