How to create an OpenVPN bastion machine in AWS

It is common practice to use SSH jumpboxes and security group to restrict SSH access to instances in AWS. This method works fine, but sometimes being able to access instances directly in a secure way can be very useful indeed, to achieve this you can use OpenVPN combined with a few AWS tricks to create a resilient VPN bastion server.

In this article we are going to create an OpenVPN server; when connected to this machine it will appear as if your local machine is within an AWS VPC - you will be able to appear to be within the same CIDR range as the machines in a VPC. This will allow you to do a few neat things:

  • Restrict SSH access on your machines to your VPCs CIDR, so SSH will not be pubicly accessible
  • Machines within private subnets will be accessible directly
  • R53 entries can point to machines and ELBs in private subnets so they will be accessible directly, very useful for restricting access to things like Kibana

Since we are in AWS we are going to need to ensure that the VPN server can be recreated with minimal human intervention as machines can die any time, so we will need to persist a few things outside of the server:

  • Server keys need to stay the same so we don't have to recreate client keys, these can be kept in S3
  • The config for the server will need to be kept the same, we use an ASG Launch Config for this
  • The server IP needs to be the same every time, we can use an Elastic IP for this
  • We should keep the machine in an auto scaling group so we will always have a server

Server keys and config

Let's create some keys for our server, please check OpenVPN's documentation for more information on this as I will gloss over this quickly, note that in addition to the usual server keys we are going to use a static tls key as well. These instructions are written for Ubuntu, please adjust them as needed for your OS:

# Install easy-rsa
sudo apt-get install openvpn easy-rsa
mkdir -p ~/aws-openvpn/key-store
cd  ~/aws-openvpn/key-store
# Generate pre-shared key
openvpn --genkey --secret static.key
cp -R /usr/share/easy-rsa/ ~/aws-openvpn
cd ~/aws-openvpn/easy-rsa
vi vars

Adjust KEY_COUNTRY, KEY_PROVINCE, KEY_CITY, KEY_ORG, KEY_EMAIL in vars as needed and ensure KEY_SIZE is 2048, before saving the file set KEY_DIR to the key store directory you created earlier eg. "$HOME/aws-openvpn/key-store".

Now we can generate the servers keys like so:

source vars
./build-ca
./build-dh
./build-key-server server

Enter y for all the options and do not enter a passphrase. After this you should have the following files in your key-store directory: ca.crt, server.crt, server.key, static.key and dh2048.pem. These keys will be used by both server and client (clients will only use ca.crt and static.key), these need to be kept the same even when a new VPN bastion machine is deployed, changing these keys will invalidate all clients.

With the need to preserve keys in mind let's push them all into an S3 bucket, go ahead and create one making sure that it has private access; here's a CloudFormation template that will create a bucket entitled {BucketPrefix}-vpn-tutorial-keys-bucket; S3 bucket names are unique across AWS so pick a distinct prefix here:

---
# vpn_s3_bucket.yaml

Description: S3 bucket
AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  BucketPrefix:
    Description: Prefix for S3 bucket
    Type: String

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Sub ${BucketPrefix}-vpn-tutorial-keys-bucket
      VersioningConfiguration:
        Status: "Enabled"

You can deploy this template using the AWS CLI tools like so (assuming you've saved the above template to vpn_s3_bucket.yaml):

aws cloudformation create-stack \
  --stack-name vpn-stack-s3 \
  --template-body file://vpn_s3_bucket.yaml
  --parameters \
    ParameterKey=BucketPrefix,ParameterValue="{SOME-UNIQUE-HANDLE}" \

Copy the keys and config into your bucket, so this should be ca.crt, server.crt, server.key, dh2048.pem and static.key.

Deployment

We are going to use CloudFormation to deploy our VPN bastion, however you can use any tool you like  such as Terraform, Ansible, whatever you want really - the principles remain the same.

The template below will create a VPC with subnets and all furniture needed, minus private subnets, I left these out to save on having a NAT, you will need this for a real working VPC though. For the bastion we have an Autoscaling Group with a launch config that will bootstrap an instance and set it up as a VPN server with keys, config and an elastic ip.

Here's the Cloudformation template:

---
# vpn_bastion.yaml

Description: Bastion stack
AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  ImageId:
    Description: AMI ID of the API Lists
    Type: AWS::EC2::Image::Id
  InstanceType:
    Description: EC2 instance type
    Type: String
    Default: t2.nano
    AllowedValues:
    - t2.nano
    - t2.micro
    ConstraintDescription: must be a valid EC2 instance type.
  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: must be the name of an existing EC2 KeyPair.
  S3VpnKeysBucketName:
    Description: Name of S3 Secrets Bucket
    Type: String

Resources:
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Stack
          Value: !Ref AWS::StackId

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/22
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      Tags:
        - Key: Stack
          Value: !Ref AWS::StackId

  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  PublicSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Sub ${AWS::Region}a
      CidrBlock: 10.0.0.0/24
      MapPublicIpOnLaunch: 'false'
      VpcId: !Ref VPC
      Tags:
        - Key: Stack
          Value: !Ref AWS::StackId

  PublicSubnetB:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Sub ${AWS::Region}b
      CidrBlock: 10.0.1.0/24
      MapPublicIpOnLaunch: 'false'
      VpcId: !Ref VPC
      Tags:
        - Key: Stack
          Value: !Ref AWS::StackId

  PublicSubnetC:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Sub ${AWS::Region}c
      CidrBlock: 10.0.2.0/24
      MapPublicIpOnLaunch: 'false'
      VpcId: !Ref VPC
      Tags:
        - Key: Stack
          Value: !Ref AWS::StackId

  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Stack
          Value: !Ref AWS::StackId

  Route:
    Type: AWS::EC2::Route
    DependsOn: AttachGateway
    Properties:
      RouteTableId: !Ref RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnetARouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnetA
      RouteTableId: !Ref RouteTable

  PublicSubnetBRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnetB
      RouteTableId: !Ref RouteTable

  PublicSubnetCRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnetC
      RouteTableId: !Ref RouteTable

  BastionSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable access to Bastion
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: 10.0.0.0/22
        - IpProtocol: udp
          FromPort: '1194'
          ToPort: '1194'
          CidrIp: 0.0.0.0/0

  BastionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - ec2.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"

  BastionAccessPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: BastionAccessPolicy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action:
              - ec2:AssociateAddress
              - ec2:DescribeInstances
            Effect: Allow
            Resource: "*"
          - Action:
              - s3:List*
              - s3:Get*
            Effect: Allow
            Resource:
              - !Sub arn:aws:s3:::${S3VpnKeysBucketName}/*
              - !Sub arn:aws:s3:::${S3VpnKeysBucketName}
      Roles:
        - !Ref BastionRole

  BastionInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: "/"
      Roles:
       - !Ref BastionRole

  BastionEIP:
    Type: "AWS::EC2::EIP"

  BastionLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /etc/openvpn/server.conf:
              content: !Sub |
                port 1194
                proto udp
                dev tun
                server 172.16.0.0 255.255.252.0
                push "route 10.0.0.0 255.255.252.0"
                ca /etc/openvpn/keys/ca.crt
                cert /etc/openvpn/keys/server.crt
                key /etc/openvpn/keys/server.key
                dh /etc/openvpn/keys/dh2048.pem
                tls-server
                tls-auth /etc/openvpn/keys/static.key 0
                tls-version-min 1.2
                tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
                cipher AES-256-CBC
                auth SHA512
                ifconfig-pool-persist ipp.txt
                keepalive 10 120
                ping-timer-rem
                comp-lzo
                persist-key
                persist-tun
                status openvpn-status.log
                log-append /var/log/openvpn.log
                verb 3
                max-clients 100
                user nobody
                group nogroup
              mode: "000644"
              owner: "root"
              group: "root"
    Properties:
      AssociatePublicIpAddress: 'true'
      IamInstanceProfile: !Ref BastionInstanceProfile
      ImageId: !Ref ImageId
      InstanceMonitoring: false
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SecurityGroups:
        - !Ref BastionSG
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          set -euo pipefail
          apt-get update && apt-get install -y curl python-pip ntp wget
          wget -O - https://swupdate.openvpn.net/repos/repo-public.gpg|apt-key add -
          echo "deb http://build.openvpn.net/debian/openvpn/stable xenial main" \
            > /etc/apt/sources.list.d/openvpn-aptrepo.list
          apt-get update && apt-get install -y openvpn
          pip install awscli https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
          cfn-init \
            --resource BastionLaunchConfig \
            --stack ${AWS::StackName} \
            --region ${AWS::Region}
          mkdir -p /etc/openvpn/keys
          aws s3 cp s3://${S3VpnKeysBucketName} \
            /etc/openvpn/keys \
            --recursive \
            --include "ca.crt" \
            --include "server.crt" \
            --include "server.key" \
            --include "static.key" \
            --include "dh2048.pem"
          chmod -R 0600 /etc/openvpn/keys
          echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
          sysctl -p
          iptables -t nat -A POSTROUTING -s 172.16.0.0/22 -o eth0 -j MASQUERADE
          INSTANCE_ID=$( curl -s http://169.254.169.254/latest/meta-data/instance-id )
          aws ec2 associate-address \
            --region ${AWS::Region} \
            --instance-id  $INSTANCE_ID \
            --public-ip ${BastionEIP}
          systemctl daemon-reload
          service openvpn start
          INSTANCE_ASG=$( aws ec2 describe-instances \
            --instance-id $INSTANCE_ID \
            --region ${AWS::Region} \
            --query "Reservations[0].Instances[0].Tags[?Key=='aws:cloudformation:logical-id'].Value" \
            --output text )
          set +e
          ps auxw | grep -P '\b'openvpn'(?!-)\b'
          OPENVPN_RUNNING=$?
          set -e
          cfn-signal \
            -e $OPENVPN_RUNNING \
            --stack ${AWS::StackName} \
            --resource $INSTANCE_ASG \
            --region ${AWS::Region}

  BastionGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      DesiredCapacity: 1
      LaunchConfigurationName: !Ref BastionLaunchConfig
      MaxSize: 2
      MinSize: 1
      VPCZoneIdentifier:
      - !Ref PublicSubnetA
      - !Ref PublicSubnetB
      - !Ref PublicSubnetC
    CreationPolicy:
      ResourceSignal:
        Count: 1
        Timeout: PT10M
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MaxBatchSize: 1
        MinInstancesInService: 1
        PauseTime: PT10M
        WaitOnResourceSignals: 'true'

You can use the following script to deploy this stack; you will need to have the s3 template from earlier to use this script saved as vpn_s3_bucket.yaml and the above template saved as vpn_bastion.yaml:

#!/bin/bash

# vpn_stack.sh

set -euo pipefail

function main {
    local AMI_ID
    local BUCKET_PREFIX
    local SSH_KEY_NAME
    local S3_STACK_EXISTS
    local VPN_STACK_EXISTS
    local USAGE
    local DIR

    USAGE="$(basename "$0") [BUCKET_PREFIX] [SSH_KEY_PAIR_NAME] [AMI_ID](optional)"
    DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
    BUCKET_PREFIX=${1-}
    SSH_KEY_NAME=${2-}
    AMI_ID=${3-}

    if [[ -z ${BUCKET_PREFIX} ]] || [[ -z ${SSH_KEY_NAME} ]]; then
        echo "Missing required arguments" >&2
        echo "$USAGE" >&2
        exit 1
    fi

    set +e
    aws cloudformation describe-stacks \
        --stack-name vpn-stack-s3 \
        &> /dev/null
    S3_STACK_EXISTS=$?
    aws cloudformation describe-stacks \
        --stack-name vpn-stack \
        &> /dev/null
    VPN_STACK_EXISTS=$?
    set -e

    if [[ -z "${AMI_ID}" ]]; then
      AMI_ID=$( aws ec2 describe-images \
        --owners 099720109477 \
        --filters "Name=name,Values=*ubuntu-xenial-16.04-amd64*" \
          "Name=virtualization-type,Values=hvm" \
          "Name=root-device-type,Values=ebs" \
          "Name=hypervisor,Values=xen" \
        --output text \
        --query "reverse(sort_by(Images, &CreationDate))|[].ImageId | [0]" )
  	fi

    echo "Using AMI ID: ${AMI_ID}"

    if [[ $S3_STACK_EXISTS -ne 0 ]]; then
        echo "Creating vpn s3 stack"

        aws cloudformation create-stack \
            --stack-name vpn-stack-s3 \
            --template-body file://"${DIR}"/vpn_s3_bucket.yaml \
            --parameters \
                ParameterKey=BucketPrefix,ParameterValue="${BUCKET_PREFIX}" \
            &> /dev/null

        aws cloudformation wait stack-create-complete \
            --stack-name vpn-stack-s3

        echo "Stack created"
    fi

    if [[ $VPN_STACK_EXISTS -ne 0 ]]; then
        echo "Creating vpn stack"

        aws cloudformation create-stack \
            --stack-name vpn-stack \
            --template-body file://"${DIR}"/vpn_bastion.yaml \
            --capabilities CAPABILITY_IAM \
            --parameters \
                ParameterKey=ImageId,ParameterValue="${AMI_ID}" \
                ParameterKey=KeyName,ParameterValue="${SSH_KEY_NAME}" \
                ParameterKey=S3VpnKeysBucketName,ParameterValue="${BUCKET_PREFIX}"-vpn-tutorial-keys-bucket \
            &> /dev/null

        aws cloudformation wait stack-create-complete \
            --stack-name vpn-stack

        echo "Stack created"
    else
        echo "Updating vpn stack"

        aws cloudformation update-stack \
            --stack-name vpn-stack \
            --template-body file://"${DIR}"/vpn_bastion.yaml \
            --capabilities CAPABILITY_IAM \
            --parameters \
                ParameterKey=ImageId,ParameterValue="${AMI_ID}" \
                ParameterKey=KeyName,ParameterValue="${SSH_KEY_NAME}" \
                ParameterKey=S3VpnKeysBucketName,ParameterValue="${BUCKET_PREFIX}"-vpn-tutorial-keys-bucket \
            &> /dev/null

        aws cloudformation wait stack-update-complete \
            --stack-name vpn-stack

        echo "Stack updated"
    fi
}

main "$@"

You can this script like so:

vpn_stack.sh [BUCKET-PREFIX] [EC2-SSH-KEY]

Once the machine has booted up OpenVPN should be up and running and ready to connect to, note that SSH is locked down on the instance, you need to connect via OpenVPN in order to SSH into the machine.

Diving into the details

Let's take a look at some of the key components in play within the Bastion Cloudformation template, I am not going to go through the foundational parts such as the VPC and Subnets as I assume that you are familiar with these.

We are using an Autoscaling Group here to maintain a single bastion instance and perform releases of new instances, for detailed information on this process please take a look at my post about High Availibilty deployments in AWS; note that in this instance we are not pre-baking everything into a dedicated AMI - instead we are starting with a base Ubuntu AMI and installing everything on boot-up via UserData, ideally though you should bake as much as possible using a tool like Packer, the UserData step should just perform configuration and start services.

Now let's take a look at the UserData contained in the LaunchConfig, this is just a bit of bash that will install and configure the bastion:

#!/bin/bash
set -euo pipefail
apt-get update && apt-get install -y curl python-pip ntp wget
wget -O - https://swupdate.openvpn.net/repos/repo-public.gpg|apt-key add -
echo "deb http://build.openvpn.net/debian/openvpn/stable xenial main" \
  > /etc/apt/sources.list.d/openvpn-aptrepo.list
apt-get update && apt-get install -y openvpn
pip install awscli https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
cfn-init \
  --resource BastionLaunchConfig \
  --stack ${AWS::StackName} \
  --region ${AWS::Region}
mkdir -p /etc/openvpn/keys
aws s3 cp s3://${S3VpnKeysBucketName} \
  /etc/openvpn/keys \
  --recursive \
  --include "ca.crt" \
  --include "server.crt" \
  --include "server.key" \
  --include "static.key" \
  --include "dh2048.pem"
chmod -R 0600 /etc/openvpn/keys
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p
iptables -t nat -A POSTROUTING -s 172.16.0.0/22 -o eth0 -j MASQUERADE
INSTANCE_ID=$( curl -s http://169.254.169.254/latest/meta-data/instance-id )
aws ec2 associate-address \
  --region ${AWS::Region} \
  --instance-id  $INSTANCE_ID \
  --public-ip ${BastionEIP}
systemctl daemon-reload
service openvpn start
INSTANCE_ASG=$( aws ec2 describe-instances \
  --instance-id $INSTANCE_ID \
  --region ${AWS::Region} \
  --query "Reservations[0].Instances[0].Tags[?Key=='aws:cloudformation:logical-id'].Value" \
  --output text )
set +e
ps auxw | grep -P '\b'openvpn'(?!-)\b'
OPENVPN_RUNNING=$?
set -e
cfn-signal \
  -e $OPENVPN_RUNNING \
  --stack ${AWS::StackName} \
  --resource $INSTANCE_ASG \
  --region ${AWS::Region}

The first few lines here deal with installing OpenVPN and some AWS tools (AWS Cli and Cloudformation scripts). The call to cfn-init activates the AWS::CloudFormation::Init portion of the Launch Config, the end result of this is writing out a config file for OpenVPN to /etc/openvpn/server.conf, we will take a detailed look at this config file shortly. With the config in place we next use the AWS Cli to download the keys that we created earlier from the S3 bucket that we provisioned for use with the bastion. In order to allow VPN clients to appear as if they are in our VPCs network we need to enable ip forwarding and setup an iptables rule that will overwrite client source IPs so this takes place in this UserData scripts. With most of the pieces now in place we associate an Elastic IP with the instance so clients can keep the same config and start up the OpenVPN service, a quick and dirty check is performed to see if OpenVPN is acually running, if it is then we send a success signal to the ASG which completes the deployment.

Now the last piece to take a look at is the OpenVPN config file for our server, as mentioned previously this is created by Cloudformation Init:

port 1194
proto udp
dev tun
server 172.16.0.0 255.255.252.0
push "route 10.0.0.0 255.255.252.0"
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
dh /etc/openvpn/keys/dh2048.pem
tls-server
tls-auth /etc/openvpn/keys/static.key 0
tls-version-min 1.2
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
cipher AES-256-CBC
auth SHA512
ifconfig-pool-persist ipp.txt
keepalive 10 120
ping-timer-rem
comp-lzo
persist-key
persist-tun
status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
max-clients 100
user nobody
group nogroup

Most of this is pretty standard OpenVPN stuff, however in order to make our server a bit more secure we specify which tls ciphers and versions we will accept, ensuring that weak ciphers are left out, we also specify the static key that we created earlier.

Creating clients

All we need to do now is generate a client key and create a config file so you can connect your local mahine to the VPN, generate a client key like so:

cd ~/aws-openvpn/easy-rsa
source vars
./build-key [YOUR_USERNAME]

You can then create a client config file like so:

client
dev tun
proto udp
remote [AWS_ELASTIC_IP] 1194
ca ca.crt
cert [YOUR_USERNAME].crt
key [YOUR_USERNAME].key
tls-client
tls-auth static.key 1
tls-version-min 1.2
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
cipher AES-256-CBC
auth SHA512
resolv-retry infinite
auth-retry none
nobind
persist-key
persist-tun
ns-cert-type server
comp-lzo
verb 3

Place your keys and the config in the same directory and push them into your VPN client of choice, if you are on a Linux based machine then I would recommend using OpenVPN, simply copy everything to /etc/openvpn. For other OS's you will need to use a GUI client, there are variuos GUI based versions of OpenVPN for windows, and for MacOS you can try Tunnelblick or Viscocity. For GUI based clients you will need to load your keys and config in (again making sure that they are in the same directory).

You should now be able to directly ssh into the bastion machine via it's private IP, if you were to add private instances in private subnets to this VPC you will be able to contact them directly as well.

Finishing touches

When you are connected to the VPN you will appear to be within your VPCs private subnets, this means that you can do a few things:

  • You can restrict SSH access on all of your machines (including the bastion itself) to your VPCs CIDR eg. 10.0.0.0/22.
  • Public R53 entries can point to private resources, you will be able to access them because you appear to be within the private subnets.

That's it for now, if you have any questions please leave a comment. Check my cloudformation examples repo for the templates used in this article as well as a few other useful tidbits.