Infrastructure

Building an Autodiscovering Apache Zookeeper Cluster in AWS using Packer, Ansible and Terraform

Following my pattern of building AMIs for applications, I create my Apache Zookeeper cluster with Packer for my AMI and Terraform for the infrastructure. This Zookeeper cluster is auto-discovering of the other nodes that are determined to be in the cluster ###Building Zookeeper AMIs with Packer The packer template looks as follows: { "variables": { "ami_id": "", "private_subnet_id": "", "security_group_id": "", "packer_build_number": "", }, "description": "Zookeeper Image", "builders": [ { "ami_name": "zookeeper-{{user `packer_build_number`}}", "availability_zone": "eu-west-1a", "iam_instance_profile": "app-server", "instance_type": "t2.

Replacing a node in a Riak Cluster

The instances that run in my infrastructure get a lifespan of 14 days. This allows me to continually test that I can replace my environment at any point. People always ask me if I follow the same principal for data nodes. I posted previously about replacing nodes is an ElasticSearch cluster, this post will detail how I replace nodes in a Riak cluster NOTE: This post assumes that you have the Riak Control console enabled for Riak.

Building a Riak Cluster in AWS with Packer and Terraform

Following my pattern of building AMIs for applications, I create my riak cluster with Packer for my AMI and Terraform for the infrastructure ###Building Riak AMIs with Packer {% raw %} { "variables": { "ami_id": "", "private_subnet_id": "", "security_group_id": "", "packer_build_number": "", }, "description": "Riak Image", "builders": [ { "ami_name": "riak-{{user `packer_build_number`}}", "availability_zone": "eu-west-1a", "iam_instance_profile": "app-server", "instance_type": "t2.small", "region": "eu-west-1", "run_tags": { "role": "packer" }, "security_group_ids": [ "{{user `security_group_id`}}" ], "source_ami": "{{user `ami_id`}}", "ssh_timeout": "10m", "ssh_username": "ubuntu", "subnet_id": "{{user `private_subnet_id`}}", "tags": { "Name": "riak-packer-image" }, "type": "amazon-ebs" } ], "provisioners": [ { "type": "shell", "inline": [ "sleep 10" ] }, { "type": "shell", "script": "install_dependencies.

Replacing the Nodes in an AWS ElasticSearch Cluster

In a previous post, I talked about how I have tended towards the philosophy of ‘Immutable Infrastructure’. As part of that philospohy, when a box is created in my environment, it has a lifespan of 14 days. On the 14th day, I get a notification to tell me that the box is due for renewal. When it comes to ElasticSearch nodes, there is a process I follow to renew a box.

Deploying Kibana Using Nginx as an SSL Proxy

In my last post, I described how I use Packer and Terraform to deploy an ElasticSearch cluster. In order to make the logs stored in ElasticSearch searchable, I use Kibana. I follow the previous pattern and deploy Kibana using Packer to build an AMI and then create the infrastructure using Terraform. The Packer template has already taken into account that I want to use nginx as a proxy. ###Building Kibana AMIs with Packer and Ansible

Building an ElasticSearch cluster in AWS with Packer and Terraform

As discussed in a previous post, I like to build separate AMIs for each of my systems. This allows me to scale up and recycle nodes easily. I have been doing this with ElasticSearch for a while now. I usually build an AMI with Packer and Ansible and I use Terraform to roll out the infrastructure ###Building ElasticSearch AMIs with Packer The packer template looks as follows: {% raw %}

Autoscaling Group Notifications with Terraform and AWS Lambda

I use Autoscaling Groups in AWS for all of my systems. The main benefit for me here was to make sure that when a node died in AWS, the Autoscaling Group policy made sure that the node was replaced. I wanted to get some visibility of when the Autoscaling Group was launching and terminating nodes and decided that posting notifications to Slack would be a good way of getting this. With Terraform and AWS Lambda, I was able to make this happen.