Friday, August 9, 2013
One of the considerations when planning a robust infrastructure architecture is avoiding a single point of failure scenario.
While you can easily create a highly available HAproxy LB using solutions such as Heartbeat or Keepalived (which are based on a floating IP) in a non-cloud environment, it's not possible to implement similar solutions in Amazon EC2/VPC because of instances networking stack limitations.
One possible solution is keeping 2 HAproxy instances configurations in sync using cron based 'rsync'.
For example, you can sync /etc/haproxy and /etc/apache2 (assuming you are using Apache as reverse proxy) with 'rsync' over ssh every X minutes (a good idea will be ssh key trust between the machines, also remember that service restart is needed for changes to become active).
Then have a monitoring node (Nagios?) which will run a health check script (curl?) ,once the health check fails the monitoring node will re-associate the failed HAproxy Elastic IP to the standby HAproxy instance using EC2 API tools:
Script can be as simple as this: The script is using another script (add_dns_record.sh), which associates a CNAME with the new active HAproxy: