Today being the first day of the re:invent we like like to write a little “thank you” dedicated to the AWS EC2 / Load Balancer Team. Last Wednesday AWS promoted a bunch of new awesome features for their managed Load Balancers. For that we’d like to say thank you with providing an additional idea what you could achieve with at least one of these features.
While you can find pretty good use-cases for “Weighted Target Groups for ALB”, “Subnet expansion on NLB”, “Shared VPC support for NLB” and “Least Outstanding Request for ALB” in this official blog post, you might ask what benefit the feature “Private IP Address Selection for Internal NLB” might bring.

Private IP Adress Selection for Internal NLB

Many of our enterprise customers are having the challenge of connecting their on-premises network with their AWS workloads through direct connect.
While resolving domain names got a lot easier for most of them after AWS launched AWS Route53 Resolver, there are still some unicorn security requirements or legacy systems having special needs, resulting in hosting their own DNS forwarders being accessible from on-prem forwarding to Route53 and vice versa.

You can get a good starting point for setting this up by following this how-to “How to Set Up DNS Resolution Between On-Premises Networks and AWS by Using Unbound“. Still, this solution comes with multiple challenges like when you think about possible machine errors:

How to distribute new IPs of custom resolver endpoints?
Best practice is leveraging your VPCs DHCP option sets, whilst some clients prefer “manually” distributing this information by custom AMI or user data.

How to determine the custom endpoints especially when an instances lifetime ends?
There are various approaches all having this foundation:
Using one ASG per AZ ensuring there is always exactly one endpoint in each AZ. When a new instance is launched – because before an instance was unhealthy – either a Lambda function is triggered or the instance itself is taking care of the following actions.

  • Option A: Distributing the new endpoint IP to all instances by either DHCP option sets or AWS Systems Manager.
  • Option B: A prior statically created ENI, that never dies along with its previous instance, is being attached to the newly launched instance, resulting in reserving always 2 ENI per DNS instance. As the IP stays static this way, there’s no need of distributing any new IP.
  • Option C: Using EC2 auto recover to reboot the instance if it is failing. This doesn’t help if a full EBS volume is the issue, though.
    … Actually there are even more options for example with Launch Templates, but you get the idea.
  • Secret Option D to the rescue:
    With the new feature of AWS Network Load Balancers, you can now just handle your DNS forwarders as you would do with any other EC2 instance with a rather

    • default Autoscaling Group spreading instances over all AZs.
    • Just put an NLB with 3 static private IPs in front,
    • distribute those with DCHP options sets
    • and share the IPs with your on-prem DNS resolver.
    • Don’t forget activating cross-zone-load-balancing as you know, there are always things to go wrong, even if it’s just an on-prem admin putting only one of 3 IPs into the DNS resolver.

DNS Forwarder with static IPs

Feature Request

While trying to express BIG THANKS to the AWS Load Balancer team, we would also like to propose and push a feature request.
If you could make it possible to ignore source and destination IPs on Network Load Balancers, giving this solution of egress traffic filtering a real benefit!

Thanks to the Load Balancing Team at AWS!

After delivering great work, I hope you can enjoy the re:Invent.