Archive for May, 2014

12
May
14

Monitoring Anycast nodes

In my previous article i mentioned the script i was working was intended for an anycast system. So I thought i would expand a bit on what i was working on.

I have operated a couple of anycast clouds and one of the problems one has is ensuring a node is serving correct dns traffic from its anycast address. There are many was to do this and our puppet policies already ensure we do not advertise our anycast prefix unless the dns server is serving correct answers. This ensures the host can answer locally but you are still left wondering if things are working up stream. you can check the unicast address from a remote system which gives you a another pice of incomplete information, although you may not want to listen on your unicast address.

With that in mind i was inspired by a friend to add another pice of incomplete information with a new idea. They had already set up the perl snippet in the aforementioned article. with a monitoring server parsing tcpdumps to preform much of what i describe on github. I have simply tarted things up and wrote things down.

The basic idea is to periodically run a script on each anycast node, this script will spoof the source address of a monitoring server and send a few queries to help us ensure the server is functioning. This allows us to check that responses from a nodes anycast address has a reverse path back to our monitoring server. however we can also use it in place of other checks to ensure serial numbers and other factors are correct.

My [work in progress] solution to this is, is a bunch of tools i have named very badly as dnsreader. I have tried to document examples on that page but feel free to comment if anything is unclear