Grid member stopped resolving queries for authorative zone

[#1]

Hi folks

So one of my grid members suddenly stopped resolving any and all quries to SOME

of the zones that it is authorative for. This particular grid member only resolves external zones at the moment and some of the subzones it is authorative for. All other grid members are working fine.

I'm battling to find the cause of the problem. I've checkd the replication status and it says its ok. If I look at statistics for that member is has processed zero queries for the zone in question.

I've restarted it, rebooted it and so on. I've made no changes recently to the grid, it just stopped working. Doing a dig returns SERVFAIL for anything in that zone/domain.

I see the following error in that members log but it's not even for the zone in question:

general: zone domaindnszones.ZONEDELETED.COM/IN: refresh: skipping zone transfer as master IPDELETED#53 (source IPDELETED#0) is unreachable (cached)

I'm running 8.2.6 across the entire grid. There is no firewall between the grid members and grid master and all network connectivity has been verified.

Any advice appreciated.

[#2]

Hmmm, sounds strange to me without seeing the complete configuration (named.conf). Better open a support ticket with infoblox so they can investigate it.

[#3]

What stands out to me here is "refresh: skipping zone transfer as master is unreachable". This tells me that this is a secondary zone on this server and the primary has stopped responding. The servfail response that you are seeing when querying this server is expected in this case.

If you can connect to the CLI for this server, you can attempt a zone transfer and see if that is successful or not. Example:

dig @<server_address> <zone.name> axfr

A servfail or refused response will confirm that the primary is no longer responding properly and you need to check that server. Verify if it is seeing zone transfer requests from this server and if so, what it is doing in response (it may no longer have the zone loaded, or its zone transfer settings/ACL no longer allows your server). If it is not seeing the zone transfer requests at all, you have a firewall issue or misconfiguration somewhere.

Hope this helps.

-Tony

[#4]

Hello Marcelv,

To follow up what Tony has stated :

By default dig uses LAN1 interface. Thus, while performing a dig as he suggested, you would also have to ensure that the dig request is initiated from the ‘transfer-source’ as in the named.conf file. So to append the request, you would :

dig @<primary_server_address> <zone.name> axfr -b <transfer-source-IP>

If your transfer source in named.conf is indeed the LAN1 IP address, you don’t have to use ‘-b’ extension. Comparing the system log pasted with the SERVFAIL responses, I am assuming that the secondary zone could have expired in those servers. If the above query is successful, you may use the following control channel signal to force a zone transfer (it does the traditional rndc underneath...) :

set dns transfer <zone_name> <DNS_view_name>

This comamnd would pull a complete set of data from the 'master' in the named.conf. Starting a packet capture during the same would help you identify additional information in case if the transfer has been Refused/Failed for any reason.

Additional note : Your statement, “I see the following error in that members log but it's not even for the zone in question:” tells me that the pasted system log may/may not be related to the zones which could be affected. If this is something which affects an active production environment causing impact to depending clients, please contact Infoblox technical support immediately to expedite resolution!

Regards.

[#5]

Thank you all for the replies.

Your input has been most valuable.

I tested from the member and it was not able to do any zone transfers at all (via dig or via

set dns transfer). There are a few zones and subzones and none of them worked.

Doing a packet capture I could not even see the zone transfer attempt (Query type 252) from that member and I captured on ALL ports.

Other members in my grid work perfectly and doing the same test and capture I can clearly see the zone transfer happening from the same primary server.

I've checked as many settings as I can find and all the ACEs etc look correct, there is no config mismatch between this particular member and the others on the grid.

The search continues......

[#6]

Not knowing more- zone transfer requests don't always happen right away so it is possible that the capture did not run long enough. That, or the DNS service has an issue. I would recommend working with Infoblox Support to troubleshoot this further and they can help review the logs to see if that gives any indication as to why it is not working as expected.

Regards,

Tony

[#7]

Hi folks

Just to bring this full circle (somewhat).

The problem has been resolved by removing the offending member from the

grid and rejoining it to the grid. This obviously forces all sorts of background

syncs to happen and that fixed the problem.

Sadly I can't say why it happened in the first place but at least it's resolved,

and rather simply at that.

Thanks again for all your inputs.

[#8]

Hi,

Apparently i came across this thread when i facing same issue, the reason to my issue is due to the primary dns zone which was in my AD server was disconnected from network, and the secondary dns zone which is my infoblox has exceed DNS zone's expire time. Suddenly all DNS query to that zone returned "Server Failure" eventhough the DNS zone still exist in Infoblox. 

In this case, I wonder if anyone has anyway to identified if a particular Infoblox DNS zone has already expired? I mean...i can't see any differences between an expired DNS zone and valid one's in GUI, except expired zone returns me "Server Failure" DNS response. I just wish to have a way to confirmed on it.

Thank you.