07-26-2017 11:28 AM
I have member with DNS and Therat Analytics and Reporting services running. This one member is all the time displaying error form the subject of this post. All other members have no issue with Reporting service.
What can be the reason? Where to look for some clue?
In syslog I can see:
Type: REPORTING, State: Red, Event: A reporting task monitoring failure has occurred.
07-26-2017 11:55 AM - edited 07-26-2017 12:04 PM
NIOS version 8.0 and above includes a new piece of code which scans the splunkd.log on grid members to understand connection failures to the Indexer (Reporting Server).
The error in question "Forwarder is running but it fails to connect to none of the indexers", is usually seen when the member in question is unable to communicate with the Reporting server via TCP port #9997.
I would suggest verifying the following:
1 - Navigate to "Grid-->Grid Manager-->Reporting-->Edit the member in question-->General sidetab, Advanced section--> Verify "Forwarding interface used for reporting traffic"
If this is set to "ANY", typically, a member would use its LAN/VIP (on an HA pair).
2 - Review your firewall rules to verify that you are allowing TCP port 9997 communication from the member to the Indexer. Grid members can use a random source port 1024 - 65535 to connect to destination 9997 on the reporting server. This is TCP traffic and hence bidirectional.
07-27-2017 12:24 AM
Well, this was my fault as I did not assign correct VLAN to MGMT port on member. All my members (vNIOS) have configured both LAN1 (192.168.80.0/24) and MGMT (192.168.90.0/24) interfaces.
Here strange things happens. When performing trace I can see that traffic to Reporting member is sourced from MGMT IP but targeted to LAN1 IP on reporting member.
Why so? Why source of the traffic is not LAN1 IP on member connecting to Reporting member? Sure I have Forwarding interface used for reporting traffic: Any, but what is reason MGMT is chosen?
All my members in Grid was added to grid with Enable grid service on the Management port set to no.
Even so when issuing show status on grid members I am getting such output (Grid Master VIP is 192.168.80.20, GM Node 1 MGMT 192.168.90.10, Node 2 (Active) 192.168.90.11):
Grid Status: ID Grid Member
HA Status: Not Configured
Grid Master IP: 192.168.90.11
Why MGMT IP of GM Node 2 is listed here?
09-01-2017 05:20 PM
Regarding the member using its MGMT IP address instead of LAN1, though Forwarding interface used for reporting traffic==Any - Have you checked whether that member has any static routes configured?
If there is nothing explicitly configured, it is possible that you running into a known defect named NIOS-60468.
The above defect has multiple variants:
1. Reporting forwarder using VIP address through the LAN1 interface to connect to the Indexer. The forwarder is configured to use MGMT as forwarding interface (This can also be vice-versa).
2. Reporting service seems to be sourcing from VIP (expected) whilst using the MAC address of LAN1
Both variants have been resolved in all the latest releases such as 8.2.1 | 8.1.6 | 8.0.9. You will find the defect code documented in the release notes.
1. Before assuming the above defects, could you set the 'Forwarding interface used for reporting traffic' to VIP and verify whether it makes a difference.
2. Even if you are running into the above mentioned defects, you could add a static route on the affected member and force its traffic to the reporting server via LAN.
10.239.40.31/32->10.231.40.1. '10.239.40.31' is the reporting server LAN IP address and '10.231.40.1' is the gateway IP for the forwarder's LAN network.
Regarding your second question, can you confirm that 192.168.90.11 is the MGMT IP address of the GM active node? Do you have 'Enable GUI/API Access via both MGMT and LAN1/VIP' enabled?
06-20-2019 09:04 AM
"If this is set to "ANY", typically, a member would use its LAN/VIP (on an HA pair)."
Thanks man, this config did the trick! Changed ANY for the correct interface in my enviroment (MGMT in my case) and everything worked.