Reply

Forwarder is running but it fails to connect to none of the indexers.

Authority
Posts: 18
831     0

Hi,

 

I have member with DNS and Therat Analytics and Reporting services running. This one member is all the time displaying error form the subject of this post. All other members have no issue with Reporting service.

 

What can be the reason? Where to look for some clue?

 

In syslog I can see:

Type: REPORTING, State: Red, Event: A reporting task monitoring failure has occurred.

 

Piotr

Re: Forwarder is running but it fails to connect to none of the indexers.

[ Edited ]
Adviser
Posts: 121
831     0

Hello,

 

NIOS version 8.0 and above includes a new piece of code which scans the splunkd.log on grid members to understand connection failures to the Indexer (Reporting Server).

 

The error in question "Forwarder is running but it fails to connect to none of the indexers", is usually seen when the member in question is unable to communicate with the Reporting server via TCP port #9997.

I would suggest verifying the following:

1 - Navigate to "Grid-->Grid Manager-->Reporting-->Edit the member in question-->General sidetab, Advanced section--> Verify "Forwarding interface used for reporting traffic"

If this is set to "ANY", typically, a member would use its LAN/VIP (on an HA pair).

2 - Review your firewall rules to verify that you are allowing TCP port 9997 communication from the member to the Indexer. Grid members can use a random source port 1024 - 65535 to connect to destination 9997 on the reporting server. This is TCP traffic and hence bidirectional.

 

Best Regards,
Bibin Thomas

Re: Forwarder is running but it fails to connect to none of the indexers.

Authority
Posts: 18
831     0

Hi,

Well, this was my fault as I did not assign correct VLAN to MGMT port on member. All my members (vNIOS) have configured both LAN1 (192.168.80.0/24) and MGMT (192.168.90.0/24) interfaces.

 

Here strange things happens. When performing trace I can see that traffic to Reporting member is sourced from MGMT IP but targeted to LAN1 IP on reporting member.

 

Why so? Why source of the traffic is not LAN1 IP on member connecting to Reporting member? Sure I have Forwarding interface used for reporting traffic: Any, but what is reason MGMT is chosen?

 

All my members in Grid was added to grid with  Enable grid service on the Management port set to no.

 

Even so when issuing show status on grid members I am getting such output (Grid Master VIP is 192.168.80.20, GM Node 1 MGMT 192.168.90.10, Node 2 (Active) 192.168.90.11):

 

Grid Status: ID Grid Member
HA Status: Not Configured
Hostname: gmemb-8-threat.test.com
Grid Master IP: 192.168.90.11

 

Why MGMT IP of GM Node 2 is listed here?

 

Piotr

Re: Forwarder is running but it fails to connect to none of the indexers.

Adviser
Posts: 121
831     0

Hello,

 

Regarding the member using its MGMT IP address instead of LAN1, though Forwarding interface used for reporting traffic==Any  -  Have you checked whether that member has any static routes configured?

 

If there is nothing explicitly configured, it is possible that you running into a known defect named NIOS-60468.

The above defect has multiple variants:

1. Reporting forwarder using VIP address through the LAN1 interface to connect to the Indexer. The forwarder is configured to use MGMT as forwarding interface  (This can also be vice-versa).

 

2. Reporting service seems to be sourcing from VIP (expected) whilst using the MAC address of LAN1

 

Both variants have been resolved in all the latest releases such as 8.2.1 | 8.1.6 | 8.0.9. You will find the defect code documented in the release notes.


Suggestion:

1. Before assuming the above defects, could you set the 'Forwarding interface used for reporting traffic' to VIP and verify whether it makes a difference.

 

2. Even if you are running into the above mentioned defects, you could add a static route on the affected member  and force its traffic to the reporting server via LAN.

Example:
10.239.40.31/32->10.231.40.1. '10.239.40.31' is the reporting server LAN IP address and '10.231.40.1' is the gateway IP for the forwarder's LAN network.

 

Regarding your second question, can you confirm that 192.168.90.11 is the MGMT IP address of the GM active node? Do you have 'Enable GUI/API Access via both MGMT and LAN1/VIP' enabled?

 

Best Regards,

Bibin Thomas

Re: Forwarder is running but it fails to connect to none of the indexers.

gerardoguizar
Techie
Posts: 2
831     0

"If this is set to "ANY", typically, a member would use its LAN/VIP (on an HA pair)."

 

Thanks man, this config did the trick! Changed ANY for the correct interface in my enviroment (MGMT in my case) and everything worked.

Highlighted

Re: Forwarder is running but it fails to connect to none of the indexers.

juliablair
Techie
Posts: 1
831     0

thank you for your help it worked

 

run 3

Showing results for 
Search instead for 
Do you mean 

Recommended for You