Forwarder Reporting Service is failed issue

[#1]

Hi Team,

Right now we are doing POC  on our customer with configuration below:

1. TE 1425 run as authoritative NS

2. TR 800 run as reporting

TE run as DNS and grid master on the POC, and then the reporting as grid member. I found issue on the TE that it cannot start the reporting service even though the Enable data forwarding to the indexer on this member already check, but the error still occur.

if i check the logs, it shows like below:

2019-03-12T06:38:47+00:00 user dmz.xxx.xxx monitor[14219]: alert Type: REPORTING, State: Red, Event: A reporting task monitoring failure has occurred.

Please your help and advice.

Thanks.

[#2 · Accepted solution]

Greetings!

Unfortunately, that message is not clear enough. However, it usually means one or more of the below.

1. The TE-1425 is unable to reach the TR-800 over TCP port# 9997 (default port but configurable). Perhaps a network trace will provide more info.

2. There is a file corruption on the TE-1425 causing the Splunk forwarder startup to fail. [Unless someone can help fix it via root or hotfix, best solution to this would be to reset the appliance to disjoin it from the grid. Then downgrade it to an older NIOS version, upgrade it to the NIOS version of the grid and join it back to the grid.]

[You could also join it back to the grid without upgrading and the member would auto-sync NIOS from the GM but if the problem lies in the splunk-forwarder.tar.gz package on the GM (which is unlikely), you would be stuck with the same issue again].
 
3. This is initial setup and you've started the reporting service on the DNS member and reporting server selectively instead of enabling it at Grid Reporting properties. Or you have never started the reporting service on the Grid Master. GM is the certificate authority who is suppose to make and distrubute the certificates to the reproting server, all members and the GM itself for securing the channel using SSL and TLS (in latest versions). Not starting the reporting service on the GM during initial configuration is a bad idea and can mess things up at times.

While I do not know what NIOS version you are on, you may want to login to the CLI of the DNS member and run "show log debug follow" to capture more information and also look at a packet capture.


Best Regards,
Bibin Thomas