Hi at all,
I have a Search Head Cluster with 3 SHs that sends alerts to an external system based on IBM NetCool.
Cluster deploys alerts between the three Search Heads and ensures that only one of them runs one alert.
My problem is to create a HeartBeat alert that runs on all the three Search Heads every period to be sure that the connection with IBM NetCool is OK.
How can I configure this alert to be executed at the same time on the three Search Heads?
Thank you.
Bye.
Giuseppe
↧
How to configure a Heartbeat alert in a Search Head Cluster
↧
Why is the "sendemail" command not sending email on one of the member in search head cluster?
when am executing `|sendemail xxxxx` command in search, the job is getting queued and is running forever on one of the search head cluster members.
the is working fine on other cluster members and is sending mail.
email server settings are all same on all cluster members i don't know why `|sendemail` is not working on one server.
↧
↧
How to access a REST endpoint on the deployment server from a search head cluster?
I'm trying to populate a lookup table with information about my deployment clients. The only place I've found to get that information is from the deployment server itself. The following rest command returns the data I want, but it only works when I run it on the deployment server (which is not a search head peer, so doesn't have access to the same lookup tables for `|outputlookup`).
If I specify "splunk_server=master" in the search, and run it on my normal search head cluster, I get no results back.
Is there a way to make the search run on my normal search heads and query against the deployment server? Or, alternatively, run on the master and write against a lookup table (kvstore, if it matters) on my search head cluster?
| rest splunk_server=local /services/deployment/server/clients
|eval serverClass=""
|foreach *.serverclasses [eval serverClass=mvdedup(mvappend(serverClass,'<< FIELD >>'))]
|rename hostname as sourceHost ip as sourceIp
|table sourceHost,sourceIp,serverClass
↧
Splunk Add-on for Kafka: How to send topic messages to my indexer and retain for a longer period?
I tried first option of manage inputs from a single node via a Search Head Cluster and I get any topic messages in UI
So I tried via the heavy forwarder - added Kafka cluster and the via data inputs added the Kafka topic. But at this step, I see only four indexers- default, main etc.
I wanna send these topic messages to my own indexer and retain for a longer period. How can I send it to my indexer???
And every time a new topic is created, should I manually add this via the data inputs????
Won't Splunk detect this topic automatically???
↧
Splunk Add-on for Kafka: How to send topic messages to my indexer ?
1)I tried first option of manage inputs from a single node via a Search Head Cluster and I'm not getting any any topic messages in UI.
Why is this so? I gave the kafka cluster details and Heavy forwader detail and topic name.
2)
So I tried via the heavy forwarder - added Kafka cluster and the via data inputs added the Kafka topic. But at this step, I see only four indexers- default,main etc.
I wanna send these topic messages to my own indexer and retain for a longer period. How can I configure my indexer here???
3) Also When i followed the above method itself for adding another topic. I couldnt see these topic messages in the UI. Im not seeing further topic messages that Im adding.
wHY IS THIS HAPPENING?
Please guide....
↧
↧
Why is the replication status on distributed search failing with this error "Gave up waiting for the captain to establish a common bundle version across all search peers"?
Hi, We are seeing replication status on search head captain and one of the search head has failed.
Also we are seeing below error when we tried to search any data.
Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead
We tried restarting search head cluster, rolling restart and resync, but still facing this issue.
Can you please advice how we can handle this issue?
Thanks.
↧
Why is the KV Store status is showing as "starting" in Search Head Cluster?
We created a KV Store in a search head in clustered architecture, by adding the files collections.conf and transformations.conf.
--But we can't access the kvstore using inputlookup command and getting the below error.
"Error in 'inputlookup' command: External command based lookup 'kvstorecoll_lookup' is not available because KV Store initialization has not completed yet. Please try again later.
The search job has failed due to an error. You may be able view the job in the Job Inspector."
When checked for the status of the KV Store using the curl command, found it to be as "starting".
"curl -k -s https://localhost:8089/services/server/info | grep kvStore"
Also, Checked the **mongod.log**, deleted log and tried, but no success.
Checked for SSL certificate validity and found it to be "notAfter=Dec 9 19:01:45 2019 GMT"
Could not really trace out exact reason for the problem. Please suggest other options that we should try?
↧
Is search head clustering now supported on Windows?
I understand that support for search head clustering was supposed to be added with version 6.3. Is that now supported?
↧
Why does dashboard drilldown display message "ERROR fetching search"?
Data is indexed properly and when trying to hit query in search query bar, it is showing results but dashboard is showing "ERROR fetching search" message. Does anyone have any idea? (Search Head cluster is there)
↧
↧
Can a Search Head outside of a Search Head Cluster use the same indexers?
In my company we set up our Splunk to use a Search Head Cluster and a Indexer Cluster but I want to make a separate Splunk instance (not in the sh-cluster) use the indexers. This is meant to be a "developers" instance and I want to keep the environment to a minimum by opting out of things like shcluster redeploys, bundle replication, scheduled searches, etc...
Is there a way to configure a search head to work like this?
↧
How to pinpoint in configuration settings why "index=_audit action=alert_fired" search only returns 100 results?
Hi Splunkers,
I am trying to pinpoint to the config settings that limits this events to a 100
Search: `index=_audit action=alert_fired`
Setup:
Indexer cluster - 6VMs
Search Head Cluster - 5VMs
I've checked the alert_actions.conf file and it has it set at 10000:
# The global maximum number of results to be emailed. Any alert level
# max-results greater than this number will be capped at this level.
# maxresults=10000
Thanks!
↧
What is best practice for keeping var/run/searchpeers clean on your indexers?
I've recently been seeing many errors related to bundle replication ....
Unable to distribute to peer named 'peername' at URI https://IP:PORT because replication was unsuccessful.
replicationStatus Failed failure info: falied_because_REMOTE_CHKSUM_UNMATCHED: blah, blah, blah.
I can usually mitigate this by bouncing splunkd on the search peer.
But when I log in to the peer, I see bundles and delta files matching my Search Cluster GID, but also many, many older directories. ( as old as 2 1/2 years old ) ... Some of them have names matching our Search Heads as if the bundles may have been prior to when this platform was set up for Search Head Clustering.
So my question is:
What's the best practice for keeping $SPLUNK_HOME/var/run/searchpeers up to date and free of stale and/or no longer useful bundles?
Thank you
↧
Using different Splunk Version for Search Head Cluster as for the Indexer Cluster
Hi ninjas
I wonder wheter if it is supported having different splunk versions of the indexer cluster and the search head cluster.
Lets say we have an existing multiside indexer cluster and search head cluster running ond version 6.3x and i want to add an additonal search head cluster to that indexer cluster - can i go with the actual version of splunk for the new shc or should i use the same version as the existing ones?
Any one made experiences in that case? I didtnt found anything useful in the docs hop someone can help me out here.
Thanks in advance
↧
↧
How to upgrade a Splunk search head and indexer cluster from 6.3.2 to 6.5.1?
Hi
We are doing upgrade from 6.3.2 to 6.5.1. We have a search head cluster and indexer cluster in our Splunk setup.
In doc http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Upgradeacluster
First It says
*"When you upgrade a 6.x indexer cluster, such as 6.2, to a later 6.x cluster, such as 6.3 or 6.4, you must take all cluster nodes offline. You cannot perform a rolling, online upgrade."*
And later it says
*"Perform the following steps:
1. Stop the master.
2. Stop all the peers and search heads.
When bringing down the peers, use the splunk stop command, not splunk offline."*
So first it says you need to take all cluster nodes (Peer nodes) offline, and then second it says do not splunk offline command. It is confusing, so please help me. Should I use the splunk offline command or splunk stop command for peer nodes and search head nodes in a search head cluster?
↧
Lookup Definition Not Replicating Across Search Head Cluster
I have pushed a static lookup file via the Deployer to all of my Search Heads.
I then configure the lookup definition on 1 Search Head, however, only that Search Head sees the definition.
Shouldn't that change be propagated to all of the Search Heads?
I also want to configure an automatic lookup, which I also thought would propagate.
Thanks!
↧
Why did my search head stop unexpectedly?
Not too sure what happened here, but today we had a search head randomly stop on us. I was alerted via email, but unfortunately was away so I didn't get around to it. Took a look at the splunkd.log file on the server and this is all I saw:
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "mi_output://" with 18 parameters: description, connection, query, query_timeout, search, is_saved_search, time_out, transacti
onal, customized_mappings, ui_mappings, ui_selected_fields, ui_saved_search_str, ui_query_sql, ui_query_mode, ui_query_catalog, ui_query_schema, ui_query_table, resource_pool
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "mi_session://" with 1 parameters: user
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 12 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, index, showZe
roValue, useEnglishOnly, formatString
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "powershell2://" with 2 parameters: script, schedule
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "powershell://" with 2 parameters: script, schedule
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "rpcstart://" with 12 parameters: javahome, options, port, bindIP, proc_pid, useSSL, protocol, cipherSuite, keystore_password,
cert_file, cert_validity, Exception
01-05-2017 13:05:51.899 -0500 INFO SpecFiles - Found external scheme definition for stanza "splunktcptoken://" with 1 parameters: token
01-05-2017 13:06:01.464 -0500 WARN HandleJobsDataProvider - Can't request search.log for search job with sid=scheduler__m64232__launcher__RMD502eb6b7c7185a95f_at_1483552800_53667_DF5065C3-81ED-41A2-8E24-EF74E8474AC6 ,
Either job doesn't exist or the user doesn't have permission to access it.
01-05-2017 13:49:55.607 -0500 INFO ServerConfig - My GUID is DF5065C3-81ED-41A2-8E24-EF74E8474AC6
01-05-2017 13:49:55.607 -0500 INFO ServerConfig - My server name is "ciMASKED0202".
01-05-2017 13:49:55.609 -0500 INFO ServerConfig - Found no site defined in server.conf
Seems that right after 13:06 the application stopped responding.
I don't see anything in splunkd-stderr either. Any pointers on where else to look?
↧
Does Splunk support two search head clusters with one indexer cluster?
Does Splunk support two search head clusters with one indexer cluster? Basically we have 3 search heads clustered. we want to add other 3 search heads which should be separately clustered. Is it possible?
↧
↧
SA-cim_validator: Can this app be installed in a Search Head Cluster environment?
I see the instructions for the SA-cim_validator app only include a single-instance install. Can this app be installed in a Search Head Cluster environment?
↧
If I run "tscollect" command multiple times on the same time range and same set of data, will data be counted multiple times?
Hi all,
Would anyone please help in following newbie questions about tscollect?
- We've search head cluster running. Is it correct that we have to put the tsidxstats folder of all search heads on shared storage in order to share the tsidx files among all search heads?
- Can we simply delete any unwanted tsidxstats files?
- If I run tscollect multiple times on the same time range and same set of data, will data be counted multiple times ?
- How can I delete the 'namespace' (or tsindx file) created by tscollect?
We're running 6.5.1.
Thanks a lot.
Regards,
/ST Wong
↧
How to resolve frequent license issue errors such as "In handler 'localslave': editTracker failed" on Search Head Cluster?
We have Search Head cluster with 3 nodes. We try to point all these nodes to my License master .
When I tried to do as the master Uri from the SH members/Deployer UI it showing error message:
"Bad Request — In handler 'localslave': editTracker failed, reason='WARN: path=/masterlm/usage: invalid signature on request from ip=x.x.xx.xx "
I have manually updated the same from CLI in server.conf, it worked but slaves are not contacting frequently license master . After 24 hrs i am getting license expired message on Splunk Web. Again, I have removed the master uri stanza and placed the same back and restarted. it worked
How to fix this one ?
Also I have done telnet from my search head members to license master and it connected without any issues.
↧