When showing the Search Head Cluster status, we get something similar to this:
./splunk show shcluster-status
Captain:
dynamic_captain : 0
elected_captain : Fri Sep 23 09:00:37 2016
id : B997E10B-0E99-4363-9887-66DE2BF8C379
initialized_flag : 1
label : shcaptain.cdn
mgmt_uri : https://10.17.240.141:8089
min_peers_joined_flag : 1
rolling_restart_flag : 0
service_ready_flag : 1
But I'm having trouble finding documentation on that output.
- What's the difference between "initialized_flag" and "service_ready_flag"?
- What are the conditions for these flags to be 0 or 1?
- How many peers have to join for the "min_peers_joined_flag" to be 1? By "peers" it mean Splunk Indexers or members of the Search Head Cluster?
Thanks!
↧
What do the status flags mean in a Search Head Cluster?
↧
How do I fix "splunk resync shcluster-replicated-config" failures on search head cluster?
I have this error:
Error pulling configurations from the search head cluster captain (https://192.168.221.101:8089); consider performing a destructive configuration resync on this search head cluster member.
On the machine generating the error (192.168.221.103), I run "splunk resync shcluster-replicated-config". I get the following error:
"ConfReplicationException: Error downloading snapshot: Network-layer error: Winsock error 10054"
In splunkd.log, I get:
"Error in RestConfRepoProxy::fetchFrom, at=: Non-200 status_code=500: refuse request without valid baseline; snapshot exists at op_id=05f70b29f3775768ee85212227c8ecd3983235c8 for repo=https://192.168.221.101:8089"
I have restarted, rebooted and reevaluated my sanity. Suggestions?
↧
↧
Splunk Enterprise Security: Why does using a search head cluster display half of the correlation searches as Saved Search?
When I deploy a DA in Splunk Enterprise Security as standalone, it shows me multiple correlation searches. But within the same app I use in a search head cluster, it shows half of the correlation searches as Saved Search.
Any possible reason for the same?
↧
How to recover a deleted dashboard after there was a change in search head cluster captain?
Hi All,
We have an Splunk environment with 6 search heads under Search Head Clustering with load balanced at front.
We have saved a dashboard in Search & Reporting app with admin privileges on GUI and it was missing (disappeared) when we tried to access on the next day.
So, on investigating we came to know that it was deleted by user n/a from the Audit logs
audit.log.4:09-21-2016 14:50:02.394 +0100 INFO AuditLogger - Audit:[timestamp=09-21-2016 14:50:02.394, user=n/a, action=delete,path="/opt/splunk/etc/apps/search/local/data/ui/views/foundation_servers_status_report.xml"][n/a]
in audit.log.4
On further investigating, we found that there was a change of search head captain just before that
09-21-2016 14:43:39.565 +0100 INFO SHPRaftConsensus - stepDown(29)
09-21-2016 14:43:39.565 +0100 INFO ServerRoles - Undeclared role=shc_captain.
09-21-2016 14:43:39.565 +0100 INFO SHPMaster - Master is being changed state = downgrading
09-21-2016 14:42:57.972 +0100 ERROR HttpListener - Exception while processing request from for /services/replication/configurat
ion/commits?output_mode=json&at=: refuse request without valid baseline; snapshot exists at op_id=7b51bac7acb6e1e030109756d1bef0746b867f5c f
or repo=https://SH Host Name:8089
I have read from the docs that, all the search heads share the same configurations all the time. Now i am not able to understand what has caused to delete the dashboard, is it the change of search head captain overwriting the local files or any other issue?
How to recover the deleted dashboard? How to resolve this issue? Please help
Thanks in advance
↧
Why can't I re-enable or re-add one member of my search head cluster after upgrading it from 6.4 to 6.5?
I am following the upgrade instructions at http://docs.splunk.com/Documentation/Splunk/6.5.0/Installation/UpgradeyourdistributedSplunkEnterpriseenvironment like so:
**Upgrade the search heads**
1. Disable one of the search heads.
2. Upgrade the search head. Do not let it restart.
3. After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.
4. Re-enable and restart the search head.
5. Test apps on the search head for operation and functionality.
6. If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment.
7. (Optional) Test each search head for operation and functionality after you bring it up.
8. After you upgrade the last search head, test all of the search heads for operation and functionality.
At step 1 I did `splunk disable shcluster-config` but I have a feeling I screwed that up, because at step 4 I cannot re-enable it with either `splunk add shcluster-member -new_member_uri https://blah:8089` or with `splunk add shcluster-member -current_member_uri https://blah:8089`
What have I messed up on this sunny Friday afternoon?
↧
↧
Why is my search head cluster not working after updating to Splunk 6.5?
My search head cluster is no longer working after an update from 6.4 to 6.5 (I think it was the update!). It seemed to work just fine after the update but then I get in today and it is not working. Here is the log messages:
10-03-2016 17:10:37.586 +0000 WARN DistributedPeerManagerHeartbeat - Send failure while pushing PK to search peer = http://10.0.8.7:8089 , Connect Timeout
10-03-2016 17:10:37.586 +0000 ERROR DistributedPeerManagerHeartbeat - Status 502 while sending public key to cluster search peer http://10.0.8.8:8089:
10-03-2016 17:10:37.586 +0000 WARN DistributedPeerManagerHeartbeat - Send failure while pushing PK to search peer = http://10.0.8.71:8089 , Connect Timeout
10-03-2016 17:10:37.586 +0000 ERROR DistributedPeerManagerHeartbeat - Status 502 while sending public key to cluster search peer http://10.0.8.7:8089:
Please advise, it seems as though something happened to SSL in the update.
↧
Why can't I re-enable or re-add one member of my search head cluster after upgrading it from 6.4 to 6.5.0?
I am following the upgrade instructions at http://docs.splunk.com/Documentation/Splunk/6.5.0/Installation/UpgradeyourdistributedSplunkEnterpriseenvironment like so:
**Upgrade the search heads**
1. Disable one of the search heads.
2. Upgrade the search head. Do not let it restart.
3. After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.
4. Re-enable and restart the search head.
5. Test apps on the search head for operation and functionality.
6. If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment.
7. (Optional) Test each search head for operation and functionality after you bring it up.
8. After you upgrade the last search head, test all of the search heads for operation and functionality.
At step 1 I did `splunk disable shcluster-config` but I have a feeling I screwed that up, because at step 4 I cannot re-enable it with either `splunk add shcluster-member -new_member_uri https://blah:8089` or with `splunk add shcluster-member -current_member_uri https://blah:8089`
What have I messed up on this sunny Friday afternoon?
↧
Why is my search head cluster not working after updating to Splunk 6.5.0?
My search head cluster is no longer working after an update from 6.4 to 6.5.0 (I think it was the update!). It seemed to work just fine after the update but then I get in today and it is not working. Here is the log messages:
10-03-2016 17:10:37.586 +0000 WARN DistributedPeerManagerHeartbeat - Send failure while pushing PK to search peer = http://10.0.8.7:8089 , Connect Timeout
10-03-2016 17:10:37.586 +0000 ERROR DistributedPeerManagerHeartbeat - Status 502 while sending public key to cluster search peer http://10.0.8.8:8089:
10-03-2016 17:10:37.586 +0000 WARN DistributedPeerManagerHeartbeat - Send failure while pushing PK to search peer = http://10.0.8.71:8089 , Connect Timeout
10-03-2016 17:10:37.586 +0000 ERROR DistributedPeerManagerHeartbeat - Status 502 while sending public key to cluster search peer http://10.0.8.7:8089:
Please advise, it seems as though something happened to SSL in the update.
↧
Can I configure a search head cluster if there is no data replication across data centers?
I have 6 standalone Splunk instances across different data centers (DCs) and data is not replicated across DCs for security reasons.
Requirement is
a) Power users - should be able to access logs into their DCs - which is possible and I can configure index-level access
b) Admin users - should have access to all the information. - This is what I need help for. What would be the best architecture?
Possible solutions
a) Have a SH in one of the DCs and configure SH as a Search peer for all indexers
b) Configure SH cluster across DCs. - But question is, can i configure SH cluster if there is no data replication and if yes, then how to configure it?
Please suggest if there is any alternate solution.
↧
↧
Website Monitoring: Why is app not working in a clustered environment?
We have a cluster environment, which has a Master node (Distributed Management console) with multiple search heads and indexers. We tried the master node deployment and pushing search head as well as pushing to each member individually, but it did not work. All our attempts to make the app working in the clustered env have failed.
Question2:
If we install this App on all search heads in the cluster, will it start running on all search heads parallel? In other words if I schedule 100 URL's to be hit every 5 mins on 6 search heads over a cluster, will it start hitting 6*100 URLs every 5 mins?
We need to take a decision on this soon hence a quick reply would be appreciated!
Thanks
↧
How to change the "From" address when an alert email is generated from a new search head server in the cluster?
We have 4 search head servers in search cluster. One of them was added recently.
When Splunk alerts come from "old" servers , they show "**Splunk Alert** splunk@hostname.acml.com" as a sender.
Splunk Alerts from a newly added server has just "splunk@hostname.acml.com". As a result, a recipient of the email sees this email address, not the name "**Splunk Alert**".
Cannot find where to change it. All servers have the same /opt/splunk/etc/system/default/alert_actions.conf
Thank you in advance for any suggestions.
↧
CSV Lookups not replicating in Search Heads
Hi All,
We are using a clustered environment with 3 indexers, 3 search-heads, a deployer and a heavy weight forwarder (all running on Splunk enterprise 6.4.1).
There are saved searches that run every minute and populate values to .csv files using ** |outputlookup** command.
a) When there is a result returned by these searches, it gets replicated in all the .csv file in all search heads irrespective of which search-head executed the search.
b) When there is no result found by the search, the csv lookup file in the search head on which the search was executed is emptied. But this is not replicated in other search heads.
Why is it that only in case of emptying the csv, it is not replicated?
Are there any constraints on CSV files being replicated by the search head cluster?
Any help would be greatly appreciated!
Thanks
↧
What's the best way to perform maintenance on a member of a search head cluster?
I need to perform some emergency maintenance on 1 member of my 4-member Search Head Cluster tonight. [From the docs][1], it looks like I need to remove the target from the SHC, clean the Splunk install, perform my maintenance (including a reboot), then re-add the target member back to the cluster. This seems insane to me. Is that really the best practice?
Would it be easier to just take down the entire cluster while working on this one machine?
[1]: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Addaclustermember
↧
↧
How to properly connect a search head cluster to a search peer?
I'm having a very hard time connecting my search head cluster to my search peer. I have stepped through the search head documentation very carefully located here: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/SHCdeploymentoverview
I have successfully installed my deployer and added the `[shclustering]` stanza to the /opt/splunk/etc/system/local/server.conf file and added the pass4SymmKey and shcluster_label.
I then ran `splunk init shcluster-config` on each of my search head members and restarted Splunk. Each one ran successfully without any reported errors. I'm also able to run `splunk bootstrap shcluster-captain` without any issues and `splunk show shcluster-status` doesn't report any problems:
[splunk@lelsplunksh02 ~]$ splunk show shcluster-status
Captain:
dynamic_captain : 1
elected_captain : Thu Oct 13 15:48:05 2016
id : C2403815-55A2-413E-AF26-4998CFD9508F
initialized_flag : 1
label : lelsplunksh03
maintenance_mode : 0
mgmt_uri : https://splunkserver:8089
min_peers_joined_flag : 1
rolling_restart_flag : 0
service_ready_flag : 1
Members:
lelsplunksh02
label : lelsplunksh02
mgmt_uri : https://splunkserver:8089
mgmt_uri_alias : https://xx.xxx.xx.xxx:8089
status : Up
lelsplunksh04
label : lelsplunksh04
mgmt_uri : https://splunkserver:8089
mgmt_uri_alias : https://xx.xxx.xx.xxx:8089
status : Up
lelsplunksh03
label : lelsplunksh03
mgmt_uri : https://splunkserver:8089
mgmt_uri_alias : https://xx.xxx.xx.xxx:8089
status : Up
My problem starts when I try to add my search peer. I only have one indexer and I'm following this doc: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Connectclustersearchheadstosearchpeers
I'm running:
splunk add search-server https://splunkserver:8089 -auth admin:pswd -remoteUsername admin -remotePassword pswd
This also runs successfully, but I'm just not getting any results when I connect to my search head and run a search. I can run the exact same search on the indexer itself and it returns results. I can't see any errors in logs on either the indexer or the search head members.
Any help would be appreciated to point me in the right direction.
↧
How do I Secure SplunkWeb on a Search Head Cluster
I am working on securing our Splunk environment. I'm starting with the Splunk Web parts. I have a few non-clustered Search Heads and they went fine with no issues. I'm adding the SSL settings to the $SPLUNK_HOME/etc/system/local/web.conf.
When I do the same thing to my Search Head Cluster the Splunk Webs do not respond. Looking in the splunkd.log I can see issues with connecting to each other. I can't find any of the documentation for securing the Search Head Cluster.
I am following the Splunk SSL Best Practices .conf session for most of this (can't post links yet). They don't really discuss the SHC side of things.
I was building certificates for each SHC member with a common Subject Alternative Name as well as the host name. Figuring on the Splunk Web parts would use common name and then I could set them up to use the host name part of the certificate to use other Splunk parts (splunkd, kvstore).
Does anyone have any good documentation or guidance on how to do secure Search Head Clusters?
With it being a Cluster can I only do this through apps pushed out through the Deployer?
Should I be using a common certificate for Splunk Web?
↧
After deploying KV_MODE = auto_escaped in props.conf to my search head cluster, why are we seeing unexpected search results?
I am trying to set up KV_MODE = auto_escaped for a particular source. The stanza looks like the following:
[source:///var/log/test.log]
KV_MODE = auto_escaped
I used the test data directly from the Splunk documentation:
field = "value with \"nested\" quotes."
The resulting search shows the field, field with a value with `\`.
I have set this in the props.conf on the deployer in the following areas:
$SPLUNKHOME/etc/master-apps/_cluster/local/props.conf
$SPLUNKHOME/etc/shcluster/apps/search/props.conf
Neither of these produce the correct results.
↧
Why is the Splunk Enterprise Security "Content Management" screen blank on 6.5.0 search head cluster members after upgrade to ES 4.5.0?
Hi,
We recently deployed ES Version 4.5.0 via Deployer to the Search Head Cluster. While testing on a stand-alone server, we can see the correlations being loaded under Configure -> Content Management, but for both SH cluster members, this screen is blank. Splunk Enterprise version is 6.5.0. Earlier, with ES 4.1.2, we were able to load the correlations on both members.
Is this by design for SHC, or did something go wrong during the deployment? I did verify that all necessary Apps/Add-on are on 4.5.0 on both Cluster members. Here is a screenshot:
![alt text][1]
Thanks,
~ Abhi
[1]: /storage/temp/165217-content-management-blank.png
↧
↧
Why is our custom dashboard reporting search errors, but only in the dashboard view in our search head cluster?
We are experiencing some weird issues with a custom developed dashboard application, and after a couple of days trying to debug the issue, I feel it is time I reached out to the community. If anyone can help, even if it's extra debug steps, that would be great!
**Background**
I developed a dashboard application with 2 views using a standalone search head to test and then deployed the custom app to the search head cluster in the standard way.
**Issue**
For certain users (as yet we cannot determine what the commonality is) the dashboard application does not work. We get a red triangle with the following text:
> Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
Clicking the spyglass and using the integrated search within the app initially shows the errors (and allows us to look at the Job Inspector), however, if we retry the search using the same search parameters and time - we get results as expected.
Within the failed search, the search log has some differences from the successful search. We see WARN messages from the AuthorizationManager about an unknown role - '', something that does not appear in the successful search. I have confirmed that our authorization.conf and authorize.conf are the same on all members of the search head cluster, and that there are no etc/system/local versions of these files either.
Lastly, the permissions on this application are pretty wide, as defined in the default.meta for the app:
> []> access = read : [ * ], write : [ admin, power ]> export = system
**Environment**
- Splunk 6.4.1 on Windows Server 2012,
- we have a working Search Head Cluster running, that connects to 2 clustered indexers
- Using LDAP for authentication and deploying authorize.conf and authentication.conf as part of the Search Head Cluster bundle.
I am really at a loss for what to do or look at next, any help is very much appreciated
**Update:** We created two users with the same roles - one works, the other does not
↧
What is the best way to add users to the search head cluster?
Hi,
We are running Splunk 6.3.3. Our search head cluster (4 search heads at the moment) is using SAML authentication.
We are looking for a way to add an user with Splunk authentication without adding it on each search head separately.
Any ideas will be appreciated!
Thanks
↧
How to verify the search head cluster deployer successfully pushed my server.conf minFreeSpace configuration to cluster members?
Hello,
I am attempting to set up a lab utilizing a search head cluster.
The two errors I am trying to resolve are that two of my search head cluster members are showing status:Detention and they also have notification messages in the Splunk Web referencing the minimum free disk space (5000MB) reached.
On the Deployer, I have changed the Index Settings->Pause indexing if free disk space falls below-> to 2000.
On the Deployer, I have verified the setting is in the server.conf under the [diskUsage] stanza -> minFreeSpace = 2000
How can I verify that the minimum free disk space configuration is successfully being pushed to the members?
Are there other settings I need to verify in Splunk Web on the Deployer or the .conf files to fix the errors?
Results of: `splunk show shcluster-status`
Captain:
dynamic_captain : 1
elected_captain : Wed Oct 19 17:28:35 2016
id : xxxxxx
initialized_flag : 1
label : ip-xxx-xxx-45-163
mgmt_uri : https://xxx.xxx.45.163:8089
min_peers_joined_flag : 1
rolling_restart_flag : 0
service_ready_flag : 1
Members:
ip-xxx-xxx-47-211
label : ip-xxx-xxx-47-211
last_conf_replication : Wed Oct 19 19:06:43 2016
mgmt_uri : https://xxx.xxx.47.211:8089
mgmt_uri_alias : https://xxx.xxx.47.211:8089
status : Detention
ip-xxx-xxx-46-17
label : ip-xxx-xxx-46-17
last_conf_replication : Wed Oct 19 19:06:45 2016
mgmt_uri : https://xxx.xxx.46.17:8089
mgmt_uri_alias : https://xxx.xxx.46.17:8089
status : Detention
ip-xxx-xxx-45-163
label : ip-xxx-xxx-45-163
mgmt_uri : https://xxx.xxx.45.163:8089
mgmt_uri_alias : https://xxx.xxx.45.163:8089
status : Up
↧