Hi,
I am trying to setup a Search Head Cluster, and during the Cluster member initialization step, `./splunk init shcluster-config` command always results in
Error, Parameters must be in the form '-parameter value'
Double checked all values and there are no typos. Only item I am not using is "shcluster_label" which the guide says is optional.
Following steps from: http://docs.splunk.com/Documentation/Splunk/6.4.0/DistSearch/SHCdeploymentoverview
Please assist.
Thanks,
~ Abhi
↧
Why are we getting "Error, Parameters must be in the form '-parameter value'" during search head cluster member initialization?
↧
In my Splunk 6.3.3 search head cluster, why is an alert email not being sent to a distribution email list?
I have a scheduled search that finds results successfully.
However, the search will NOT email the results as part of an alert action when the "to" field is set to a distribution email list. EG:
"security_employees@mycompany.com".
It works just fine when I set the "to" field to an actual user's email address. EG: "john.doe@mycompany.com".
What is even more strange, is that when I run the SAME search in the search app and simply append the "sendemail" command to the end and I set the "to" field to the original distribution list "security_employees@mycompany.com" then it DOES work. Example below:
index=ABC sourcetype=123 "find this event" | stats count by host | sendemail to="security_employees@mycompany.com" subject="Email Alert"
Anybody have any ideas here?
NOTE: We are running 6.3.3 in a Search Head Cluster environment.
↧
↧
Why do I not see HTTP Event Collector under data inputs in a Splunk 6.3 search head cluster?
Hi
I have a similar issue. I do not see HTTP Event Collector, under data inputs.
/opt/splunk/etc/apps/splunk_httpinput/default
inputs.conf
$ more inputs.conf
[http]
disabled=1
port=8088
enableSSL=1
dedicatedIoThreads=2
maxThreads = 0
maxSockets = 0
useDeploymentServer=0
I have the same version in non prod but the only difference is, prod is a search head cluster and non prod has only one search head.
It is available in non prod, the file contents of inputs.conf are the same.
What needs to be modified in inputs.conf?
↧
Is this the correct way to use appOrder in user-prefs.conf on a search head cluster?
I want to sort everyone's apps in the Launcher, so on my deployer I created an app called MY_user-prefs. It contains a single file `MY_user-prefs\local\user-prefs.conf` that looks like this:
[general_default]
appOrder = MY_Docs,search,pingfederate,data_curator,SplunkAppForWebAnalytics
After I apply the shcluster-bundle, the order of apps in the Launcher is unchanged. Any suggestions?
**EDIT**: I forgot to mention that if I manually go to each search head and add the `appOrder` parameter to the standard user-prefs app, then it works. I just would like to do it centrally from the deployer whenever I need to make adjustments.
↧
Search Head Clustering: Artifact proxying fails for real time alerts
Hello,
We have 5 search heads in cluster and have a few (5) alerts in real-time.
I know it is better to have scheduled searches, but please understand these alerts must be in real-time.
So, according to Splunk:
> The cluster only replicates search artifacts resulting from scheduled> saved searches. It does not replicate results from these other search types:> Scheduled real-time searches> Ad hoc searches of any kind (realtime or> historical)> Instead, the cluster proxies these results, if they are requested by a non-originating search head.> They appear on the requesting member after a short delay.
Does anyone know how long is this "short delay"? and actually, this is not happening in our environment.
When these real-time alerts trigger, I cannot simply bring the result by typing `|loadjob $sid$`.
Instead, I have to log in to the originating search head to bring the job.
Does this require a different port open other than the usual 8089?
reference: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCarchitecture
↧
↧
How to configure and deploy the Splunk Support for Active Directory add-on to a search head cluster?
Hello
I configured my SA-ldapsearch app on the Search head cluster deployer and deployed the app to the cluster members.
However, when I go to the "App" menu in the upper left corner of the screen, after the add-on loads, I select "Configuration" from the menu." and I don't get the screen with all the fields to test with. I just get the screen that lists configuration information.
I followed the steps in the guide, but something isn't right.
Ideas?
Thanks!
↧
Why we are unable to add a cluster member via CLI to our existing search head cluster?
Hi,
We created a new Search Head Cluster that includes one Deployer and 2 Cluster members with one being the captain. Deployment went well and the cluster members can recognize each other.
Captain:
dynamic_captain : 1
elected_captain : Thu May 12 15:37:23 2016
id : *******************************
initialized_flag : 1
label : splunk03.x.y.z
maintenance_mode : 0
mgmt_uri : https://splunk03.x.y.z:8089
min_peers_joined_flag : 1
rolling_restart_flag : 0
service_ready_flag : 1
Members:
splunk03.x.y.z
label : splunk03.x.y.z
mgmt_uri : https://splunk03.x.y.z:8089
mgmt_uri_alias : https://X.X.X.56:8089
status : Up
splunk04.x.y.z
label : splunk04.x.y.z
mgmt_uri : https://splunk04.x.y.z:8089
mgmt_uri_alias : https://X.X.X.57:8089
status : Up
But now when we are trying to add another member, it is giving errors. We tried both options. From an existing member using `splunk add shcluster-member -new_member_uri :` and from the new member, using `splunk add shcluster-member -current_member_uri :`
While trying from the new member, packet capture shows communication between splunk05 (New member) and splunk04 (existing member). In splunkd.log on splunk05, following messages are repeated.
05-20-2016 16:04:59.958 -0400 WARN SHClusterHandler - Failed to proxy call to member. https://splunk04.x.y.z:8089 WARN: call not properly authenticated
05-20-2016 16:05:00.081 -0400 WARN SHClusterHandler - Failed to proxy call to member. https://splunk04.x.y.z:8089 WARN: call not properly authenticated
- Verified server.conf for all members and made sure mgmt_uri is correct.
- All members have same value for replication_factor, replication_port, shcluster_lable and pass4SymmKey.
- Firewall rules allow communication on management port.
- Admin credential being used to authenticate are correct.
I could not find any articles referring to this proxy error. Are we missing anything obvious? Are these only warnings which can be ignored?
Thanks in advance..
~ Abhi
↧
Getting error "insufficient permission to access this resource" when I use "splunk apply shcluster-bundle" on the search head cluster deployer
Dear Splunker,
In our Splunk environment, we built a search head cluster (SHC) with 3 search heads and 1 deployer (both version 6.3).
For some reason, after I upgraded my deployer to version 6.4, when I try to use command:
splunk apply shcluster-bundle -target https:x.x.x.x:8089 -auth admin:changeme
to push my bundles, it gives me this error message:
insufficient permission to access this resource
I then downgraded the deployer to version 6.3 (by untar splunk 6.3 tgz to override the installed directory) and tried again, but it still gives me that message.
Can anyone help me out?
Thank you very much.
PS : admin user on SHC has default privileges.
↧
How to distribute Distributed Search configuration using a deployer for a Search Head Cluster?
Hi,
We recently set up a SH Cluster which includes 3 members and one deployer. Basic replication seems to be working fine(tested by creating a dashboard on one member), but running into issues when deploying configuration changes. What are the best practices when it comes to deploy a system configuration, e.g. distributed search peer's, from the Deployer to all the SH members?
If I understood the steps correctly, the only way to deploy anything from a deployer is to create an app under `/opt/splunk/etc/shcluster/apps`.
For this, I created a new folder called "configuration" and copied distsearch.conf from `/opt/splunk/etc/system/local/distsearch.conf
`
Deployment was initiated using `splunk apply shcluster-bundle`. I can see the changes were accepted on the SH Member under `/opt/splunk/etc/apps/configuration`, but SH member is still unable to search any peer. Most likely these changes did not take effect.
Is this a wrong way to deploy any system changes using deployer?
Please advise.
Thanks,
~Abhi
↧
↧
Why are PIDs hanging once a week on random search heads in our search head cluster?
About once a week, and not at any particular time or even a particular search head, we will get hung processes on a search head. When I do a splunk status, I see all of the PIDs that are trying to run. In the Distributed Management Console, everything shows as up. No reports run while in this state. A Splunk restart takes some time and it clears the PIDs, but it occurs again in a matter of days. Any help or suggestions is greatly appreciated.
↧
Can I run two Splunk instances on one physical server with one instance in a Search Head Cluster and the other in an Indexer Cluster?
I am developing a Disaster Recover solution for my Splunk environment and only have four physical servers (all 32 CPU 128 GB memory) and two VMs.
I was hoping that I would be able to run two Splunk instances on one of the physical servers and have one instance in a Search Head Cluster and the other in an Indexer Cluster (Search and Replication factor of 2).
Unfortunately, I cannot find the answer in the documentation and I cannot test in the VM world.
↧
After applying a shcluster bundle, why am I getting splunkd startup parsing errors for Splunk_App_DB_Connect?
Today I was doing maintenance on my shcluster bundle and I did an apply bundle. The apply returned an error (timeout) talking to a member. I ended up restarting splunkd on the cluster master. Now on startup, I get an error on the SH Cluster Master. I reviewed hashes on the file and nothing recently changed. So I safely ignored the error and everything appears to be working. Any idea why this is happening?
/opt/splunk/bin/splunk start
Splunk> Like an F-18, bro.
Checking prerequisites...
Checking http port [8443]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Error while parsing '/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/output/scheme.xml':
mismatched tag: line 28, column 74
Error while parsing '/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/input/scheme.xml':
mismatched tag: line 16, column 73
Error while parsing '/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/lookup/scheme.xml':
mismatched tag: line 25, column 73
There were problems with the configuration files.
Would you like to ignore these errors? [y/n]:y
Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _thefishbucket history main perfmon summary windows wineventlog
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-6.3.3-f44afce176d0-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
[ OK ]
↧
What are the minimum Resource requirements for Splunk Enterprise Security in a Search Head Cluster
Hi,
We currently have a Search Head Cluster consisting of 3 members. There is also a deployer configured to push configuration bundles to these 3 members. All servers are VM's and will be used for Splunk Enterprise Security.
Minimum resources required for ES app, as on the website are 16 CPU cores with 32 GB RAM. When we have a cluster situation, does it mean that each member should meet the minimum requirement, or the cluster as a whole? Since the Captain will be load-balancing all the search queries, would it be possible to deploy ES on these 3 members with slightly less resources on each?
If we use the deployer to configure/test the App and then deploy, does this server also needs to meet the minimum specified requirements in terms of CPU/ RAM?
We also have few custom dashboards which we are planning to make available on the cluster. Is it a good idea to have them run on these 3 members already hosting ES?
Please advise..
Thanks,
~ Abhi
↧
↧
How to deploy and configure the Slack Notification Alert app in a search head clustering environment?
Installing the Slack Notification Alert app works fine when I install it as a stand alone, but my servers are clustered and the app doesn't work when pushed out with shcluster-bundle. I noticed that it takes the local directory and puts that into the default directory when deployed. The app will install, but will not let you add a Channel or Message when editing Trigger Actions. Is there a fix for this and does it work with clustered services?
Thank you,
↧
How to delete reports via CLI in a search head cluster?
All,
I have a search head cluster. A user has a couple of reports that he does not have a delete option for. I went in as admin and myself and there is no delete option in the GUI. Is there a way for me to delete these reports from the Command line? If so how? I would of course need SHC replication to not fail.
↧
Why am I unable to set up a search head cluster and getting error "Failed to bootstrap this node as a captain"?
I am trying to set up a search head cluster, but failed.
Below are my settings:
1) on Search head1 (xx.xx.xx.aa)
run below command then restart splunk
splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.aa:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1
2) on Search head2 (xx.xx.xx.bb)
run below command then restart splunk
splunk init shcluster-config -auth admin:changeme -mgmt_uri https://xx.xx.xx.bb:8089 -replication_port 8888 -replication_factor 2 -conf_deploy_fetch_url https://xx.xx.xx.cc:8089 -secret changeme -shcluster_label shcluster1
3) on Deployer host (xx.xx.xx.cc), set below in server.conf
[shclustering]
shcluster_label = shcluster1
4) Bring up captain on SH1
/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://xx.xx.xx.aa:8089,https://xx.xx.xx.bb:8089" -auth admin:changeme
I'm getting the error below:
[root@splunksh1hk1 ~]# tail -f /opt/splunk/var/log/splunk/splunkd.log
06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-01-2016 20:50:48.977 +0000 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-01-2016 20:50:48.984 +0000 INFO ServerConfig - Using REMOTE_SERVER_NAME=5453F6EB-0F41-49FA-9203-F6A6FAED2D85
06-01-2016 20:50:48.987 +0000 INFO ServerRoles - Declared role=search_head.
06-01-2016 20:51:44.987 +0000 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
06-01-2016 20:53:43.271 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.
06-01-2016 21:00:23.340 +0000 ERROR SHCRaftConsensus - Failed to bootstrap this node as a captain.
↧
Can I upgrade a search head cluster by a rolling upgrade?
Hello Community,
I want to upgrade an indexer cluster and a search head cluster from 6.3.3 to 6.3.4.
From Splunk docs, I could understand that I can upgrade an indexer cluster by a rolling upgrade.
http://docs.splunk.com/Documentation/Splunk/6.3.3/Indexer/Upgradeacluster#Upgrade_to_a_new_maintenance_release
But I cannot find information about a search head cluster. Splunk docs only describes upgrading a search head.
Can I upgrade a search head cluster by a rolling upgrade?
↧
↧
How to set a new pass4SymmKey password on a search head cluster deployer?
Hello,
We have a Search head cluster in our environment and the person who set up the Deployer initially forgot the pass4SymmKey. Now , as a result, it's not letting me deploy content and throws the following message
Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}
Now the Cluster is running fine, but it's just that I can't deploy apps/content to the SHC members. Can I set a new password on the server.conf under the shclustering stanza (On Deployer) and add the same pass4SymmKey = new password to SHC members? Does it work, or do I need to re-initialize SHC members after adding the new password?
Appreciate your inputs...I just want to hear if you experts have an alternative before I do it the hard way :(
Thanks,
Raghav
↧
Do user accounts and and dashboards get replicated in a 6.4.1 search head cluster?
Hi,
I am using a search head cluster on 6.4.1. I have a customer who created a dashboard, but for some reason, can't modify the permissions. (Different issue). When I investigated and logged into one of the search heads in that cluster, his account was not there. Are these immediately created, or only created when the individual actually logs into that server (we are using a load-balancer across 8 servers).
↧
Should I be worried about these bundle replication messages??
Splunk Version 6.3.4
Search head cluster of 3 nodes
Indexer distributed search peers 4 nodes
I see the following error messages in _internal:
WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/F7521905-DA3E-4B9B-B2FE-08B911826B00-1465250902.b469fbba316fbf76.tmp -> /opt/splunkp/splunk/var/run/searchpeers/F7521905-DA3E-4B9B-B2FE-08B911826B00-1465250902: File exists
ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/F7521905-DA3E-4B9B-B2FE-08B911826B00-1465250902.bundle
WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 4 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=48188, tar_elapsed_ms=10311, bundle_file_size=344190KB, replication_id=1465250902, replication_reason="async replication allowed"
↧
