Hi
Can ES 4.7 be installed on a Windows SH? I know the documentation excludes ES with SHC on Windows, but it does not state anything (that I have found) about single SH for ES on a Windows machine.
Anyone?
Thanks
↧
Can ES 4.7 be installed on a Windows SH?
↧
Is there a way to ensure a report is run on a specific search head cluster member?
All,
Is there a way to get a report to run on a specific member of a search head cluster? Mainly I want to know for sure which member has the outputcsv.
thanks
-Daniel
↧
↧
Upgrade from distributed to clustered environment retaining configurations and data?
Hi ,
Is there a way to upgrade a distributed environment consisting of 1 x SH, 2 x IDX and a DS to a HA clustered env consisting of
3 x SH, 1 X Deployer, 3 X IDX and a Cluster Node ? (assume cluster node also LM)
with an aim to keep data already in the distributed indexes
is this possible and which steps should be carried out in what order ?
gratzi.
↧
Splunk Web GUI Slow (search performance is fine, however)
The GUI is sooooo slow. If you run a search results are pulled in fast, the page is already loaded. The problem is when navigating around the GUI loading new pages, dashboards or even configuration settings. There is a significant delay in the server sending the html page back to the user.
Our old search head which is still online 6.3.2 is blazin fast. I'm not talking about searches I'm talking about page loads. Purely the hosting of the GUI.
We built a new search head cluster on new systems which are physical, SSD loaded, tons of RAM awesomeness.
Currently these systems are 6.5.0.
Even our stand alone non-clustered 6.5.0 search heads are slow.
Does anyone have an idea to this issue?
Does anyone have an idea where the web-hosting component of splunk is and how to adjust performance settings.
Else is there a way to decouple the hosting of the splunk GUI and move it to Apache etc?
↧
Indexer is going down when running large number of searches
Hi,
I have one indexer and 3 search heads in cluster mode. And I have developed too many dashboards with graphic representation.
Now when ever i open 2 or more dashboards from my search head, which will be having total 30 or 40 searches, my Indexer is gong down for some time and again its coming back up.
Is there any connection restriction from Search Head to Indexer? Or Could this be some other issue?
↧
↧
Why is my search head cluster captain logging KV Store replication errors?
The log is repeating at sub-second intervals:
`2017-10-27T20:44:53.389Z I REPL [ReplicationExecutor] Error in heartbeat request to shccaptain:8191; InvalidReplicaSetConfig Our replica set configuration is invalid or does not include us`
The kvstore appears to be healthy otherwise.
`> curl -sku admin:password https://shcmember:8089/services/server/info | grep -i kvready `
`> curl -sku admin:password https://shccaptain:8089/services/server/info | grep -i kvready `
What's the problem? What's the fix?
EDIT: The error started after creating a collection by running this curl command:
`curl -ku admin -d name=userid https://shcmember:8089/servicesNS/nobody/alpha_search/storage/collections/config`
↧
SAML SSO on search head cluster behind load balancer
I have been trying to configure SAML SSO for the search head clusters running behind the LB. Our setup is Splunk WIP (wide IP Port 80) --> two VIPs in each DC which has Splunk search head servers under then listening on port 8000.It is working fine with LDAP settings.
- We are able to get SSO working by generating the metadata from the individual search head server listening on port 8000. However Load Balancing is not working since it always redirects to the same server where we generated the metadata. How we have generate a saml metadata file such that SAML SSO works with Wide IP? like how it is working with LDAP.
- I tried changing the saml/acs URL to the WIP but it doesn't work.
Does anyone come across this situation? any ideas how to deal with this>
Thanks in Advance,
↧
Search head cluster deployer -- "SHC deployer" isn't indicated as a server role after deploying via /shcluster
Hello, could you let me know if it's a GUI bug? I use /shcluster to deploy SH configurations but the role "SHC deployer" isn't indicated on our 6.5.2 Enterprise:
![alt text][1]
Thanks.
[1]: /storage/temp/219650-capture.png
↧
Fixing 502 errors when front-ending Search Heads with an AWS application load balancer?
We have a Splunk deployment in AWS and have our Search Head Cluster front-ended with an ALB (not ELB). Users frequently have the screen say "502 bad gateway", which usually goes away after a refresh or two. Has anyone else seen this, and figured out how to fix it?
↧
↧
Search fails in search head clustering.
In my environment, there are search head clustering consisting of three search heads and one deployer, and indexer clustering consisting of one cluster master and three indexers.
All of these are unified to ver6.5.5.
Up to now it was working normally,
Starting around 11/5, sometimes the following error occurred and search failed.
---------------------------------
Streamed search execute failed because:
Error in 'litsearch' command: Your Splunk license expired or
You have exceeded your license limit too many times. Renew
your Splunk license by visiting www.splunk.com/store or calling
866. GET.SPLUNK
---------------------------------
However, when I checked with the license master, the license has not expired, each indexer being license slave, and the number of excess license was also 0.
Is this a known problem?
And is there a workaround?
↧
Search head cluster member does not come up after exec restart cmd from master
Hi,
I have a three search head SHC.
I see that one SHC member going for restart but never comes back up. This is the log line.
`INFO SHCSlave - event=SHPSlave::handleHeartbeatDone master has instructed peer to restart`
SHC has three members with Dynamic captain.
What could be going wrong.
Please help.
↧
Search job execution in search head cluster environment
I have two questions about search job execution when search head cluster is used.
1. In a search head cluster, when I access a specific search head from a browser and search there, is the job execute with the search head that I access?
Or, if there are members that have fewer load, will the member responsible for processing?
2. Is there a way to check which members performed the schedule search?
↧
Search Head Cluster bundle push failed with error Network-layer error: Read Timeout
Issue: We are on Splunk version 6.6.3 and when we push the Search Head Cluster Bundle from the Deployer to Search Head Cluster Member it fails with error
Deployer's *"splunk apply shcluster-bundle"* output
{noformat}
[splunk662T]# ./bin/splunk apply shcluster-bundle -target "https://10.140.54.186:55821"
Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.
Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while deploying apps to first member: Error while updating app=unix on target=https://10.140.54.187:55821: Network-layer error: Read Timeout
↧
↧
About details of behavior on setting bundle from SH cluster
In the following manual, when distributing the configuration file from the deployer, there is a describe that the local directory existing under the app of the SH cluster member will remain.
http://docs.splunk.com/Documentation/Splunk/6.5.5/DistSearch/PropagateSHCconfigurationchanges
-------------------------------------------------
The deployer never deploys files to the members' local app directories, $SPLUNK_HOME/etc/apps//local. Instead, it deploys both local and default settings from the configuration bundle to the members' default app directories, $SPLUNK_HOME/etc/apps//default. This ensures that deployed settings never overwrite local or replicated runtime settings on the members. Otherwise, for example, app upgrades would wipe out runtime changes.
-------------------------------------------------
However, as a result of setting bundle, the local directory existing under SH cluster member's apps disappeared.
If settings that is same with settings distributed by the setting bundle exists in the local directory of the SH cluster member, will it be deleted?
↧
How to replace a search-head deployer?
Hi,
Is there a document on replacing a search-head deployer? My existing server is being decommed and I need to replace it.
↧
Search head cluster data replication for summary index
Hi All,
We have 3 indexes which are in cluster and 3 search heads which are in cluster and we are forwarding summary index data from each search head to the indexers which are in cluster. But when we are searching "index="index name" we are getting 3 records for each event. and the reason is 3 are having different index time.
So can any one help me in how we can push summary index data from the search head cluster to index cluster and when we search the index it should only show 1 record in search head not the 3 records for the same event.
Regards,
Santosh
↧
Search Head Cluster - Scheduled Search Running only in one instance
Hi All,
We have a search head cluster with 3 search heads along with a deployer. We have a scheduled search which runs a query every 8 hours and pushes the data to the "summary indexes". Though the search is scheduled on all the search heads (SH1, SH2, SH3), we observed that the summary index calculation is happening only in one of the search heads. Is this a default setting?
↧
↧
How do you report on the search head cluster availability score (99.9999% score)?
I am tasked with reporting on our Splunk environment. I am running a Search Head cluster with 3 Search Heads and an index cluster with 6 indexers (single site). They are all on 6.6.3.
Does anyone know a search that would return a result indicating the availability of the SH cluster and the Index cluster independent of each other. I am looking for a (99.99999%) 5x9's availability score specifically. Up until now I am just showing up times on my search heads (based on last restart of splunkd).
I wasn't able to find this in the monitoring console either (DMC). Any help would be greatly appreciated :)
↧
Search Head Cluster captain confiugred to run ad hoc searches only still execute savedseaches
Hello there,
On a Search Head Cluster (6.5.3), when performing an Health Check, I have had a warning for having a high skip ratio - between 60 & 80 %.
It seemed like it only affected the SHC captain.
I found out that, in order to reduce the load on the SHC captain - which is executing savedsearches, ad hoc searches and delegating savedsearches between other peers -, it was recommended to configure the captain to run ad hoc searches only.
It is documented here :
https://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Adhocclustermember#Configure_the_captain_to_run_ad_hoc_searches_only
This way, the captain only launches ad hoc searches & still delegates seavedsearches between cluster members.
I believed this would resolve our issue.
However, the skip ratio is now of 100%, still only on the captain.
It is always 100% which is weird.
To me it is more like an issue in the way internal log are being generated.
Logs are saying : this savedsearch has been skipped on this host which is the captain, reason : the max number of auto summarization has been reached ...
While it should rather be saying : this savedsearch has been skipped on this host which is the captain, reason : captain configured to run ad hoc searches only.
The thing that makes me doubt about this is the reason savedsearches are being skipped on the captain : "the max number of auto summarization has been reached"
So I am wondering if :
it is really a good practice & it's more like a logging issue
or
it is not a good practice
Note that there are no errors on the Data Models savedsearches are flagged as being skipped -> Build 100 %
Would anyone have an idea on this ?
Thanks for any feedback!
↧
Search Head Cluster concurrency context configuration (instance-wide) vs logs (cluster-wide)
Hello there,
We have a Search Head Cluster in 6.5.3 which configured by default in "member by member".
Our configuration :
shc_role_quota_enforcement="0"
shc_local_quota_check="1"
Splunk Documentation :
To enforce quotas on a member-by-member basis, use this configuration:
shc_role_quota_enforcement=false
shc_local_quota_check=true
Version Default enforcement
6.3-6.4 cluster-wide
6.5+ member-by-member
However, while investigating skip ratio, it appears that logs are saying the opposite :
index=_internal sourcetype=scheduler status=skipped | stats count by concurrency_context
cluster-wide 1636
saved-search_cluster-wide 144
Does anyone knows if this is a logging issue or else ?
Thanks for any hint!
↧