Hi -
My site has some standalone 6.2 search heads and recently implemented a new cluster of 6.6.1 search heads as well.
I've enabled LDAP authentication, defined a default strategy, and mapped LDAP groups to roles on the cluster, but there are some puzzling differences between 6.2 stand-alone and 6.6.1 clustered that I'm hoping to learn more about. Specifically, there are a number of strategies (settings -> access controls -> authentication method -> LDAP strategies) listed that I didn't create and can't delete/clone/enable but seem to be related to my "default" strategy. Their names are: authenticaiton, cacheTiming,roleMap_default, secrets. And while I can create additional strategies, the only one I can "enable" is "default". I've tried these operations from all cluster members with the same results on all.
I've read lots of docs about 6.6.1 search head clusters and LDAP authentication, but nothing I saw discussed automatically created strategies. Anyone got any pointers that'll help me understand this ?
Thanks,
-Rob
↧
Seeking documentation re: LDAP strategies on a search head cluster in 6.6.1
↧
How to troubleshoot why our lookup that is updated by "outputlookup append=true" missing data added from the last 2 months?
We have small lookup updated in search by `outputlookup append=true`
This is a SMALL size
Our users noticed the lookup lost the added data from the last 2 months.
Any clue?
We have a search head and indexer cluster.
↧
↧
A customer with admin privileges and a search head cluster cannot see "Setup" Action in Manage Apps. Is this expected behavior?
A customer of ours is unable to see our App's Setup Action when using a Search Head Clustered environment.
They have *admin* permissions and are able to navigate to the Setup page via a link, so it appears that they should see it in this page.
1. Is this expected behavior?
2. If not, is it most likely a problem with our App configuration or with the customer's configuration of their Search Head Clustered environment?
Thanks!
↧
How to automatically create Splunk roles in a Search Head Cluster?
We are implementing an IDM (LDAP) solution with our Splunk search head (SH) cluster. We are doing role based access control in a way that there will be a Splunk role per index (thank god we only have 9 indexes). How can i programmatically add roles as I add indexes? So if I create an index called new_prod_data, i need to have a role with access to new_prod_data. REST API? How will this work in a shcluster?
Any guidance is MUCH appreciated!
↧
Search Query consuming high memory utilization on indexers
Hi,
I am trying to find a list of search queries in a specific time frame that consumed high memory on the indexers.
We have an indexer cluster of 40 indexers and search head cluster of 4 SHs, suddenly for a short span of time we experienced high memory utilization on 33 indexers and consequently 2 SHs also went down.
Please help in generating the query and understanding the cause of such behavior.
↧
↧
Why is the deployment server unable to push apps to search head clusters?
Hello !!
I have created multiple apps under $SPLUNK_HOME/etc/shcluster/apps/ on the deployment server and checked the permissions on the directories and files under the apps. I have placed the files under local subdirectory in the SPLUNK_HOME/etc/shcluster/apps/{apps}/local/. Also checked the pass4Symmkey in the server.conf under /opt/splunk/etc/system/local/ on the search head deploy and on the search head cluster.
When executed the command that is `/splunk apply shcluster-bundle -target https://{search_head_cluster_captain_ip_address}:port -auth {username}:{password} `
Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://{search_head_cluster_captain_ip_address}:{port}: Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}.
I can see the apps staged under $SPLUNK_HOME/var/run/splunk/deploy/ but I'm unable to resolve it.
I have restarted the Splunk service and also the instance too but no luck. Every time I try to execute the command it always gives the above error.
When checked in the documentation they referred to this error because of the mismatch of Pass4Symmkey between the search head deploy and search head cluster.
Let me know where I can start debugging this issue. I have checked in the logs too on the search head deploy but it doesn't help since it only says about the above error.
Thanks
↧
How to calculate dispatch directory size in Splunk?
Hi,
I want to calculate dispatch directory size in Splunk to help in Splunk performance monitoring. Can anyone please help in writing the Splunk query to calculate the size of dispatch directory?
Thanks!
↧
Splunk Add-on for Microsoft Cloud Services: Where to install when using search head cluster and universal forwarder?
The Splunk Add-on for Microsoft Cloud Services documentation at http://docs.splunk.com/Documentation/AddOns/released/MSCloudServices/Install seems to be stating that you must configure the input on the search head if you are using a Universal Forwarder. Underneath, however, it says if installing on a search head cluster you should configure the input to be on the forwarder.
What are you supposed to do when you are using a search head cluster but the (unsupported) Universal Forwarder?
↧
how to find out if all our search heads in search head cluster environment is in sync?
Hello,
We are running 6.3.3 with search head clustering and 4 search heads in the cluster.
Some times users complain that, few of the saved items are not visible when they come back. We believe its due to the item saved earlier did not get synced with other search heads.
Is there way/command we can easily find out if all the search heads are in sync?
Thanks,
Simon Mandy
↧
↧
What is the best order to perform an indexer cluster migration and expansion of both Search Head cluster and Indexer cluster?
What is the best order to perform the above? Our current Splunk environment consists of 5 clustered Indexers and 4 clustered Search Heads load balanced, both single-site. We will be adding 11 Indexers equally distributed between two sites, and an additional 4 Search Heads, equally distributed between two sites, all utilizing an existing Load Balancer.
Indexer cluster will become multisite, but remain single cluster. Search Head cluster will remain a logical single site cluster.
My issue is the order in which the migration and expansion steps should be done. Data migration should not be an issue; we do not currently have an official data retention policy; I'm fairly certain we will let existing single-site data age out. Here is my current outline:
Indexer expansion and Migration
1. Install and configure additional 11 Indexers
2. Add indexers to current site
3. Migrate from single-site to multisite
Search Head expansion
1. Install and configure additional 4 Search Heads
2. Add Search Heads to existing cluster
3. Add Search Heads to Load Balancer
Please advise if this is the best way to proceed, and if not please give recommendations.
↧
How to set up a search head and indexer clustering from a standalone setup?
Hi,
My current Splunk setup is
1- stand alone search
1 - master node
3 - indexer(clustering)
Future Splunk setup
3- search head (clustering)
1- master node
3 - indexer (clustering)
I would like to implement clustering setup for search head. i need your opinion to do this without affecting the service. If there is any wiki please let me know. thank you.
↧
Why am I having issues pushing configurations to/from captain?
Error pushing configurations to captain=, consecutiveErrors=1 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=b654ee1c551ee32d816a7e8fb737ed6cd0553e7e; current_baseline_op_id=4a96d3dbb548a27fa02bbc4e6550edf01d0a7728"
Typically I get a few of these and then the captain or server becomes very busy and a destructive resync was necessary.
↧
unable to search static captain during search head clustering.
We have followed the link to create a search head cluster.
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.splunk.com%2FDocumentation%2FSplunk%2F6.5.3%2FDistSearch%2FStaticcaptain&data=02%7C01%7CVikram.Mawinkatti%40honeywell.com%7C11d72e1b5dcb4346aeb108d4d88f552d%7C96ece5269c7d48b08daf8b93c90a5d18%7C0%7C0%7C636371557512280453&sdata=nYIWxLt5gCOx0CdmZnE0HtM4vS6tl%2Bmh%2BkwG3k6321E%3D&reserved=0
however it says finally we need to initiate a captain then captain election will be random.
On all the servers we connected them to the deployment server and when I hit the final command to elect a captain for the first time I see below error.
I checked all 8089 and the mgmt port for Search replication are opened on VM level on all the search heads but it giving following erorr.
[splunk@search1 bin]$ ./splunk bootstrap shcluster-captain -servers_list ":8089,:8089,:8089 " -auth admin:******
Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch
Thanks for your reply.
Vikram.
↧
↧
Various problems with bundle replication; unauthorized and generic error
Running a 2 site indexer cluster configuration, with 2 search head clusters.
Site 1 shows a search head with the following error:
Search peer site1sh08u has the following message: Bundle Replication: Problem replicating config (bundle) to search peer ' 172.30.63.42:8089 ', error while transmitting bundle data.
Site 2 is showing the following error:
Search peer site2sh01u has the following message: Bundle Replication: Problem replicating config (bundle) to search peer ' 172.31.133.16:8089 ', got http response code 401 HTTP/1.1 401 Unauthorized.
Search peer site2sh01u has the following message: Bundle Replication: Problem replicating config (bundle) to search peer ' 172.31.133.17:8089 ', got http response code 401 HTTP/1.1 401 Unauthorized.
Search peer site2sh01u has the following message: Bundle Replication: Problem replicating config (bundle) to search peer ' 172.31.133.18:8089 ', got http response code 401 HTTP/1.1 401 Unauthorized.
What authentication is the message from site 2 referring to? Where can I find more information about these messages?
↧
Adding users via REST API on search head cluster
Is it possible to add users via the Splunk REST API in a SH cluster? I thought that I had tested this but now when I try to add a user it just hangs. Any help is much appreciated!
↧
Splunk Apps that are installed on a deployment client running a universal forwarder (From Distributed SH)
As per somesoni2 answer in https://answers.splunk.com/answers/426786/is-there-a-way-to-get-a-list-of-splunk-apps-that-a-1.html (which works perfectly) from a deployment server Manager, it is NOT working for a search member of the cluster. I have tried putting physical splunk deployment server too, but still no luck. Is there a way to query REST endpoint of another splunk tier via UI? something like ..
| rest /services/deployment/server/clients splunk_server=my_deployment_manager
The reason for this is to provide UI self catering capability for customers so they can check the status of Apps and they don't have access to Master servers. Any tricks/tips which can make this information from Search Head members (SHC) in a cluster would be highly appreciated.
↧
xmlutils app in clustered environment?
Has anybody experienced installing xmlutils app in Splunk clustered environment? I receive below error with 'xmlprettyprint' command only shcluster installation:
[myindexer1insite1] Search Factory: Unknown search command 'xmlprettyprint'.
[myindexer2insite2] Search Factory: Unknown search command 'xmlprettyprint'.
thanks up front for your time.
↧
↧
Is there a way to ensure a report is run on a specific search head cluster member?
All,
Is there a way to get a report to run on a specific member of a search head cluster? Mainly I want to know for sure which member has the outputcsv.
thanks
-Daniel
↧
unable to search static captain during search head clustering. - Resolved
We have followed the link to create a search head cluster.
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.splunk.com%2FDocumentation%2FSplunk%2F6.5.3%2FDistSearch%2FStaticcaptain&data=02%7C01%7CVikram.Mawinkatti%40honeywell.com%7C11d72e1b5dcb4346aeb108d4d88f552d%7C96ece5269c7d48b08daf8b93c90a5d18%7C0%7C0%7C636371557512280453&sdata=nYIWxLt5gCOx0CdmZnE0HtM4vS6tl%2Bmh%2BkwG3k6321E%3D&reserved=0
however it says finally we need to initiate a captain then captain election will be random.
On all the servers we connected them to the deployment server and when I hit the final command to elect a captain for the first time I see below error.
I checked all 8089 and the mgmt port for Search replication are opened on VM level on all the search heads but it giving following erorr.
[splunk@search1 bin]$ ./splunk bootstrap shcluster-captain -servers_list ":8089,:8089,:8089 " -auth admin:******
Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch
Thanks for your reply.
Vikram.
↧
Are lookup backup files supposed to replicate across servers?
Lookup File Editor with Search Head Clustering : Backup files not replicating
As the title said, I have a cluster with 3 servers. When user 1 saved a lookup, the lookup (CSV file) is replicated but the backup file is not. If user 2 wants to restore the backup of user 1 he can't (if he is on another server than user 1)
Is this a normal behavior ? I'm using lookup editor v2.5.0 and Splunk 6.6.1
↧