There was an extra incorrect A record in DNS for one of my search heads that I am building. As a result when I tried to elect a captain the wrong name was coming back. I have had the network team correct DNS but now I can't seem to get the Cluster master to see the search head as the correct name.
on the cluster master the name is showing up as myserver-D.mydomain.net
When I try to elect a captain I get the below error: (the correct name should be https://myserver-A:8089 )
04-05-2019 11:52:51.054 -0400 ERROR SHCRaftConsensus - failed appendEntriesRequest err: uri=https://myserver-C:8089/services/shcluster/member/consensus/pseudoid/raft_append_entries?output_mode=json, error=400 - Mismatch in mgmt_uri and server URI provided to LEADER. Check URI strings in set_configuration mgmt_uri = https://myserver-A:8089 remote_server_name =
When I look at the cluster master the server is showing up in the Search Head list as the incorrect myserver-D.mydomain.net name. Can anyone tell me how to fix this? Where to go delete or remove and correct this on the cluster master.
↧
Search head had bad DNS entry - Now can't delete it from the cluster
↧
Splunk Search Head Cluster member doen't get up again
Hello together,
we have a 3 node SH-Cluster where one member is not getting up again.
If we want to restart the Splunk daemon it will stuck on the very last task to start the web server.
After a while we are getting a WARNING: web interface does not seem to be available!
On the newly selected captain node I've checked the kv status for the specific host:
configVersion : -1
hostAndPort : :8191
lastHeartbeat : Mon Apr 15 ....
lastHeartbeatRecv : ZERO_TIME
lastHeartbeatRecvSec: 0
.
.
.
replicationStatus : Down
uptime : 0
When I search for error logs in the _internal logs I can see following messages in mongod logs:
REPL [ReplicationExecutor] Error in heartbeat request to .8191; HostUnreachable: Connection refused
ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to :8191 - HostUnreachable: Connection refused
should this ip address be the address of the captain?
splunkd logs doesn't indicate any errors.
For me it seems like the syncronisation of the kv store doesn't work.
I've tried this already, but it didn't help:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/ResyncKVstore
any suggestions? Thanks!
↧
↧
On a three node SH cluster, why does one member doesn't get up again?
Hello together,
we have a 3 node SH-Cluster where one member is not getting up again.
If we want to restart the Splunk daemon it will stuck on the very last task to start the web server.
After a while we are getting a WARNING: web interface does not seem to be available!
On the newly selected captain node I've checked the kv status for the specific host:
configVersion : -1
hostAndPort : :8191
lastHeartbeat : Mon Apr 15 ....
lastHeartbeatRecv : ZERO_TIME
lastHeartbeatRecvSec: 0
.
.
.
replicationStatus : Down
uptime : 0
When I search for error logs in the _internal logs I can see following messages in mongod logs:
REPL [ReplicationExecutor] Error in heartbeat request to .8191; HostUnreachable: Connection refused
ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to :8191 - HostUnreachable: Connection refused
Should this ip address be the address of the captain?
splunkd logs doesn't indicate any errors.
For me it seems like the syncronisation of the kv store doesn't work.
I've tried this already, but it didn't help:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/ResyncKVstore
any suggestions? Thanks!
↧
Why is the email icon not shown properly on Edit Alert screen when alert_actions.conf is deployed from the deployer?
I created an App, and deployed it with alert_actions.conf to Search Heads.
When I tried to set up an alert on a Search Head by the below procedures, Send email icon was not shown properly.
[Procedures to create an alert]
(1) Create a search
(2) Save As -> Alert
(3) On Save As Alert, click Add Actions
(4) Send email icon is not shown properly
[alert_actions.conf in the App]
[email]
reportCIDFontList = jp
use_ssl = 0
footer.text = My Footer
...
↧
Why are there errors when configuring search head clustering with a deployer?
Hi , I am configuring a search head clustering with a deployer but ran in many issues:
as per the Splunk docs these are the steps I did :
First I set the search head deployer:
I have added a pass4symmkey on deployer under shcluster stanza
[general]
serverName = sh2
pass4SymmKey = $1$ir3GnxLQSyQCIHmqN+mx
[shclustering]
pass4SymmKey = $1$1WrJE7i8jQ+THZ0MWxYh
I kept this same key across all search heads on server.conf ([shclustering]
pass4SymmKey = $1$1WrJE7i8jQ+THZ0MWxYh)
Setting up search head cluster:
I ran this command in all search heads (total 3):
./splunk init shcluster-config -auth admin:password -mgmt_uri "https://vvvvv:8089" -replication_port 8079 -replication_factor 3 -conf_deploy_fetch_url https:/// (deployer_ip_address):8089 -secret -shcluster_label
I run this command in all search heads (Total 3)
then I did restart.
Choosing captain:
after completing all the above steps I choose any one search head to make a captain;
Then I run this command :
./splunk bootstrap shcluster-captain -servers_list "URI:8089". -autgh admin:password
But after running its saying ,
error=This node seems to have already joined another cluster with below members: 'https://xxxx:8089'. First remove the member from the old cluster. Then run 'splunk clean
raft' on the member to reuse it in a new cluster; server=https://xxxxx:8089, error=This node seems to have already joined another cluster with below members: 'https://xxxxx:8089'. First remove the me
mber from the old cluster. Then run 'splunk clean raft' on the member to reuse it in a new cluster.
I am not clear why I am getting this error first time I am setting on these new servers , when I am running to make a captain getting this error. Please help me
2) And what the difference between general pass4symmkey and shcluster pass4symmkey , my pass4symmkey the shclustering one is same across all cluster members not the pass4symmkey which is under general stanza , which one should need to same across all search heads ?
Thanks
↧
↧
Deployer から アプリを配布後、Alert を作成しようとしたところ Alert 作成画面にて Send email アイコンが正常に表示されません。
アプリを作成し、alert_actions.conf と一緒に Search Head に配布しました。
その後、Search Head 上にて Alert を以下の手順にて作成しようとしたところ、Send email アイコンが正常に表示されませんでした。
[Alert 作成方法]
(1) サーチの作成
(2) Save As にて Alert を選択
(3) Save As Alert 画面にて Add Actions を選択
(4) Send email アイコンが正常に表示されない
[アプリ内の alert_actions.conf 設定]
[email]
reportCIDFontList = jp
use_ssl = 0
footer.text = My Footer
...
↧
Deployment Server - local folder not being pushed to Search Head cluster
On my Deployment server in the /opt/splunk/etc/deployment-apps directory, I have the Splunk_TA_f5-bigip app with a local directory that contains a transforms.conf file. I reloaded the deploy server and then went to my Search Head cluster deployment server and verified that the transforms.conf file exists in /opt/splunk/etc/shcluster/apps/Splunk_TA_f5-bigip/local/ directory (and verified the contents of the file).
I then applied the shcluster bundle to my three search heads and waited a few minutes before checking to see if the local folder and the transforms.conf file made it over, and it did not. Running ls shows no local folder under the Splunk_TA_f5-bigip directory on either of the three search heads.
Am I missing a step to get the local folder with the transforms.conf file over from the Deployment server to the search head deploy serves and then finally to the three search head servers?
Thx
↧
Custom conf files replication in search head cluster
I've one App which has Add-on builder created configuration page with API key, proxy settings, etc. This configuration is going to store in _settings.conf in local.
If I configure this on one of the search head, will this configuration replicates to other search heads?
If Yes - Any document that can say that custom conf files replicates to other search heads.
If No - How should I configure App?
↧
Search head keeps failing in search head cluster
This was the search head that kept failing:
splunk > /appl/splunk/bin/splunk show shcluster-status -auth admin:adminpassword
Encountered some errors while trying to obtain sh cluster status.
This node is not the captain of the search head cluster, and we could not determine the current captain. The cluster is either in the process of electing a new captain, or this member hasn't joined the pool
splunk > /appl/splunk/bin/splunk show shcluster-status -auth admin:Adm\!n4Splk
On the other hand, the other 2 SHs look ok when I issued shcluster-status command on CLI:
Captain:
dynamic_captain : 1
elected_captain : Tue Jun 4 12:47:03 2019
id : A6F265E7-5FEC-448A-9ACD-8FE901D045D5
initialized_flag : 0
label : pgv013d27
mgmt_uri : https://172.26.42.160:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0
Members:
pgv013aba
label : pgv013aba
last_conf_replication : Tue Jun 4 14:04:23 2019
mgmt_uri : https://172.26.96.216:8089
mgmt_uri_alias : https://172.26.96.216:8089
status : Up
pgv013d27
label : pgv013d27
mgmt_uri : https://172.26.42.160:8089
mgmt_uri_alias : https://172.26.42.160:8089
status : Up
I did check with the forum and made sure that the mgmt_url was correct in server.conf..
It worked for a while but it started failing again after a while.
And I see the captain was selected and up and running. not sure why on the failing SH, it shows cannot determine the current captain.. Any advise where else shall I check?
Thanks.
↧
↧
Upgrade 5.2.2 to 5.3 - is the documentation wrong or is it me ?
Hello,
I'm using Splunk 7.2.6 and ES 5.2.2 (on a SHC) and I want to upgrade ES to 5.3 on this SHC environment.
According to the install documentation, I did the following :
- install ES 5.2.2 on Master Deployment server (ES was never installed before on the deployer, only on SHC members)
- restart, blabla, then "splunk apply shcluster-bundle"
As long as I already had ES 5.2.2 on SHC members, nothing was changed.
According to the UPGRADE documentation now, I did the following :
- install ES 5.3 on Deployer (via the GUI, as explained)
- restart blabla, splunk apply shcluster-bundle.
And 5.3 was **NOT deployed** on my SHC members, just as I expected.
In fact, as far as I understand Splunk deployment, installing something on the deployer via GUI will install the app (here ES) in etc/apps.
For any app to be deployed by deployer, it has to be present in etc/**shcluster**/apps.
So here is my point : how is it possible for ES to be deployed anywhere if it's only installed in etc/apps ?
Did I miss anything, or is it something missing in the documentation ?
Link to th docs :
Install : https://docs.splunk.com/Documentation/ES/5.3.0/Install/InstallEnterpriseSecuritySHC
Upgrade :
https://docs.splunk.com/Documentation/ES/5.3.0/Install/UpgradeEnterpriseSecuritySHC
Thanks for the help.
Regards.
↧
How to tear down a search head cluster?
Good evening all.
I would like to know exactly how to properly tear down a search head cluster.
I am rebuilding / upgrading a Splunk environment I inherited and I need to re-utilize some of the servers in the seach head cluster for dedicated purposes; such as installed Enterprise security app on a dedicated search head.
I have thought this through long a hard and decided this is the direction I want to go.
I have already disabled search head cluster on the members (set disabled stanza to 1 in the shclustering stanzas.)
As soon as I did this, I am now getting KV store failed errors on all 3 search heads.
I did the KVstore status command and the status show failed on the status details.
I am not sure exactly where to go from here but I am sure that disabling search head clustering on my search heads caused the KV store issue.
How do you "undo/bring down" a search head cluster completely; or maybe its better to say "revert" the search head cluster back into individual search heads?
Could the KVStore errors really be associated with my disabling the search head members?
If not, where could this KV error be coming from?
Thank you.
mgiddens
↧
How can I connect two Splunk silos together ?
hi,
I have two teams each running their Splunk deployments and I need to have a centralized manner and woud like to be able of accessing the data and run my ML algorithms on subdata sets from both splunks.
how is it possible ?
how can I connect to two Splunk at the same time ?
thanks
-Bill
↧
Can multiple search head clusters connected to the same index use port 8080?
Hello,
Currently we have Single Search Head Cluster with Enterprise Security and single Indexer Cluster. As part of the platform uplift, we would like to introduce a new Search Head Cluster (with no ES) and it will be connected to the same Indexer Cluster.
Currently we are using port 8080 for the SHC communication, can we use the same port for the new SHC as well?
All (old and new SHC) are on the same subnet.
↧
↧
Disable visibility of app on Search Head Cluster
Hey All,
I would like to turn of visibility for a number of apps on our search head cluster and could of sworn I could do it from the GUI then the change to replicate to the other cluster members but that doesn't appear to be working. I made the changes on the search head captain. Am I missing something or do I have to make the changes in the app.conf for all the apps and then push to the cluster?
Thanks,
Andrew
↧
How to sync the cluster members of a search head?
I have message popping on my UI messages " Search head cluster member (sh3) is having problems pushing configurations to the search head cluster captain (sh4). Changes on this member are not replicating to other members. We have 4 search heads with dynamic captain
I have ran the ./splunk resync shcluster-replicated-config command and also ran the /splunk clean kvstore --local . Still it is showing the error
Any other workaround for this
↧
How to add a new search head to existing SHC
Hello , we are planning to add a new search head to our existing search head cluster.
What are the steps I need to follow to properly to add a new member to our existing search head cluster?
We are pushing configs through deployer, so what is a command that I can use to update the new member with existing deployer and other search head settings?
↧
How to add a new search head to an existing SHC?
Hello , we are planning to add a new search head to our existing search head cluster.
What are the steps I need to follow to properly to add a new member to our existing search head cluster?
We are pushing configs through deployer, so what is a command that I can use to update the new member with existing deployer and other search head settings?
↧
↧
What does this search head cluster function alert WARN messages mean?
Every other day, we are getting following error on the internal index. Nearly 65,000 messages are generated for less than 15mins. What does this error actually mean?
_WARN SHCFunctions - alert csv wrong action csv = key,expire,ACTION,MD5,"__mv_key","__mv_expire","__mv_ACTION","__mv_MD5"\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n_
↧
Are there any search head to search peer recommended latency?
Hi there,
There are recommendations around latency between Search Head cluster members or IDX cluster members but are there any guidelines on latency requirements between search heads and search peers?
Thanks,
Saeed
↧
Minimum disk usage(5000MB) on Search head error
Hi Guys,
I have my searches disabled on Search heads as the default minimum free disk space is 5000MB.
Problem is my splunk configuration. I have splunk installed on two different file system. One FS is for Search head pooling(NAS).
Where and what changes should be make to avoid this error.
265G 905M 251G 1% **/APPLICATIONS/SPLUNKT**
9.9G 5.0G 5.0G 50% **/APPLICATIONS/SPLUNKT/Global_Storage** --- Search head pooling
↧