We've recently moved our production search heads to a search head cluster, since last week (6.2.6?) I have noticed that any ad-hoc jobs (via REST API or WEB UI) are not expiring and quickly stack up.
I've checked the limits.conf and savedsearches.conf, and have confirmed that the ttl's are set to 600 seconds or less.
This only happens in a clustered environment. We have dev servers running the exact same searches without issue.
In the job inspector info below, I can see that the job was created yesterday. It has completed successfully and has TTLs of 600 seconds, so why is it still there?? The expiration time just updates to now whenever I refresh the jobs list.
Is there some config specific to SHC that sets the TTL for completed jobs?
This is an example from the job inspector
Search job inspector
This search has completed and has returned 1,192 results by scanning 4,986 events in 8.404 seconds.
The following messages were returned by the search subsystem:
INFO: Your timerange was substituted based on your search string
(SID: 1443331931.9_ECBEC051-E014-4F98-95CC-90307C8D43D7) search.log
Execution costs
Duration (seconds) Component Invocations Input count Output count
0.16 command.addinfo 158 4,808 4,808
0.02 command.eval 5 19,877 19,877
0.07 command.fields 158 4,808 4,808
0.00 command.presort 1 1,192 1,192
0.91 command.prestats 158 4,808 4,742
21.69 command.search 317 8,347 10,808
6.22 command.search.rawdata 149 - -
0.74 command.search.kv 149 - -
0.46 command.search.typer 149 4,808 4,808
0.32 command.search.filter 309 - -
0.15 command.search.calcfields 149 4,986 4,986
0.15 command.search.fieldalias 149 4,986 4,986
0.09 command.search.lookups 149 4,986 4,986
0.08 command.search.tags 149 4,808 4,808
0.05 command.search.summary 157 - -
0.00 command.search.index.usec_1_8 38,306 - -
0.00 command.search.index.usec_32768_262144 2 - -
0.00 command.search.index.usec_4096_32768 838 - -
0.00 command.search.index.usec_512_4096 68 - -
0.00 command.search.index.usec_64_512 153 - -
0.00 command.search.index.usec_8_64 1,053 - -
0.00 command.sort 1 1,192 1,192
0.12 command.stats 1 - 3,539
0.61 command.stats.execute_input 159 - -
0.15 command.stats.execute_output 1 - -
0.00 command.table 1 1,192 2,384
0.00 dispatch.check_disk_usage 1 - -
0.08 dispatch.createdSearchResultInfrastructure 1 - -
0.06 dispatch.evaluate 1 - -
0.06 dispatch.evaluate.search 2 - -
0.00 dispatch.evaluate.eval 5 - -
0.00 dispatch.evaluate.stats 2 - -
0.00 dispatch.evaluate.sort 1 - -
0.00 dispatch.evaluate.table 1 - -
7.32 dispatch.fetch 159 - -
0.00 dispatch.localSearch 1 - -
0.32 dispatch.parserThread 157 - -
0.00 dispatch.stream.local 1 - -
22.39 dispatch.stream.remote 157 - 32,716,802
0.03 dispatch.writeStatus 12 - -
0.26 startup.configuration 9 - -
3.49 startup.handoff 9 - -
Search job properties
bundleVersion 4206439116757466412
canSummarize 1
**createTime 2015-09-27T15:32:11.000+10:00**
cursorTime 1970-01-01T10:00:00.000+10:00
defaultSaveTTL 604800
**defaultTTL 600**
delegate None
diskUsage 188416
**dispatchState DONE**
doneProgress 1.0
dropCount 0
eai:acl
{
"app": "apm_snpm",
"can_write": "1",
"modifiable": "1",
"owner": "username",
"perms": {
"read": [
"username"
],
"write": [
"username"
]
},
"sharing": "global",
**"ttl": "600"**
}
earliestTime 2015-09-13T00:00:00.000+10:00
eventAvailableCount 0
eventCount 4808
eventFieldCount 0
eventIsStreaming True
eventIsTruncated True
eventSearch search (eventtype="summary_cvc_util") eventtype=summary_sanitized earliest=1442066400 latest=1443276000 CVC_ID="CVC000000123456"
eventSorting none
isBatchModeSearch True
isDone True
isFailed False
isFinalized False
isGoodSummarizationCandidate 1
isPaused False
isPreviewEnabled False
isRealTimeSearch False
isRemoteTimeline False
isSaved False
isSavedSearch False
isTimeCursored 1
isZombie False
keywords cvc_id::cvc000000123456 earliest::1442066400 eventtype::summary_cvc_util eventtype::summary_sanitized latest::1443276000 tclass::4
label None
latestTime 2015-09-27T00:00:00.000+10:00
modifiedTime 2015-09-28T10:10:59.478+10:00
normalizedSearch litsearch foo bar
numPreviews 0
pid 19020
priority 5
reduceSearch foo bar
request
{
"namespace": "apm_snpm",
"search": "| savedsearch cvc_util_up_down_green cvcid=\"CVC000000123456\" startdate=\"1442066400\" enddate=\"1443276000\" | search tclass=4 | sort 0 date | table date, ACCESS_SEEKER_ID, CSA_ID, POI_CODE, POI_STATE, CVC_ID, tclass, bandwidth, inboundUtilizationPcnt, inboundThroughputMbps, inboundDroppedPcnt, inboundDroppedMbps, outboundUtilizationPcnt, outboundThroughputMbps, outboundDroppedPcnt, outboundDroppedMbps"
}
resultCount 1192
resultIsStreaming False
resultPreviewCount 1192
runDuration 8.404
runtime
{
"auto_cancel": "0",
"auto_pause": "0"
}
scanCount 4986
search | savedsearch cvc_util_up_down_green cvcid="CVC000000123456" startdate="1442066400" enddate="1443276000" | search tclass=4 | sort 0 date | table date, ACCESS_SEEKER_ID, CSA_ID, POI_CODE, POI_STATE, CVC_ID, tclass, bandwidth, inboundUtilizationPcnt, inboundThroughputMbps, inboundDroppedPcnt, inboundDroppedMbps, outboundUtilizationPcnt, outboundThroughputMbps, outboundDroppedPcnt, outboundDroppedMbps
searchCanBeEventType 0
searchEarliestTime 1442066400.000000000
searchLatestTime 1443276000.000000000
searchProviders
[
"indexer1-heavy",
"indexer2-heavy",
"indexer3-heavy",
"indexer4-heavy",
"indexer5-heavy",
"indexer6-heavy",
"indexer7-heavy",
"indexer8-heavy",
"searchead1-heavy"
]
sid 1443331931.9_ECBEC051-E014-4F98-95CC-90307C8D43D7
statusBuckets 0
ttl 600
Additional info search.log
Server info: Splunk 6.2.6, foo.bar.local:8000, Mon Sep 28 10:10:59 2015 User: keithmuggleton
↧