String Query
A query may consist of one or more words or a phrase. A phrase is a group of words surrounded by double quotation marks, such as “test search”.
Syntax: – index= query= AND type:”accesslog”
To search for all the transactions that contain the following message:
“Data audited successfully”
Note: To search for an exact string, you need to wrap the string in double quotation marks. Without quotation marks, the search in the example would match any documents containing one of the following words: “Data” OR “audited” OR “successfully”
The asterisk “*” can be used to match the preceding shortest pattern zero-or-more times. For string “aaabbb”:
Field Based Queries
All the fields available for search are listed in the left panel of the search view or (settings => Index Patterns).
To search for a value of any specific field syntax is:
Env: env
For example:
server:h121618vaps2307
server:h*vaps*
Logical Operator-Based Queries
Logical operators for example: AND, OR and NOT also can be used in queries.
For example:
server:h121618vaps2307 OR
server:h121618vaps2307 OR host:h121618vaps2307 AND instance: kls-api-07
Server is a must; the instance is optional:
index= query= AND httpstatuscode:”200″ AND @timestamp:”2022-09-30T06:26:07.430Z”
To exclude logs that match some specific criteria, you can put a minus (-), an exclamation mark (!) or the word NOT. For example,
index= query= AND httpstatuscode:”200″ NOT @timestamp:”2022-09-30T06:26:07.430Z”
Range Queries
On number fields:
Greater than or equal to 10 ms:
resptime: [10 TO *]
resptime:>=10
Between 10 and 20:
responsetime: [10 TO 20]
Regular Expression (Regex Query)
To write regex-based queries, the user has to write queries in between slash (/).
server:/server [13]/
Above query gives logs having server as server1 or server3.
type:/a{3}b{3}/
Above query search for type having aaabbb.
type:/a{3,4}b{3,5}/
Above query gives logs for type having at least 3 a’s and at most 4 a’s followed by at least 3 b’s followed by at most 5 b’s.
type:/[abc]+/
Above query filters out logs in which type field starts with one or more a’s or b’s or c’s.
type:/[a-e]+/
Above query filters out logs in which the type field starts with one or more characters in the range a to e, both inclusive.
type:/[abc\-]+/
Above query filters out logs in which type field starts with one or more a’s or b’s or c’s or -. Since – is used as range character in regex it needs escaping.
type:/[^abc]+/
Above query filters out logs in which type field does not start with one or more a’s or b’s or c’s.
type:/[^a-e]+/
Above query filters out logs in which type field doesn’t start with one or more characters in range a to e, both inclusive.
name:/john~athon<1-3>/
Above query contains two operators COMPLEMENT i.e. ~ and range <>. This query return logs where the name starts with john followed by some character other than a and ends with thon1 or thon2 or thon3.
type:/a~(cb)def/
The above query returns logs where the type field starts with a followed by any character other than cb and ends with def.
type:/aaa.+&.+bbb/
Above query filters out logs where the type field starts with aaa followed by one or more occurrences of any character and ends with one or more occurrences of any character and ends with bbb.
type:/@&~(foo.+)/
Note: The ‘at’ sign “@” matches any string in its entirety. Above query filters out logs where type field contains anything except string beginning with “foo”.
Removing Duplicity
Users can remove duplicity from the indices using an inbuilt functionality of NetForest. Below is the syntax along with an example to illustrate this feature.
Syntax: *|dedup| <Field Name>
Example: *|dedup tier
Output: If the query is executed successfully and there is any duplicate record (tier in this case), the result would be displayed as follows:
The number of unique records (hits) is displayed at the top. In this case, it is 1 as displayed.
Dedup Query
Dedup query is used to remove duplicates based on the value of a field in the document.
Index=* query=* NOT(“error”) | dedup node.
The above query filters out logs where the error string is not present and returns unique logs based on value of server field.
Number after dedup keyword tells number of duplicates allowed. Eg: 1 means unique values.
Output
Custom/Dynamic Field Extraction using Rex
Rex query doesn’t work on analyzed fields.
“/product?id=10” | rex api = “^\/([a-zA-Z]+)\/.*$” path
Above query extracts api field from the path field (already extracted).
ciscreensize|rex screen=”.*\”(\d{2,6}\sx\s\d{2,6}).*” message.keyword
The above query extracts out screen field from message. In message, it searches for a pattern having integer of length 2 to 6 followed by x and ends with integer of length 2 to 6. e.g. 1024×768.
uripath:\/search*|rex keyword=”^/.*?keyword=([a-zA-Z0-9+%]+)&.*$” uripath|rex corrid=”^/.*&corrid=([a-zA-Z0-9\:\-\.]+)&.*$” uripath | table uripath,keyword,corrid
The above query filters logs where uri path starts with /search and extracts multiple fields e.g. keyword, corrid from uri path and plots a table from these fields.
_exists_:type|rex abh=”(cloud_alert)” type
The above query is used to match the string which has “cloud_alert” text.
Output
_exists_: type|rex abh=”(cloud.*)” type
The above query is used to match the string which starts with ‘cloud’ followed by any character.
_exists_: type|rex abh=”(. *alert)” type
The above query is used to match the string which starts with anything but should end with ‘alert’.
_exists_: type|rex abh=”(. *oud_ale. *)” type
Above query is used to match the string which starts and ends with anything but should contain ‘oud_ale’.
_exists_: type|rex abh=”([a-z]+)” format
Above query is used to match the string from a to z of any length with at least one or more occurrences.
_exists: type|rex abh=”([a-z]*)” format
Above query is used to match the string from a to z of any length with zero or more occurrences.
_exists: format|rex abh=”(def{1}ault)” format
To match the string of ‘default’ where the number of character occurrences of ‘f’ is one.
_exists: format|rex abh=”^([a-z]+)$” format
To match the string from a to z of any length with at least one or more occurrence, where ^ and $ indicates the start and end of regex pattern.
*|rex abh=”([^0-9]+)”format
To match the string of all numbers except 0 to 9.
*|rex abh=”([^0]+)”format
To match the string of all numbers except the numbers starting with ‘0’.
Custom/Dynamic Field Extraction using Eval
Eval query doesn’t work on analyzed fields.
OrderNo:3410 | eval status=case (doc.httpstatuscode.value==200, “Ok”,doc.httpstatuscode.value>=400,”Error”,”Success”)
Above query returns logs where the OrderNo field is equal to 3410 and creates custom status field based on value of httpstatuscode field. E.g. If its value is equal to 200 statuses will be, Ok, if its value is greater than or equal to 400 statuses will be, Error otherwise status will be, Success.
uripath:”/product? id=10″ | rex api = “^\/([a-zA-Z]+)\/.*$” path | eval apisubstr=substr(api,1,4)
Above query filters out logs where uri path is/product? id=10 and extracts out custom field api from path field and further extracts out abisubstr field from extracted field api.
*|eval num=$$return doc. amountInWallet.value$$ | eval x = abs(num)
Above query gives all the logs and create a custom field num and x with absolute value of field num.
type: accesslog | eval channel=case (KEY like “1c9d5%”,”Kiosk”,KEY like “%8c718%”,”Mobile Apps”, KEY like “%9694d”, “Old Mcom”, KEY like “KXnAJ%”, “Tablet”, KEY like “d1c2a%”, “IPad App”,KEY like “oAEItD%”,”New MCOM”)
Above query filters out logs where the type is access log and creates a custom channel field based on the value of KEY field.
type: accesslog | eval ceilRespTime=ceiling (doc. resptime.value)
Above query filters out logs where the type is accesslog and creates a custom field ceilRespTime which is the ceiling value of the resptime field.
type: accesslog | eval floorRespTime=floor (doc. resptime. Value)
The above query filters out logs where type is accesslog and creates a custom field floorRespTime which is floor value of resptime field.
type: accesslog | eval responseStatus = if(doc.resptime.value>4,”Slow”,”Fast”)
The above query filters out logs where type is accesslog and creates a responseStatus field based on value of resptime field.
*|eval status = exec (“if(doc[‘thread’].value.isNull()) {return ‘stuck_thread’} elseif(doc[‘FP’].value.isEmpty()){return ‘With Flow Path Info’} else(doc[‘TOPO’].value.isEmpty()) {return ‘With Topology Info’} else {return ‘Without Topology and Flowpath Info’}”)
Above query gives all logs and create a custom field status based on value of FP and TOPO field.
OrderNo:3410 | eval indexvalue=exec (“return doc[‘path’].value.indexOf(‘log’)”)
Above query filters out logs where OrderNo is 3410 and create an indexvalue field based on the index of “log” string in path field.
uripath:”/product? id=10″ | eval squareAmount =pow(amount,2)
Above query filters out logs where uripath is /product? id=10 and extract a custom field squareAmount from amount field, which is square of amount field.
“ERROR” | eval randVal = random ()
Above query filters logs having ERROR string and creates a custom field called randVal which is a random number.
OrderNo:3410 | eval pathSubstr=substr (doc. path.value,1,4)
Above query filters out logs where OrderNo is 3410 and creates a custom field pathSubstr from path field.
“log” | eval roundedPrice = round (doc. price.value, 3)
Above query filters logs having log string and create a custom field roundedPrice from price field by rounding off upto 3 decimal places.
OrderNo:3410 | eval pathLength = len (doc. path. Value)
The above query filters out logs where OrderNo is 3410 and creates a custom field pathLength from the path field length.