 |
VOOZH | about |
|
VOOZH | about |
Logs | Metrics
Use this processor with Vector Remap Language (VRL) to modify and enrich your logs. VRL is an expression-oriented, domain specific language designed for transforming logs. It features built-in functions for observability use cases. You can use custom functions in the following ways:
See Custom functions for the full list of available functions.
See Remap Reserved Attributes on how to use the Custom Processor to manually and dynamically remap attributes.
To set up this processor:
The functions are organized into the following categories:
Appends each item in the items array to the end of the value array.
argument
Type
Description
default
required
value
array
The initial array.
N/A
yes
items
array
The items to append.
N/A
yes
Source:
append([1, 2], [3, 4])
Return:
[1,2,3,4]Chunks value into slices of length chunk_size bytes.
argument
Type
Description
default
required
value
array, string
The array of bytes to split.
N/A
yes
chunk_size
integer
The desired length of each chunk in bytes. This may be constrained by the host platform architecture.
N/A
yes
chunk_size must be at least 1 byte.chunk_size is too large.Source:
chunks("abcdefgh", 4)
Return:
["abcd","efgh"]Source:
chunks("ab你好", 4)
Return:
["ab�","�好"]Removes the last item from the value array, returning a new array without the last element.
argument
Type
Description
default
required
value
array
The array to pop from.
N/A
yes
Source:
pop([1, 2, 3])
Return:
[1,2]Adds the item to the end of the value array.
argument
Type
Description
default
required
value
array
The target array.
N/A
yes
item
any
The item to push.
N/A
yes
Source:
push([1, 2], 3)
Return:
[1,2,3]Iterate over several arrays in parallel, producing a new array containing arrays of items from each source.
The resulting array will be as long as the shortest input array, with all the remaining elements dropped.
This function is modeled from the zip function in Python,
but similar methods can be found in Ruby
and Rust.
If a single parameter is given, it must contain an array of all the input arrays.
argument
Type
Description
default
required
array_0
array
The first array of elements, or the array of input arrays if no other parameter is present.
N/A
yes
array_1
array
The second array of elements. If not present, the first parameter contains all the arrays.
N/A
no
array_0 and array_1 must be arrays.Source:
zip([1, 2, 3], [4, 5, 6, 7])
Return:
[[1,4],[2,5],[3,6]]Source:
zip([[1, 2], [3, 4], [5, 6]])
Return:
[[1,3,5],[2,4,6]]Calculates a CRC of the value.
The CRC algorithm used can be optionally specified.
This function is infallible if either the default algorithm value or a recognized-valid compile-time
algorithm string literal is used. Otherwise, it is fallible.
argument
Type
Description
default
required
value
string
The string to calculate the checksum for.
N/A
yes
algorithm
string
The CRC algorithm to use.
CRC_32_ISO_HDLC
no
value is not a string.algorithm is not a supported algorithm.Source:
crc("foo")
Return:
"2356372769"Source:
crc("foo", algorithm: "CRC_32_CKSUM")
Return:
"4271552933"Decodes the value (a Base16 string) into its original string.
argument
Type
Description
default
required
value isn’t a valid encoded Base16 string.Source:
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")
Return:
"you have successfully decoded me"Decodes the value (a Base64 string) into its original string.
argument
Type
Description
default
required
charset
string
The character set to use when decoding the data.
standard
no
value isn’t a valid encoded Base64 string.Source:
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return:
"you have successfully decoded me"Source:
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return:
"you can't make your heart feel something it won't"Decodes the value (a non-UTF8 string) to a UTF8 string using the specified character set.
argument
Type
Description
default
required
value
string
The non-UTF8 string to decode.
N/A
yes
from_charset isn’t a valid character set.Source:
decode_charset!(decode_base64!("vsiz58fPvLy/5A=="), "euc-kr")
Return:
"안녕하세요"Source:
decode_charset!(decode_base64!("pLOk86TLpMGkzw=="), "euc-jp")
Return:
"こんにちは"Source:
decode_charset!(decode_base64!("xOO6ww=="), "gb2312")
Return:
"你好"Decodes the value (a Gzip string) into its original string.
argument
Type
Description
default
required
value isn’t a valid encoded Gzip string.Source:
encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)
Return:
"you have successfully decoded me"Decodes the value (an lz4 string) into its original string.
argument
Type
Description
default
required
value
string
The lz4 block data to decode.
N/A
yes
value unable to decode value with lz4 decoder.Source:
encoded_text = decode_base64!("LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4=")
decode_lz4!(encoded_text)
Return:
"The quick brown fox jumps over 13 lazy dogs."Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.
argument
Type
Description
default
required
value has invalid encoded encoded-word string.Source:
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")
Return:
"Hello, World!"Source:
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")
Return:
"From: Hello, World! <hello_world@example.com>"Source:
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")
Return:
"Hello, World!"Decodes a percent-encoded value like a URL.
argument
Type
Description
default
required
value
string
The string to decode.
N/A
yes
Source:
decode_percent("foo%20bar%3F")
Return:
"foo bar?"Decodes a punycode encoded value, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.
argument
Type
Description
default
required
value
string
The string to decode.
N/A
yes
validate
boolean
If enabled, checks if the input string is a valid domain name.
true
no
value is not valid punycodeSource:
decode_punycode!("www.xn--caf-dma.com")
Return:
"www.café.com"Source:
decode_punycode!("www.cafe.com")
Return:
"www.cafe.com"Source:
decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return:
"١٠.٦٦.٣٠.٥."Decodes the value (a Snappy string) into its original string.
argument
Type
Description
default
required
value
string
The Snappy data to decode.
N/A
yes
value isn’t a valid encoded Snappy string.Source:
encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)
Return:
"The quick brown fox jumps over 13 lazy dogs."Decodes the value (a Zlib string) into its original string.
argument
Type
Description
default
required
value isn’t a valid encoded Zlib string.Source:
encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)
Return:
"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."Decodes the value (a Zstandard string) into its original string.
argument
Type
Description
default
required
value isn’t a valid encoded Zstd string.Source:
encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)
Return:
"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."Encodes the value to Base16.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
Source:
encode_base16("please encode me")
Return:
"706c6561736520656e636f6465206d65"Encodes the value to Base64.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
charset
string
The character set to use when encoding the data.
standard
no
Source:
encode_base64("please encode me")
Return:
"cGxlYXNlIGVuY29kZSBtZQ=="Source:
encode_base64("please encode me, no padding though", padding: false)
Return:
"cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g"Source:
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return:
"cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM="Encodes the value (a UTF8 string) to a non-UTF8 string using the specified character set.
argument
Type
Description
default
required
value
string
The UTF8 string to encode.
N/A
yes
to_charset isn’t a valid character set.Source:
encode_base64(encode_charset!("안녕하세요", "euc-kr"))
Return:
"vsiz58fPvLy/5A=="Source:
encode_base64(encode_charset!("こんにちは", "euc-jp"))
Return:
"pLOk86TLpMGkzw=="Source:
encode_base64(encode_charset!("你好", "gb2312"))
Return:
"xOO6ww=="Encodes the value to Gzip.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
compression_level
integer
The default compression level.
6
no
Source:
encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)
Return:
"H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA"Encodes the value to JSON.
argument
Type
Description
default
required
value
any
The value to convert to a JSON string.
N/A
yes
pretty
boolean
Whether to pretty print the JSON string or not.
N/A
no
Source:
.payload = encode_json({"hello": "world"})
Return:
"{\"hello\":\"world\"}"Encodes the value into key-value format with customizable delimiters. Default delimiters match
the logfmt format.
argument
Type
Description
default
required
value
object
The value to convert to a string.
N/A
yes
fields_ordering
array
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.
N/A
no
key_value_delimiter
string
The string that separates the key from the value.
=
no
field_delimiter
string
The string that separates each key-value pair.
N/A
no
flatten_boolean
boolean
Whether to encode key-value with a boolean value as a standalone key if true and nothing if false.
false
no
fields_ordering contains a non-string element.Source:
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return:
"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"Source:
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return:
"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"Source:
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return:
"agent.name=foo event=log log.file.path=my.log"Source:
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return:
"event=log log.file.path=my.log agent.name=foo"Source:
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
field_delimiter: ",",
key_value_delimiter: ":"
)
Return:
"lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"Source:
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
field_delimiter: ",",
key_value_delimiter: ":",
flatten_boolean: true
)
Return:
"beta,lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"Encodes the value to logfmt.
argument
Type
Description
default
required
value
object
The value to convert to a logfmt string.
N/A
yes
fields_ordering
array
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.
N/A
no
fields_ordering contains a non-string element.Source:
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return:
"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"Source:
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return:
"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"Source:
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return:
"agent.name=foo event=log log.file.path=my.log"Source:
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return:
"event=log log.file.path=my.log agent.name=foo"Encodes the value to Lz4.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
Source:
encoded_text = encode_lz4!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)
Return:
"LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4="Encodes a value with percent encoding to safely be used in URLs.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
ascii_set
string
The ASCII set to use when encoding the data.
NON_ALPHANUMERIC
no
Source:
encode_percent("foo bar?")
Return:
"foo%20bar%3F"Source:
encode_percent("foo bar", ascii_set: "CONTROLS")
Return:
"foo %09bar"Encodes the value into a protocol buffer payload.
argument
Type
Description
default
required
value
object
The object to convert to a protocol buffer payload.
N/A
yes
desc_file
string
The path to the protobuf descriptor set file. Must be a literal string.
This file is the output of protoc -o
N/A
yes
message_type
string
The name of the message type to use for serializing.
Must be a literal string.
N/A
yes
desc_file file does not exist.message_type message type does not exist in the descriptor file.Source:
.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))
Return:
"Cgdzb21lb25lIggKBjEyMzQ1Ng=="Encodes a value to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
validate
boolean
Whether to validate the input string to check if it is a valid domain name.
true
no
value can not be encoded to punycodeSource:
encode_punycode!("www.café.com")
Return:
"www.xn--caf-dma.com"Source:
encode_punycode!("www.CAFé.com")
Return:
"www.xn--caf-dma.com"Source:
encode_punycode!("www.cafe.com")
Return:
"www.cafe.com"Source:
encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return:
"xn--8hbb.xn--fiba.xn--8hbf.xn--eib."Encodes the value to Snappy.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
value cannot be encoded into a Snappy string.Source:
encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)
Return:
"LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg=="Encodes the value to Zlib.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
compression_level
integer
The default compression level.
6
no
Source:
encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)
Return:
"eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7"Encodes the value to Zstandard.
argument
Type
Description
default
required
value
string
The string to encode.
N/A
yes
compression_level
integer
The default compression level.
3
no
Source:
encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)
Return:
"KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ=="Coerces the value into a boolean.
argument
Type
Description
default
required
value
boolean, integer, float, null, string
The value to convert to a Boolean.
N/A
yes
value is not a supported boolean representation.Source:
to_bool!("yes")
Return:
trueSource:
to_bool(0.0)
Return:
falseSource:
to_bool(0)
Return:
falseSource:
to_bool(null)
Return:
falseSource:
to_bool(true)
Return:
trueCoerces the value into a float.
argument
Type
Description
default
required
value
integer, float, boolean, string, timestamp
The value to convert to a float. Must be convertible to a float, otherwise an error is raised.
N/A
yes
value is not a supported float representation.Source:
to_float!("3.145")
Return:
3.145Source:
to_float(t'2020-12-30T22:20:53.824727Z')
Return:
1609366853.824727Coerces the value into an integer.
argument
Type
Description
default
required
value
integer, float, boolean, string, timestamp, null
The value to convert to an integer.
N/A
yes
value is a string but the text is not an integer.value is not a string, int, or timestamp.Source:
to_int!("2")
Return:
2Source:
to_int(t'2020-12-30T22:20:53.824727Z')
Return:
1609366853Coerces the value into a regex.
argument
Type
Description
default
required
value
string
The value to convert to a regex.
N/A
yes
value is not a string.Source:
to_regex("^foo$") ?? r''
Return:
"^foo$"Coerces the value into a string.
argument
Type
Description
default
required
value
integer, float, boolean, string, timestamp, null
The value to convert to a string.
N/A
yes
value is not an integer, float, boolean, string, timestamp, or null.Source:
to_string(true)
Return:
"true"Source:
to_string(52)
Return:
"52"Source:
to_string(52.2)
Return:
"52.2"Converts the value integer from a Unix timestamp to a DPL timestamp.
Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
argument
Type
Description
default
required
value
integer
The Unix timestamp to convert.
N/A
yes
unit
string
The time unit.
seconds
no
Source:
from_unix_timestamp!(5)
Return:
"1970-01-01T00:00:05Z"Source:
from_unix_timestamp!(5000, unit: "milliseconds")
Return:
"1970-01-01T00:00:05Z"Source:
from_unix_timestamp!(5000, unit: "nanoseconds")
Return:
"1970-01-01T00:00:00.000005Z"Converts the value, a Syslog facility code, into its corresponding
Syslog keyword. For example, 0 into "kern", 1 into "user", etc.
argument
Type
Description
default
required
value
integer
The facility code.
N/A
yes
value is not a valid Syslog facility code.Source:
to_syslog_facility!(4)
Return:
"auth"Converts the value, a Syslog facility keyword, into a Syslog integer
facility code (0 to 23).
argument
Type
Description
default
required
value
string
The Syslog facility keyword to convert.
N/A
yes
value is not a valid Syslog facility keyword.Source:
to_syslog_facility_code!("authpriv")
Return:
10Converts the value, a Syslog severity level, into its corresponding keyword,
i.e. 0 into "emerg", 1 into "alert", etc.
argument
Type
Description
default
required
value
integer
The severity level.
N/A
yes
value isn’t a valid Syslog severity level.Source:
to_syslog_level!(5)
Return:
"notice"Converts the value, a Syslog log level keyword, into a Syslog integer
severity level (0 to 7).
argument
Type
Description
default
required
value
string
The Syslog level keyword to convert.
N/A
yes
value is not a valid Syslog level keyword.Source:
to_syslog_severity!("alert")
Return:
1Converts the value timestamp into a Unix timestamp.
Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
argument
Type
Description
default
required
value
timestamp
The timestamp to convert into a Unix timestamp.
N/A
yes
unit
string
The time unit.
seconds
no
Source:
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return:
1609459200Source:
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return:
1609459200000Source:
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return:
1609459200000000000Decrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
argument
Type
Description
default
required
ciphertext
string
The string in raw bytes (not encoded) to decrypt.
N/A
yes
algorithm
string
The algorithm to use.
N/A
yes
key
string
The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
N/A
yes
iv
string
The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
The value should match the one used during encryption.
N/A
yes
algorithm is not a supported algorithm.key length does not match the key size required for the algorithm specified.iv length does not match the iv size required for the algorithm specified.Source:
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)
Return:
"super_secret_message"Encrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
argument
Type
Description
default
required
plaintext
string
The string to encrypt.
N/A
yes
algorithm
string
The algorithm to use.
N/A
yes
key
string
The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
N/A
yes
iv
string
The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
N/A
yes
algorithm is not a supported algorithm.key length does not match the key size required for the algorithm specified.iv length does not match the iv size required for the algorithm specified.Source:
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
Return:
"GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE="Calculates a HMAC of the value using the given key.
The hashing algorithm used can be optionally specified.
For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.
This function is infallible if either the default algorithm value or a recognized-valid compile-time
algorithm string literal is used. Otherwise, it is fallible.
argument
Type
Description
default
required
value
string
The string to calculate the HMAC for.
N/A
yes
key
string
The string to use as the cryptographic key.
N/A
yes
algorithm
string
The hashing algorithm to use.
SHA-256
no
Source:
encode_base64(hmac("Hello there", "super-secret-key"))
Return:
"eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI="Source:
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))
Return:
"42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90"Source:
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
.hmac = encode_base16(hmac_bytes)
}
Return:
"78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662"Calculates an md5 hash of the value.
argument
Type
Description
default
required
value
string
The string to calculate the hash for.
N/A
yes
Source:
md5("foo")
Return:
"acbd18db4cc2f85cedef654fccc4a4d8"Calculates a Seahash hash of the value.
Note: Due to limitations in the underlying DPL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.
argument
Type
Description
default
required
value
string
The string to calculate the hash for.
N/A
yes
Source:
seahash("foobar")
Return:
5348458858952426000Source:
seahash("bar")
Return:
-2796170501982571500Calculates a SHA-1 hash of the value.
argument
Type
Description
default
required
value
string
The string to calculate the hash for.
N/A
yes
Source:
sha1("foo")
Return:
"0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"Calculates a SHA-2 hash of the value.
argument
Type
Description
default
required
value
string
The string to calculate the hash for.
N/A
yes
variant
string
The variant of the algorithm to use.
SHA-512/256
no
Source:
sha2("foo", variant: "SHA-512/224")
Return:
"d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be"Calculates a SHA-3 hash of the value.
argument
Type
Description
default
required
value
string
The string to calculate the hash for.
N/A
yes
variant
string
The variant of the algorithm to use.
SHA3-512
no
Source:
sha3("foo", variant: "SHA3-224")
Return:
"f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a"Asserts the condition, which must be a Boolean expression. The program is aborted with
message if the condition evaluates to false.
argument
Type
Description
default
required
condition
boolean
The condition to check.
N/A
yes
message
string
An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples below
for a fully formed log message sample.
N/A
no
condition evaluates to false.Source:
assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return:
trueSource:
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")
Asserts that two expressions, left and right, have the same value. The program is
aborted with message if they do not have the same value.
argument
Type
Description
default
required
left
any
The value to check for equality against right.
N/A
yes
right
any
The value to check for equality against left.
N/A
yes
message
string
An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples
below for a fully formed log message sample.
N/A
no
Source:
assert_eq!(1, 1)
Return:
trueSource:
assert_eq!(127, [1, 2, 3])
Source:
assert_eq!(1, 0, message: "Unequal integers")
Logs the value to stdout at the specified level.
argument
Type
Description
default
required
value
any
The value to log.
N/A
yes
level
string
The log level.
info
no
rate_limit_secs
integer
Specifies that the log message is output no more than once per the given number of seconds.
Use a value of 0 to turn rate limiting off.
1
no
Source:
log("Hello, World!", level: "info", rate_limit_secs: 60)
Return:
nullSource:
_, err = to_int(.field)
if err != null {
log(err, level: "error")
}
Return:
nullSearches an enrichment table for rows that match the provided condition.
For file enrichment tables, this condition needs to be a DPL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity
can be specified using the case_sensitive argument. An exact match search can use an
index directly into the dataset, which should make this search fairly “cheap” from a
performance perspective.
Date range search. The given field must be greater than or equal to the from date
and/or less than or equal to the to date. A date range search involves
sequentially scanning through the rows that have been located using any exact match
criteria. This can be an expensive operation if there are many rows returned by any exact
match criteria. Therefore, use date ranges as the only criteria when the enrichment
data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
ISP databases:
autonomous_system_numberautonomous_system_organizationisporganizationCity databases:
city_namecontinent_codecountry_codecountry_nameregion_coderegion_namemetro_codelatitudelongitudepostal_codetimezoneConnection-Type databases:
connection_typeTo use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
argument
Type
Description
default
required
table
string
The enrichment table to search.
N/A
yes
condition
object
The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.
N/A
yes
select
array
A subset of fields from the enrichment table to return. If not specified, all fields are returned.
N/A
no
case_sensitive
boolean
Whether text fields need to match cases exactly.
true
no
Source:
find_enrichment_table_records!("test",
{
"surname": "smith",
},
case_sensitive: false)
Return:
[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]Source:
find_enrichment_table_records!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})
Return:
[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.
For file enrichment tables, this condition needs to be a DPL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity
can be specified using the case_sensitive argument. An exact match search can use an
index directly into the dataset, which should make this search fairly “cheap” from a
performance perspective.
Date range search. The given field must be greater than or equal to the from date
and/or less than or equal to the to date. A date range search involves
sequentially scanning through the rows that have been located using any exact match
criteria. This can be an expensive operation if there are many rows returned by any exact
match criteria. Therefore, use date ranges as the only criteria when the enrichment
data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
ISP databases:
autonomous_system_numberautonomous_system_organizationisporganizationCity databases:
city_namecontinent_codecountry_codecountry_nameregion_coderegion_namemetro_codelatitudelongitudepostal_codetimezoneConnection-Type databases:
connection_typeTo use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
argument
Type
Description
default
required
table
string
The enrichment table to search.
N/A
yes
condition
object
The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.
N/A
yes
select
array
A subset of fields from the enrichment table to return. If not specified, all fields are returned.
N/A
no
case_sensitive
boolean
Whether the text fields match the case exactly.
true
no
Source:
get_enrichment_table_record!("test",
{
"surname": "bob",
"firstname": "John"
},
case_sensitive: false)
Return:
{"id":1,"firstname":"Bob","surname":"Smith"}Source:
get_enrichment_table_record!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})
Return:
{"id":1,"firstname":"Bob","surname":"Smith"}Compacts the value by removing empty values, where empty values are defined using the
available parameters.
argument
Type
Description
default
required
value
array, object
The object or array to compact.
N/A
yes
recursive
boolean
Whether the compaction be recursive.
true
no
null
boolean
Whether null should be treated as an empty value.
true
no
string
boolean
Whether an empty string should be treated as an empty value.
true
no
object
boolean
Whether an empty object should be treated as an empty value.
true
no
array
boolean
Whether an empty array should be treated as an empty value.
true
no
Source:
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return:
["foo","bar","buzz"]Source:
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return:
{"field1":1}Filter elements from a collection.
This function currently does not support recursive iteration.
The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
argument
Type
Description
default
required
value
array, object
The array or object to filter.
N/A
yes
Source:
filter(array!(.tags)) -> |_index, value| {
# keep any elements that aren't equal to "foo"
value != "foo"
}
Return:
["bar","baz"]Flattens the value into a single-level representation.
argument
Type
Description
default
required
value
array, object
The array or object to flatten.
N/A
yes
separator
string
The separator to join nested keys
.
no
Source:
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return:
[1,2,3,4,5,6,7,8,9]Source:
flatten({
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
})
Return:
{"parent1.child1":1,"parent1.child2":2,"parent2.child3":3}Iterate over a collection.
This function currently does not support recursive iteration.
The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
argument
Type
Description
default
required
value
array, object
The array or object to iterate.
N/A
yes
Source:
tally = {}
for_each(array!(.tags)) -> |_index, value| {
# Get the current tally for the `value`, or
# set to `0`.
count = int(get!(tally, [value])) ?? 0
# Increment the tally for the value by `1`.
tally = set!(tally, [value], count + 1)
}
tally
Return:
{"foo":2,"bar":1,"baz":1}Determines whether the value array includes the specified item.
argument
Type
Description
default
required
value
array
The array.
N/A
yes
item
any
The item to check.
N/A
yes
Source:
includes(["apple", "orange", "banana"], "banana")
Return:
trueReturns the keys from the object passed into the function.
argument
Type
Description
default
required
value
object
The object to extract keys from.
N/A
yes
Source:
keys({"key1": "val1", "key2": "val2"})
Return:
["key1","key2"]Returns the length of the value.
value is an array, returns the number of elements.value is an object, returns the number of top-level keys.value is a string, returns the number of bytes in the string. If
you want the number of characters, see strlen.argument
Type
Description
default
required
value
array, object, string
The array or object.
N/A
yes
Source:
length({
"portland": "Trail Blazers",
"seattle": "Supersonics"
})
Return:
2Source:
length({
"home": {
"city": "Portland",
"state": "Oregon"
},
"name": "Trail Blazers",
"mascot": {
"name": "Blaze the Trail Cat"
}
})
Return:
3Source:
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return:
3Source:
length("The Planet of the Apes Musical")
Return:
30Map the keys within an object.
If recursive is enabled, the function iterates into nested
objects, using the following rules:
The above rules mean that map_keys with
recursive enabled finds all keys in the target,
regardless of whether nested objects are nested inside arrays.
The function uses the function closure syntax to allow reading the key for each item in the object.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
argument
Type
Description
default
required
value
object
The object to iterate.
N/A
yes
recursive
boolean
Whether to recursively iterate the collection.
false
no
Source:
map_keys(.) -> |key| { upcase(key) }
Return:
{"FOO":"foo","BAR":"bar"}Source:
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
Return:
{"labels":{"app_kubernetes_io/name":"mysql"}}Map the values within a collection.
If recursive is enabled, the function iterates into nested
collections, using the following rules:
The function uses the function closure syntax to allow mutating the value for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
argument
Type
Description
default
required
value
array, object
The object or array to iterate.
N/A
yes
recursive
boolean
Whether to recursively iterate the collection.
false
no
Source:
map_values(.) -> |value| { upcase!(value) }
Return:
{"foo":"FOO","bar":"BAR"}Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.
argument
Type
Description
default
required
value
array
The array.
N/A
yes
pattern
regex
The regular expression pattern to match against.
N/A
yes
all
boolean
Whether to match on all elements of value.
false
no
Source:
match_array(["foobar", "bazqux"], r'foo')
Return:
trueSource:
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return:
trueSource:
match_array(["bazqux", "xyz"], r'foo')
Return:
falseSource:
match_array(["foo", "foobar", "baz"], r'foo', all: true)
Return:
falseReturns the number of UTF-8 characters in value. This differs from
length which counts the number of bytes of a string.
Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.
argument
Type
Description
default
required
value
string
The string.
N/A
yes
Source:
strlen("ñandú")
Return:
5Unflattens the value into a nested representation.
argument
Type
Description
default
required
value
object
The array or object to unflatten.
N/A
yes
separator
string
The separator to split flattened keys.
.
no
recursive
boolean
Whether to recursively unflatten the object values.
true
no
Source:
unflatten({
"foo.bar.baz": true,
"foo.bar.qux": false,
"foo.quux": 42
})
Return:
{"foo":{"bar":{"baz":true,"qux":false},"quux":42}}Source:
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
})
Return:
{"flattened":{"parent":{"foo":{"bar":true,"baz":false}}}}Source:
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
}, recursive: false)
Return:
{"flattened":{"parent":{"foo.bar":true,"foo.baz":false}}}Source:
unflatten({
"a": 3,
"a.b": 2,
"a.c": 4
})
Return:
{"a":{"b":2,"c":4}}Returns the unique values for an array.
The first occurrence of each element is kept.
argument
Type
Description
default
required
value
array
The array to return unique elements from.
N/A
yes
Source:
unique(["foo", "bar", "foo", "baz"])
Return:
["foo","bar","baz"]Returns the values from the object passed into the function.
argument
Type
Description
default
required
value
object
The object to extract values from.
N/A
yes
Source:
values({"key1": "val1", "key2": "val2"})
Return:
["val1","val2"]Decrypts an IP address encrypted with encrypt_ip, returning the original IP address.
Supported modes:
aes128 - Reverses AES-128 ipcrypt-deterministic encryption. Key must be exactly 16 bytes.pfx - Reverses prefix-preserving ipcrypt-pfx encryption. Key must be exactly 32 bytes.The key and mode must match those used during encryption.
argument
Type
Description
default
required
ip
string
The encrypted IP address to decrypt (IPv4 or IPv6).
N/A
yes
key
string
The decryption key as raw bytes. Must match the key used for encryption: 16 bytes for aes128, 32 bytes for pfx.
N/A
yes
mode
string
The decryption mode: aes128 or pfx. Must match the mode used for encryption.
N/A
yes
ip is not a valid IP address.mode is not a supported mode (aes128 or pfx).key length does not match the mode requirements (16 bytes for aes128, 32 bytes for pfx).Source:
decrypt_ip!("72b9:a747:f2e9:72af:76ca:5866:6dcf:c3b0", key: "sixteen byte key", mode: "aes128")
Return:
"192.168.1.1"Encrypts an IP address, transforming it into a different valid IP address of the same version.
Supported modes:
aes128 - Scrambles the IP using AES-128 encryption (ipcrypt-deterministic). Accepts IPv4 or IPv6; key must be exactly 16 bytes.pfx - Prefix-preserving encryption (ipcrypt-pfx), that maintains network hierarchy so addresses in the same subnet encrypt to addresses sharing the same prefix. Key must be exactly 32 bytes.Encryption is deterministic: the same input IP, key, and mode always produce the same output.
argument
Type
Description
default
required
ip
string
The IP address to encrypt (IPv4 or IPv6).
N/A
yes
key
string
The encryption key as raw bytes. Must be 16 bytes for aes128 mode or 32 bytes for pfx mode.
N/A
yes
mode
string
The encryption mode: aes128 or pfx.
N/A
yes
ip is not a valid IP address.mode is not a supported mode (aes128 or pfx).key length does not match the mode requirements (16 bytes for aes128, 32 bytes for pfx).Source:
encrypt_ip!("192.168.1.1", key: "sixteen byte key", mode: "aes128")
Return:
"72b9:a747:f2e9:72af:76ca:5866:6dcf:c3b0"Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.
This behavior mimics inet_aton.
argument
Type
Description
default
required
value
string
The IP address to convert to binary.
N/A
yes
value is not a valid IPv4 address.Source:
ip_aton!("1.2.3.4")
Return:
16909060Determines whether the ip is contained in the block referenced by the cidr.
argument
Type
Description
default
required
cidr
string, array
The CIDR mask (v4 or v6).
N/A
yes
ip
string
The IP address (v4 or v6).
N/A
yes
cidr is not a valid CIDR.ip is not a valid IP address.Source:
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return:
trueSource:
ip_cidr_contains!(["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"], "192.168.10.32")
Return:
trueSource:
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return:
trueConverts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.
This behavior mimics inet_ntoa.
argument
Type
Description
default
required
value
string
The integer representation of an IPv4 address.
N/A
yes
value cannot fit in an unsigned 32-bit integer.Source:
ip_ntoa!(16909060)
Return:
"1.2.3.4"Converts IPv4 and IPv6 addresses from binary to text form.
This behavior mimics inet_ntop.
argument
Type
Description
default
required
value
string
The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long.
N/A
yes
value must be of length 4 or 16 bytes.Source:
ip_ntop!(decode_base64!("wKgAAQ=="))
Return:
"192.168.0.1"Source:
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))
Return:
"2001:db8:85a3::8a2e:370:7334"Converts IPv4 and IPv6 addresses from text to binary form.
This behavior mimics inet_pton.
argument
Type
Description
default
required
value
string
The IP address (v4 or v6) to convert to binary form.
N/A
yes
value is not a valid IP (v4 or v6) address in text form.Source:
encode_base64(ip_pton!("192.168.0.1"))
Return:
"wKgAAQ=="Source:
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))
Return:
"IAENuIWjAAAAAIouA3BzNA=="Extracts the subnet address from the ip using the supplied subnet.
argument
Type
Description
default
required
ip
string
The IP address (v4 or v6).
N/A
yes
subnet
string
The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask
like 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.
N/A
yes
ip is not a valid IP address.subnet is not a valid subnet.Source:
ip_subnet!("192.168.10.32", "255.255.255.0")
Return:
"192.168.10.0"Source:
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return:
"2404:6800::"Converts the ip to an IPv6 address.
argument
Type
Description
default
required
ip
string
The IP address to convert to IPv6.
N/A
yes
ip is not a valid IP address.Source:
ip_to_ipv6!("192.168.10.32")
Return:
"::ffff:192.168.10.32"Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is
currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.
argument
Type
Description
default
required
ip
string
The IPv4-mapped IPv6 address to convert.
N/A
yes
ip is not a valid IP address.ip is an IPv6 address that is not compatible with IPv4.Source:
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return:
"192.168.0.1"Check if the string is a valid IPv4 address or not.
An IPv4-mapped or IPv4-compatible IPv6 address is not considered valid for the purpose of this function.
argument
Type
Description
default
required
value
string
The IP address to check
N/A
yes
Source:
is_ipv4("10.0.102.37")
Return:
trueSource:
is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return:
falseSource:
is_ipv4("foobar")
Return:
falseCheck if the string is a valid IPv6 address or not.
argument
Type
Description
default
required
value
string
The IP address to check
N/A
yes
Source:
is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return:
trueSource:
is_ipv6("10.0.102.37")
Return:
falseSource:
is_ipv6("foobar")
Return:
falseCalculates the haversine great-circle distance and bearing between two geographic coordinates.
argument
Type
Description
default
required
latitude1
float
Latitude of the first point in decimal degrees.
N/A
yes
longitude1
float
Longitude of the first point in decimal degrees.
N/A
yes
latitude2
float
Latitude of the second point in decimal degrees.
N/A
yes
longitude2
float
Longitude of the second point in decimal degrees.
N/A
yes
measurement_unit
string
Unit for the returned distance: kilometers or miles.
kilometers
no
Source:
haversine(0.0, 0.0, 10.0, 10.0)
Return:
{"bearing":44.561,"distance":1568.5227233}Source:
haversine(0.0, 0.0, 10.0, 10.0, measurement_unit: "miles")
Return:
{"bearing":44.561,"distance":974.6348468}Computes the absolute value of value.
argument
Type
Description
default
required
value
integer, float
The number to calculate the absolute value.
N/A
yes
Source:
abs(-42)
Return:
42Source:
abs(-42.2)
Return:
42.2Rounds the value up to the specified precision.
argument
Type
Description
default
required
value
integer, float
The number to round up.
N/A
yes
precision
integer
The number of decimal places to round to.
0
no
Source:
ceil(4.345)
Return:
5Source:
ceil(4.345, precision: 2)
Return:
4.35Rounds the value down to the specified precision.
argument
Type
Description
default
required
value
integer, float
The number to round down.
N/A
yes
precision
integer
The number of decimal places to round to.
0
no
Source:
floor(4.345)
Return:
4Source:
floor(4.345, precision: 2)
Return:
4.34Formats the integer value into a string representation using the given base/radix.
argument
Type
Description
default
required
value
integer
The number to format.
N/A
yes
base
integer
The base to format the number in. Must be between 2 and 36 (inclusive).
10
no
Source:
format_int!(42, 16)
Return:
"2a"Source:
format_int!(-42, 16)
Return:
"-2a"Formats the value into a string representation of the number.
argument
Type
Description
default
required
value
integer, float
The number to format as a string.
N/A
yes
scale
integer
The number of decimal places to display.
N/A
no
decimal_separator
string
The character to use between the whole and decimal parts of the number.
.
no
grouping_separator
string
The character to use between each thousands part of the number.
,
no
Source:
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return:
"1,234,567.890"Calculates the remainder of value divided by modulus.
argument
Type
Description
default
required
value
integer, float
The value the modulus is applied to.
N/A
yes
modulus
integer, float
The modulus value.
N/A
yes
value is not an integer or float.modulus is not an integer or float.modulus is equal to 0.Source:
mod(5, 2)
Return:
1Rounds the value to the specified precision.
argument
Type
Description
default
required
value
integer, float
The number to round.
N/A
yes
precision
integer
The number of decimal places to round to.
0
no
Source:
round(4.345)
Return:
4Source:
round(4.345, precision: 2)
Return:
4.35Matches an object against a Datadog Search Syntax query.
argument
Type
Description
default
required
value
object
The object.
N/A
yes
query
string
The Datadog Search Syntax query.
N/A
yes
Source:
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return:
trueSource:
match_datadog_query({"message": "contains only this"}, "this AND that")
Return:
falseSource:
match_datadog_query({"name": "foobar"}, "@name:foo*")
Return:
trueSource:
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return:
trueMerges the from object into the to object.
argument
Type
Description
default
required
to
object
The object to merge into.
N/A
yes
from
object
The object to merge from.
N/A
yes
deep
boolean
A deep merge is performed if true, otherwise only top-level fields are merged.
false
no
Source:
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
}
)
Return:
{"parent1":{"child2":4,"child5":5},"parent2":{"child3":3}}Source:
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
},
deep: true
)
Return:
{"parent1":{"child1":1,"child2":4,"child5":5},"parent2":{"child3":3}}Iterate over either one array of arrays or a pair of arrays and create an object out of all the key-value pairs contained in them.
With one array of arrays, any entries with no value use null instead.
Any keys that are null skip the corresponding value.
If a single parameter is given, it must contain an array of all the input arrays.
argument
Type
Description
default
required
values
array
The first array of elements, or the array of input arrays if no other parameter is present.
N/A
yes
keys
array
The second array of elements. If not present, the first parameter must contain all the arrays.
N/A
no
values and keys must be arrays.keys is not present, values must contain only arrays.Source:
object_from_array([["one", 1], [null, 2], ["two", 3]])
Return:
{"one":1,"two":3}Source:
object_from_array([1, 2, 3], keys: ["one", null, "two"])
Return:
{"one":1,"two":3}Unnest an array field from an object to create an array of objects using that field; keeping all other fields.
Assigning the array result of this to . results in multiple events being emitted from remap. See the
remap transform docs for more details.
This is also referred to as explode in some languages.
argument
Type
Description
default
required
path
path
The path of the field to unnest.
N/A
yes
Source:
. = unnest!(.messages)
Source:
. = unnest!(.event.messages)
Parses Apache access and error log lines. Lines can be in common,
combined, or the default error format.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
timestamp_format
string
The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone.
%d/%b/%Y:%T %z
no
format
string
The format to use for parsing the log.
N/A
yes
value does not match the specified format.timestamp_format is not a valid format string.value fails to parse using the provided timestamp_format.Source:
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}Source:
parse_apache_log!(
s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
"combined",
)
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326,"referrer":"http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth","agent":"Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"}Source:
parse_apache_log!(
s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
"error"
)
Return:
{"client":"147.159.108.175","message":"I will bypass the haptic COM bandwidth, that should matrix the CSS driver!","module":"ab","pid":4803,"port":24259,"severity":"alert","thread":"3814","timestamp":"2021-03-01T12:00:19Z"}Parses value in the Elastic Load Balancer Access format.
argument
Type
Description
default
required
value
string
Access log of the Application Load Balancer.
N/A
yes
value is not a properly formatted AWS ALB log.Source:
parse_aws_alb_log!(
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return:
{"type":"http","timestamp":"2018-11-30T22:23:00.186641Z","elb":"app/my-loadbalancer/50dc6c495c0c9188","client_host":"192.168.131.39:2817","target_host":null,"request_processing_time":0,"target_processing_time":0.001,"response_processing_time":0,"elb_status_code":"200","target_status_code":"200","received_bytes":34,"sent_bytes":366,"request_method":"GET","request_url":"http://www.example.com:80/","request_protocol":"HTTP/1.1","user_agent":"curl/7.46.0","ssl_cipher":null,"ssl_protocol":null,"target_group_arn":"arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067","trace_id":"Root=1-58337364-23a8c76965a2ef7629b185e3","traceability_id":null,"domain_name":null,"chosen_cert_arn":null,"matched_rule_priority":"0","request_creation_time":"2018-11-30T22:22:48.364000Z","actions_executed":"forward","redirect_url":null,"error_reason":null,"target_port_list":[],"target_status_code_list":[],"classification":null,"classification_reason":null}Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the
aws_kinesis_firehose source.
argument
Type
Description
default
required
value
string
The string representation of the message to parse.
N/A
yes
value is not a properly formatted AWS CloudWatch Log subscription message.Source:
parse_aws_cloudwatch_log_subscription_message!(.message)
Return:
{"owner":"111111111111","message_type":"DATA_MESSAGE","log_group":"test","log_stream":"test","subscription_filters":["Destination"],"log_events":[{"id":"35683658089614582423604394983260738922885519999578275840","message":"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}","timestamp":"2020-09-14T19:09:29.039Z"}]}Parses value in the VPC Flow Logs format.
argument
Type
Description
default
required
value
string
VPC Flow Log.
N/A
yes
format
string
VPC Flow Log format.
N/A
no
value is not a properly formatted AWS VPC Flow log.Source:
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return:
{"version":2,"account_id":"123456789010","interface_id":"eni-1235b8ca123456789","srcaddr":null,"dstaddr":null,"srcport":null,"dstport":null,"protocol":null,"packets":null,"bytes":null,"start":1431280876,"end":1431280934,"action":null,"log_status":"NODATA"}Source:
parse_aws_vpc_flow_log!(
"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return:
{"instance_id":null,"interface_id":"eni-1235b8ca123456789","srcaddr":"10.0.1.5","dstaddr":"10.0.0.220","pkt_srcaddr":"10.0.1.5","pkt_dstaddr":"203.0.113.5"}Source:
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")
Return:
{"account_id":"123456789012","action":"ACCEPT","az_id":"apse2-az3","bytes":15044,"dstaddr":"10.0.0.71","dstport":34210,"end":1616729349,"flow_direction":"ingress","instance_id":"i-0c50d5961bcb2d47b","interface_id":"eni-1235b8ca123456789","log_status":"OK","packets":14,"pkt_dst_aws_service":null,"pkt_dstaddr":"10.0.0.71","pkt_src_aws_service":"S3","pkt_srcaddr":"52.95.128.179","protocol":6,"region":"ap-southeast-2","srcaddr":"52.95.128.179","srcport":80,"start":1616729292,"sublocation_id":null,"sublocation_type":null,"subnet_id":"subnet-aaaaaaaa012345678","tcp_flags":19,"traffic_path":null,"type":"IPv4","version":5,"vpc_id":"vpc-abcdefab012345678"}Parses the value into a human-readable bytes format specified by unit and base.
argument
Type
Description
default
required
value
string
The string of the duration with either binary or SI unit.
N/A
yes
unit
string
The output units for the byte.
N/A
yes
base
string
The base for the byte, either 2 or 10.
2
no
value is not a properly formatted bytes.Source:
parse_bytes!("1024KiB", unit: "MiB")
Return:
1Source:
parse_bytes!("4TB", unit: "MB", base: "10")
Return:
4000000Source:
parse_bytes!("1GB", unit: "B", base: "2")
Return:
1073741824Parses the value as CBOR.
argument
Type
Description
default
required
value
string
The CBOR payload to parse.
N/A
yes
value is not a valid CBOR-formatted payload.Source:
parse_cbor!(decode_base64!("oWVmaWVsZGV2YWx1ZQ=="))
Return:
{"field":"value"}Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
translate_custom_fields
boolean
Toggles translation of custom field pairs to key:value.
N/A
no
value is not a properly formatted CEF string.Source:
parse_cef!(
"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
Return:
{"cefVersion":"0","deviceVendor":"CyberArk","deviceProduct":"PTA","deviceVersion":"12.6","deviceEventClassId":"1","name":"Suspected credentials theft","severity":"8","suser":"mike2@prod1.domain.com","shost":"prod1.domain.com","src":"1.1.1.1","duser":"andy@dev1.domain.com","dhost":"dev1.domain.com","dst":"2.2.2.2","cs1Label":"ExtraData","cs1":"None","cs2Label":"EventID","cs2":"52b06812ec3500ed864c461e","deviceCustomDate1Label":"detectionDate","deviceCustomDate1":"1388577900000","cs3Label":"PTAlink","cs3":"https://1.1.1.1/incidents/52b06812ec3500ed864c461e","cs4Label":"ExternalLink","cs4":"None"}Source:
parse_cef!(
"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
Return:
{"cefVersion":"1","deviceVendor":"Security","deviceProduct":"threatmanager","deviceVersion":"1.0","deviceEventClassId":"100","name":"worm successfully stopped","severity":"10","src":"10.0.0.1","dst":"2.1.2.2","spt":"1232"}Source:
parse_cef!(
"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
translate_custom_fields: true
)
Return:
{"cefVersion":"0","deviceVendor":"Dev","deviceProduct":"firewall","deviceVersion":"2.2","deviceEventClassId":"1","name":"Connection denied","severity":"5","Device IPv6 Address":"2345:0425:2CA1:0000:0000:0567:5673:23b5"}Parses the value using the Common Log Format (CLF).
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
value does not match the Common Log Format.timestamp_format is not a valid format string.value fails to parse using the provided timestamp_format.Source:
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}Source:
parse_common_log!(
"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"%+"
)
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
delimiter
string
The field delimiter to use when parsing. Must be a single-byte utf8 character.
,
no
value is not a valid CSV string.Source:
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return:
["foo","bar","foo \", bar"]Source:
parse_csv!("foo bar", delimiter: " ")
Return:
["foo","bar"]Parses the value as base64 encoded DNSTAP data.
argument
Type
Description
default
required
value
string
The base64 encoded representation of the DNSTAP data to parse.
N/A
yes
lowercase_hostnames
boolean
Whether to turn all hostnames found in resulting data lowercase, for consistency.
false
no
value is not a valid base64 encoded string.valueSource:
parse_dnstap!("ChVqYW1lcy1WaXJ0dWFsLU1hY2hpbmUSC0JJTkQgOS4xNi4zGgBy5wEIAxACGAEiEAAAAAAAAAAAAAAAAAAAAAAqECABBQJwlAAAAAAAAAAAADAw8+0CODVA7+zq9wVNMU3WNlI2kwIAAAABAAAAAAABCWZhY2Vib29rMQNjb20AAAEAAQAAKQIAAACAAAAMAAoACOxjCAG9zVgzWgUDY29tAGAAbQAAAAByZLM4AAAAAQAAAAAAAQJoNQdleGFtcGxlA2NvbQAABgABAAApBNABAUAAADkADwA1AAlubyBTRVAgbWF0Y2hpbmcgdGhlIERTIGZvdW5kIGZvciBkbnNzZWMtZmFpbGVkLm9yZy54AQ==")
Return:
{"dataType":"Message","dataTypeId":1,"extraInfo":"","messageType":"ResolverQuery","messageTypeId":3,"queryZone":"com.","requestData":{"fullRcode":0,"header":{"aa":false,"ad":false,"anCount":0,"arCount":1,"cd":false,"id":37634,"nsCount":0,"opcode":0,"qdCount":1,"qr":0,"ra":false,"rcode":0,"rd":false,"tc":false},"opt":{"do":true,"ednsVersion":0,"extendedRcode":0,"options":[{"optCode":10,"optName":"Cookie","optValue":"7GMIAb3NWDM="}],"udpPayloadSize":512},"question":[{"class":"IN","domainName":"facebook1.com.","questionType":"A","questionTypeId":1}],"rcodeName":"NoError"},"responseData":{"fullRcode":16,"header":{"aa":false,"ad":false,"anCount":0,"arCount":1,"cd":false,"id":45880,"nsCount":0,"opcode":0,"qdCount":1,"qr":0,"ra":false,"rcode":16,"rd":false,"tc":false},"opt":{"do":false,"ednsVersion":1,"extendedRcode":1,"ede":[{"extraText":"no SEP matching the DS found for dnssec-failed.org.","infoCode":9,"purpose":"DNSKEY Missing"}],"udpPayloadSize":1232},"question":[{"class":"IN","domainName":"h5.example.com.","questionType":"SOA","questionTypeId":6}],"rcodeName":"BADSIG"},"responseAddress":"2001:502:7094::30","responsePort":53,"serverId":"james-Virtual-Machine","serverVersion":"BIND 9.16.3","socketFamily":"INET6","socketProtocol":"UDP","sourceAddress":"::","sourcePort":46835,"time":1593489007920014000,"timePrecision":"ns","timestamp":"2020-06-30T03:50:07.920014129Z"}Parses the value into a human-readable duration format specified by unit.
argument
Type
Description
default
required
value
string
The string of the duration.
N/A
yes
unit
string
The output units for the duration.
N/A
yes
value is not a properly formatted duration.Source:
parse_duration!("1005ms", unit: "s")
Return:
1.005Source:
parse_duration!("1s 1ms", unit: "ms")
Return:
1001Parses the eTLD from value representing domain name.
argument
Type
Description
default
required
value
string
The domain string.
N/A
yes
plus_parts
integer
Can be provided to get additional parts of the domain name. When 1 is passed, eTLD+1 will be returned, which represents a domain registrable by a single organization. Higher numbers will return subdomains.
false
no
psl
string
Can be provided to use a different public suffix list.
By default, https://publicsuffix.org/list/public_suffix_list.dat is used.
false
no
valueSource:
parse_etld!("sub.sussex.ac.uk")
Return:
{"etld":"ac.uk","etld_plus":"ac.uk","known_suffix":true}Source:
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
Return:
{"etld":"ac.uk","etld_plus":"sussex.ac.uk","known_suffix":true}Source:
parse_etld!("vector.acmecorp")
Return:
{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}Source:
parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")
Return:
{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}Parses the value using the glog (Google Logging Library) format.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
value does not match the glog format.Source:
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return:
{"level":"info","timestamp":"2021-01-31T14:48:54.411655Z","id":15520,"file":"main.c++","line":9,"message":"Hello world!"}Parses the value using the grok format. All patterns listed here
are supported.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
value fails to parse using the provided pattern.Source:
parse_grok!(
"2020-10-02T23:22:12.223222Z info Hello world",
"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return:
{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}Parses the value using multiple grok patterns. All patterns listed here
are supported.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
aliases
object
The shared set of grok aliases that can be referenced in the patterns to simplify them.
true
no
alias_sources
string
Path to the file containing aliases in a JSON format.
N/A
no
value fails to parse using the provided pattern.patterns is not an array.aliases is not an object.alias_sources is not a string or doesn’t point to a valid file.Source:
parse_groks!(
"2020-10-02T23:22:12.223222Z info Hello world",
patterns: [
"%{common_prefix} %{_status} %{_message}",
"%{common_prefix} %{_message}",
],
aliases: {
"common_prefix": "%{_timestamp} %{_loglevel}",
"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
"_loglevel": "%{LOGLEVEL:level}",
"_status": "%{POSINT:status}",
"_message": "%{GREEDYDATA:message}"
}
)
Return:
{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}Source:
parse_groks!(
"username=foo",
patterns: [ "%{PATTERN_A}" ],
alias_sources: [ "path/to/aliases.json" ]
)
# aliases.json contents:
# {
# "PATTERN_A": "%{PATTERN_B}",
# "PATTERN_B": "username=%{USERNAME:username}"
# }
Parses the value as an InfluxDB line protocol
string, producing a list of Vector-compatible metrics.
argument
Type
Description
default
required
value
string
The string representation of the InfluxDB line protocol to parse.
N/A
yes
value is not a valid InfluxDB line protocol string.string.NaN field value.Source:
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")
Return:
[{"name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":64}},{"name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":10}},{"name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":50.5}},{"name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":1}},{"name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":0}}]Parses the string value representing a number in an optional base/radix to an integer.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
base
integer
The base the number is in. Must be between 2 and 36 (inclusive).
If unspecified, the string prefix is used to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise.
N/A
no
Source:
parse_int!("-42")
Return:
-42Source:
parse_int!("0b1001")
Return:
9Source:
parse_int!("0o42")
Return:
34Source:
parse_int!("0x2a")
Return:
42Source:
parse_int!("2a", 17)
Return:
44Parses the value as JSON.
argument
Type
Description
default
required
value
string
The string representation of the JSON to parse.
N/A
yes
max_depth
integer
Number of layers to parse for nested JSON-formatted documents. The value must be in the range of 1 to 128.
N/A
no
lossy
boolean
Whether to parse the JSON in a lossy manner. Replaces invalid UTF-8 characters
with the Unicode character � (U+FFFD) if set to true, otherwise returns an error
if there are any invalid UTF-8 characters present.
true
no
value is not a valid JSON-formatted payload.Source:
parse_json!("{\"key\": \"val\"}")
Return:
{"key":"val"}Source:
parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)
Return:
{"top_level":"{\"key\": \"val\"}"}Parses the value in key-value format. Also known as logfmt.
"." characters can be escaped using \.argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
key_value_delimiter
string
The string that separates the key from the value.
=
no
field_delimiter
string
The string that separates each key-value pair.
N/A
no
whitespace
string
Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter.
lenient
no
accept_standalone_key
boolean
Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value true.
true
no
value is not a properly formatted key-value string.Source:
parse_key_value!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return:
{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}Source:
parse_key_value!(
"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
field_delimiter: ",",
key_value_delimiter: ":"
)
Return:
{"path":"/cart_link","host":"store.app.com","fwd":"102.30.171.16","dyno":"web.1","connect":"0ms","service":"87ms","status":"304","bytes":"632","protocol":"https"}Source:
parse_key_value!(
"env:prod,service:backend,region:eu-east1,beta",
field_delimiter: ",",
key_value_delimiter: ":",
)
Return:
{"env":"prod","service":"backend","region":"eu-east1","beta":true}Source:
parse_key_value!(
"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
field_delimiter: ",",
key_value_delimiter: "=",
)
Return:
{"at":"info","method":"GET","path":"/index","status":"200","tags":["dev","dummy"]}Parses the value using the klog format used by Kubernetes components.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
value does not match the klog format.Source:
parse_klog!("I0505 17:59:40.692994 28133 klog.go:70] hello from klog")
Return:
{"file":"klog.go","id":28133,"level":"info","line":70,"message":"hello from klog","timestamp":"2025-05-05T17:59:40.692994Z"}Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or
/var/log/secure (for RedHat-based systems) according to Syslog format.
argument
Type
Description
default
required
value
string
The text containing the message to parse.
N/A
yes
value is not a properly formatted Syslog message.Source:
parse_linux_authorization!(
s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return:
{"appname":"sshd","hostname":"localhost","message":"Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar","procid":1111,"timestamp":"2025-03-23T01:49:58Z"}Parses the value in logfmt.
" character." characters can be escaped by the \ character.parse_logfmt function
accepts standalone keys and assigns them a Boolean value of true.argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
value is not a properly formatted key-value stringSource:
parse_logfmt!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return:
{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}Parses Nginx access and error log lines. Lines can be in [`combined`](https://nginx.org/en/docs/http/ngx_http_log_module.html),
[`ingress_upstreaminfo`](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/), [`main`](https://hg.nginx.org/pkg-oss/file/tip/debian/debian/nginx.conf) or [`error`](https://github.com/nginx/nginx/blob/branches/stable-1.18/src/core/ngx_log.c#L102) format.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
timestamp_format
string
The date/time format to use for encoding the timestamp. The time is parsed
in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for
combined logs and %Y/%m/%d %H:%M:%S for error logs.
%d/%b/%Y:%T %z
no
format
string
The format to use for parsing the log.
N/A
yes
value does not match the specified format.timestamp_format is not a valid format string.value fails to parse using the provided timestamp_format.Source:
parse_nginx_log!(
s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
"combined",
)
Return:
{"agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36","client":"172.17.0.1","compression":"2.75","referer":"http://localhost/somewhere","request":"POST /not-found HTTP/1.1","size":153,"status":404,"timestamp":"2021-04-01T12:02:31Z","user":"alice"}Source:
parse_nginx_log!(
s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
"error"
)
Return:
{"timestamp":"2021-04-01T13:02:31Z","severity":"error","pid":31,"tid":31,"cid":1,"message":"open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)","client":"172.17.0.1","server":"localhost","request":"POST /not-found HTTP/1.1","host":"localhost:8081"}Source:
parse_nginx_log!(
s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
"ingress_upstreaminfo"
)
Return:
{"body_bytes_size":12312,"http_referer":"https://10.0.0.1/some/referer","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","proxy_alternative_upstream_name":"some-other-upstream-5000","proxy_upstream_name":"some-upstream-service-9000","remote_addr":"0.0.0.0","remote_user":"bob","req_id":"752178adb17130b291aefd8c386279e7","request":"GET /some/path HTTP/2.0","request_length":462,"request_time":0.05,"status":200,"timestamp":"2023-03-18T15:00:00Z","upstream_addr":"10.0.50.80:9000","upstream_response_length":19437,"upstream_response_time":0.049,"upstream_status":200}Source:
parse_nginx_log!(
s'172.24.0.3 - alice [31/Dec/2024:17:32:06 +0000] "GET / HTTP/1.1" 200 615 "https://domain.tld/path" "curl/8.11.1" "1.2.3.4, 10.10.1.1"',
"main"
)
Return:
{"body_bytes_size":615,"http_referer":"https://domain.tld/path","http_user_agent":"curl/8.11.1","http_x_forwarded_for":"1.2.3.4, 10.10.1.1","remote_addr":"172.24.0.3","remote_user":"alice","request":"GET / HTTP/1.1","status":200,"timestamp":"2024-12-31T17:32:06Z"}Parses the value as a protocol buffer payload.
argument
Type
Description
default
required
value
string
The protocol buffer payload to parse.
N/A
yes
desc_file
string
The path to the protobuf descriptor set file. Must be a literal string.
This file is the output of protoc -o
N/A
yes
message_type
string
The name of the message type to use for serializing.
Must be a literal string.
N/A
yes
value is not a valid proto payload.desc_file file does not exist.message_type message type does not exist in the descriptor file.Source:
parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")
Return:
{"name":"someone","phones":[{"number":"123456"}]}Parses the value as a query string.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
Source:
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return:
{"foo":"+1","bar":["2","3"],"xyz":""}Source:
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return:
{"foo[]":["1","2"]}Parses the value using the provided Regex pattern.
This function differs from the parse_regex_all function in that it returns only the first match.
argument
Type
Description
default
required
value
string
The string to search.
N/A
yes
pattern
regex
The regular expression pattern to search against.
N/A
yes
numeric_groups
regex
If true, the index of each group in the regular expression is also captured. Index 0
contains the whole match.
false
no
value fails to parse using the provided pattern.Source:
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return:
{"number":"first"}Source:
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return:
{"0":"first group","1":"first"}Parses the value using the provided Regex pattern.
This function differs from the parse_regex function in that it returns all matches, not just the first.
argument
Type
Description
default
required
value
string
The string to search.
N/A
yes
pattern
regex
The regular expression pattern to search against.
N/A
yes
numeric_groups
regex
If true, the index of each group in the regular expression is also captured. Index 0
contains the whole match.
false
no
value is not a string.pattern is not a regex.Source:
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return:
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]Parses the value as ruby hash.
argument
Type
Description
default
required
value
string
The string representation of the ruby hash to parse.
N/A
yes
value is not a valid ruby hash formatted payload.Source:
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return:
{"test":"value","testNum":0.2,"testObj":{"testBool":true,"testNull":null}}Parses the value in Syslog format.
argument
Type
Description
default
required
value
string
The text containing the Syslog message to parse.
N/A
yes
value is not a properly formatted Syslog message.Source:
parse_syslog!(
s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
Return:
{"severity":"notice","facility":"user","timestamp":"2020-03-13T20:45:38.119Z","hostname":"dynamicwireless.name","appname":"non","procid":2426,"msgid":"ID931","message":"Try to override the THX port, maybe it will reboot the neural interface!","exampleSDID@32473":{"eventID":"1011","eventSource":"Application","iut":"3"},"version":1}Parses the value in strptime format.
argument
Type
Description
default
required
value
string
The text of the timestamp.
N/A
yes
timezone
string
The TZ database format. By default, this function parses the timestamp by global timezone option.
This argument overwrites the setting and is useful for parsing timestamps without a specified timezone, such as 16/10/2019 12:00:00.
N/A
no
value fails to parse using the provided format.value fails to parse using the provided timezone.Source:
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return:
"2020-10-10T16:00:00Z"Source:
parse_timestamp!("16/10/2019 12:00:00", format: "%d/%m/%Y %H:%M:%S", timezone: "Asia/Taipei")
Return:
"2019-10-16T04:00:00Z"Parses the value in token format. A token is considered to be one of the following:
"..". Quotes can be included in the token if they are escaped by a backslash (\).[..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).argument
Type
Description
default
required
value
string
The string to tokenize.
N/A
yes
value is not a properly formatted tokenized string.Source:
parse_tokens(
"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)
Return:
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]Parses the value in URL format.
argument
Type
Description
default
required
value
string
The text of the URL.
N/A
yes
default_known_ports
boolean
If true and the port number is not specified in the input URL
string (or matches the default port for the scheme), it is
populated from well-known ports for the following schemes:
http, https, ws, wss, and ftp.
false
no
value is not a properly formatted URL.Source:
parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")
Return:
{"scheme":"ftp","username":"foo","password":"bar","host":"example.com","port":4343,"path":"/foobar","query":{"hello":"world"},"fragment":"123"}Source:
parse_url!("https://example.com", default_known_ports: true)
Return:
{"scheme":"https","username":"","password":"","host":"example.com","port":443,"path":"/","query":{},"fragment":null}Source:
parse_url!("https://www.café.com")
Return:
{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}Source:
parse_url!("https://www.CAFé.com")
Return:
{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}Parses the value as a user agent string, which has a loosely defined format
so this parser only provides best effort guarantee.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
mode
string
Determines performance and reliability characteristics.
fast
no
Source:
parse_user_agent(
"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return:
{"browser":{"family":"Firefox","version":"1.0.1"},"device":{"category":"pc"},"os":{"family":"Linux","version":null}}Source:
parse_user_agent(
"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
mode: "reliable"
)
Return:
{"browser":{"family":"Internet Explorer","version":"7.66"},"device":{"category":"pc"},"os":{"family":"Windows XP","version":"NT 5.1"}}Source:
parse_user_agent(
"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
mode: "enriched"
)
Return:
{"browser":{"family":"Opera Mini","major":"4","minor":"3","patch":"24214","version":"10.54"},"device":{"brand":"Apple","category":"smartphone","family":"iPhone","model":"iPhone"},"os":{"family":"iOS","major":"4","minor":"2","patch":"1","patch_minor":null,"version":"4.2.1"}}Parses the value as XML.
argument
Type
Description
default
required
value
string
The string representation of the XML document to parse.
N/A
yes
include_attr
boolean
Include XML tag attributes in the returned object.
true
no
attr_prefix
string
String prefix to use for XML tag attribute keys.
@
no
text_key
string
Key name to use for expanded text nodes.
text
no
always_use_text_key
boolean
Always return text nodes as {"<text_key>": "value"}.
false
no
parse_bool
boolean
Parse “true” and “false” as boolean.
true
no
parse_null
boolean
Parse “null” as null.
true
no
parse_number
boolean
Parse numbers as integers/floats.
true
no
value is not a valid XML document.Source:
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';
parse_xml!(value, text_key: "value", parse_number: false)
Return:
{"book":{"@category":"CHILDREN","author":"J K. Rowling","title":{"@lang":"en","value":"Harry Potter"},"year":"2005"}}Parses the value as YAML.
argument
Type
Description
default
required
value
string
The string representation of the YAML to parse.
N/A
yes
value is not a valid YAML-formatted payload.Source:
parse_yaml!("key: val")
Return:
{"key":"val"}Removes the field specified by the static path from the target.
For dynamic path deletion, see the remove function.
argument
Type
Description
default
required
path
path
The path of the field to delete.
N/A
yes
compact
boolean
After deletion, if compact is true and there is an empty object or array left,
the empty object or array is also removed, cascading up to the root. This only
applies to the path being deleted, and any parent paths.
false
no
Source:
del(.field1)
Source:
.new_field = del(.old_field)
Checks whether the path exists for the target.
This function distinguishes between a missing path
and a path with a null value. A regular path lookup,
such as .foo, cannot distinguish between the two cases
since it always returns null if the path doesn’t exist.
argument
Type
Description
default
required
path
path
The path of the field to check.
N/A
yes
Source:
exists(.field)
Return:
trueSource:
exists(.array[2])
Return:
trueDynamically get the value of a given path.
If you know the path you want to look up, use
static paths such as .foo.bar[1] to get the value of that
path. However, if you do not know the path names,
use the dynamic get function to get the requested
value.
argument
Type
Description
default
required
value
object, array
The object or array to query.
N/A
yes
path
array
An array of path segments to look for the value.
N/A
yes
path segment must be a string or an integer.Source:
get!(value: { "foo": "bar" }, path: ["foo"])
Return:
"bar"Source:
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return:
"baz"Source:
get!(value: ["foo", "bar", "baz"], path: [-2])
Return:
"bar"Dynamically remove the value for a given path.
If you know the path you want to remove, use
the del function and static paths such as del(.foo.bar[1])
to remove the value at that path. The del function returns the
deleted value, and is more performant than remove.
However, if you do not know the path names, use the dynamic
remove function to remove the value at the provided path.
argument
Type
Description
default
required
value
object, array
The object or array to remove data from.
N/A
yes
path
array
An array of path segments to remove the value from.
N/A
yes
compact
boolean
After deletion, if compact is true, any empty objects or
arrays left are also removed.
false
no
path segment must be a string or an integer.Source:
remove!(value: { "foo": "bar" }, path: ["foo"])
Return:
{}Source:
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return:
{"foo":{}}Source:
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return:
["foo","baz"]Source:
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return:
{"foo":{"baz":true}}Dynamically insert data into the path of a given object or array.
If you know the path you want to assign a value to,
use static path assignments such as .foo.bar[1] = true for
improved performance and readability. However, if you do not
know the path names, use the dynamic set function to
insert the data into the object or array.
argument
Type
Description
default
required
value
object, array
The object or array to insert data into.
N/A
yes
path
array
An array of path segments to insert the value into.
N/A
yes
data
any
The data to be inserted.
N/A
yes
path segment must be a string or an integer.Source:
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return:
{"foo":"baz"}Source:
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return:
{"foo":{"bar":"qux"}}Source:
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return:
["foo",42,"baz"]Returns a random boolean.
Source:
is_boolean(random_bool())
Return:
trueA cryptographically secure random number generator. Returns a string value containing the number of random bytes requested.
argument
Type
Description
default
required
length
integer
The number of bytes to generate. Must not be larger than 64k.
N/A
yes
length is negative.length is larger than the maximum value (64k).Source:
encode_base64(random_bytes(16))
Return:
"LNu0BBgUbh7XAlXbjSOomQ=="Returns a random float between [min, max).
argument
Type
Description
default
required
min
float
Minimum value (inclusive).
N/A
yes
max
float
Maximum value (exclusive).
N/A
yes
max is not greater than min.Source:
f = random_float(0.0, 10.0)
f >= 0 && f < 10
Return:
trueReturns a random integer between [min, max).
argument
Type
Description
default
required
min
integer
Minimum value (inclusive).
N/A
yes
max
integer
Maximum value (exclusive).
N/A
yes
max is not greater than min.Source:
i = random_int(0, 10)
i >= 0 && i < 10
Return:
trueConvert a Friendly ID (base62 encoding a 128-bit word) to a UUID.
argument
Type
Description
default
required
value
timestamp
A string that is a Friendly ID
N/A
yes
value is a string but the text uses characters outside of class [0-9A-Za-z].value is a base62 encoding of an integer, but the integer is greater than or equal to 2^128.Source:
uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")
Return:
"7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2"Generates a random UUIDv4 string.
Source:
uuid_v4()
Return:
"1d262f4f-199b-458d-879f-05fd0a5f0683"Generates a random UUIDv7 string.
argument
Type
Description
default
required
timestamp
timestamp
The timestamp used to generate the UUIDv7.
now()
no
now()Source:
uuid_v7()
Return:
"06338364-8305-7b74-8000-de4963503139"now()Source:
uuid_v7(now())
Return:
"018e29b3-0bea-7f78-8af3-d32ccb1b93c1"Source:
uuid_v7(t'2020-12-30T22:20:53.824727Z')
Return:
"0176b5bd-5d19-7394-bb60-c21028c6152b"Takes the value string, and turns it into camelCase. Optionally, you can
pass in the existing case of the function, or else an attempt is made to determine the case automatically.
argument
Type
Description
default
required
value
string
The string to convert to camelCase.
N/A
yes
original_case
string
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
N/A
no
Source:
camelcase("input-string")
Return:
"inputString"Source:
camelcase("input-string", "kebab-case")
Return:
"inputString"Generates an ID based on the Community ID Spec.
argument
Type
Description
default
required
source_ip
string
The source IP address.
N/A
yes
destination_ip
string
The destination IP address.
N/A
yes
protocol
integer
The protocol number.
N/A
yes
source_port
integer
The source port or ICMP type.
N/A
no
destination_port
integer
The destination port or ICMP code.
N/A
no
seed
integer
The custom seed number.
N/A
no
Source:
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)
Return:
"1:wCb3OG7yAFWelaUydu0D+125CLM="Determines whether the value string contains the specified substring.
argument
Type
Description
default
required
value
string
The text to search.
N/A
yes
substring
string
The substring to search for in value.
N/A
yes
case_sensitive
boolean
Whether the match should be case sensitive.
true
no
Source:
contains("The Needle In The Haystack", "Needle")
Return:
trueSource:
contains("The Needle In The Haystack", "needle", case_sensitive: false)
Return:
trueDetermines whether the value string contains all the specified substrings.
argument
Type
Description
default
required
value
string
The text to search.
N/A
yes
substrings
array
An array of substrings to search for in value.
N/A
yes
case_sensitive
boolean
Whether the match should be case sensitive.
N/A
no
Source:
contains_all("The Needle In The Haystack", ["Needle", "Haystack"])
Return:
trueSource:
contains_all("the NEEDLE in the haystack", ["needle", "haystack"])
Return:
falseDowncases the value string, where downcase is defined according to the
Unicode Derived Core Property Lowercase.
argument
Type
Description
default
required
value
string
The string to convert to lowercase.
N/A
yes
Source:
downcase("Hello, World!")
Return:
"hello, world!"Determines whether the value string ends with the specified substring.
argument
Type
Description
default
required
value
string
The string to search.
N/A
yes
substring
string
The substring with which value must end.
N/A
yes
case_sensitive
boolean
Whether the match should be case sensitive.
true
no
Source:
ends_with("The Needle In The Haystack", "The Haystack")
Return:
trueSource:
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return:
trueDetermines from left to right the start position of the first found element in value
that matches pattern. Returns -1 if not found.
argument
Type
Description
default
required
value
string
The string to find the pattern in.
N/A
yes
pattern
regex, string
The regular expression or string pattern to match against.
N/A
yes
from
integer
Offset to start searching.
0
no
Source:
find("foobar", "foo")
Return:
0Source:
find("foobar", r'b.r')
Return:
3Source:
find("foobar", "baz")
Return:
-1Source:
find("foobarfoobarfoo", "bar", 4)
Return:
9Joins each string in the value array into a single string, with items optionally separated from one another
by a separator.
argument
Type
Description
default
required
value
array
The array of strings to join together.
N/A
yes
separator
string
The string separating each original element when joined.
N/A
no
Source:
join!(["bring", "us", "together"])
Return:
"bringustogether"Source:
join!(["sources", "transforms", "sinks"], separator: ", ")
Return:
"sources, transforms, sinks"Takes the value string, and turns it into kebab-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
argument
Type
Description
default
required
value
string
The string to convert to kebab-case.
N/A
yes
original_case
string
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
N/A
no
Source:
kebabcase("InputString")
Return:
"input-string"Source:
kebabcase("InputString", "PascalCase")
Return:
"input-string"Determines whether the value matches the pattern.
argument
Type
Description
default
required
value
string
The value to match.
N/A
yes
pattern
regex
The regular expression pattern to match against.
N/A
yes
Source:
match("I'm a little teapot", r'teapot')
Return:
trueSource:
match("I'm a little teapot", r'.*balloon')
Return:
falseDetermines whether value matches any of the given patterns. All
patterns are checked in a single pass over the target string, giving this
function a potential performance advantage over the multiple calls
in the match function.
argument
Type
Description
default
required
value
string
The value to match.
N/A
yes
patterns
array
The array of regular expression patterns to match against.
N/A
yes
Source:
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return:
trueParses the string value representing a floating point number in base 10 to a float.
argument
Type
Description
default
required
value
string
The string to parse.
N/A
yes
value is not a string.Source:
parse_float!("-42")
Return:
-42Source:
parse_float!("42.38")
Return:
42.38Source:
parse_float!("2.5e3")
Return:
2500Takes the value string, and turns it into PascalCase. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
argument
Type
Description
default
required
value
string
The string to convert to PascalCase.
N/A
yes
original_case
string
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
N/A
no
Source:
pascalcase("input-string")
Return:
"InputString"Source:
pascalcase("input-string", "kebab-case")
Return:
"InputString"Redact sensitive data in value such as:
This can help achieve compliance by ensuring sensitive data does not leave your network.
argument
Type
Description
default
required
value
string, object, array
The value to redact sensitive data from.
The function’s behavior depends on value’s type:
For arrays and objects, the function recurses into any nested arrays or objects. Any non-string elements are skipped.
Redacted text is replaced with [REDACTED].
N/A
yes
filters
array
List of filters applied to value.
Each filter can be specified in the following ways:
type key that corresponds to a named filter and additional keys for customizing that filter.Named filters can be a:
pattern: Redacts text matching any regular expressions specified in the patterns
key, which is required. This is the expanded version of just passing a regular expression as a filter.us_social_security_number: Redacts US social security card numbers.See examples for more details.
This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it.
N/A
yes
redactor
string, object
Specifies what to replace the redacted strings with.
It is given as an object with a “type” key specifying the type of redactor to use and additional keys depending on the type. The following types are supported:
full: The default. Replace with the string “[REDACTED]”.text: Replace with a custom string. The replacement key is required, and must
contain the string that is used as a replacement.sha2: Hash the redacted text with SHA-2 as with sha2. Supports two optional parameters:variant: The variant of the algorithm to use. Defaults to SHA-512/256.encoding: How to encode the hash as text. Can be base16 or base64.
Defaults to base64.sha3: Hash the redacted text with SHA-3 as with sha3. Supports two optional parameters:variant: The variant of the algorithm to use. Defaults to SHA3-512.encoding: How to encode the hash as text. Can be base16 or base64.
Defaults to base64.As a convenience you can use a string as a shorthand for common redactor patterns:
"full" is equivalent to {"type": "full"}"sha2" is equivalent to {"type": "sha2", "variant": "SHA-512/256", "encoding": "base64"}"sha3" is equivalent to {"type": "sha3", "variant": "SHA3-512", "encoding": "base64"}This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it.
N/A
no
Source:
redact("my id is 123456", filters: [r'\d+'])
Return:
"my id is [REDACTED]"Source:
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return:
{"name":"John Doe","ssn":"[REDACTED]"}Source:
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})
Return:
"my id is ***"Source:
redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")
Return:
"my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I="Source:
redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")
Return:
"my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw=="Source:
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
Return:
"my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92"Replaces all matching instances of pattern in value.
The pattern argument accepts regular expression capture groups.
Note when using capture groups:
$ by using $$ to avoid Vector interpreting it as an
environment variable when loading configuration$ in the replacement pattern, you will also need to escape this
with $$. When combined with environment variable interpolation in config files this
means you will need to use $$$$ to have a literal $ in the replacement pattern.argument
Type
Description
default
required
value
string
The original string.
N/A
yes
pattern
regex, string
Replace all matches of this pattern. Can be a static string or a regular expression.
N/A
yes
with
string
The string that the matches are replaced with.
N/A
yes
count
integer
The maximum number of replacements to perform. -1 means replace all matches.
-1
no
Source:
replace("Apples and Bananas", "and", "not")
Return:
"Apples not Bananas"Source:
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return:
"Apples and Pineapples"Source:
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return:
"Pineapples and Bananas"$$num in config files)Source:
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")
Return:
"123"Takes the value string, and turns it into SCREAMING_SNAKE case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
argument
Type
Description
default
required
value
string
The string to convert to SCREAMING_SNAKE case.
N/A
yes
original_case
string
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
N/A
no
Source:
screamingsnakecase("input-string")
Return:
"INPUT_STRING"Source:
screamingsnakecase("input-string", "kebab-case")
Return:
"INPUT_STRING"Generates Shannon entropy from given string. It can generate it based on string bytes, codepoints, or graphemes.
argument
Type
Description
default
required
value
string
The input string.
N/A
yes
segmentation
string
Defines how to split the string to calculate entropy, based on occurrences of segments.
Byte segmentation is the fastest, but it might give undesired results when handling UTF-8 strings, while grapheme segmentation is the slowest, but most correct in these cases.
byte
no
Source:
floor(shannon_entropy("vector.dev"), precision: 4)
Return:
2.9219Source:
floor(shannon_entropy("test123%456.فوائد.net."), precision: 4)
Return:
4.0784Source:
floor(shannon_entropy("test123%456.فوائد.net.", segmentation: "grapheme"), precision: 4)
Return:
3.9362Keeps only matches of pattern in value.
This can be used to define patterns that are allowed in the string and remove everything else.
argument
Type
Description
default
required
value
string
The original string.
N/A
yes
pattern
regex
Keep all matches of this pattern.
N/A
yes
replace_single
string
The string to use to replace single rejected characters.
N/A
no
replace_repeated
string
The string to use to replace multiple sequential instances of rejected characters.
N/A
no
Source:
sieve("test123%456.فوائد.net.", r'[a-z0-9.]')
Return:
"test123456..net."Source:
sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")
Return:
"test123X456.<REMOVED>.net."Returns a slice of value between the start and end positions.
If the start and end parameters are negative, they refer to positions counting from the right of the
string or array. If end refers to a position that is greater than the length of the string or array,
a slice up to the end of the string or array is returned.
argument
Type
Description
default
required
value
array, string
The string or array to slice.
N/A
yes
start
integer
The inclusive start position. A zero-based index that can be negative.
N/A
yes
end
integer
The exclusive end position. A zero-based index that can be negative.
String length
no
Source:
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return:
"califrag"Source:
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return:
"califragilistic"Takes the value string, and turns it into snake-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
argument
Type
Description
default
required
value
string
The string to convert to snake-case.
N/A
yes
original_case
string
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
N/A
no
Source:
snakecase("input-string")
Return:
"input_string"Source:
snakecase("input-string", "kebab-case")
Return:
"input_string"Splits the value string using pattern.
argument
Type
Description
default
required
value
string
The string to split.
N/A
yes
pattern
string, regex
The string is split whenever this pattern is matched.
N/A
yes
limit
integer
The maximum number of substrings to return.
N/A
no
Source:
split("apples and pears and bananas", " and ")
Return:
["apples","pears","bananas"]Source:
split("apples and pears and bananas", " and ", limit: 2)
Return:
["apples","pears and bananas"]Determines whether value begins with substring.
argument
Type
Description
default
required
value
string
The string to search.
N/A
yes
substring
string
The substring that the value must start with.
N/A
yes
case_sensitive
boolean
Whether the match should be case sensitive.
true
no
Source:
starts_with("The Needle In The Haystack", "The Needle")
Return:
trueSource:
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return:
trueStrips ANSI escape codes from value.
argument
Type
Description
default
required
value
string
The string to strip.
N/A
yes
Source:
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")
Return:
"foo bar"Strips whitespace from the start and end of value, where whitespace is defined by the Unicode
White_Space property.
argument
Type
Description
default
required
value
string
The string to trim.
N/A
yes
Source:
strip_whitespace(" A sentence. ")
Return:
"A sentence."Truncates the value string up to the limit number of characters.
argument
Type
Description
default
required
value
string
The string to truncate.
N/A
yes
limit
integer, float
The number of characters to truncate the string after.
N/A
yes
ellipsis
boolean
This argument is deprecated. An ellipsis (...) is appended if the parameter is set to true and the value string
is truncated because it exceeded the limit.
N/A
no
suffix
string
A custom suffix (...) is appended to truncated strings.
If ellipsis is set to true, this parameter is ignored for backwards compatibility.
N/A
no
Source:
truncate("A rather long sentence.", limit: 11, suffix: "...")
Return:
"A rather lo..."Source:
truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")
Return:
"A rather lo[TRUNCATED]"Upcases value, where upcase is defined according to the Unicode Derived Core Property
Uppercase.
argument
Type
Description
default
required
value
string
The string to convert to uppercase.
N/A
yes
Source:
upcase("Hello, World!")
Return:
"HELLO, WORLD!"Returns the local system’s hostname.
Source:
.hostname = get_hostname!()
Returns the name of the timezone in the Vector configuration (see
global configuration options).
If the configuration is set to local, then it attempts to
determine the name of the timezone from the host OS. If this
is not possible, then it returns the fixed offset of the
local timezone for the current time in the format "[+-]HH:MM",
for example, "+02:00".
Source:
.vector_timezone = get_timezone_name!()
Formats value into a string representation of the timestamp.
argument
Type
Description
default
required
value
timestamp
The timestamp to format as text.
N/A
yes
timezone
string
The timezone to use when formatting the timestamp. The parameter uses the TZ identifier or local.
N/A
no
Source:
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return:
"2020-10-21T16:00:00+00:00"Source:
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return:
"21-Oct-2020 16:00"Returns the current timestamp in the UTC timezone with nanosecond precision.
Source:
now()
Return:
"2021-03-04T10:51:15.928937Z"Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an array and can be used in any function that expects an array.
argument
Type
Description
default
required
value
any
The value to check if it is an array.
N/A
yes
value is not an array.Source:
array!(.value)
Return:
[1,2,3]Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a Boolean and can be used in any function that expects a Boolean.
argument
Type
Description
default
required
value
any
The value to check if it is a Boolean.
N/A
yes
value is not a Boolean.Source:
bool!(.value)
Return:
falseReturns value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a float and can be used in any function that expects a float.
argument
Type
Description
default
required
value
any
The value to check if it is a float.
N/A
yes
value is not a float.Source:
float!(.value)
Return:
42Returns value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an integer and can be used in any function that expects an integer.
argument
Type
Description
default
required
value
any
The value to check if it is an integer.
N/A
yes
value is not an integer.Source:
int!(.value)
Return:
42Check if the value’s type is an array.
argument
Type
Description
default
required
value
any
The value to check if it is an array.
N/A
yes
Source:
is_array([1, 2, 3])
Return:
trueSource:
is_array("a string")
Return:
falseCheck if the value’s type is a boolean.
argument
Type
Description
default
required
value
any
The value to check if it is a Boolean.
N/A
yes
Source:
is_boolean(false)
Return:
trueSource:
is_boolean("a string")
Return:
falseCheck if the object, array, or string has a length of 0.
argument
Type
Description
default
required
value
object, array, string
The value to check.
N/A
yes
Source:
is_empty([])
Return:
trueSource:
is_empty("a string")
Return:
falseSource:
is_empty({"foo": "bar"})
Return:
falseCheck if the value’s type is a float.
argument
Type
Description
default
required
value
any
The value to check if it is a float.
N/A
yes
Source:
is_float(0.577)
Return:
trueSource:
is_float("a string")
Return:
falseCheck if the value`’s type is an integer.
argument
Type
Description
default
required
value
any
The value to check if it is an integer.
N/A
yes
Source:
is_integer(1)
Return:
trueSource:
is_integer("a string")
Return:
falseCheck if the string is a valid JSON document.
argument
Type
Description
default
required
value
string
The value to check if it is a valid JSON document.
N/A
yes
variant
string
The variant of the JSON type to explicitly check for.
N/A
no
Source:
is_json("{}")
Return:
trueSource:
is_json("{")
Return:
falseSource:
is_json("{}", variant: "object")
Return:
trueSource:
is_json("{}", variant: "array")
Return:
falseCheck if value’s type is null. For a more relaxed function,
see is_nullish.
argument
Type
Description
default
required
value
any
The value to check if it is null.
N/A
yes
Source:
is_null(null)
Return:
trueSource:
is_null("a string")
Return:
falseDetermines whether value is nullish. Returns true if the specified value is null,
an empty string, a string containing only whitespace, or the string "-". Returns false otherwise.
argument
Type
Description
default
required
value
any
The value to check for nullishness, for example, a useless value.
N/A
yes
Source:
is_nullish("")
Return:
trueSource:
is_nullish("-")
Return:
trueSource:
is_nullish("
")
Return:
trueCheck if value’s type is an object.
argument
Type
Description
default
required
value
any
The value to check if it is an object.
N/A
yes
Source:
is_object({"foo": "bar"})
Return:
trueSource:
is_object("a string")
Return:
falseCheck if value’s type is a regex.
argument
Type
Description
default
required
value
any
The value to check if it is a regex.
N/A
yes
Source:
is_regex(r'pattern')
Return:
trueSource:
is_regex("a string")
Return:
falseCheck if value’s type is a string.
argument
Type
Description
default
required
value
any
The value to check if it is a string.
N/A
yes
Source:
is_string("a string")
Return:
trueSource:
is_string([1, 2, 3])
Return:
falseCheck if value’s type is a timestamp.
argument
Type
Description
default
required
value
any
The value to check if it is a timestamp.
N/A
yes
Source:
is_timestamp(t'2021-03-26T16:00:00Z')
Return:
trueSource:
is_timestamp("a string")
Return:
falseReturns value if it is an object, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an object and can be used in any function that expects an object.
argument
Type
Description
default
required
value
any
The value to check if it is an object.
N/A
yes
value is not an object.Source:
object!(.value)
Return:
{"field1":"value1","field2":"value2"}Returns value if it is a string, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a string and can be used in any function that expects a string.
argument
Type
Description
default
required
value
any
The value to check if it is a string.
N/A
yes
value is not a string.Source:
string!(.message)
Return:
"{\"field\": \"value\"}"Adds type information to all (nested) scalar values in the provided value.
The type information is added externally, meaning that value has the form of "type": value after this
transformation.
argument
Type
Description
default
required
value
any
The value to tag with types.
N/A
yes
Source:
tag_types_externally(123)
Return:
{"integer":123}Source:
tag_types_externally({
"message": "Hello world",
"request": {
"duration_ms": 67.9
}
})
Return:
{"message":{"string":"Hello world"},"request":{"duration_ms":{"float":67.9}}}Source:
tag_types_externally(["foo", "bar"])
Return:
[{"string":"foo"},{"string":"bar"}]Source:
tag_types_externally(null)
Return:
nullReturns value if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that
the returned value is a timestamp and can be used in any function that expects a timestamp.
argument
Type
Description
default
required
value
any
The value to check if it is a timestamp.
N/A
yes
value is not a timestamp.Source:
timestamp(t'2020-10-10T16:00:00Z')
Return:
"2020-10-10T16:00:00Z"Additional helpful documentation, links, and articles:
| |
