DCR transformkql not parsing custom log field containing special character

Praveesh Shetty 0 Reputation points

Hi I'm trying to parse logs which is coming from fluentbit in json format

raw log contains in json format:

{

"method": "saml",

"addr.remote": "101.2.10.142"

}

Everything parsed except addr.remote, tried multiple DCR transformation which didn't work

"transformKql": "source\n| extend SourceIP = tostring('addr.remote')",

need suggestion if anyone successfully parsed key filed containing special character like addr.remote

0 comments No comments
Sign in to comment

2 answers

  1. SAI JAGADEESH KUDIPUDI 3,470 Reputation points • Microsoft External Staff • Moderator

    Hi Praveesh Shetty,

    Thank you for checking on this. I’ve reviewed your scenario, and I can confirm that the suggested approach is correct and supported for resolving the issue with parsing the addr.remote field in a DCR transformation.

    The behavior you’re seeing is expected. In Kusto Query Language (KQL), a dot (.) in a field name is interpreted as a property accessor, not as part of the column name itself. Because of this, fields such as addr.remote cannot be accessed using standard dot notation or by treating the name as a string literal.

    To handle field names that contain special characters (including dots), KQL requires the use of bracket notation. In the context of a Data Collection Rule (DCR) transformation, source represents the incoming record, and the field must be referenced exactly as it appears using brackets.

    For example: KQL

    source
| extend SourceIP = tostring(source['addr.remote'])

    If the JSON payload is stored as a string inside another column (for example, RawData), the payload must first be parsed, and then the field accessed using the same bracket notation:

    source
| extend parsed = parse_json(RawData)
| extend SourceIP = tostring(parsed['addr.remote

    This is the supported and recommended pattern for accessing JSON properties with special characters during ingestion-time transformations. No changes are required on the Fluent Bit side if the payload structure remains the same.
    Conclusion
    Once the transformation is updated using bracket notation, the field should ingest correctly and be queryable in Log Analytics without further changes.
    Microsoft reference documentation

    Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

    Please do not forget to "Accept Answer" and "up-vote" wherever the information provided helps you, as this can be beneficial to other community members.

    1. Praveesh Shetty 0 Reputation points

      Hi Jagadeesh,

      Very much appreciate your quick response and the detailed explaination.

      however when i used the below transformation kql in my DCR ( i'm modifying the DCR in json and redeploying it )

      "transformKql": "source\n | extend SourceIP = tostring(source['addr.remote'])",

      Getting the error: i've observed transformkql 

      {
 "code": "InvalidPayload",
 "message": "Data collection rule is invalid",
 "details": [
 {
 "code": "InvalidTransformQuery",
 "target": "properties.dataFlows[0]",
 "message": "Error occurred while compiling query in query: SemanticError:0x00000007 at 2:30 : Operand number 1 of node PathExpression has invalid type shape: Tabular. Expected: Scalar"
 }
 ]
}
    2. SAI JAGADEESH KUDIPUDI 3,470 Reputation points • Microsoft External Staff • Moderator

      Hi Praveesh Shetty,

      Thank you for the detailed follow-up and for sharing the exact error message — that really helps narrow this down.

      From what you’ve described, the behavior you’re seeing is expected in DCR transformation KQL, and the issue is not with the bracket notation itself, but with how it’s being applied on the source object.

      In DCR transformations, source represents a tabular input (rowset) and not a dynamic JSON object. Because of this, expressions like:

      tostring(source['addr.remote'])

      will fail with the semantic error you observed:

      Operand number 1 of node PathExpression has invalid type shape: Tabular. Expected: Scalar

      This happens because bracket notation (['...']) is valid only for:

      Dynamic (JSON) objects, or

      Column name escaping (without prefixing with source)

      If addr.remote is already being ingested as a column (which seems to be your case since other fields are parsing correctly), you should reference it using column escaping syntax, like below:

      source
| extend SourceIP = tostring(['addr.remote'])

      This tells KQL to treat addr.remote as a column name containing a special character, rather than attempting to interpret it as a property path.

      Alternate Scenario (if JSON is stored as a string)

      If your payload is actually inside a column (for example RawData), then you would first parse it and then access the field:

      source
| extend parsed = parse_json(RawData)
| extend SourceIP = tostring(parsed['addr.remote'])
      • source['addr.remote'] → Not valid (source is tabular, not dynamic)
      • ['addr.remote'] → Correct way to reference a column with special characters
      • parsed['addr.remot'] → Correct when working with parsed JSON

      The error occurs because source is treated as a table, not a JSON object. By switching to column escaping syntax (['addr.remote']), or parsing the JSON first if needed, the transformation will compile successfully and the field should be ingested correctly.

      Microsoft Documentation

      Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
      Please do not forget to "Accept Answer" and "up-vote" wherever the information provided helps you, as this can be beneficial to other community members.

    3. SAI JAGADEESH KUDIPUDI 3,470 Reputation points • Microsoft External Staff • Moderator

      Hi **Praveesh Shetty,
      **I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

    Sign in to comment

  2. AI answer

    In a DCR transformation, source already represents the incoming JSON as columns. To access a JSON property whose name contains a dot (like addr.remote), use bracket notation instead of a bare identifier or string literal.

    Use this KQL in transformKql:

    "transformKql": "source | extend SourceIP = tostring(source['addr.remote'])"

    Key points:

    • source is the virtual table for incoming data in transformations.
    • addr.remote is not a valid bare column name because of the dot, so it must be referenced as ['addr.remote'].
    • 'addr.remote' in the original query is just a string literal, not a column reference, so it never reads the field from the log.

    If the JSON is nested inside another column (for example, all JSON is in a column called RawData), first parse that column, then use the same bracket notation on the parsed object:

    source
| extend parsed = parse_json(RawData)
| extend SourceIP = tostring(parsed['addr.remote'])

    Ensure the final query is flattened to a single line when placed in the DCR transformKql property.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.
Sign in to answer

Your answer