Supported Logs and Discovered Fields

Overview

CloudWatch Logs Insights provides structured analysis of different AWS service logs by automatically generating system fields and discovering relevant log fields. It supports advanced querying of both standard and custom logs using a purpose-built query language.

Key System Fields Automatically Generated:

1. @message: Raw, unparsed log event.

2. @timestamp: Timestamp when the event occurred.

3. @ingestionTime: Time the log was ingested by CloudWatch Logs.

4. @logStream: Name of the log stream where the event resides.

5. @log: Identifier in the form account-id:log-group-name (helps identify log groups across multiple queries).

Log Field Discovery by Log Type:

• VPC Flow Logs: Includes fields like accountId, interfaceId, srcAddr, dstPort, packets, and protocol.

• Route 53 Logs: Captures fields such as edgeLocation, queryName, responseCode, and resolverIp.

• Lambda Logs: Includes Lambda-specific metrics such as @requestId, @duration, @memorySize, and optional X-Ray fields (@xrayTraceId).

• CloudTrail Logs: Handled as JSON logs with fields accessible using dot notation.

• Custom Logs: For logs without auto-discovered fields, use the parse command to extract data.

Dot Notation for JSON Logs:

• Access Nested Fields: Example: "userIdentity.type" retrieves the value of the type field inside userIdentity.

• Access Array Elements: Example: "requestParameters.instancesSet.items.0.instanceId" retrieves the first instanceId.

Sample Query Using Dot Notation:

fields @timestamp, @message
| filter requestParameters.instancesSet.items.0.instanceId="i-abcde123"
| sort @timestamp desc

This query retrieves log events where the instanceId matches "i-abcde123" and sorts them by timestamp in descending order.

Additional Notes:

• Log Class Limitation: Field discovery is available only in Standard log class groups.

• Naming Convention: CloudWatch Logs Insights adds @ to its generated fields. If a log field name already contains @, it will be displayed with an additional @ (e.g., @@example.com).