☁️
CTHFM: AWS
  • Welcome
  • Getting Started
    • Account Setup
  • AWS CLI
    • AWS CLI Overview
    • Installation
  • AWS Fundamentals
    • AWS Documentation
    • AWS Shared Responsibility Model
    • Organizational Hierarchy
    • AWS Principals
    • IAM Fundamentals
      • IAM Policy Components
      • IAM Documentation References
    • AWS Security Services Overview
    • AWS Core Services
    • AWS Frameworks
    • Regions and Availability Zones
  • SQL
    • SQL Refresher for Threat Hunting
  • Logging Reference
    • Cloudtrail
      • What is Cloudtrail?
      • Setting Up Cloudtrail
      • Cloudtrail Events Structure
      • Filtering and Searching CloudTrail Logs
      • IAM ID Prefixes
      • Additional Resources
      • API References
    • VPCFlow Logs
    • GuardDuty
      • Multi-Account Setup
      • GuardDuty Concepts
      • GuardDuty Finding References
      • S3 Protection
      • Malware Protection
        • EC2 Malware Protection
          • EC2 Protection Resources
          • Monitoring Scans
          • EC2 Malware Protection Events: CloudWatch
        • S3 Malware Protection
          • Enabling S3 Malware Protection
          • After Enabling S3 Malware Protection
          • S3 Malware Resource Plan Status
          • S3 Malware Protection Quotas
      • RDS Protection Enablement
      • Lambda Protection Enablement
      • Trusted IP Lists and Threat Lists in Amazon GuardDuty
      • Remediation Recommendations
      • GuardDuty API Reference
      • GuardDuty Quotas
    • Access Analyzer
      • Setup
      • External Access and Unused Access Analyzer Findings
      • Review Findings
      • Access Analyzer Resources
      • Access Analyzer API Reference
    • AWS Network Firewall
      • Permissions
      • Firewall Log Contents
      • Logging Destinations
      • CloudWatch Firewall Metrics
    • AWS Config
      • Resource Management in AWS Config
      • AWS Config Integrations
      • AWS Config Resources
      • Configuration Item
      • Config Rules
        • Evaluation Modes
  • CloudWatch
    • Amazon CloudWatch
      • CloudWatch Concepts
      • CloudWatch Metrics
        • Filter Pattern Syntax
      • CloudWatch Alarms
        • Alarm Recommendations
      • Subscriptions
      • CloudWatch Agent
      • CloudWatch Insights
        • Supported Logs and Discovered Fields
        • CloudWatch Insights Query Syntax
      • Anomaly Detection
        • Create Anomaly Detector
        • Alarms for Anomaly Detections
      • CloudWatch Filter Syntax
      • CloudWatch Service Quota
  • Athena For Threat Hunting
    • Introduction to Athena
    • Setting Up Athena
    • SQL For Threat Hunters
    • Automated Response
    • Query Best Practices
  • AWS Security Research and Resources
    • AWS Security Blog
    • AWS Goat
    • Cloud Goat
    • Pacu
    • Prowler
    • Scout Suite
  • Threat Hunting in AWS
    • Threat Hunting in AWS
    • Threat Hunting Introduction
    • Threat Hunting Process
      • Hypothesis Generation
      • Investigation
      • Identification
      • Resolution & Follow Up
    • Pyramid of Pain
    • MITRE Att&ck
      • MITRE Att&ck Concepts
      • MITRE Att&CK Data Sources
      • MITRE Att&CK Mitigations
    • MITRE Att&ck: AWS
      • MITRE Att&CK Matrix
      • Amazon Web Services Security Control Mappings
    • AWS Threat Hunting Ideas
      • AWS Threat Hunting Ideas: EC2
      • AWS Threat Hunting Ideas: Lambda
      • AWS Threat Hunting Ideas: SQS
      • AWS Threat Hunting Ideas: SNS
      • AWS Threat Hunting Ideas: RDS
Powered by GitBook
On this page
  • Overview
  • Key System Fields Automatically Generated:
  • Log Field Discovery by Log Type:
  • Dot Notation for JSON Logs:
  • Sample Query Using Dot Notation:
  1. CloudWatch
  2. Amazon CloudWatch
  3. CloudWatch Insights

Supported Logs and Discovered Fields

Overview

CloudWatch Logs Insights provides structured analysis of different AWS service logs by automatically generating system fields and discovering relevant log fields. It supports advanced querying of both standard and custom logs using a purpose-built query language.

Key System Fields Automatically Generated:

  1. @message: Raw, unparsed log event.

  2. @timestamp: Timestamp when the event occurred.

  3. @ingestionTime: Time the log was ingested by CloudWatch Logs.

  4. @logStream: Name of the log stream where the event resides.

  5. @log: Identifier in the form account-id:log-group-name (helps identify log groups across multiple queries).

Log Field Discovery by Log Type:

  • VPC Flow Logs: Includes fields like accountId, interfaceId, srcAddr, dstPort, packets, and protocol.

  • Route 53 Logs: Captures fields such as edgeLocation, queryName, responseCode, and resolverIp.

  • Lambda Logs: Includes Lambda-specific metrics such as @requestId, @duration, @memorySize, and optional X-Ray fields (@xrayTraceId).

  • CloudTrail Logs: Handled as JSON logs with fields accessible using dot notation.

  • Custom Logs: For logs without auto-discovered fields, use the parse command to extract data.

Dot Notation for JSON Logs:

  • Access Nested Fields: Example: "userIdentity.type" retrieves the value of the type field inside userIdentity.

  • Access Array Elements: Example: "requestParameters.instancesSet.items.0.instanceId" retrieves the first instanceId.

Sample Query Using Dot Notation:

fields @timestamp, @message
| filter requestParameters.instancesSet.items.0.instanceId="i-abcde123"
| sort @timestamp desc

This query retrieves log events where the instanceId matches "i-abcde123" and sorts them by timestamp in descending order.

Additional Notes:

  • Log Class Limitation: Field discovery is available only in Standard log class groups.

  • Naming Convention: CloudWatch Logs Insights adds @ to its generated fields. If a log field name already contains @, it will be displayed with an additional @ (e.g., @@example.com).

PreviousCloudWatch InsightsNextCloudWatch Insights Query Syntax

Last updated 8 months ago