☁️
CTHFM: AWS
  • Welcome
  • Getting Started
    • Account Setup
  • AWS CLI
    • AWS CLI Overview
    • Installation
  • AWS Fundamentals
    • AWS Documentation
    • AWS Shared Responsibility Model
    • Organizational Hierarchy
    • AWS Principals
    • IAM Fundamentals
      • IAM Policy Components
      • IAM Documentation References
    • AWS Security Services Overview
    • AWS Core Services
    • AWS Frameworks
    • Regions and Availability Zones
  • SQL
    • SQL Refresher for Threat Hunting
  • Logging Reference
    • Cloudtrail
      • What is Cloudtrail?
      • Setting Up Cloudtrail
      • Cloudtrail Events Structure
      • Filtering and Searching CloudTrail Logs
      • IAM ID Prefixes
      • Additional Resources
      • API References
    • VPCFlow Logs
    • GuardDuty
      • Multi-Account Setup
      • GuardDuty Concepts
      • GuardDuty Finding References
      • S3 Protection
      • Malware Protection
        • EC2 Malware Protection
          • EC2 Protection Resources
          • Monitoring Scans
          • EC2 Malware Protection Events: CloudWatch
        • S3 Malware Protection
          • Enabling S3 Malware Protection
          • After Enabling S3 Malware Protection
          • S3 Malware Resource Plan Status
          • S3 Malware Protection Quotas
      • RDS Protection Enablement
      • Lambda Protection Enablement
      • Trusted IP Lists and Threat Lists in Amazon GuardDuty
      • Remediation Recommendations
      • GuardDuty API Reference
      • GuardDuty Quotas
    • Access Analyzer
      • Setup
      • External Access and Unused Access Analyzer Findings
      • Review Findings
      • Access Analyzer Resources
      • Access Analyzer API Reference
    • AWS Network Firewall
      • Permissions
      • Firewall Log Contents
      • Logging Destinations
      • CloudWatch Firewall Metrics
    • AWS Config
      • Resource Management in AWS Config
      • AWS Config Integrations
      • AWS Config Resources
      • Configuration Item
      • Config Rules
        • Evaluation Modes
  • CloudWatch
    • Amazon CloudWatch
      • CloudWatch Concepts
      • CloudWatch Metrics
        • Filter Pattern Syntax
      • CloudWatch Alarms
        • Alarm Recommendations
      • Subscriptions
      • CloudWatch Agent
      • CloudWatch Insights
        • Supported Logs and Discovered Fields
        • CloudWatch Insights Query Syntax
      • Anomaly Detection
        • Create Anomaly Detector
        • Alarms for Anomaly Detections
      • CloudWatch Filter Syntax
      • CloudWatch Service Quota
  • Athena For Threat Hunting
    • Introduction to Athena
    • Setting Up Athena
    • SQL For Threat Hunters
    • Automated Response
    • Query Best Practices
  • AWS Security Research and Resources
    • AWS Security Blog
    • AWS Goat
    • Cloud Goat
    • Pacu
    • Prowler
    • Scout Suite
  • Threat Hunting in AWS
    • Threat Hunting in AWS
    • Threat Hunting Introduction
    • Threat Hunting Process
      • Hypothesis Generation
      • Investigation
      • Identification
      • Resolution & Follow Up
    • Pyramid of Pain
    • MITRE Att&ck
      • MITRE Att&ck Concepts
      • MITRE Att&CK Data Sources
      • MITRE Att&CK Mitigations
    • MITRE Att&ck: AWS
      • MITRE Att&CK Matrix
      • Amazon Web Services Security Control Mappings
    • AWS Threat Hunting Ideas
      • AWS Threat Hunting Ideas: EC2
      • AWS Threat Hunting Ideas: Lambda
      • AWS Threat Hunting Ideas: SQS
      • AWS Threat Hunting Ideas: SNS
      • AWS Threat Hunting Ideas: RDS
Powered by GitBook
On this page
  • Overview
  • Key Features:
  • Examples of Flagged Anomalies:
  1. CloudWatch
  2. Amazon CloudWatch

Anomaly Detection

Overview

CloudWatch Logs Anomaly Detection leverages machine learning and pattern recognition to detect anomalies in log events. It identifies unusual behaviors by establishing baselines based on past log patterns and monitoring deviations from expected trends.

Key Features:

  1. Anomaly Detection Process:

    • Uses two weeks of historical logs for initial training, which takes up to 15 minutes.

    • Analyzes incoming logs continuously to identify and display anomalies in the CloudWatch Logs console.

    • Automatically extracts patterns by recognizing static and dynamic content (e.g., timestamps or request IDs).

  2. Pattern Recognition & Tokens:

    • Log patterns are expressed using tokens; dynamic tokens, such as varying request IDs, are represented as <*>.

    • Example Pattern:

      <*> <*> [INFO] Calling DynamoDB to store for resource id <*>

    • Patterns simplify the analysis of large log sets, compressing many log events into a few patterns.

  3. Anomaly Visibility & Suppression:

    • Default visibility period for anomalies is 21 days (can be adjusted). After this period, recurring anomalies are accepted as normal behavior.

    • Users can suppress anomalies temporarily or permanently to prevent unnecessary alerts. Suppression can apply to individual anomalies or entire patterns.

  4. Severity & Priority:

    • Severity is based on keywords like ERROR, WARN, FATAL; logs without such keywords have a severity of NONE.

    • Priority considers both severity and deviation from expected values. For example, a 500% spike in a token value could trigger a HIGH priority anomaly.

  5. Suitable Log Types:

    • Best suited for application logs with typical patterns and log levels (INFO, ERROR).

    • Less effective for network or access logs (e.g., VPC flow logs) and long JSON logs (e.g., CloudTrail logs). Pattern analysis is limited to the first 1500 characters of each log event.

  6. Privacy & Data Usage:

    • Anomaly detection models are trained per log group and account-specific; AWS does not use customer logs to train models for other customers.

Examples of Flagged Anomalies:

  • Unseen Patterns: A log event with a new structure.

  • Pattern Deviations: Unexpected changes in a known pattern.

  • New Token Values: A dynamic token value outside its usual set.

  • Sudden Spikes: A sharp increase in token occurrences (e.g., a surge in HTTP 200 responses).

PreviousCloudWatch Insights Query SyntaxNextCreate Anomaly Detector

Last updated 8 months ago