Filter Pattern Syntax

Overview

Metric filters in CloudWatch allow you to search log data for specific patterns and convert matching events into numerical metrics. These metrics are useful for monitoring trends, setting alarms, and visualizing key metrics in dashboards.

---

How Metric Filters Differ from Logs Insights Queries:

• Metric Filters: Each match increments a metric with a specified numerical value.

• Logs Insights Queries: Used to query historical logs but do not generate metrics.

---

How Metric Filters Work:

1. Filter Patterns: Define how CloudWatch interprets log data (e.g., matching keywords, JSON fields, or space-delimited entries).

2. Metrics Generation: A metric is created for every match, with an increment value specified.

3. Dimensions: Key-value pairs (e.g., Service:API) provide context for metrics.

4. Default Value: If no match is found during a period, CloudWatch reports a default value (e.g., 0) to prevent missing data.

Examples of Filter Patterns:

• JSON Log Events: Extract specific values, like latency:

  { $.latency = * } metricValue: $.latency

  Example Log Entry:

  {
    "latency": 50,
    "requestType": "GET"
  }

  This filter publishes a latency value of 50 to the metric.

• Space-Delimited Log Events: Example log:

  127.0.0.1 Prod frank [10/Oct/2000:13:25:15 -0700] "GET /index.html HTTP/1.0" 404 1534

  This type of log can be filtered using logical operators (e.g., AND, OR) to extract metrics based on different fields.

Creating a Metric Filter:

1. Open the CloudWatch console.

2. Navigate to Logs > Log Groups and select or create a log group.

3. Choose Actions > Create metric filter.

4. Define the filter pattern (e.g., { $.latency = * }).

5. Set the metric name and value (e.g., latency).

6. (Optional) Set a default value (e.g., 0) to ensure continuous reporting.

7. Create the filter, and CloudWatch will aggregate the metric every minute.

Important Considerations:

• JSON and Space-Delimited Logs: You can create filters for up to three dimensions from these logs.

• Avoid High-Cardinality Fields: Using fields like IPAddress or requestID as dimensions may lead to unexpected costs and potential filter deactivation.

• Default Value: Always set a default value (e.g., 0) to avoid gaps in metrics.

Benefits of Metric Filters:

• Track Key Events: Monitor occurrences of keywords like "ERROR."

• Extract Values Dynamically: Use JSON logs to publish specific metrics (e.g., latency).

• Set Alarms: Use metrics to trigger alarms based on thresholds or trends.