AWSEventBridge

Filtering using Events Patterns – EventBridge

Amazon EventBridge as the name suggest is a serverless pub/sub allowing applications to connect via an “event bus”. It helps build loosely coupled and distributed event driven architecture. EventBridge was formerly called CloudWatch Events.

In this blog, I will give an example of setting filter based event pattern in Amazon EventBridge to send SNS notification.

I have been testing AWS Glue jobs with Job Bookmark enabled. These jobs are scheduled to run every hour, reading the data from cross account S3 bucket. Glue job fails with “AnalysisException” error when there are no new files in source S3 to be read causing EventBridge to trigger SNS topic to send a message. But as the source does not have any new data, no action is required. This is where EventBridge event pattern content filtering is handy, triggering under very specific conditions.

You can set the event pattern in Define pattern section of EventBridge under Amazon EventBridge > Events > Rules > Create rule.

Below is the event pattern set in EventBridge –

{
    "detail-type": [
        "Glue Job State Change"
    ],
    "source": [
        "aws.glue"
    ],
    "detail": {
        "jobName": [
            {
                "prefix": "DataLake-"
            }
        ],
        "state": [
            "FAILED"
        ],
        "message": [
            {
                "anything-but": {
                    "prefix": "AnalysisException"
                }
            }
        ]
    }
}

This event is triggered when glue job state change to failed. But target will receive the message only if it matches anything except what’s provided in the “anything-but” matching rule. It will match on any event, except those that have the prefix  AnalysisException for the "message" field. This helps reduce the number of alert you get.

Probably I need to handle the error in the Glue pyspark code itself, such that if source does have any new records to be processed, Glue job run state should be Succeeded. But until then EventBridge pattern matching is useful.

Content-based Filtering in EventBridge has lots of great options allowing you to create complex rules in event patterns.

Reference – https://docs.aws.amazon.com/eventbridge/latest/userguide/content-filtering-with-event-patterns.html#filtering-anything-but

Leave a Reply