Guest Post: Developing a Serverless Scheduler Using DynamoDB TTL and Filtered Streams

Today's guest post is from Frédéric Barthelet. He has led Theodo’s serverless initiative since 2017 and contributes to the Serverless Framework and is building the expertise with his team. He is part of the AWS Community Builder program and helps the Typescript community with an improved developer-experience. Frédéric also is a DIY-er lover and an IoT enthusiastic. Learn more about him on Github and Twitter!

Let us know if you'd like to be a guest author on the Serverless blog!

AWS announced Lambda event filtering during re:Invent 2021. This new feature helps reduce the quantity of Lambda's invocation to exactly match your requirements and possibly save on costs. We'll deep dive into a specific example in this article.

We're happy to announce that Serverless Framework now supports event filtering for both stream and sqs event types via the new filterPatterns option:

functions:
  filteredEventHandler:
    handler: handler.main
    events:
      - stream:
          arn: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000
          filterPatterns:
          - eventName: [INSERT]
      - sqs:
        arn: arn:aws:sqs:region:XXXXXX:myQueue
        filterPatterns:
          - a: [1, 2]

Upgrade to v2.68.0 or greater to start using it. In case you are trying out Serverless Framework v3 beta, make sure to update the v3 beta with "npm -g i serverless@pre-3".

Scheduler Pattern

Applications that need to perform a specific action at a certain time (sending out a marketing communication regarding a new menu just before lunch time i.e.) usually rely on a polling strategy, periodically checking in a database if any upcoming tasks are due. Polling however unnecessarily requires compute resources to not perform anything the majority of the time.

An alternative is the scheduler pattern. It leverages DynamoDB TTL (Time To Live) and DynamoDB stream features to be much more efficient in resource consumption.

Scheduled tasks are inserted in DynamoDB in the form of an item with a timestamp attribute mapped to the DynamoDB TTL attribute. This timestamp corresponds to the moment you'd like this task to be processed. Anything can schedule a task as long as it can write a new item in the DynamoDB table.

When the task scheduled time is reached, DynamoDB deletes the corresponding item. A REMOVE event is then published in the table stream, event that can be processed by Lambda to execute the actions corresponding to the task.

Additional information can be found related to this pattern, its advantages and drawbacks in Yan Cui's article.

It is critical that the task processor logic filters out events corresponding to INSERT or MODIFY operations within the table. Only REMOVE events correspond to scheduled tasks and must be processed accordingly. Newly inserted scheduled task will result in a INSERT event being published in DynamoDB stream, triggering the processor as well. Those events must be discarded. A common way to achieve this is the following:

export const main = async (event) => {
  event.Records.forEach(({ eventName, dynamodb }) => {
    // filtering out Records that are not REMOVE operations
    if (eventName !== 'REMOVE') {
      return;
    }

    // Unmarshalling removed item to get task details
    const task = Converter.unmarshall(dynamodb.OldImage);
    // Task processing logic
    return processTask(task);
  });
};

Improving scheduler pattern with the event filtering feature

Using the new event filtering feature, you can ensure only REMOVE operations actually trigger the Lambda. The following corresponding configuration leverage the new filterPatterns property:

functions:
  scheduledTasksProcessor:
    handler: handler.main
    events:
      - stream:
          arn: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000
          filterPatterns:
            - eventName: [REMOVE]

The scheduledTasksProcessor can then be simplified, removing the need to check for the  eventName:

export const main = async (event) => {
  // Thanks to filterPatterns, all Records are REMOVE records
  event.Records.forEach(({ dynamodb }) => {
    // Unmarshalling deleted item to get task details
    const task = Converter.unmarshall(dynamodb.OldImage);
    // Task processing logic
    return processTask(task);
  });
};

This improved pattern both reduces processor code base and improves readability. It also reduces the quantity of invocation for this processor, thus reducing overall costs of the pattern.

If you want to learn more, check out the official AWS documentation on Lambda event filters as well as the Framework documentation.