Serverless Plugin Pubsub

homepage icon https://github.com/fivepapertigers/serverless-plugin-pubsub
Follow @fivepapertigers

Tracked

NPM Downloads Last Month
50
Issues
0
Stars
0
Forks
0
Watchers
0
Watch Star Fork Issue Download License NPM Build Status Coverage Status Contributors

Repo README Contents:

Serverless PubSub

Motivation

Publish-subscribe is a pattern used in message mediated architectures that is meant to decouple logical units in a codebase or software system. Serverless computing, or more specifically, Function-as-a-Service computing, caters rather well to a messaging infrastructure, since “serverless” functions are configured and priced to be ephemeral units of execution that allow for easy chaining.

Many of the major platforms offer their own tooling and services for implementing pub/sub in your serverless stacks. In the AWS landscape, pub/sub is made possible through the Simple Notification Service, which exposes topics that to which functions can either publish or subscribe. As one of AWS’s older offerings, SNS is well-understood, feature-rich, and has a native with the Serverless framework.

However, there are a number of headaches that one encouters when using SNS as a serverless pub/sub engine, particularly when used with the Serverless framework:

With those limitations in mind, this plugin aims to provide the following for those looking to implement easy and readable pub/sub into their configurations:

Installation

Install the plugin with:

npm install --save-dev serverless-plugin-pubsub

And add the following to your Serverless confing:

plugins:
  - serverless-plugin-pubsub

Usage

Subscriber

To add a pub/sub event, simply use the pubSub event syntax on a new or existing function. In its simplest form, all you need to provide is a topic:

functions:
  myTopicConsumer:
    handler: mymodule.myhandler
    events:
      - pubSub: my-first-topic

You can also use the topic key to provide the name:

functions:
  myTopicConsumer:
    handler: mymodule.myhandler
    events:
      - pubSub:
          topic: my-first-topic

It is often advantageous to have a message queue as an intermediary between your topic and your function. This can be used to throttle message activity (using reserved function concurrency) during times of heavy load on the system.

To place a queue as an intermediary between your topic and your , simply set the queue or queue.name parameter to true (it is false, by default) or to the desired name of the queue. If you do not specify a queue name, one will be generated based on the function name.

functions:
  myTopicConsumer:
    handler: mymodule.myhandler
    events:
      - pubSub:
          topic: my-first-topic
          queue: true # resolves to myTopicConsumer-queue

Publisher

To inject a SNS topic Arn into the runtime environment of a function, simply use the ${pubSubTopic:<topic key>} syntax in the environment section of the function config.

functions:
  myPublisher:
    handler: mymodule.myhandler
    environment:
      PUBLISH_TOPIC_ARN: ${pubSubTopic:my-first-topic}

In the Lambda runtime, this will resolve as the SNS Topic Arn. The following is an example of using in a NodeJS publisher:

//handler.js
const { SNS } = require('aws-sdk');

module.exports = {
  handler: async () => {
    await SNS.publish({
      Message: 'hello world',
      TopicArn: process.env.PUBLISH_TOPIC_ARN // e.g. arn:aws:sns:us-east-1:123456789:my-first-topic
    }).promise())
  }
};

Note: if a topic is referenced with the pubSubTopic anywhere in the stack, it will be created regardless of whether it has a corresponding subscriber. This allows for a lot of important configurations, such as cross-service subscription and future extensibility.

Advanced Configuration

More detailed CloudFormation configuration can be added to both topics and queues in order to tweak the performance or behavior of the service. These are configured in the custom section of the serverless configuration, and not at the function level. This ensures that if a topic or queue is referenced in multiple places, they will be deployed using only one configuration. The topic and queue accept parameters for AWS::SNS::Topic and AWS::SQS::Queue respectively.

functions:
  myTopicConsumer:
    handler: mymodule.myhandler
    events:
      - pubSub:
          topic: my-first-topic
          queue: myTopicConsumer-queue
  myPublisher:
    handler: mymodule.myhandler
    environment:
      PUBLISH_TOPIC_ARN: ${pubSubTopic:my-first-topic}

custom:
  pubSub:
    topics:
      my-first-topic:
        DisplayName: MyFirstTopic
    queues:
      myTopicConsumer-queue:
        MessageRetentionPeriod: 60

Cloudformation configuration can also be added to the topic/queue subscriptions, by using the subscription keyword. This should be specified at the functions since a subscription is inherently unique. The topic.subscription and queue.subscription accept Cloudformation properties for AWS::SNS::Subscription and AWS::Lambda::EventSourceMapping respectively. Note that if a queue is not specified, the AWS::SNS::Subscription will use the lambda protocol, not the sqs protocol.

functions:
  myTopicConsumer:
    handler: mymodule.myhandler
    events:
      - pubSub:
          topic:
            name: my-first-topic
            subscription:
              DeliveryPolicy:
                throttlePolicy:
                  maxReceivesPerSecond: 3
          queue:
            name: myTopicConsumer-queue
            subscription:
              BatchSize: 1

It is often the case that you want to set default values for Cloudformation configuration across the entire service. You can do that using custom.pubSub.defaults.(queues|topics|queueSubscriptions|topicSubscriptions). Note that defaults will be overwritten if defined at the resource level.

custom:
  pubSub:
    defaults:
      topics:
        KmsMasterKeyId: my-key-id
      queues:
        MessageRetentionPeriod: 60
      queueSubscriptions:
        BatchSize: 1
      topicSubscriptions:
        DeliveryPolicy:
          throttlePolicy:
            maxReceivesPerSecond: 3

Contributing

Please open a Github issue with any bug reports or feature suggestions.

Feel free to open a pull request with any additions or enhancements. A failing test would be appreciated to understand the issue better and prevent it from recurring in the future.

To install:

npm install

To run tests:

npm test
# or
npm test --- --watch