Serverless Ephemeral

homepage icon https://github.com/Accenture/serverless-ephemeral
Follow @Accenture

Tracked

NPM Downloads Last Month
113
Issues
0
Stars
23
Forks
3
Watchers
23
Watch Star Fork Issue Download License NPM Build Status Coverage Status Contributors

Repo README Contents:

Serverless Ephemeral

NPM Version Build Status

Serephem

Serverless Ephemeral (or Serephem) is a Serverless Framework plugin that helps bundle any stateless library into a Lambda deployment artifact.

Pre-requirements

Examples

Add the plugin

  1. Install it

     npm i --save-dev serverless-ephemeral
    
  2. Add it to your serverless.yml file and exclude the .ephemeral directory

         plugins:
             - serverless-ephemeral
    
         package:
             exclude:
                 - package.json
                 - package-lock.json
                 - node_modules/**
                 - .ephemeral/**
    
  3. Add the .ephemeral directory to .gitignore

# Serverless Framework
.serverless
.ephemeral

Configuration

The configuration for the Ephemeral plugin is set inside the custom section of the serverless.yml file. In it, you can define the list of stateless libraries you wish to pull into the final Lambda artifact.

There are two types of configuration:

Both can be enhanced with global configuration options.

Build a library

You can build a specific library during runtime. This is achieved via a Docker container that outputs a zip library.

The Serepehm plugin provides some useful packagers out of the box. However, you can create your own packager via Docker files.

Serephem packagers

You can use one of the Docker packagers provided with the Serephem plugin.

TensorFlow
custom:
  ephemeral:
    libraries:
      - packager:
          name: tensorflow
          version: 1.4.0

Build your own packager

You can create your own packager via Docker. To do so:

  1. Create a directory where you will store all your Docker files:

     mkdir my-packager
     cd my-packager
    
  2. Create a docker-compose.yml file. For example:

     version: '3'
     services:
       packager:
         build: .
    

    Keep note of the name of your packager service, in this case packager.

  3. Create a Dockerfile and any other support files. For example:

    Dockerfile

     FROM amazonlinux
    
     COPY scripts/build.sh scripts/build.sh
     RUN yum -y install zip && \
         chmod +x scripts/build.sh
    
     CMD [ "scripts/build.sh" ]
    

    scripts/build.sh

     # create zip destination directory
     mkdir -p /tmp/lambda-libraries
    
     # download library files
     mkdir /tmp/files
     cd /tmp/files
     curl http://example.com/file-1.py --output file-1.py
     curl http://example.com/file-2.py --output file-2.py
     zip -9rq /tmp/lambda-libraries/library-a.zip *
    

    IMPORTANT: the container must generate a zip file containing the stateless library files. Thus:

    • Your container must zip the stateless library files.

    • You must create a directory where the final zip(s) will be stored. This directory will be mounted to the Serephem’s libraries directory, so add only the necessary zip files.

    • It is recommended that your Docker container extends from amazonlinux image to maximize compatibility with the Lambda environment.

  4. Add this configuration to your serverless.yml:

     custom:
       ephemeral:
         libraries:
         - packager:
             compose: my-packager/docker-compose.yml
             service: packager
             output: /tmp/lambda-libraries/library-a.zip
    

    Notice how each of the values correspond to a setting previously created:

    • compose: points to your Docker compose file, inside the directory you created

    • service: the name of the service inside the docker-compose.yml file

    • output: the output path for the zip file in the Docker container

Download a library

custom:
  ephemeral:
    libraries:
      - url: https://xxxxx.s3.amazonaws.com/tensorflow-1.3.0-cp27-none-linux_x86_64.zip

Documentation explaining how to create the deployable TensorFlow zipped package can be found here: docs/build-tensorflow-package.md. This approach can be used as a base to create other stateless libraries.

Global options

custom:
  ephemeral:
    libraries:
      - packager:
          name: tensorflow
          version: 1.4.0
        directory: tfpackage
      - url: https://xxxxx.s3.amazonaws.com/boto3.zip
        nocache: true

Deploy

  1. Deploy your service normally with the serverless deploy (or sls deploy) command. If you use the -v option, Ephemeral will show more information about the process.

     sls deploy -v
    

    If the Serverless deployment is timing out, use the AWS_CLIENT_TIMEOUT environment variable: https://github.com/serverless/serverless/issues/490#issuecomment-204976134

The .ephemeral directory

During the deployment process, the .ephemeral directory will be created. The purpose of this directory is:


Contribute

This plugin is created with Node and uses the Serverless Framework hooks to execute the necessary actions.

Installation

  1. Clone this repository

     git clone https://github.com/Accenture/serverless-ephemeral.git
    
  2. Install the node dependencies

     npm i
    

Running Lint

The plugin code uses the AirBnB ESLint rule set with some enhancements (see .eslintrc file). To run the linter:

npm run lint

Tests

The unit tests are coded with Ava and SinonJS. They can be found inside the spec folder.

To run the tests:

npm test

To run tests on “watch” mode and add verbosity:

npm test -- --watch -v

Test via examples

Refer to the examples directory, for instance the TensorFlow example.