Skip to main content

Distributed tests on AWS Lambda

What you'll learn
  • How to scale out your Artillery tests using built-in AWS Lambda support
  • What AWS resources Artillery creates on your behalf to run your tests
  • Current limitations in AWS Lambda support in Artillery

Overview​

This guide describes Artillery's support for running highly-distributed serverless load tests on AWS Lambda.

Getting Started​

Install Artillery​

AWS Lambda support is available in Artillery v2.0.0-21 or later.

You can the version of Artillery installed on your system with:

artillery version

Folllow the Artillery installation guide for details on how to install the latest version of Artillery.

AWS credentials​

To execute tests in AWS Lambda the Artillery CLI makes use of the official AWS SDK to create the resources needed to run your tests (see the AWS Resources seciton for details on what Artillery creates).

The SDK requires AWS credentials to be present to work. Please refer to the official AWS documentation if you don't have one set up already: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Please see the IAM Permissions section for details on permissions required to run tests from AWS Lambda.

Running tests from AWS Lambda​

To run an existing test script from AWS Lambda, add --platform and --platform-opt flags to the run command.

For example, if you have a test script saved in blitz.yml and you want to run it from eu-west-1 region, run the following command:

artillery run \
   --platform aws:lambda \
   --platform-opt region=eu-west-1 \
   blitz.yml

AWS resources created​

Artillery will create a number of AWS resources behind the scenes to be able to execute your tests. All resources created by Artillery are serverless and created on-demand. There are no long-running infrastructure components involved.

  • An S3 bucket to store an AWS Lambda deployment package generated from your test script
  • An AWS Lambda function. This function is deleted once the test run completes.
  • An SQS queue for communication between Lambda workers executing your test and the Artillery CLI. This queue is deleted once the test run completes.
  • (Optional) An IAM role named artilleryio-default-lambda-role for Lambda functions running your test. This role uses the official AWS-managed AWSLambdaBasicExecutionRole and AWSLambdaVPCAccessExecutionRole policies, and an additional policy which grants access to the SQS queues created for these tests. If a role with that name already exists, Artillery will use it instead of creating it.

IAM permissions​

The AWS profile that the Artillery CLI runs under when you run artillery run needs to have sufficient permissions to be able to create the resources listed above.

An example IAM policy definition is below. The AWS IAM user you're using will need permissions to assume a role which makes use of the policy created with this template.

Note that 123456789000 will need to be replaced with the id of the AWS account you'll be using.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CreateOrGetLambdaRole",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole"
            ],
            "Resource": "arn:aws:iam::123456789000:role/artilleryio-default-lambda-role"
        },
        {
            "Sid": "CreateLambdaPolicy",
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:AttachRolePolicy"
            ],
            "Resource": "arn:aws:iam::123456789000:policy/artilleryio-lambda-policy"
        },
        {
            "Sid": "SQSPermissions",
            "Effect": "Allow",
            "Action": [
                "sqs:*"
            ],
            "Resource": "arn:aws:sqs:*:123456789000:artilleryio*"
        },
        {
            // ListQueues does cannot be scoped to individual resources
            // https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonsqs.html#amazonsqs-queue
            "Sid": "SQSListQueues",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": "*"
        },
        {
            "Sid": "LambdaPermissions",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeAsync",
                "lambda:CreateFunction",
                "lambda:DeleteFunction"
            ],
            "Resource": "arn:aws:lambda:*:123456789000:function:artilleryio-*"
        },
        {
            "Sid": "S3Permissions",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetLifecycleConfiguration",
                "s3:PutLifecycleConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::artilleryio-test-data-*",
                "arn:aws:s3:::artilleryio-test-data-*/*"
            ]
        }
    ]
}

Limitations​

AWS Lambda support is in preview. There are some limitations to what's possible, and you may run into bugs. Please report any issues via GitHub issues on https://github.com/artilleryio/artillery/issues

Test duration​

Each AWS Lambda is limited to 15 minutes of running time, which means that the entire load test cannot run for longer than 15 minutes at the moment.

A running test cannot be stopped​

Once an AWS Lambda starts running, there is no way to stop it. Neither the AWS SDK, nor the AWS Console provide that ability.

This means that once a load test starts, it will run to completion. Be mindful of this, and ramp up load on your applications gradually.

Unavailable Artillery features​

The following Artillery features are not supported yet:

  • Only built-in engines (HTTP, WebSocket and Socket.io) may be used
  • Only built-in publish-metrics and metrics-by-endpoint plugins may be used
  • Custom JS code provided with config.processor cannot yet use any third-party npm packages
  • Using any third-party plugins or engines
  • --target, and --insecure flags for the run command
  • ensure specs
  • before and after hooks run once in each Lambda worker rather than once per test run

Questions?​

Table of contentsOn this page