Distributed Load Testing on AWS Lambda
What you'll learn
- How to scale out your Artillery tests using built-in AWS Lambda support
- What AWS resources Artillery creates on your behalf to run your tests
- Current limitations in AWS Lambda support in Artillery
Overview
This guide describes Artillery's support for running highly-distributed serverless load tests on AWS Lambda.
AWS credentials
To execute tests in AWS Lambda the Artillery CLI makes use of the official AWS SDK (opens in a new tab) to create the resources needed to run your tests (see the AWS Resources seciton for details on what Artillery creates).
The SDK requires AWS credentials (opens in a new tab) to be present to work. Please refer to the official AWS documentation if you don't have one set up already: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html (opens in a new tab)
Please see the IAM Permissions section for details on permissions required to run tests from AWS Lambda.
Running tests from AWS Lambda
To run an existing test script from AWS Lambda, add --platform
and --platform-opt
flags to the run
command.
For example, if you have a test script saved in blitz.yml
and you want to run it from eu-west-1
region, run the following command:
artillery run-lambda \
--region eu-west-1 \
blitz.yml
How it works
AWS resources created
Artillery will create a number of AWS resources behind the scenes to be able to execute your tests. All resources created by Artillery are serverless and created on-demand. There are no long-running infrastructure components involved.
- An S3 bucket to store an AWS Lambda deployment package generated from your test script
- An AWS Lambda function. This function is deleted once the test run completes.
- An SQS queue for communication between Lambda workers executing your test and the Artillery CLI. This queue is deleted once the test run completes.
- (Optional) An IAM role named
artilleryio-default-lambda-role
for Lambda functions running your test. This role uses the official AWS-managedAWSLambdaBasicExecutionRole
andAWSLambdaVPCAccessExecutionRole
(opens in a new tab) policies, and an additional policy which grants access to the SQS queues created for these tests. If a role with that name already exists, Artillery will use it instead of creating it.
IAM permissions
The AWS profile that the Artillery CLI runs under when you run artillery run
needs to have sufficient permissions to be able to create the resources listed above.
An example IAM policy definition is below. The AWS IAM user you're using will need permissions to assume a role which makes use of the policy created with this template.
Note that 123456789000
will need to be replaced with the id of the AWS account you'll be using.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CreateOrGetLambdaRole",
"Effect": "Allow",
"Action": ["iam:CreateRole", "iam:GetRole", "iam:PassRole"],
"Resource": "arn:aws:iam::123456789000:role/artilleryio-default-lambda-role-*"
},
{
"Sid": "CreateLambdaPolicy",
"Effect": "Allow",
"Action": ["iam:CreatePolicy", "iam:AttachRolePolicy"],
"Resource": "arn:aws:iam::123456789000:policy/artilleryio-lambda-policy"
},
{
"Sid": "SQSPermissions",
"Effect": "Allow",
"Action": ["sqs:*"],
"Resource": "arn:aws:sqs:*:123456789000:artilleryio*"
},
{
// ListQueues does cannot be scoped to individual resources
// https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonsqs.html#amazonsqs-queue
"Sid": "SQSListQueues",
"Effect": "Allow",
"Action": ["sqs:ListQueues"],
"Resource": "*"
},
{
"Sid": "LambdaPermissions",
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction",
"lambda:CreateFunction",
"lambda:DeleteFunction",
"lambda:GetFunctionConfiguration"
],
"Resource": "arn:aws:lambda:*:123456789000:function:artilleryio-*"
},
{
"Sid": "S3Permissions",
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket",
"s3:GetLifecycleConfiguration",
"s3:PutLifecycleConfiguration"
],
"Resource": [
"arn:aws:s3:::artilleryio-test-data-*",
"arn:aws:s3:::artilleryio-test-data-*/*"
]
}
]
}
Setting up the IAM Policy and Role in AWS
To help you get started, here's a small guide to set up the necessary role and policy in AWS. It's especially important to have this created if you're planning on using Artillery in CI/CD.
IAM Policy: You should first create the policy. You can do that from the AWS UI or CLI. To do it from the UI:
- Go to
IAM -> Policies
and selectCreate Policy
. SelectJSON
. - Copy the JSON policy from the section above. You’ll need to change the following:
- AWS account from
123456789000
to our account ID (e.g. thetest1
account ID); - Remove any comments from the JSON policy, as that is not allowed;
- AWS account from
- Press
Next
and give the policy a meaningful name.
IAM Role: After the policy created, you'll need to use that policy in a role:
- Go to IAM -> Roles and select Create Role;
- Select
Custom Trust Policy
. You can keep the default generated or customise it based on your organisational needs; - After, on the
Add Permissions
UI, select the policy created in the previous step; - Finally, press
Next
and give the role a meaningful name.
That's it! You should now be able to use this role to run Artillery tests in Lambda.
If you're setting up OIDC, for example to interact with Github Actions (opens in a new tab), then you might need to also set a Custom Trust Policy. You can do so by editing the Custom Trust Policy in step 2 with the appropriate policy from the GH Actions guide linked above.
Limitations
AWS Lambda support is in preview. There are some limitations to what's possible, and you may run into bugs. Please report any issues via GitHub issues on https://github.com/artilleryio/artillery/issues (opens in a new tab)
Test duration
Each AWS Lambda is limited to 15 minutes (opens in a new tab) of running time, which means that the entire load test cannot run for longer than 15 minutes at the moment.
A running test cannot be stopped
Once an AWS Lambda starts running, there is no way to stop it. Neither the AWS SDK, nor the AWS Console provide that ability.
This means that once a load test starts, it will run to completion. Be mindful of this, and ramp up load on your applications gradually.
Unavailable Artillery features
The following Artillery features are not supported yet:
- Only built-in engines (HTTP, WebSocket and Socket.io) may be used
- Only built-in
publish-metrics
andmetrics-by-endpoint
plugins may be used - Custom JS code provided with
config.processor
cannot yet use any third-party npm packages - Using any third-party plugins or engines
--target
, and--insecure
flags for therun
commandensure
specsbefore
andafter
hooks run once in each Lambda worker rather than once per test run
Questions?
- Post your questions on the community forum on GitHub at https://github.com/artilleryio/artillery/discussions (opens in a new tab)
- Report a bug or raise an issue via https://github.com/artilleryio/artillery/issues (opens in a new tab)