Distributed Load Testing on AWS Fargate
What you'll learn
- How to scale out your Artillery tests using built-in AWS Fargate support
- What AWS resources Artillery creates on your behalf to run your tests
Overview
This guide describes how to run high-scale distributed load tests with Artillery on AWS Fargate.
AWS credentials
To execute tests on AWS Fargate the Artillery CLI makes use of the official AWS SDK (opens in a new tab) to create the resources needed to run your tests (see the AWS Resources seciton for details on what Artillery creates).
The SDK requires AWS credentials (opens in a new tab) to be present to work. Please refer to the official AWS documentation if you don't have one set up already: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html (opens in a new tab)
Please see the IAM Permissions section for details on permissions required to run tests from AWS Fargate.
Running tests from AWS Fargate
To run an existing test script from AWS Fargate, use run-fargate
command instead of the run
command.
For example, if you have a test script saved in blitz.yml
and you want to run it from eu-west-1
region, run the following command:
artillery run-fargate \
--region eu-west-1 \
blitz.yml
How it works
When you run a load test with AWS Fargate, Artillery tries to make it as seamless as possible.
Bundle load test
Artillery analyzes the test script and packages up all of its dependencies into a bundle to make it available to Fargate containers executing the test. Those dependencies include:
- The test script itself, which may be split across more than one YAML file
- CSV data files used by the test
- Third-party plugins and engines
- Custom JavaScript code and all of its dependencies
Create AWS resources
If required, Artillery creates an IAM role for Fargate containers and an empty Fargate cluster for the test to execute on.
Create load generators
Artillery spins up a number of Fargate containers (specified with the --count
flag by the user), distributed the test bundle to all of them, and waits for all of them to sync up and become available.
Aggregate results from load generators
Once load generators are running, Artillery aggregates reports from individual containers and aggregates them in a statistically correct way to present to the user.
Clean up AWS resources
When the test finishes, Artillery removes AWS resources that are no longer needed.
AWS resources created
Artillery will create a number of AWS resources behind the scenes to be able to execute your tests. All resources created by Artillery are serverless and created on-demand. There are no long-running infrastructure components involved.
- An S3 bucket to store test bundles
- An SQS queue for communication between Fargate containers executing your test and the Artillery CLI. This queue is deleted once the test run completes.
- (Optional) An IAM role named
artilleryio-ecs-worker-role
for Fargate tasks that execute the test. If a role with that name already exists, Artillery will use it instead of creating it.
IAM permissions
The AWS profile that the Artillery CLI runs under needs to have sufficient permissions to be able to create the resources listed above.
If running in a sandbox/developer account, the easiest way is to run with admin
privileges using the default AWS AdministratorAccess
(opens in a new tab) managed policy.
An definition of IAM policy to use instead of AdministratorAccess
is below. The AWS IAM user you're using will need permissions to assume a role which makes use of the policy created with this template.
Note that 123456789000
will need to be replaced with the id of the AWS account you'll be using.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CreateOrGetECSRole",
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:GetRole"
],
"Resource": "arn:aws:iam::123456789000:role/artilleryio-ecs-worker-role"
},
{
"Sid": "CreateECSPolicy",
"Effect": "Allow",
"Action": [
"iam:CreatePolicy",
"iam:AttachRolePolicy"
],
"Resource": "arn:aws:iam::123456789000:policy/ecs-worker-policy"
},
// Allow Artillery CLI to create AWS service role for ECS when creating a Fargate cluster
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html#create-service-linked-role
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:aws:iam::*:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS*"
],
"Condition": {
"StringLike": {
"iam:AWSServiceName": "ecs.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": ["iam:PassRole"],
"Resource": ["arn:aws:iam::123456789000:role/artilleryio-ecs-worker-role"]
},
{
"Sid": "SQSPermissions",
"Effect": "Allow",
"Action": [
"sqs:*"
],
"Resource": "arn:aws:sqs:*:123456789000:artilleryio*"
},
{
// ListQueues cannot be scoped to individual resources
// https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonsqs.html#amazonsqs-queue
"Sid": "SQSListQueues",
"Effect": "Allow",
"Action": [
"sqs:ListQueues"
],
"Resource": "*"
},
{
"Sid": "ECSPermissionsGeneral",
"Effect": "Allow",
"Action": [
"ecs:ListClusters",
"ecs:CreateCluster",
"ecs:RegisterTaskDefinition",
"ecs:DeregisterTaskDefinition"
],
"Resource": "*"
},
{
"Sid": "ECSPermissionsScopedToCluster",
"Effect": "Allow",
"Action": [
"ecs:DescribeClusters",
"ecs:ListContainerInstances"
],
"Resource": "arn:aws:ecs:*:123456789000:cluster/*"
},
{
"Sid": "ECSPermissionsScopedWithCondition",
"Effect": "Allow",
"Action": [
"ecs:SubmitTaskStateChange",
"ecs:DescribeTasks",
"ecs:ListTasks",
"ecs:ListTaskDefinitions",
"ecs:DescribeTaskDefinition",
"ecs:StartTask",
"ecs:StopTask",
"ecs:RunTask"
],
"Condition": {
"ArnEquals": {
"ecs:cluster": "arn:aws:ecs:*:123456789000:cluster/*"
}
},
"Resource": "*"
},
{
"Sid": "S3Permissions",
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectTagging",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetBucketLogging",
"s3:GetBucketPolicy",
"s3:GetBucketTagging",
"s3:PutBucketPolicy",
"s3:PutBucketTagging",
"s3:PutMetricsConfiguration",
"s3:GetLifecycleConfiguration",
"s3:PutLifecycleConfiguration"
],
"Resource": [
"arn:aws:s3:::artilleryio-test-data-*",
"arn:aws:s3:::artilleryio-test-data-*/*"
]
},
{
"Effect": "Allow",
"Action": ["secretsmanager:GetSecretValue"],
"Resource": ["arn:aws:secretsmanager:*:123456789000:secret:artilleryio/*"]
},
{
"Effect": "Allow",
"Action": [
"ssm:PutParameter",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:DeleteParameter",
"ssm:DescribeParameters",
"ssm:GetParametersByPath"
],
"Resource": [
"arn:aws:ssm:us-east-1:123456789000:parameter/artilleryio/*",
"arn:aws:ssm:us-west-1:123456789000:parameter/artilleryio/*",
"arn:aws:ssm:eu-west-1:123456789000:parameter/artilleryio/*",
"arn:aws:ssm:eu-central-1:123456789000:parameter/artilleryio/*",
"arn:aws:ssm:ap-south-1:123456789000:parameter/artilleryio/*",
"arn:aws:ssm:ap-northeast-1:123456789000:parameter/artilleryio/*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeRouteTables",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets"
],
"Resource": ["*"]
}
]
}
Setting up the IAM Policy and Role from AWS
To help you get started, here's a small guide to set up the necessary role and policy in AWS. It's especially important to have this created if you're planning on using Artillery in CI/CD.
IAM Policy: You should first create the policy. You can do that from the AWS UI or CLI. To do it from the UI:
- Go to
IAM -> Policies
and selectCreate Policy
. SelectJSON
. - Copy the JSON policy from the section above. You’ll need to change the following:
- AWS account from
123456789000
to our account ID (e.g. thetest1
account ID); - Remove any comments from the JSON policy, as that is not allowed;
- AWS account from
- Press
Next
and give the policy a meaningful name.
IAM Role: After the policy created, you'll need to use that policy in a role:
- Go to IAM -> Roles and select Create Role;
- Select
Custom Trust Policy
. You can keep the default generated or customise it based on your organisational needs; - After, on the
Add Permissions
UI, select the policy created in the previous step; - Finally, press
Next
and give the role a meaningful name.
That's it! You should now be able to use this role to run Artillery tests in Fargate.
If you're setting up OIDC, for example to interact with Github Actions (opens in a new tab), then you might need to also set a Custom Trust Policy. You can do so by editing the Custom Trust Policy in step 2 with the appropriate policy from the GH Actions guide linked above.
Limitations
A running test can only be stopped from the AWS Console
Once an AWS Fargate tests starts running, it can only be stopped by stopping Fargate tasks via the AWS Console or the AWS CLI.
We will add the ability to stop a currently running tests to the Artillery CLI in future.
Troubleshooting
Unable to assume the service linked role
`InvalidParameterException`: Unable to assume the service linked role. Please verify that the ECS service linked role exists
You may get the error above if, in some situations, your AdministratorAccess
role may be misconfigured from AWS. This seems more likely to happen when no ECS cluster has been created before in that AWS account.
To resolve this, try to run the ECS wizard to create a Fargate cluster manually from the AWS console (by going to ECS in the console). Doing so seems to do something internally in AWS to resolve the misconfiguration.
Could not find public subnets in default VPC
If you're getting this error from the CLI, then it's likely that there isn't a default VPC or that the default VPC doesn't have public subnets in the region you're trying to run Fargate tests from.
AWS Regions should come with a default VPC, but if for some reason you don't have it, you can create one from the AWS console yourself (opens in a new tab). Artillery needs it to be there for Fargate to work.
Questions?
- Post your questions on the community forum on GitHub at https://github.com/artilleryio/artillery/discussions (opens in a new tab)
- Report a bug or raise an issue via https://github.com/artilleryio/artillery/issues (opens in a new tab)