Skip to main content

· 8 min read

GraphQL is everywhere

GraphQL is becoming one of the standards among the different types of APIs available, thanks to their flexibility in allowing consumers to choose the data they need. Artillery can help you regularly test your GraphQL APIs to detect and eliminate potential problems, as shown in the article Using Artillery to Load Test GraphQL APIs. However, as seen in the article's examples, GraphQL's flexibility also makes it easy to have a less-performant API.

Site reliability engineers and developers can spend weeks hunting down and fixing GraphQL performance issues, trying to squeeze as much efficiency as possible. Some problems like improperly fetching data associations or optimizing inefficient database queries can take you a long way to improve the performance of your APIs. However, minor incremental performance updates can quickly become a time suck.

Can we cache GraphQL?

On most GraphQL APIs, the bulk of the work they perform is fetching data from a data source. Usually, it's the same data fetched repeatedly. If your API is primarily read-heavy, an alternative for performance is to use a Content Delivery Network (CDN). A CDN temporarily caches your API responses and serves them much quicker to consumers since the data gets retrieved and sent through globally distributed servers much closer than your servers.

Caching in GraphQL is less straightforward than with RESTful HTTP because GraphQL lacks an URL-like primitive that can serve as a unique ID for caching (see Caching in the official docs, GraphQL & Caching: The Elephant in the Room and HTTP caching in GraphQL for more details on the challenges of caching in GraphQL)

However, there's now a CDN specifically built for GraphQL services called Stellate (formerly known as GraphCDN). In this article, we'll check out Stellate by setting it up for an existing GraphQL API and load testing it with Artillery to verify whether it can boost performance of our API without any changes to the backend service.

Setting our baseline

Setting up our test

Our example GraphQL API allows consumers to fetch a list of users in a database or retrieve a single user based on their ID. For this example, we'll use two GraphQL queries to get this information from the backend service and include a couple of fields related to the data we find. These queries are read-only, a good use case for load testing the API through a CDN since the underlying data won't change between requests.

We'll use the following Artillery test script to go through these queries:

config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 25
processor: "./helper-functions.js"

scenarios:
- name: "Fetch user data"
flow:
- post:
url: "/"
json:
query: |
query UsersQuery {
users(limit: 100) {
id
}
}
capture:
json: "$.data.users"
as: "users"

- loop:
- post:
url: "/"
beforeRequest: "selectRandomUserId"
json:
query: |
query UserQuery($userId: ID!) {
user(id: $userId) {
id
username
email
}
}
variables:
userId: "{{ selectedUser.id }}"
count: 10

The test script will generate 25 virtual users per second for 10 minutes. Each VU will first make a query to retrieve 100 users from the API and get their IDs using the users GraphQL query. Next, it will go through a loop ten times to retrieve a single user using the user GraphQL query. Inside our loop, we'll use the selectRandomUserId function to take a random ID from the previous data retrieval to make the API query.

The selectRandomUserId function comes from custom JavaScript code that we load to use in our Artillery tests. We'll invoke the function in a beforeRequest hook. Our custom JavaScript code inside the helper-functions.js file contains the following:

module.exports = {
selectRandomUserId: selectRandomUserId
};

function selectRandomUserId(requestParams, context, ee, next) {
// Select a random user from the `users` variable set in a prior request.
const users = context.vars["users"];
context.vars["selectedUser"] = users[Math.floor(Math.random() * users.length)];

return next();
};

Load testing the origin

This basic test will give us a good idea of how much read-only traffic our GraphQL server can withstand. Assuming the test script is under users-test.yaml, we will run an Artillery load testing from the eu-west-2 AWS region with the following command:

artillery run-test --region eu-west-2 users-test.yaml
info

We're using Artillery Pro to run the test from our own AWS account rather than from a local machine for more realistic traffic generation. We're using AWS Fargate for a serverless experience to avoid needing to set up any infrastructure for load testing from the cloud.

The test will send virtual user traffic directly to our GraphQL endpoint for ten minutes. When the test wraps up, we'll see our results:

All VUs finished. Total time: 12 minutes, 0 seconds

--------------------------------
Summary report @ 08:31:17(+0100)
--------------------------------

vusers.created_by_name.Fetch user data: ..................... 15000
vusers.created.total: ....................................... 15000
vusers.completed: ........................................... 14999
vusers.failed: .............................................. 1
http.request_rate: .......................................... 274/sec
http.requests: .............................................. 164990
http.codes.200: ............................................. 164988
http.responses: ............................................. 164989
http.codes.502: ............................................. 1
http.response_time:
min: ...................................................... 66
max: ...................................................... 750
median: ................................................... 80.6
p95: ...................................................... 89.1
p99: ...................................................... 96.6
errors.ETIMEDOUT: ........................................... 1

For read-only queries, these results aren't great. With a median response time of >80 milliseconds for each virtual user's flow and the 99th percentile hitting >96 milliseconds, the GraphQL API looks to struggle a bit under load. A few VUs also hit some timeouts, which doesn't give us the confidence that we can scale this service any further. Remember that we're testing a read-only endpoint, i.e. even though our queries vary the underlying data hasn't really changed. Let's see how Stellate can help here without spending too much engineering time fixing the issue.

Setting up Stellate

Setting up Stellate for an existing GraphQL API is a straightforward process. At a high level, Stellate will set up an edge cache between your GraphQL service and any consumers, caching all the query results that go through. Consumers will communicate with your API through Stellate instead of directly to your server. When a query has a cache hit, Stellate serves it up without going to your service — much faster than a direct query. How much faster? We'll see in a bit.

First, we'll sign up for a Stellate account. Stellate provides a generous free tier supporting up to 5 million CDN requests per month so that you can get started quickly. After signing up and setting up your organization, Stellate asks you to create a new service by entering the URL of your GraphQL API. It will then attempt to fetch your GraphQL backend schema. Don't worry if you don't, as you can proceed without it. If your GraphQL service requires authentication, you'll need to enter the information for Stellate to work correctly. You'll also need to enter your desired subdomain that will serve as your service URL.

Once you finish this single step, Stellate will create a new edge cache with your service URL ready for you to use immediately. It can't get any simpler than that. The only thing you need to do is replace your service's URL with the provided Stellate service URL for any consumers of your API, and you can begin using the CDN instantly.

Load testing Stellate

Let's see how the CDN helps with performance by running the same Artillery test script pointing at our new Stellate edge cache. The only change we need to make is to switch the config.target URL in our Artillery test script.

Let's go back to running the tests, but this time we'll run the tests on ten workers (up from one that we ran the first test with) to increase the amount of traffic to the service by 10x:

artillery run-test --region eu-west-2 --count users-test.yaml
All VUs finished. Total time: 12 minutes, 3 seconds

--------------------------------
Summary report @ 08:44:21(+0100)
--------------------------------

vusers.created_by_name.Fetch user data: ..................... 150000
vusers.created.total: ....................................... 150000
vusers.completed: ........................................... 150000
http.request_rate: .......................................... 2773/sec
http.requests: .............................................. 1650000
http.codes.200: ............................................. 1650000
http.responses: ............................................. 1650000
http.response_time:
min: ...................................................... 1
max: ...................................................... 81
median: ................................................... 2
p95: ...................................................... 4
p99: ...................................................... 7

How about that? the median and 99th percentile response times are an order of magnitude lower when using Stellate, despite throwing ten times more traffic at it.

Conclusion

If you're a developer and you're worried about running into performance issues with your GraphQL APIs, it sure looks like Stellate can help with intelligent GraphQL-specific caching to reduce load on your origin server and improve the experience of users of your service.

If you're an SRE, you know that testing critical dependencies of the services you're looking after is part of Testing For Reliability. "If you haven't tried it, assume it's broken." is a a great guiding principle, which often extends to "Trust but verify" when it comes to performance & reliability claims by third-party services. Artillery is there for you for those scenarios. In this case, we've verified that Stellate is doing a stellar job.

(It goes without saying that you should always make sure to check and comply with ToS of any hosted services when it comes to load testing.)

· One min read
Ezo Saleh

We've recently integrated Prometheus (via the Pushgateway) as a publish-metrics plugin target.

This makes it super easy for you to collect your test metrics on Prometheus. A logical next step would be to visualise those metrics to better make sense of how your tests performed.

Prometheus ♥️ Grafana. To that end, checkout our Artillery Grafana dashboards.

They should help you get a leg up with your metric visualisations. Use them as is, or treat them as templates to customise to run your own analysis.

Just import the dashboards and get visualising!

Here's a sneak peek of our HTTP metrics dashboard.

Artillery Grafana HTTP metrics dashboard

· One min read
Ezo Saleh

Our new kubectl-artillery plugin helps developers and testers boostrap Artillery testing on Kubernetes.

See it in action in the video below. This video walks through:

  • The k8s-testing-with-kubectl-artillery example.
  • Scaffolding Artillery test-scripts from running Kubernetes services.
  • Generating tests that target Kubernetes services.
  • Running the generated tests.

To use the plugin we assume you have kubectl installed and some familiarity with Kubernetes in general.

Install the plugin by following the instructions here for your target OS.

Here's an example of installing it on macOS (Darwin) amd64:

curl -L -o kubectl-artillery.tar.gz https://github.com/artilleryio/kubectl-artillery/releases/download/v0.2.2/kubectl-artillery_0.2.2_darwin_amd64_2022-05-11T16.49.36Z.tar.gz
tar -xvf kubectl-artillery.tar.gz
sudo mv kubectl-artillery /usr/local/bin

Cloud native testing for the win! 🔌☁️

· One min read
Hassy Veldstra

See Artillery Probe in action in the video below. This video walks through:

  • Sending HTTP requests
  • Checking request performance waterfalls
  • Pretty-printing and querying JSON responses
  • Sending POST requests with JSON payloads

To install Artillery Probe, grab the latest version of Artillery from npm with:

npm install -g artillery@latest

Happy testing! 🛰

· 6 min read
Hassy Veldstra
Artillery Probe

Introducing Artillery Probe

Say hello to artillery probe, a Swiss army knife for testing HTTP. A smart and user-friendly HTTP client for the terminal which shows request performance waterfalls, and lets you set up expectations/checks on responses for quick-n-easy automated testing.

Think of it as curl with better UX for common use-cases, and a couple of extra powers.

Artillery Probe

Features

  • Send HTTP requests from the command line. 📺 Use any HTTP method (GET, POST, PUT etc). With HTTP/2 support.
  • Send JSON or arbitrary request bodies, custom headers, forms, file uploads, query strings, Basic Auth etc.
  • See response headers and bodies with syntax highlighting & pretty-printing 🌈
  • See request performance waterfalls 〰〰 (inspired by reorx/httpstat)
  • Built-in support for JSON querying with the same syntax as AWS and Azure CLIs (via JMESPath). You may not need jq at all!
  • Set expectations & checks for easy acceptance testing (testing locally, or for continuous verification and post-deployment checks in CI/CD) 🔁
    • These are powered by the official expect plugin under the hood

Let's look at some examples

Probe an URL quickly

Start with something simple - just give probe an URL. (if protocol is omitted, it will default to HTTPS, it's 2022)

artillery probe www.artillery.io

Et voilá, we can see response headers, and a waterfall chart.

Connected to https://www.artillery.io (76.76.21.61)

HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
connection: keep-alive
x-matched-path: /
cache-control: public, max-age=0, must-revalidate
content-length: 275503
date: Fri, 29 Apr 2022 21:40:51 GMT
x-powered-by: Next.js
etag: "4342f-g4C79gzSagz298qfv0rAclmRmks"
x-vercel-cache: STALE
age: 90
server: Vercel
x-vercel-id: dub1::iad1::7szlw-1651268541693-cce911e1815d
strict-transport-security: max-age=63072000


DNS Lookup | TCP Connection | SSL Handshake | Time to First Byte | Content Transfer
56ms | 14ms | 19ms | 181ms | 88ms |
| | | | |
56ms | | | |
70ms | | |
89ms | |
270ms |
total:358ms


Body stored in: /var/folders/yd/nyh8vtm17w92h5hm4pdhs8j00000gn/T/2022329-40081-18t3wn9.8m17l.html

Response body is saved to a file by default in case you want to look at it, with the right extension for the MIME type.

info

Pro-tip! artillery probe may also be called as artillery http.

You can go one step further and pop something like this into your shell config:

alias http='artillery http'

Which will let you run just http www.artillery.io for example. The rest of our examples will use just http alias for brevity.

See response bodies

To display response body in line just add a -b to the command. We'll use another URL for this example, which returns some data about movies as JSON:

http http://lab.artillery.io/movies -b

The output will be similar to that above, but we will see color-highlighted JSON body too:

[{"id":1,"releaseDate":"Dec 18 1985","director":"Terry Gilliam","title":"Brazil","genre":"Black Comedy","imdbRating":8,"runningTimeMin":136},{"id":2,"releaseDate":"Feb 16 1996","director":"Harold Becker","title":"City Hall","genre":"Drama","imdbRating":6.1,"runningTimeMin":111},{"id":3,"releaseDate":"Jul 12 1996","director":"Edward Zwick","title":"Courage Under Fire","genre":"Drama","imdbRating":6.6,"runningTimeMin":115},{"id":4,"releaseDate":"May 31 1996","director":"Rob Cohen","title":"Dragonheart","genre":"Adventure","imdbRating":6.2,"runningTimeMin":108},{"id":5,"releaseDate":"Jan 19 1996","director":"Robert Rodriguez","title":"From Dusk Till Dawn","genre":"Horror","imdbRating":7.1,"runningTimeMin":107},{"id":6,"releaseDate":"Mar 08 1996","director":"Joel Coen","title":"Fargo","genre":"Thriller/Suspense","imdbRating":8.3,"runningTimeMin":87},{"id":7,"releaseDate":"Oct 11 1996","director":"Stephen Hopkins","title":"The Ghost and the Darkness","genre":"Action","imdbRating":6.6,"runningTimeMin":109},{"id":8,"releaseDate":"Feb 16 1996","director":"Dennis Dugan","title":"Happy Gilmore","genre":"Comedy","imdbRating":6.9,"runningTimeMin":92},{"id":9,"releaseDate":"Jul 02 1996","director":"Roland Emmerich","title":"Independence Day","genre":"Adventure","imdbRating":6.5,"runningTimeMin":145},{"id":10,"releaseDate":"Jun 13 1980","director":"Michael Ritchie","title":"The Island","genre":"Adventure","imdbRating":6.9,"runningTimeMin":138}]

Let's pretty-print that JSON

That's a lovely blob of JSON we got there, but let's add a -p to pretty-print it.

http http://lab.artillery.io/movies -b -p

and there we go:

[
{
"id": 1,
"releaseDate": "Dec 18 1985",
"director": "Terry Gilliam",
"title": "Brazil",
"genre": "Black Comedy",
"imdbRating": 8,
"runningTimeMin": 136
},
{
"id": 2,
"releaseDate": "Feb 16 1996",
"director": "Harold Becker",
"title": "City Hall",
"genre": "Drama",
"imdbRating": 6.1,
"runningTimeMin": 111
},
(... rest of the JSON body omitted for brevity)
]

Query JSON with JMESPath

JMESPath is a powerful querying syntax for JSON, used by AWS and Azure CLIs among many others.

The objects returned by our /movies endpoint contain an imdbRating field. Let's get filter down the list from our API endpoint to highly-rated movies only:

http http://lab.artillery.io/movies -q '[?imdbRating>`8`] | [*].title'

We're asking for titles of all movies with a rating of greater than 8. This is what we'll see:

[
"Fargo",
"Trainspotting",
"American Beauty",
"Le Fabuleux destin d'AmÈlie Poulain",
"Batman Begins",
"The Dark Knight",
"Big Fish",
"The Big Lebowski",
"The Bourne Ultimatum",
"Inglourious Basterds",
"Children of Men"
]

POST some JSON

We can use all standard HTTP methods such as POST, PUT, PATCH etc.

Let's send a POST request to another one of our test API endpoints.

http post \
http://lab.artillery.io/login \
--json "{username: testuser, password: testpassword}"

--json sets request body to JSON we provide... but did you notice how we didn't have to quote everything fully?

If you've ever tried sending JSON from the command-line, especially with some values set dynamically from environment variables, you'll know how gnarly it can look.

With curl:

export NAME=tiki
export TYPE=pony

curl \
-XPOST \
-H 'Content-Type: application/json' \
-d "{\"name\":\"$NAME\",\"type\":\"$TYPE\"}" \
https://httpbin.org/post

With Artillery:

export NAME=tiki
export TYPE=pony

http \
--json "{name:$NAME,type:$TYPE}" \
https://httpbin.org/post

Much easier on the eyes, and less error-prone.

Set checks & expectations

We can set expectations on the response and have artillery probe verify them for us.

For example, let's run a check on Artillery Docs to make sure the URL returns 200 OK, that an ETag header is set, that the response is served by Vercel, and that the body contains the string "Artillery Docs".

http www.artillery.io/docs \
-e "statusCode: 200" \
-e "hasHeader: etag" \
-e "{headerEquals: [server, Vercel]}" \
-e "matchesRegexp: 'Artillery Docs'"

We'll see expectation checks at the end:

Artillery Probe Expectations

These checks are powered by the official expect plugin under the hood, which supports a number of different types of checks such as statusCode, contentType, hasHeader, matchesRegexp etc.

This is automation-friendly - if an expectation fails, the command will exit with a non-zero code. That means that if you use artillery probe in a CI/CD job, a failing expectation will make the CI/CD pipeline go red. Continuous verification super-fast! Just throw an artillery probe command into your CI/CD pipeline.

Get Artillery Probe

Artillery Probe is included in the latest release of Artillery:

npm install -g artillery@latest

We'd love to hear your thoughts & feedback, and learn more about your use-cases. Drop us a message on our GitHub Discussions board.

This is an early release of Artillery Probe, but we're excited about the possibilities it has for making developers' lives easier, and we need help from our community to make it more awesome!

· 7 min read
Ezo Saleh

State can get very messy in a distributed environment like Kubernetes.

In simple environments, state machines are very adequate to define and track state. You quickly get to understand if your computation is misbehaving, and report progress to users or other programs.

But, In the context of an Operator, you can’t just pay attention to your own workload. You also need to be aware of external factors monitored by Kubernetes itself.

Here’s how to think about state when building your own Operator (based on lessons learnt from building our own).

Let’s get into it!

Spec is desired, status is actual

Let’s start with the basics. All Kubernetes objects define spec and status sub-objects to manage State. This includes custom resources monitored by your own Operator’s custom controller.

Here, spec is the specification of the object where you declare the desired state of the resource. On the other hand, the status is where Kubernetes provides visibility into the actual state of the resource.

For example, our Artillery Operator monitors a custom resource named LoadTest. As above, our LoadTest has a spec field specifying the number of load tests to run in parallel. Our custom controller processes our load test. It creates the required parallel workers (Pods) to match the spec’s desired state therefore updating the actual state.

State in Artillery Kubernetes Operator

The key takeaway from this diagram is that observing state is a constant happenstance to ensure desired becomes actual.

But we need to understand how our custom resource is progressing. So, what exactly should we track in the status sub-object?

State machines are rigid

The answer to the previous question is: do not use single fields with state machine like values to track state.

Kubernetes advocates for managing state using Conditions in our status sub-object. This gives us a less rigid and more ‘open-world’ perspective.

This makes sense. Processes in a distributed system interact in unforeseen ways creating unforeseen states. It’s hard to preemptively determine what these states will be.

Monitoring aggregate state

Let’s look at the example of a typical custom controller.

This controller creates and manages a few child objects. It is aware of state across all of these objects. And, it constantly examines its ‘aggregate’ state to inform its actual state.

We can define these aggregate states using a state machine. As an example, let's say we have the following states: initializing, running, completed, stoppedFailed

We’ll create a field called CurrentState in our status object, and update it with the current state. Our CurrentState is now running.

Unforeseen states

... Out of the blue a Kubernetes node running our workload fails!!

This drops a child object into an unforeseen (to us) state. Our custom controller’s ‘aggregate' state has now shifted. But, we continue to report it as running to our users and downstream clients.

We need a fix! We update our custom controller with logic to better examine node failure for a child object. We introduce a new CurrentState for this scenario: awaitingRestart .

Some downstream clients will find this new controller state useful. We now have to inform them to monitor for this new state. All is well in the world, for now .

... What!?? out of the blue another an accounted for state happens!

Our CurrentState can’t explain the issue. So, we update controller logic, add an adequate state for ... etc.. etc..

You can see how this state machine based approach quickly becomes rigid.

Conditions

Conditions 👀, provide a richer set of information for a controller’s actual state over time. You can think of it as a controller’s personal changelog.

Conditions are present in all Kubernetes objects. Here’s an example of a Deployment. Have a read I’ll wait... 👀

If our typical custom controller, from the last section, had a Deployment child object that exhibited the following conditions.

status:
availableReplicas: 2

conditions:
- lastTransitionTime: 2016-10-04T12:25:39Z
lastUpdateTime: 2016-10-04T12:25:39Z
message: Replica set "nginx-deployment-4262182780" is progressing.
reason: ReplicaSetUpdated
status: "True"
type: Progressing

- lastTransitionTime: 2016-10-04T12:25:42Z
lastUpdateTime: 2016-10-04T12:25:42Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available

It can use the following logic to figure out its own actual state,

  • Go through the Deployment’s .status.conditions
  • Find the latest .status.conditions[..].type and it’s .status.conditions[..].status
  • Weigh if the other Deployment .status.conditions had any alarming statuses

Here, it determines the Deployment is still progressing and not yet completed. Then it can add a condition with a ‘Progress’ like type to its Conditions list.

Creating your own Conditions

So, how do we create conditions for our own custom controller?

We’ve just seen an example of a Deployment’s Conditions, specifically how open ended they are. E.g type: Progressing may mean either of progressing and complete.

As a general rule of thumb,

  • We have a smaller set of types .status.conditions[..].type that explain general behaviour.
  • Unforeseen states, can be easily signalled using the Unknown .status.conditions[..].status value.
  • Failure for a condition type .status.conditions[..].type can be explained using False as .status.conditions[..].status value with an informative .status.conditions[..].reason.

Let’s checkout an example and see how these tips play out.

Conditions by example

If you recall, earlier, we used the Artillery Operator to explore spec and status sub-objects. Let’s get into the details of how it makes use of Conditions to manage state.

The Artillery Operator uses a LoadTest custom resource to spec out a load test run. The spec includes the number of workers to run in parallel. This information is used to generate a Kubernetes Job and Pods to run the actual load test workloads.

Artillery Kubernetes Operator Architecture diagram

The Job is a key component here. Tracking our aggregate state relies on the progress of the Job.

We also observed that,

  • There are three statuses we care about based on a Job’s status...

Namely, LoadTestInactive, LoadTestActive and LoadTestCompleted.

  • We should add extra fields to status ...

These fields will give us a fine grained view of how many workers are active, succeeded or failed.

// The number of actively runningLoadTest worker pods.
Active int32 `json:"active,omitempty"`

// The number ofLoadTest worker pods which reached phaseSucceeded.
Succeeded int32 `json:"succeeded,omitempty"`

// The number ofLoadTest worker pods which reached phaseFailed.
Failed int32 `json:"failed,omitempty"`

Using our Conditions rule of thumb, two Condition types are required:

  • Progressing, with True, False or Unknown as a value.
  • Completed, with True, False or Unknown as a value.

These easily explained the progress and completed states. But, how do explain a Load Test has failed?

In our distributed load testing domain, a load test always completes. And, will only flag failed workers. Restarting a failed worker/Pod messes with the load test metrics, so we avoid it at all costs.

So, for the controller, the Completed condition set to true was more than enough. Failed workers were flagged in the status field (using a failed field to track failed count).

Other implementations may treat the failure condition differently (e.g. Deployment uses the Progressing Condition type with value false).

Our Conditions ensure users can track a Load Test’s observed status to infer progress. While helping clients monitor the conditions that matter to them.

Feel free to check the full Conditions implementation. 👀

Conditions all the way down

The more you deal with Kubernetes object to track any form of state, the more you deal with Conditions. They’re everywhere.

They help create a simple reactive system that is open to change. Hopefully, this articles gives a good understanding of how to get started.

· 3 min read
Ezo Saleh
Artillery Kubernetes Operator

As an SRE, you’d like to be helpful. But, you have SLOs to monitor and a long list of apps to look after. Self-service to the rescue!

With self-service, you empower your developers to get stuff done. While you keep an eye on the big picture making sure apps are within performance and availability thresholds.

If you're running on Kubernetes, the Artillery Operator can be the load testing piece of that self-service puzzle.

Using it, your developers can run (distributed) load tests as first class citizens on Kubernetes.

These load tests are based on CRDs and utilise existing Artillery test scripts. Meaning, you can interact with them via kubectl, and they can be monitored with Kubernetes UI tools like any other Kubernetes object.

How does it work?

For the SRE

Simply deploy the Artillery Operator to a Kubernetes cluster.

For the Developer

Provided the Operator has already been deployed, you create a Load Test custom resource and deploy it to the cluster.

This is a simple YAML file similar to a Kubernetes manifest. The file describes a load test, the number of required workers to run in parallel. And, access to the Artillery test script.

As an aside, you’ll need to package your test script as a ConfigMap. Then, deploy it to your cluster in the correct namespace. It is now available to the Load Test custom resource when it is deployed.

If this sounds laborious, don’t fret, just follow this example, and you will be set.

Documentation and examples

Check out the README. Everything you need should be there.

Bonus: aggregated test reports on Prometheus

SREs, yup, you get this one for free. That is, if you have a Prometheus instance already up and running in your stack.

Your developers will just have to update their Load Test's test script to point to a Prometheus Pushgateway. Then every worker will publish its test reports to Prometheus.

You can then set up the necessary monitoring/alerts to ensure performance and availability are at acceptable levels.

This example documents all the details.

By the way, the Operator relies on the artillery-publish-metrics plugin to do all the heavy lifting. So, if you want to, you can swap publishing to Prometheus with any other supported target, including Datadog or others.

Feedback

Please bear in mind that this is an Early Alpha release.

We still haven't implemented all the existing test script features yet. We also might have a few kinks to iron out.

Your feedback, bug reports and any suggestions at all are most welcome.

· 6 min read
Hassy Veldstra

Artillery is a performance testing platform for DevOps and SRE. We make it easy to run planet-scale load tests from your own AWS infrastructure. Developers love Artillery because it's modern and comes with batteries included. SRE & Platform teams love Artillery because they can provide a self-service load testing platform to developer teams. Security teams love Artillery because it runs on prem (or rather cloud-prem).

While we love, love, love load testing, and have a lot (like, A LOT) of new ideas to shake up the space, the Big Vision is much bigger than load testing. The state of testing tools for today's complex production systems is abysmal. We'd like you to join us to help us build a better future.

Anyways, we're hiring full-stack product engineers.

Full-stack product engineers?

As a product engineer you will be working on our products - Artillery, Artillery Pro, and yet-unannounced-but-we-think-its-cool stuff. We treat our website and our docs as products too. You will own features end-to-end from dev to prod, and spend most of your time writing code.

Our users interact with Artillery primarily through its CLI today, but we're building an ambitious UI and UX for all things performance testing.

Our code is mostly JS and TypeScript, but there is some Go too. When building a UI we reach for React + Next.js + Tailwind. Build and test pipelines run mostly on Circle. To host stuff we use either AWS (serverless whenever possible) or Vercel. Our collaboration stack is Slack, Notion, Linear, and GSuite. We will never use Jira or Confluence. Design is done in Figma + prototypes in actual code.

We're a small team and we wear a lot of hats. Here are some of the hats you may find yourself wearing (never worn some of those hats before? that's OK):

  • Writing code. This week you may be adding a new feature to Artillery's extension APIs and doing user research to figure out which existing plugins may be affected; next week you may be working on improving one of the monitoring/observability integrations we support, and the week after you may be building a new UI + API endpoints to support it.
  • Building end to end: you'll go from a user story/PRD, to working with our designer to refine what the UI/UX is going to look like, to writing code, tests, build & release scripts, infra automation scripts, and documentation. You may find yourself writing a launch announcement for the new feature on our blog too, and dipping into our template library in Figma to make a nice header image for the blog + Twitter. You will think about ways to measure if & how the new feature is used, and how to get feedback from users to improve it.
  • Helping our community on Github, responding to and helping triage Issues, and answering questions in Discussions.
  • Helping improve the docs and examples, and spend time thinking about improving onboarding experience and DX of our products
  • Helping our customers in shared Slack channels
  • Building an internal tool to make us more productive, such as adding an integration to our Slack bot

You will have plenty of opportunities to move between projects, and will likely end up contributing to several of them, but we'd expect you to settle and focus on mostly one or two areas after a couple of months on the job.

Who is this role right for?

You're a good fit if you like the idea of:

  • Building tools for other developers to help them be more productive and get their jobs done faster and better
  • Talking to people who use the software you build every day
  • Working on open source, and helping grow our open source community
  • Working across the stack - from frontend, to backend, to infrastructure. We don't expect a unicorn who can do everything (none of us are), but you should be very comfortable in one area, and know enough to be dangerous elsewhere + be curious and willing to get involved with everything
  • Spending time thinking about UX, building intuition and deep empathy about the problems our users face, and how we can solve them
  • Working on tooling to improve the performance and reliability of other software systems
  • A chaotic fast-paced environment, which emphasises shipping, and with lots of opportunity for ownership and impact

Career level-wise, we expect this to be a good role for mid-to-senior and senior developers.

You should be in GMT ±4 hours. We are currently spread across Ireland, Germany, England, and UAE.

We are a distributed company, and we try to be as async as possible to optimize for deep work. That means we rely heavily on written communication, and on communicating proactively.

Why work with us? (or the bit where we sell the role)

  • We're a small technical team building tools for other developers, and we talk to our users every day.
  • We're a commercial open source company, a lot of your work will be open source.
  • We have paying customers who love our product.
  • This is the proverbial ground floor. You will be among the first 20 people at Artillery, not engineer no. 200 on team no. 17 at $YOUR_FRIENDLY_NEIGHBORHOOD_BIG_CO. You will have a huge amount impact and the opportunity to help build the kind of team and company you want to work for.
  • Meaningful equity and ownership.
  • We're a distributed team with a very lightweight process that optimizes for shipping and deep work.

The compensation range for this role is $80-125k plus equity. This is a full-time job with vacation time, hardware budget, learning budget, and all the usual stuff.

Next steps?

  1. Drop us a line on team+jobs@artillery.io. Tell us a little bit about yourself, and about something you built (a project at $DAY_JOB, a side project on Github, etc). Feel free to include a resume too, but it's not necessary. If you have any questions you want to ask straightaway, just shoot.
  2. We'll get back to you with some times to connect on a call. This is an opportunity for us to get to know each other, and for us to pitch the company and answer all your questions about the company, team, and the work we do.
  3. If there's mutual interest, we'll go forward from there. Our hiring process is lightweight and we don't do the typical tech company interview gauntlet here.

· 6 min read
Hassy Veldstra
Load testing with real browsers

Load testing non-trivial dynamic web applications is no walk in the park.

If you've ever had to do it, you know. Creating test scripts is slow, error-prone, cumbersome, brittle, and just downright annoying. You have to deal with forms, track down which on-page actions kick off which request to the backend, and if any part of the flow depends on in-page Javascript... forget it.

Traditional load testing tools don't work well

The root cause of it all is simple: a mismatch in the level of abstraction. Load testing tools are designed to work with API endpoints, whereas a page is a better and more natural abstraction when testing web apps.

When testing an API, there's usually a spec available (e.g. an OpenAPI spec). Testing a web app? It's unlikely there's even a list of all API endpoints used on different pages. Different APIs may be called depending on in-page actions. Some calls may be made by in-page JS — and these have to be tracked down and emulated. Hours of fun in Chrome DevTools!

There hasn't been a good solution... up until now. We've been busy at Artillery HQ1 cooking something up.

Launching 10,000 browsers for fun and profit

What if you could run load tests with real browsers, at scale, and just reuse existing end-to-end testing scripts?

Artillery Playwright load testing with browsers demo

(the demo is also on YouTube)

A few things to highlight in the demo above:

  • We're re-using Playwright scripts, which we can also generate with playwright codegen, i.e. we can click around in a browser to create a test script
  • We get both backend and frontend performance metrics in the report, so we can see how load affects LCP or FCP over time
  • We're running thousands of browsers with zero infrastructure setup, and they run from your own AWS account - not a hosted cloud service. Did "ridiculous cost efficiency" cross your mind? -- so efficient you could run these tests in CICD multiple times a day. All sorts of nice security and data privacy-related thoughts too perhaps, since this would run in your own VPC. Why, yes, that's why we like cloud-native software that we can run ourselves too.

Are we also possibly talking 10x developer productivity for load testing non-trivial applications? Yes, yes, quite possibly.

How does that work?

How does that work? Playwright provides the browsers. Artillery provides the ability to launch a ridiculous number of them from your own AWS account, with no infra to manage.

Behind the scenes, we expose the Playwright API to Artillery via Artillery's extension interface, which is the same API that powers all of Artillery's many engines. This lets Artillery treat Playwright scripts as a description of what a virtual user scenario is. From that point onwards, as far as Artillery is concerned there is no difference between opening a TCP connection and exchanging Socket.IO messages (if we're testing Socket.IO) or launching a browser and running page-based actions (with the Playwright engine).

Artillery Pro can then pick up from there to launch swarms of Artillery workers running the Playwright engine on AWS ECS or Fargate for testing at scale.

OK, so we can but should we?

Just because we can do something doesn't mean we should, does it? We're running real browsers here, which is obviously going to be resource-intensive. Is it going to scale? Is it worth it?

Does it scale?

Yes, it certainly does. It's 2021, cloud compute is cheap & plentiful, and "just throw more containers at it" can be a perfectly legitimate tactic.

Again, we are launching whole browsers here, which is going to be resource-hungry. Let's look at Fargate with the beefiest configuration per-container we can get: 4 vCPUs and 12GB of memory. Running a container with that spec is us-east-2 for a whole hour is going to cost us a whopping 22 cents. If we run a 1,000 of those containers for an hour, we're looking at $220.

How much load can those 1,000 containers generate? In our unscientific tests running a simple but not-trivial Playwright flow, we could launch one Chromium instance every other second in each container, making around 20 HTTP requests per second each.

Your particular results will of course vary. Memory is going to be the main bottleneck, which will put a ceiling on how many concurrent browser instances we can run in a single container. The number of those will depend on the web app being tested and your Playwright scenario. The slower the app and the longer the test scirpt, the more concurrent Chrome instances you'll be running in each container.

The trade-off is developer productivity vs AWS costs. If you're testing a complex web app, especially on a tight schedule, you can be up and running much faster with a Playwright-based approach. Developer time is usually expensive, and cloud compute isn't - your call.

And hey, you don’t have to go full hog and load test exclusively with browsers, as fun as that may be. We can now run hybrid tests2 and smoke tests with browsers too.

Hybrid tests

In a hybrid test you’d generate most of the load on the backend with a traditional approach of sending HTTP requests to verious endpoints. You also run a Playwright-based test alongside as part of the same test (using scenario weights), to see how load on a key API endpoint can affect frontend metrics.

Smoke tests

Smoke testing is a common use-case for Artillery, and there's nothing stopping us from running our Playwright scripts with Artillery as smoke tests (also known as synthetic checks and E2E checks - think health checks on steroids). These are particularly useful for running in CICD to check that everything is still OK after every deployment.

Try it yourself!

The engine is open source with the code available on Github at artilleryio/artillery-engine-playwright.

You can try an end-to-end example locally by following along the README in the e2e example repo we put together - artillery-examples/browser-load-testing-playwright. The official repo includes a Dockerfile for running these tests in CICD as well.

Playwright support is available out-of-the-box to all Artillery Pro customers, including the free Dev plan.

We’d love to hear how you might use this. Hit us up on Twitter or on GitHub discussions. ✌️


  1. there isn't really an HQ, we're an all-remote distributed team
  2. You can’t do hybrid testing with anything else but Artillery so we’re going to say we’re coining a new term here. "Hybrid testing" - has a bit of a ring to it, doesn't it? Expect to see hybridtestingmanifesto.dev landing any day soon.

· 11 min read

Load testing GraphQL

See end-to-end example with Apollo and Prisma - artillery-examples/graphql-api-server.


The popularity of GraphQL for building and querying APIs has been on the rise for years, with no sign of slowing down. Developers can now build robust and flexible APIs that give consumers more choices when accessing their data. This flexibility, in turn, can help with application performance compared to traditional RESTful APIs. Providing API consumers the flexibility to fetch the data they need avoids the overhead of processing unnecessary information.

Despite all the advantages GraphQL provides to developers, it's also easy to introduce potential performance issues when constructing APIs:

  • Its flexibility can allow clients to craft inefficient queries when fetching server-side data, especially when using nested queries that need to get data from multiple database tables or sources. This problem is more prevalent when using ORMs with GraphQL since it can mask any inefficiencies in your database calls.
  • Since GraphQL allows consumers to build different queries as they need, it isn't easy to implement caching effectively. Since each request made to the API is potentially unique, developers can't use standard caching techniques. Some libraries provide built-in caching, but it may not work in all scenarios.

Most developers building GraphQL APIs eventually run into one or both of these issues. While these performance problems are simple to deal with, they're also easily overlooked. It's also not uncommon to find yourself with a sluggish API after deploying to production when dealing with actual data. That's why load testing your GraphQL APIs before and after deploying is more crucial than ever.

With Artillery, you can create test scripts that can test common workflows for GraphQL to smoke out any potential performance issues that can creep up when you least expect. Running regular load tests on your APIs both during development and against your production servers will keep your APIs performant and resilient for your customers. This article shows you how to use Artillery to keep your GraphQL services in check.

GraphQL API example

For this article, let's assume we have a GraphQL API with a data store, some queries to fetch this data, and mutations to manage data manipulation based on this diagram:

GraphQL example schema

The data store consists of two tables to store user information and messages associated with that user for performing standard CRUD operations. There's a single query to fetch a specific user by their data store ID. The GraphQL API also has some mutations to create, update, and delete a user, along with creating messages associated with a single user.

Let's imagine that most consumers of the GraphQL API perform two primary flows after creating and logging in as the new user. The first flow will create new messages through the API and later fetch the user's account information along with their messages. The other flow handles updating a user. We want to make sure these flows remain performant under load for whoever accesses the API. Let's create a few load tests using Artillery.

Creating our first GraphQL load test

Most GraphQL APIs handle requests via standard HTTP GET and POST methods, meaning we can easily use Artillery's built-in HTTP engine to make requests for load testing. Both methods work similarly, with some differences related to how HTTP works. For instance, GET requests are easier to cache, while POST responses aren't cacheable unless they include specific headers. For simplicity in our load tests, we'll use the POST method to make our GraphQL API requests.

When served over HTTP, a GraphQL POST request accepts a JSON-encoded body with a query field containing the GraphQL query to execute on the server. The body can also include an optional variables field to set and use variables in the specified GraphQL query. We'll use both of these fields for our load test.

For our first test, we'll go through one of the flows described above:

  • First, we'll create a new user record through the GraphQL API and capture their email address from the GraphQL response. We'll generate randomized usernames and email addresses using Artillery's $randomString() method to ensure we don't run into uniqueness constraints when creating users during the test.
  • Next, we'll log in as the created user to test the login process and capture their ID from the data store.
  • With the user's ID in hand, we'll then use the loop construct to create 100 new messages associated with the user.
  • After creating those messages, we'll make a request to return the user data and their messages to verify how the API performs when fetching associated data between different database tables.
  • Finally, to clean up the database, we'll delete the user at the end of the flow, which will also delete the user's associated messages.

Let's see how this test script looks for executing this flow:

config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 50

scenarios:
- name: "Create and fetch messages flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"

- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"

- loop:
- post:
url: "/"
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: "{{ userId }}"
body: "Message Body {{ $loopCount }}"
count: 100

- post:
url: "/"
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: "{{ userId }}"

- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"

For this example, we have our GraphQL API set up at https://graphql.test-app.dev. The Artillery test will run for ten minutes, generating 50 virtual users per second. In our scenario, we have our defined flow with a few POST requests made to the root URL (/) where the GraphQL API handles our requests.

Each step in our flow section works similarly. We'll have a JSON request to the GraphQL server with the query and variables fields (where necessary). The query field contains the GraphQL query or mutation sent to the server for processing. We'll use a pipe (|) when sending the data to make the body more readable in our test script.

Each query and mutation in our script is set up to handle variables since we'll use them for keeping track of the dynamic data generated throughout the test phase. That's where the variables field comes into play. This field can set up the variables we'll use through a standard JSON object. The GraphQL server will handle setting up the values where specified in a query or mutation.

For instance, in the step where we create a user, the createUser mutation expects an input argument, which is an object containing the username, email, and password fields. The variables field will construct the input object using fields required in the mutation argument and pass it to the query. It allows us to generate unique usernames and emails for each virtual user dynamically.

Assuming we save the test script in a file called graphql.yml, we can run it with Artillery using artillery run graphql.yml. After Artillery completes all the requests, you'll see how the GraphQL API performs under this load:

All VUs finished. Total time: 10 minutes, 39 seconds

--------------------------------
Summary report @ 15:31:12(+0900)
--------------------------------

vusers.created_by_name.Create and fetch messages flow: ...... 30000
vusers.created.total: ....................................... 30000
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 947.6
max: ...................................................... 20767.2
median: ................................................... 1465.6
p95: ...................................................... 20136.3
p99: ...................................................... 20136.3
http.request_rate: .......................................... 29/sec
http.requests: .............................................. 3120000
http.codes.200: ............................................. 3120000
http.responses: ............................................. 3120000
http.response_time:
min: ...................................................... 173
max: ...................................................... 4870
median: ................................................... 186.8
p95: ...................................................... 1312
p99: ...................................................... 3899

This example shows how straightforward it is to create a fully-fledged load test for GraphQL services. Artillery's HTTP engine has everything you need out of the box to check the most important actions of your GraphQL API. It's a simple yet powerful way to ensure your GraphQL services are resilient to any spikes in traffic and remain performant throughout. Although the performance of our GraphQL service isn't close to what we'd like, it gives us an excellent starting point for improvement.

Expanding the GraphQL load test

Let's add another scenario to validate the GraphQL API's performance further to expand our load test. For this flow, we want to validate that the service correctly handles password updates. For security purposes, the GraphQL service uses bcrypt hashing to ensure we don't store plain-text passwords in the database. Depending on how a service uses bcrypt, the hashing function can slow down password-related functionality, so it's an excellent candidate to check for performance purposes.

The second scenario for our test goes through the following steps:

  • We'll create a new user record and log in as that user, the same way we did in the first scenario.
  • Next, we'll log in as that user to test the login process and capture their ID from the data store.
  • With the user's ID in hand, we'll call the mutation for updating a user's database record to update their password. This step helps us verify that the request works as expected along with testing its performance.
  • Just like before, we'll delete the user at the end of the flow to keep the database clean of any test records.

The entire test script with both flows is as follows:

config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 50

scenarios:
- name: "Create and fetch messages flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"

- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"

- loop:
- post:
url: "/"
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: "{{ userId }}"
body: "Message Body {{ $loopCount }}"
count: 100

- post:
url: "/"
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: "{{ userId }}"

- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"

- name: "Update password flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"

- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"

- post:
url: "/"
json:
query: |
mutation UpdateUserMutation($updateUserId: ID!, $input: UpdateUserInput) {
updateUser(id: $updateUserId, input: $input) {
id
}
}
variables:
updateUserId: "{{ userId }}"
password: "my-new-password"

- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"

Running the test script with Artillery as we did previously (artillery run graphql.yml) shows the generated VUs going through both scenarios:

All VUs finished. Total time: 11 minutes, 20 seconds

--------------------------------
Summary report @ 15:43:19(+0900)
--------------------------------

vusers.created_by_name.Update password flow: ................ 18469
vusers.created.total: ....................................... 30000
vusers.created_by_name.Create and fetch messages flow: ...... 11531
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 899.1
max: ...................................................... 30451.4
median: ................................................... 1300.1
p95: ...................................................... 29445.4
p99: ...................................................... 29445.4
http.request_rate: .......................................... 28/sec
http.requests: .............................................. 1966900
http.codes.200: ............................................. 1966900
http.responses: ............................................. 1966900
http.response_time:
min: ...................................................... 172
max: ...................................................... 5975
median: ................................................... 186.8
p95: ...................................................... 1620
p99: ...................................................... 4867

As seen in our less-than-ideal performance results, GraphQL helps developers build flexible APIs with ease, but that flexibility comes at a cost if you're not careful. It's easy to accidentally introduce N+1 queries or implement ineffective caching that brings your GraphQL API to a grinding halt with only a handful of consumers. This example test script gives you an idea of how to use Artillery to run robust load tests against your GraphQL services to find any bottlenecks or other issues during periods of high traffic.

End-to-end example

An end-to-end example with Apollo and Prisma can be seen on artillery-examples/graphql-api-server.