Artillery is an open-source command-line tool purpose-built for load testing and smoke testing web applications. It is written in JavaScript and it supports testing HTTP, Socket.io, and WebSockets APIs.
This article will get you started with load testing your Node.js APIs using Artillery. You'll be able to detect and fix critical performance issues before you deploy code to production.
Before we dive in and set up Artillery for a Node.js app, though, let's first answer the question: what is load testing and why is it important?
Why Should You Do Load Tests in Node.js?
Load testing is essential to quantify system performance and identify breaking points at which an application starts to fail. A load test generally involves simulating user queries to a remote server.
Load tests reproduce real-world workloads to measure how a system responds to a specified load volume over time. You can determine if a system behaves correctly under loads it is designed to handle and how adaptable it is to spikes in traffic. It is closely related to stress testing, which assesses how a system behaves under extreme loads and if it can recover once traffic returns to normal levels.
Load testing can help validate if an application can withstand realistic load scenarios without a degradation in performance. It can also help uncover issues like:
- Increased response times
- Memory leaks
- Poor performance of various system components under load
As well as other design issues that contribute to a suboptimal user experience.
In this article, we'll focus on the free and open-source version of Artillery to explore load testing. However, bear in mind that a pro version of Artillery is also available for those whose needs exceed what can be achieved through the free version. It provides added features for testing at scale and is designed to be usable even if you don't have prior DevOps experience.
Installing Artillery for Node.js
Artillery is an npm package so you can
install it through npm or yarn:
1$ yarn global add artilleryIf this is successful, the artillery program should be accessible from
the command line:
1$ artillery -V
2 ___ __ _ ____ _
3 _____/ | _____/ /_(_) / /__ _______ __ (_)___ _____
4 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
5/____/ ___ |/ / / /_/ / / / __/ / / /_/ / / / /_/ /____/
6 /_/ |_/_/ \__/_/_/_/\___/_/ \__, (_)_/\____/
7 /____/
8
9------------ Version Info ------------
10Artillery: 1.7.7
11Artillery Pro: not installed (https://artillery.io/pro)
12Node.js: v16.7.0
13OS: linux/x64
14--------------------------------------
Basic Artillery Usage
Once you've installed the Artillery CLI, you can start using it to send traffic
to a web server. It provides a quick subcommand that lets you run a test
without writing a test script first.
You'll need to specify:
- an endpoint
- the rate of virtual users per second or a fixed amount of virtual users
- how many requests should be made per user
1$ artillery quick --count 20 --num 10 http://localhost:4000/exampleThe --count parameter above specifies the total number of virtual users, while
--num indicates the number of requests that should be made per user. Therefore,
200 (20*10) GET requests are sent to the specified endpoint. On successful completion of the test, a report
is printed out to the console.
1All virtual users finished
2Summary report @ 14:46:26(+0100) 2021-08-29
3 Scenarios launched: 20
4 Scenarios completed: 20
5 Requests completed: 200
6 Mean response/sec: 136.99
7 Response time (msec):
8 min: 0
9 max: 2
10 median: 1
11 p95: 1
12 p99: 2
13 Scenario counts:
14 0: 20 (100%)
15 Codes:
16 200: 200
This shows several details about the test run, such as the requests completed, response times, time taken for the test, and more. It also displays the response codes received on each request so that you can determine if your API handles failures gracefully in cases of overload.
While the quick subcommand is handy for performing one-off tests from
the command line, it's quite limited in what it can achieve. That's
why Artillery provides a way to configure different load testing scenarios
through test definition files in YAML or JSON formats. This allows great
flexibility to simulate the expected flows at one or more of your application's endpoints.
Writing Your First Artillery Test Script
In this section, I'll demonstrate a basic test configuration that you can apply to any application. If you want to follow along, you can set up a test environment for your project, or run the tests locally so that your production environment is not affected. Ensure you install Artillery as a development dependency so that the version you use is consistent across all deployments.
1$ yarn add -D artilleryAn Artillery test script consists of two main sections: config and
scenarios. config includes the general configuration settings for
the test such as the target, response timeouts, default HTTP headers, etc.
scenarios consist of the various requests that virtual users should make
during a test. Here's a script that tests an endpoint by
sending 10 virtual users every second for 30 seconds:
1config:
2 target: "http://localhost:4000"
3 phases:
4 - duration: 30
5 arrivalRate: 10
6
7scenarios:
8 - name: "Retrieve data"
9 flow:
10 - get:
11 url: "/example"In the above script, the config section defines the base URL for the
application that's being tested in the target property. All the endpoints
defined later in the script will run against this base URL.
The phases property is then
used to set up the number of virtual users generated in a
period of time and how frequently these users are sent to specified endpoints.
In this test, duration determines that virtual users will be generated
for 30 seconds and arrivalRate determines the number of virtual users
sent to the endpoints per second (10 users).
On the other hand, the scenarios section defines the various operations that a virtual user should perform. This is controlled through the flow
property, which specifies the exact steps that should be executed in order. In
this case, we have a single step: a GET request to the /example
endpoint on the base URL. Every virtual user that
Artillery generates will make this request.
Now that we've written our first script, let's dive into how to run a load test.
Running a Load Test in Artillery
Save your test script to a file (such as load-test.yml) and
execute it through the command below:
1$ artillery run path/to/script.ymlThis command will start sending virtual users to the specified endpoints at a rate of 10 requests per second. A report will be printed to the console every 10 seconds, informing you of the number of test scenarios launched and completed within the time period, and other statistics such as mean response time, HTTP response codes, and errors (if any).
Once the test concludes, a summary report (identical to the one we examined earlier) is printed out before the command exits.
1All virtual users finished
2Summary report @ 15:38:48(+0100) 2021-09-02
3 Scenarios launched: 300
4 Scenarios completed: 300
5 Requests completed: 300
6 Mean response/sec: 9.87
7 Response time (msec):
8 min: 0
9 max: 1459
10 median: 1
11 p95: 549.5
12 p99: 1370
13 Scenario counts:
14 Retrieve data: 300 (100%)
15 Codes:
16 200: 300
How to Create Realistic User Flows
The test script we executed in the previous section is not very different from the
quick example in that it makes requests to only a single endpoint. However, you can use Artillery to test more complex user flows in an application.
In a SaaS product, for example, a user flow could be: someone lands on your homepage, checks out the pricing page, and then signs up for a free trial. You'll definitely want to find out how this flow will perform under stress if hundreds or thousands of users are trying to perform these actions at the same time.
Here's how you can define such a user flow in an Artillery test script:
1config:
2 target: "http://localhost:4000"
3 phases:
4 - duration: 60
5 arrivalRate: 20
6 name: "Warming up"
7 - duration: 240
8 arrivalRate: 20
9 rampTo: 100
10 name: "Ramping up"
11 - duration: 500
12 arrivalRate: 100
13 name: "Sustained load"
14 processor: "./processor.js"
15
16scenarios:
17 - name: "Sign up flow"
18 flow:
19 - get:
20 url: "/"
21 - think: 1
22 - get:
23 url: "/pricing"
24 - think: 2
25 - get:
26 url: "/signup"
27 - think: 3
28 - post:
29 url: "/signup"
30 beforeRequest: generateSignupData
31 json:
32 email: "{{ email }}"
33 password: "{{ password }}"In the above script, we define three test phases in config.phases:
- The first phase sends 20 virtual users per second to the application for 60 seconds.
- In the second phase, the load will start at 20 users per second and gradually increase to 100 users per second over 240 seconds.
- The third and final phase simulates a sustained load of 100 users per second for 500 seconds.
By providing several phases, you can accurately simulate real-world traffic patterns and test how adaptable your system is to a sudden barrage of requests.
The steps that each virtual user takes in the
application are under scenarios.flow. The first request is GET / which leads to the
homepage. Afterward, there is a pause for 1 second (configured with think) to
simulate user scrolling or reading before making the next GET request to
/pricing. After a further delay of 2 seconds, the virtual user makes a GET request to
/signup. The last request is POST /signup, which sends a JSON payload in the
request body.
The {{ email }} and {{ password }} placeholders are populated through the
generateSignupData function, which executes before the request is made. This
function is defined in the processor.js file referenced in
config.processor. In this way, Artillery lets you specify custom hooks
to execute at specific points during a test run. Here are the
contents of processor.js:
1const Faker = require("faker");
2
3function generateSignupData(requestParams, ctx, ee, next) {
4 ctx.vars["email"] = Faker.internet.exampleEmail();
5 ctx.vars["password"] = Faker.internet.password(10);
6
7 return next();
8}
9
10module.exports = {
11 generateSignupData,
12};
The generateSignupData function uses methods provided by
Faker.js to generate a random email
address and password each time it is called. The results are then set on the
virtual user's context, and next() is called so that the scenario can continue to
execute. You can use this approach to inject dynamic random content into your
tests so they're as close as possible to real-world requests.
Note that other
hooks
are available aside from beforeRequest, including the following:
afterResponse- Executes one or more functions after a response has been received from the endpoint:
1- post:
2 url: "/login"
3 afterResponse:
4 - "logHeaders"
5 - "logBody"beforeScenarioandafterScenario- Used to execute one or more functions before or after each request in a scenario:
1scenarios:
2 - beforeScenario: "setData"
3 afterScenario: "logResults"
4 flow:
5 - get:
6 url: "/auth"function- Can execute functions at any point in a scenario:
1- post:
2 url: "/login"
3 function: "doSomething"Injecting Data from a Payload File
Artillery also lets you inject custom data through a payload file in CSV format. For example, instead of generating fake email addresses and passwords on the fly as we did in the previous section, you can have a predefined list of such data in a CSV file:
1Dovie32@example.net,rwkWspKUKy
2Allen.Fay@example.org,7BaFHbaWga
3Jany30@example.org,CWvc6Bznnh
4Dorris47@example.com,1vlT_02i6h
5Imani.Spencer21@example.net,1N0PRraQU7
To access the data in this file, you need to reference it in the test script
through the config.payload.path property. Secondly, you need to specify the
names of the fields you'd like to access through config.payload.fields. The
config.payload property provides several other
options
to configure its behavior, and it's also possible to specify multiple payload
files in a single script.
1config:
2 target: "http://localhost:4000"
3 phases:
4 - duration: 60
5 arrivalRate: 20
6 payload:
7 path: "./auth.csv"
8 fields:
9 - "email"
10 - "password"
11
12scenarios:
13 - name: "Authenticating users"
14 flow:
15 - post:
16 url: "/login"
17 json:
18 email: "{{ email }}"
19 password: "{{ password }}"Capturing Response Data From an Endpoint
Artillery makes it easy to capture the response of a request and reuse certain fields in a subsequent request. This is helpful if you're simulating flows with requests that depend on an earlier action's execution.
Let's assume you're providing a geocoding API that accepts the name of a place and returns its longitude and latitude in the following format:
1{
2 "longitude": -73.935242,
3 "latitude": 40.73061
4}
You can populate a CSV file with a list of cities:
1Seattle
2London
3Paris
4Monaco
5Milan
Here's how you can configure Artillery to use each city's longitude and latitude values in another request. For example, you can use the values to retrieve the current weather through another endpoint:
1config:
2 target: "http://localhost:4000"
3 phases:
4 - duration: 60
5 arrivalRate: 20
6 payload:
7 path: "./cities.csv"
8 fields:
9 - "city"
10
11scenarios:
12 - flow:
13 - get:
14 url: "/geocode?city={{ city }}"
15 capture:
16 - json: "$.longitude"
17 as: "lon"
18 - json: "$.latitude"
19 as: "lat"
20 - get:
21 url: "/weather?lon={{ lon }}&lat={{ lat }}"The capture property above is where all the magic happens. It's where you can
access the JSON response of a request and store it in a variable to reuse in subsequent requests. The longitude
and latitude properties from the /geocode response body (with the aliases lon
and lat, respectively) are then passed on as query parameters to the /weather
endpoint.
Using Artillery in a CI/CD Environment
An obvious place to run your load testing scripts is in a CI/CD pipeline so that your application is put through its paces before being deployed to production.
When using Artillery in such environments, it's necessary to set failure
conditions that cause the program to exit with a non-zero code. Your
deployment should abort if performance objectives are not met. Artillery provides
support for this use case through its config.ensure property.
Here's an example that uses the ensure setting to assert that 99% of all
requests have an aggregate response time of 150 milliseconds or less and that
1% or less of all requests are allowed to fail:
1config:
2 target: "https://example.com"
3 phases:
4 - duration: 60
5 arrivalRate: 20
6 ensure:
7 p99: 150
8 maxErrorRate: 1Once you run the test, it will continue as before, except that assertions are verified at the end of the test and cause the program to exit with a non-zero exit code if requirements are not met. The reason for a test failure is printed at the bottom of the summary report.
1All virtual users finished
2Summary report @ 07:45:48(+0100) 2021-09-03
3 Scenarios launched: 10
4 Scenarios completed: 10
5 Requests completed: 20
6 Mean response/sec: 4
7 Response time (msec):
8 min: 1
9 max: 487
10 median: 2
11 p95: 443.5
12 p99: 487
13 Scenario counts:
14 0: 10 (100%)
15 Codes:
16 200: 20
17
18ensure condition failed: ensure.p99 < 200
Aside from checking the aggregate latency, you can also run assertions on min,
max, and median — the minimum, maximum, and median response
times, respectively. Here's how to assert that requests never take more than 500
milliseconds to complete during a test run:
1config:
2 ensure:
3 max: 500The report for a failed test will indicate the reason for failure:
1All virtual users finished
2Summary report @ 08:29:59(+0100) 2021-09-03
3 Scenarios launched: 10
4 Scenarios completed: 10
5 Requests completed: 20
6 Mean response/sec: 3.64
7 Response time (msec):
8 min: 1
9 max: 603
10 median: 305.5
11 p95: 602.5
12 p99: 603
13 Scenario counts:
14 0: 10 (100%)
15 Codes:
16 200: 20
17
18ensure condition failed: ensure.max < 500Generating Status Reports in Artillery
Artillery prints a summary report for each test run to the standard output, but
it's also possible to output detailed statistics for a test run into a JSON file
by utilizing the --output flag:
1$ artillery run config.yml --output test.jsonOnce the test completes, its report is placed in a test.json file in the
current working directory. This JSON file can be visualized through Artillery's
online report viewer or converted into an
HTML report through the report subcommand:
1$ artillery report --output report.html test.json
2Report generated: report.html
You can open the report.html file in your browser to view a full report of the
test run. It includes tables and several charts that should give you a good idea
of how your application performed under load:
Extending Artillery With Plugins
Artillery's built-in tools for testing HTTP, Socket.io, and Websocket APIs can take you quite far in your load testing process. However, if you have additional requirements, you can search for plugins on NPM to extend Artillery's functionality.
Here are some official Artillery plugins that you might want to check out:
- artillery-plugin-expect: Helps with adding expectations to HTTP requests for functional or acceptance testing.
- artillery-plugin-publish-metrics: Used to send statistics from test runs to some external monitoring and observability systems.
- artillery-plugin-fuzzer: Helps you fuzz test your APIs with random and unexpected payloads to your API endpoints so you can catch errors. It is based on the Big List Of Naughty Strings.
- artillery-plugin-metrics-by-endpoint: Breaks down response time metrics by endpoint rather than displaying aggregate values across all endpoints.
You can also extend Artillery by creating your own plugins.
Use Artillery for Node.js Apps to Avoid Downtime
In this article, we've described how you can set up a load testing workflow for your Node.js applications with Artillery. This setup will ensure that your application performance stays predictable under various traffic conditions. You'll be able to account well for traffic-heavy periods and avoid downtime, even when faced with a sudden influx of users.
We've covered a sizeable chunk of what Artillery can do for you, but there's still lots more to discover. Ensure you read the Artillery official documentation to learn about the other features on offer.
Thanks for reading, and happy coding!
P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.
P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.
