How to Use gRPC in Elixir

In today's post, we'll learn what gRPC is, when you should reach for such a tool, and some of the pros and cons of using it. After going over an introduction of gRPC, we'll dive right into a sample application where we'll build an Elixir backend API powered by gRPC.

Let's jump right in!

What Is gRPC and How Does It Work?

gRPC is a framework used to enable a remote procedure call (RPC) style of communication. RPC is a style of system communication where a client can directly invoke exposed methods on a server. From the client's perspective, it feels no different than making a call to a local function or method as long as you provide the applicable parameters and handle the return type appropriately. gRPC facilitates this communication by providing 2 things for you:

A client library which can be used to invoke allowed procedures
A consistent data serialization standard via Protocol Buffers

Let's break these two items down so we can appreciate the inner workings of gRPC. When creating a gRPC server, you'll have to define what procedures are invokable from the client, what inputs they accept, and what outputs they return. This interface specification is what allows client libraries (generally called gRPC stubs) to be automatically generated for various languages and runtimes as the contract for the remote procedure call is explicitly defined. This gRPC client library can then communicate with the server using Protocol Buffers. Protocol Buffers provide a mechanism for serializing and deserializing the payloads (both request and response) so that you can operate on the data with types native to your language. The diagram available in the gRPC documentation can help visualize this interaction:

Source: https://grpc.io

One thing that we haven't discussed yet is how data is transmitted between the client and the server. For this, gRPC leans on the HTTP/2 protocol. By using HTTP/2 as the underlying protocol, gRPC is able to support features such as bi-directional data streaming and several other features that are not available in HTTP/1.1.

When Would You Use gRPC Over REST/GraphQL?

The obvious question that may come to mind is: "How does gRPC compare to REST/GraphQL, and when do I use one over the other?".

In general, if you plan to use gRPC for a frontend application, there are a couple of caveats that you need to keep in mind. In order to serialize and deserialize Protocol Buffer payloads from your Javascript application, you'll have to leverage grpc-web. In addition, you'll also need to run a proxy on the backend (the default supported gRPC proxy is Envoy) since browsers cannot talk directly to gRPC servers. Depending on your resources and time constraints, this may be a show stopper; in which case, REST and GraphQL will do just fine.

If frontend application communication is not a requirement, and instead what you require is inter-microservice communication from within your service cluster, then the barrier to entry for gRPC gets lowered considerably. At the time of writing, gRPC currently has client libraries and tooling for most mainstream languages including Elixir, Python, C++, Go, Ruby, to name a few. I would argue that the barrier to entry for consuming a RESTful API is still much lower than consuming a gRPC service given that all you need for the former is an HTTP client, which is baked into most languages and runtimes these days.

On the other hand, if you are willing to make the investment, you do get the added benefits of having your responses and requests checked against the Protocol Buffer specification that is used for code generation. This, in turn, provides you with some guarantees as to what you can expect on the client-side and the server-side. This guarantee and introspection is also something that you get with GraphQL when you define your server schemas. An added benefit of GraphQL over gRPC is that you are able to dynamically request embedded properties from within your schema depending on the query that you make to your backend server.

Like most things in the software engineering field, the technology you choose will largely depend on your application. Below are my personal TL;DR rules of thumb regarding the various technologies:

gRPC:

Use: When communicating between microservices in my service cluster or if performance is a requirement
Don't use: When I need to transmit data from the browser to the backend

GraphQL:

Use: When I need to aggregate data from multiple microservices for the purposes of streamlining frontend development, or if my frontend data requirements are dynamic
Don't use: When communicating between microservices in my service cluster, or if my API needs to be used by the lowest common denominator of consumers

REST:

Use: When I need to put something together quickly or if I need to cater to the lowest common denominator of consumers
Don't use: If I require any kind of type checking or if I want to reduce payload sizes over the wire

Experimenting With gRPC in Elixir

With all the theory out of the way, it's time to get our hands dirty and experiment with wiring up a gRPC server. In order to keep us focused on the gRPC experience, we'll opt for having a backend powered by an Agent versus an actual database, but all of the concepts should be easily transferable to an application backed by Postgres, for example. Our gRPC application will be a simple user management service where we can create and fetch users. After creating our Elixir service, we'll interact with it via grpcurl, which is effectively cURL, but for gRPC. If at any point you get stuck or require the source code, feel free to check it out on GitHub. With all that being said, let's dive right in!

Before creating our Elixir project, there are a couple of things that we require on our machine in order to properly develop and test our application. We'll first need to install protoc so that .proto files can be compiled appropriately. If you are on an OSX machine, you can run brew install protobuf, otherwise, see instructions specific to your platform. Now, with protoc available on your machine, you'll also want to install grpcurl so that you can interact with the application. Once again, if you are on an OSX machine, you can run brew install grpcurl, otherwise, check for instructions specific to your platform.

Lastly, you'll want to run mix escript.install hex protobuf and ensure that protoc-gen-elixir script is available on your path (if you use ASDF as your runtime version manager, this requires running asdf reshim elixir). With all that boilerplate done, you can run mix new sample_app --sup to get a new application started.

Once inside your sample application directory, you'll want to update your mix.exs file to include our gRPC related dependencies. For this application, we will be leveraging https://github.com/elixir-grpc/grpc and https://github.com/tony612/protobuf-elixir. In order to bring these two dependencies into your project, ensure that your deps/0 function looks like this:

1defp deps do
2  [
3    {:grpc, "~> 0.5.0-beta"},
4    {:cowlib, "~> 2.8.0", hex: :grpc_cowlib, override: true}
5  ]
6end

With that in place, run mix deps.get from the terminal to pull down all necessary project dependencies. Next, you'll want to create a configuration file at config/config.exs with the following content:

1use Mix.Config
2
3# Configures Elixir's Logger
4config :logger, :console, format: "$time $metadata[$level] $message\n"
5
6config :grpc, start_server: true

Next, we'll want to create the required Protocol Buffer definitions for our application. The Protocol Buffer specification is fairly large and we'll only be using a small subset of it to keep things simple. Create a file sample_app.proto at the root of your project with the following content:

1syntax = "proto3";
2
3package sample_app;
4
5service User {
6  rpc Create (CreateRequest) returns (UserReply) {}
7  rpc Get (GetRequest) returns (UserReply) {}
8}
9
10message UserReply {
11  int32 id = 1;
12  string first_name = 2;
13  string last_name = 3;
14  int32 age = 4;
15}
16
17message CreateRequest {
18  string first_name = 1;
19  string last_name = 2;
20  int32 age = 3;
21}
22
23message GetRequest {
24  int32 id = 1;
25}

As you can see, our Protocol Buffer definition is fairly straightforward and easy to read. We define a service that exposes two RPC methods—Create and Get. We also define the types that each of those RPC calls takes as input and returns as a result. With the sample_app.proto file in place, we'll want to open up a terminal and run the following:

1$ protoc --elixir_out=plugins=grpc:./lib sample_app.proto

You'll notice that this command produces a file lib/sample_app.pb.ex with several modules within it. If you look carefully, you'll see that the code that was generated is the Elixir representation of the sample_app.proto file that we wrote. It contains all of the types that we defined along with the RPC method definitions. With our auto-generated code in place, let's get to work on the actual RPC handlers.

As previously mentioned, we'll be using an Agent to persist state across gRPC calls instead of a database, for the sake of simplicity. Our agent will have the ability to look up users via their ID, and will also be able to create new users. Create a file lib/user_db.ex with the following code which provides that functionality:

1defmodule UserDB do
2  use Agent
3
4  def start_link(_) do
5    Agent.start_link(
6      fn ->
7        {%{}, 1}
8      end,
9      name: __MODULE__
10    )
11  end
12
13  def add_user(user) do
14    Agent.get_and_update(__MODULE__, fn {users_map, next_id} ->
15      updated_users_map = Map.put(users_map, next_id, user)
16
17      {Map.put(user, :id, next_id), {updated_users_map, next_id + 1}}
18    end)
19  end
20
21  def get_user(id) do
22    Agent.get(__MODULE__, fn {users_map, _next_id} ->
23      Map.get(users_map, id)
24    end)
25  end
26end

With that in place, we can create our RPC handlers for creating and getting users. Create a file lib/sample_app.ex with the following content:

1defmodule SampleApp.Endpoint do
2  use GRPC.Endpoint
3
4  intercept GRPC.Logger.Server
5  run SampleApp.User.Server
6end
7
8defmodule SampleApp.User.Server do
9  use GRPC.Server, service: SampleApp.User.Service
10
11  def create(request, _stream) do
12    new_user =
13      UserDB.add_user(%{
14        first_name: request.first_name,
15        last_name: request.last_name,
16        age: request.age
17      })
18
19    SampleApp.UserReply.new(new_user)
20  end
21
22  def get(request, _stream) do
23    user = UserDB.get_user(request.id)
24
25    if user == nil do
26      raise GRPC.RPCError, status: :not_found
27    else
28      SampleApp.UserReply.new(user)
29    end
30  end
31end

Our file defines two modules. The SampleApp.Endpoint module defines the gRPC server and provides the handler module to service requests. The SampleApp.User.Server module contains the actual implementations of the two RPC calls that we defined. You'll notice that for each of the handlers, we provide the correct return type (as defined in our Protocol Buffer file). When we encounter an error (in this case, looking up a user that doesn't exist), we raise a GRPC.RPCError with the appropriate status code.

All that is left now is to start up our Agent and our gRPC server, and we're good to go. Open up lib/sample_app/application.ex and ensure that your process children list looks like this:

1children = [
2  UserDB,
3  {GRPC.Server.Supervisor, {SampleApp.Endpoint, 50051}}
4]

With that in place, you should be able to run mix grpc.server from the terminal to start your gRPC server. In another terminal session (and from within the project directory), you should be able to use grpcurl commands to interact with your application:

1$ grpcurl -plaintext -proto sample_app.proto -d '{"first_name": "Bob", "last_name": "Smith", "age": 40}' localhost:50051 sample_app.User.Create
2{
3  "id": 1,
4  "firstName": "Bob",
5  "lastName": "Smith",
6  "age": 40
7}
8
9$ grpcurl -plaintext -proto sample_app.proto -d '{"id": 1}' localhost:50051 sample_app.User.Get
10{
11  "firstName": "Bob",
12  "lastName": "Smith",
13  "age": 40
14}
15
16$ grpcurl -plaintext -proto sample_app.proto -d '{"id": 2}' localhost:50051 sample_app.User.Get
17ERROR:
18  Code: NotFound
19  Message: Some requested entity (e.g., file or directory) was not found

Conclusion

Thanks for sticking with me to the end. Hopefully, you learned a thing or two about gRPC and how to go about using it within an Elixir application. If you would like to learn more about gRPC or any of the tools that I mentioned, I suggest going through the following resources:

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!