Introduction to Elixir by Example
Here at WatchSumo we are big fans of Ruby and Erlang, we've been using Ruby and Ruby on Rails for many years and Erlang for the last four years.
This is why we are very excited with the launch of Elixir 1.0.0.
Elixir is a language created by José Valim which runs on top of the Erlang Virtual machine and provides extra features like macros, better string handling, tools, etc...
I won't go into more detail because it has been explained much better by Devin Torres in his Elixir Conf talk, The Excitement of Elixir:
Elixir borrows from Ruby, and the syntax might look familiar but aside from the syntax it also borrows concepts from other languages like Lisp or Clojure, aiming to make development productive by providing great tools out of the box.
Elixir brings a new approach to the Erlang Virtual machine wich has been designed to build highly concurrent and fault tolerant systems, hopefully it will bring new people to the Erlang ecosystem which might have been previously scared or intimidated by it's syntax or different concepts.
Enough Introduction, Let's write some Elixir!
Let's do it!
The goal of this series is to write from scratch a very simple Elixir script that will allow to perform a number of requests to a url and where we will be able to configure how many requests we want to run per second. By the end of this series we should be able to run something like:
$ ./experf --n=4 --rps=2 --url=http://www.example.com
which will execute 4 requests against http://www.example.com but running only 2 per second.
I'll try to explain some basic Elixir concepts along the way and provide detailed steps so you can build this little application with me.
Let's start!
Make sure you have Elixir installed.
$ elixir -v Elixir 1.0.0
Let's create a new Elixir project called experf using the build tool Mix which was also installed with Elixir.
$ mix new experf
You should see the following output:
* creating README.md * creating .gitignore * creating mix.exs * creating config * creating config/config.exs * creating lib * creating lib/experf.ex * creating test * creating test/test_helper.exs * creating test/experf_test.exs
Your mix project was created successfully. You can use mix to compile it, test it, and more:
cd experf mix test
Run mix help for more commands.
You can now cd into the newly created experf directory, from now on I'll assume that's our working directory.
Let's edit mix.exs, we will configure escript which will allow us to run the application as an executable file.
def project do [app: :experf, version: "0.0.1", elixir: "~> 1.0.0", escript: [main_module: Experf], deps: deps] end
This way when we run ./escript the function main in the module Experf will be run, so we need to implement it experf/lib/experf.ex.
require Logger defmodule Experf do def main(_args) do Logger.info "Hello World!" end end
This is defining a module called Experf with a function main which just logs "Hello World!". Notice that we need to require Logger before calling Logger.info, this is because info is implemented as a macro.
We can now compile the code and generate the executable file:
$ mix escript.build
And run it:
$ ./experf
23:31:35.234 [info] Hello World!
You can see what we have done so far here
Parsing command line options
Now we will make sure that we can parse a bunch of options from the command line. We will use OptionParser for this and define a private function (that's why we use defp instead of def) called parse_args that will parse the arguments received from the command line(--n for the number of total requests and --url for the url we will be requesting).
require Logger defmodule Experf do def main(args) do options = parse_args(args) Logger.info inspect(options) end defp parse_args(args) do {options, _, _} = OptionParser.parse(args, switches: [n: :integer, url: :string] ) options end end
parse_args is taking advantage of Elixir's pattern_matching to assign the variable options (and ignore the other two elements in the tuple returned by OptionParser.parse) and return it.
We can run this now:
$ mix escript.build $ ./experf --n=4 --url=http://www.example.com 11:19:09.950 [info] [n: 4, url: "http://www.example.com"]
You can see what we have done so far here.
Sequential Requests
The next step is to make sure that we can make n requests to url. We will start by making sequential requests.
To make the requests we will use an Elixir library called HTTPoison which is available in Elixir's package manager Hex.
We only need to add the dependency in the deps function in mix.exs.
defp deps do [ {:httpoison, "~> 0.4.2"} ] end
And run $ mix deps.get to download it.
HTTPoison runs as an OTP application so we need to make sure it's started before using it, we can do this in the application function in mix.exs.
def application do [applications: [:logger, :httpoison]] end
In order to run the requests we will create a new module Experf.Http in lib/experf/http.ex that will make the request.
require Logger defmodule Experf.Http do def request(id, url) do try do HTTPoison.get(url) |> handle_response(id) rescue error in HTTPoison.HTTPError -> Logger.info "#{id}: error (#{inspect error.message})" end end defp handle_response(%HTTPoison.Response{status_code: 200}, id) do Logger.info "#{id}: success" end defp handle_response(%HTTPoison.Response{status_code: status_code}, id) do Logger.info "#{id}: error (#{status_code})" end end
What we do here in request is simply call HTTPoison.get and pipe the response (this is what the |> operator does, similar to Unix |) to the function handle_response which uses pattern matching to determine if the struct returned by HTTPoison has a status_code of 200 or not and log the result.
We also need to handle the case where HTTPoison raises an exception, that's why we wrap the call within try rescue.
Now we need to call Experf.Http.request from our main function in lib/experf.ex
require Logger defmodule Experf do def main(args) do options = parse_args(args) do_requests(options[:n], options[:url]) end defp parse_args(args) do {options, _, _} = OptionParser.parse(args, switches: [n: :integer, url: :string] ) options end defp do_requests(n, url) do Enum.each(1..n, fn(i) -> Experf.Http.request(i, url) end) end end
We can run this now:
$ mix escript.build $ ./experf --n=4 --url=http://www.example.com 12:43:09.900 [info] 1: success 12:43:10.019 [info] 2: success 12:43:10.135 [info] 3: success 12:43:10.254 [info] 4: success
You can see what we have done so far here.
Concurrent Requests
Obviously one of the main features of the Erlang Virtual machine (BEAM) is concurrency, you can read a lot more about this on Fred Hébert's excellent Learn You Some Erlang for Great Good!.
The basic idea is that in Elixir/Erlang we can spawn as many lightweight processes as we need which will be run concurrently making use of as many cores as available.
Under the hood the Erlang virtual machine will only use one operating system thread per core and will schedule each lightweight process on them, this is why they are very cheap to create compared to Threads or OS Processes.
Another feature of the Erlang VM is that all I/O operations are asynchronous which means that we can spawn hundreds or thousands of processes and all of them can start an I/O operation (like an HTTP request) without blocking other processes.
In order to do this we will use Elixir's spawn which will create a new process every time it's called.
We only need to change the function do_requests in lib/experf.ex from:
Experf.Http.request(i, url)
to:
spawn Experf.Http, :request, [i, url]
And we can now run our concurrent program:
$ mix escript.build $ ./experf --n=4 --url=http://www.example.com
Wow! No output! What's going on here?
The problem is that now we have to think about our problem in terms of concurrency. Our script exits as soon as the main function in lib/experf.exhas finished its execution. Because now we are just spawning new processes nowhere in our code we are actually saying that we want to wait until those processes are finished.
Let's see if we can fix this!
The way that Elixir's processes have to comunicate with each other is by sending and receiving messages, in order to solve our problem we are going to create a new process, which will be responsible for coordinating all the Http processes and making sure that all of them finish before exiting our program.
We will create a new module called Experf.Coordinator in lib/experf/coordinator.ex.
defmodule Experf.Coordinator do def start(n) do Process.register(self, Experf.Coordinator) coordinate(%{finished: 0, num_requests: n}) end defp coordinate(%{finished: n, num_requests: n}) do :ok end defp coordinate(status = %{finished: f, num_requests: n}) do receive do {:finished, _i} -> coordinate(%{status | finished: f + 1}) end end end
We will start the coordinator by calling the start function with the number of requests that it will wait for.
We register the name of this process as Experf.Coordinator so that the Experf.Http processes can send it a message.
Then we use a pattern that's very common in Elixir programming, it consists of passing around some state, in this case a Map with the number of finished requests finished and the number of total requests num_requests and using recursion combined with pattern matching to determine when we are done.
In this particular case defp coordinate(%{finished: n, num_requests: n}) is saying that when we call coordinate and the number of finished requests is the same as the number of total requests then we are done and return the atom :ok.
In any other case that f != n we call receive which will just wait until it receives a finished message(we ignore any other message which is not {:finished, i}) and it calls coordinate again with an increased value for finished.
We now need to make sure that Experf.Http sends the message to the coordinator once it's done:
lib/experf/http.ex
def request(id, url) do try do HTTPoison.get(url) |> handle_response(id) rescue error in HTTPoison.HTTPError -> Logger.info "#{id}: error (#{inspect error.message})" after send Experf.Coordinator, {:finished, id} end end
Another thing we have to do is to make sure that the coordinator process starts before the Experf.Http processes and that the main function waits for it to finish, otherwise we wouldn't have solved the problem at all!
In order to do this we will use an Elixir Task.
The main function in lib/experf.ex will look like this:
def main(args) do options = parse_args(args) coord_task = Task.async(Experf.Coordinator, :start, [options[:n]]) do_requests(options[:n], options[:url]) Task.await(coord_task, :infinity) end
A Task is an abstraction that allows to spawn a process then do some other work and then wait until the process has returned. This is exactky what we do here by spawning the Experf.Coordinator process, then spawning the Experf.Http workers and then waiting for the Experf.Coordinator process to come back.
We can now run this code:
$ ./experf --n=4 --url=http://www.example.com 15:20:36.026 [info] 2: success 15:20:36.030 [info] 1: success 15:20:36.033 [info] 3: success 15:20:36.040 [info] 4: success
As you can see by looking at the response the order of the requests is not sequential and they all returned in the same tenth of a second. This is because they were all run concurrently.
You can see what we have done so far here.
In the next post we will see how we can limit the number of requests that run per second. We need to make the Coordinator a bit smarter and it needs to not only know when the workers have finished but also tell them when to start depending on how many of them have already run in the same second.
Disclaimers
I'm learning Elixir myself (who isn't? 1.0.0 just came out!), so this might not be the best way to solve this problem but I think it might be interesting enough to cover some basic concepts. If you can think of a better approach or you think I'm doing something silly, please let me know!
There are a few things that we are not covering in this post and some of the techniques describe here wouldn't be used in a production application. In practice you would almost never want to call spawn directly but use some of the abstractions provided by OTP, like GenServer and Supervision Trees that make sure that processes are restarted when they crash.
At least I hope this has served as a basic introduction to Elixir!
If you are really impatient, you can take a look at the master branch of vicmargar/experf It is a bit more complicated that what I described here since it also takes into account the number of concurrent requests and calculates some statistics around the response times.