CubDB great fit for Nerves-powered embedded Elixir projects

Today I learned how to use CubDB and some alternatives.

CubDB is a disk-based key-value database written in the Elixir language and it can be used as part of an Elixir application without any configuration. Its API is so simple and intuitive to Elixir programmers. This type database is a perfect fit for Nerves-powered embedded Elixir projects.

Here are the versions Erlang and Elixir that I use as of writing.

elixir          1.12.1-otp-24
erlang          24.0.2

Playing with CubDB in IEx shell

With Elixir 1.12, we can play with it using Mix.install in the IEx shell.

# Start an Interactive Elixir shell.
❯ iex

iex> :ok = Mix.install([{:cubdb, "~> 1.0"}])
:ok

iex> {:ok, cubdb} = CubDB.start_link(data_dir: "tmp")
{:ok, #PID<0.166.0>}

iex> CubDB.put(cubdb, :word, "hello")
:ok

iex> CubDB.get(cubdb, :word)
"hello"

iex> CubDB.delete(cubdb, :word)
:ok

iex> CubDB.get(cubdb, :word)
nil

iex> ls "tmp"
0.cub     data

It is very intuitive.

Starting as a child when the app starts

According to the official documentation:

Important: avoid starting multiple CubDB processes on the same data directory. Only one CubDB process should use a specific data directory at any time.

so it seems to be a good idea to name the database process. Also why not start the database when the application is starting?

In a Nerves project, we can write a file in /data directory. Don't forget the leading thrash (/). It is not data.

Because the root filesystem is read-only, we also add a read/write partition by default, called app_data and mounted at /data (the root user's home directory).

defmodule HelloNerves.Application do

  ...

  @nerves_data_dir "/data"

  def children(_target) do
    [
      # Children for all targets except host
      {CubDB, [data_dir: @nerves_data_dir, name: CubDB]}
    ]
  end

  ...

Then we can use CubDB anywhere in the app anytime.

CubDB.put(CubDB, :word, "hello")

It is worth noting that we will get an error when we cannot access the specified file.

iex> CubDB.start_link(data_dir: "/secret_dir", name: CubDB)
{:error, :erofs}
** (EXIT from #PID<0.105.0>) shell process exited with reason: :erofs

Alternatives to CubDB

The CubDB author is so kind that he lists some alternative solutions for similar use cases.

• ETS
• DETS
• Mnesia
• SQLite, LevelDB, LMDB, etc
• Writing to plain files directly

The list explains the key characteristics of each item succinctly, which is a great educational resource to me.

I also found a few Elixir wrappers of ETS:

• TheFirstAvenger/ets - Elixir wrapper for the Erlang :ets module
  • :ets, the Elixir way
  • design goals
• michalmuskala/persistent_ets
  • Ets table backed by a persistence file
  • Does not use DETS for these reasons
• fireproofsocks/pockets
  • Elixir wrapper around Erlang ETS and DETS
  • provide a simple and familiar interface for caching and persisting data by implementing many of the functions found in the built-in Map and Keyword modules
• whitfin/cachex
  • in-memory key/value store with support for many useful features
  • built on top of ETS
• whitfin/stash
  • small and user-friendly ETS wrapper for caching in Elixir
  • inactive?

Wrapping ETS and DETS

If all we want is a simple key-value store, we could just write a plain Elixir module that thinly wraps ETS and/or DETS. This might suffice in many situations.

defmodule HelloNerves.MemoryStore do
  @ets_config [
    {:read_concurrency, true},
    {:write_concurrency, true},
    :public,
    :set,
    :named_table
  ]

  def create_table() do
    :ets.new(__MODULE__, @ets_config)
  end

  def get(key) do
    case :ets.lookup(__MODULE__, key) do
      [] -> nil
      [{_key, value} | _rest] -> value
    end
  end

  def put(key, value) do
    :ets.insert(__MODULE__, [{key, value}])
    |> ok_or_error_response
  end

  def delete(key) do
    :ets.delete(__MODULE__, :word)
    |> ok_or_error_response
  end

  def delete_table do
    :ets.delete(__MODULE__)
    |> ok_or_error_response
  end

  defp ok_or_error_response(ets_result) do
    if ets_result, do: :ok, else: :error
  end
end

defmodule HelloNerves.FileStore do
  def open(opts \\ []) do
    data_dir = opts[:data_dir] || "tmp"
    file = :binary.bin_to_list(Path.join(data_dir, "file_store"))

    :dets.open_file(__MODULE__, file: file, type: :set)
  end

  def get(key) do
    case :dets.lookup(__MODULE__, key) do
      [] -> nil
      [{_key, value} | _rest] -> value
    end
  end

  def put(key, value) do
    :dets.insert(__MODULE__, [{key, value}])
  end

  def delete(key) do
    :dets.delete(__MODULE__, key)
  end

  def close do
    :dets.close(__MODULE__)
  end
end

But when something goes wrong, ETS argument error is very unfriendly. We might end up wanting more robust features.

# When table does not exist for example
** (ArgumentError) argument error
    (stdlib 3.15.1) dets.erl:1259: :dets.delete(:my_table, :name)

Final thoughts

I think CubDB is one of the most intuitive to many Elixir programmers among all the solutions. It is written in Elixir. Although we have Erlang builtin solutions like ETS or DETS, we might need some cognitive overhead for understanging how they work unless we are already familiar with them. While there are some Elixir library that wrap ETS, I could not find anything similar for DETS that is actively maintained.

If one is not sure which one to use, CubDB can be a good default for file-based key-value store in Elixir. It can help us develop things quickly and it just works.

After playing with CubDB, ETS, DETS etc and ended up with this library DBKV, which is inspired by CubDB.

That's it!