Getting started with Yokan

Installing Yokan

Yokan can be installed using Spack as follows.

spack install mochi-yokan +bedrock

The +bedrock variant will enable Bedrock support, which will be useful for spinning up a Yokan server without having to write code. The spack info mochi-yokan command can be used to show the list of variants available. Many variants refer to database backend types. They are disabled by default. In this tutorial we will use the map backend, which is natively available in Yokan, but feel free to try these tutorials with other backends!

In the following sections, the code can be compiled and linked against the yokan-server, yokan-client, and yokan-admin libraries, which can be found either by calling find_package(yokan) in CMake, or pkg-config --libs --cflags yokan-server (respectively yokan-client and yokan-admin) with PkgConfig.

Instantiating a Yokan provider

Yokan adopts the typical Mochi microservice architecture (used for instance in the Margo microservice template), with a server library providing the microservice’s provider implementation, a client library providing access to its capabilities (e.g., putting and getting key/value pairs), and an admin library providing control over the providers (creating and destroying databases). Hence the first thing we need to do is instantiate a provider.

Since we have enabled Bedrock support, let’s take advantage of that and write a config.json file for Bedrock to use (if you are not familiar with Bedrock, I highly recommand you to read the Bedrock section. Using Bedrock will save you development time since it allows you to bootstrap a Mochi service using a JSON file instead of writing code).

{
    "libraries": {
        "yokan": "libyokan-bedrock-module.so"
    },
    "providers": [
        {
            "name" : "my_yokan_provider",
            "type" : "yokan",
            "provider_id" : 42,
            "pool" : "__primary__",
            "config" : {},
            "dependencies" : {}
        }
    ]
}

We can now give this config file to Bedrock as follows.

$ bedrock na+sm -c config.json
[2021-10-14 10:16:17.529] [info] [yokan] YOKAN provider registration done
[2021-10-14 10:16:17.530] [info] Bedrock daemon now running at na+sm://8551-0

We now have a Yokan provider running, with a provider id of 42.

If you need to create a provider in C (either because you don’t want to use Bedrock or because you want your provider to be embedded into an existing application), the following code shows how to do that.

server.c (show/hide)

#include <assert.h>
#include <stdio.h>
#include <margo.h>
#include <yokan/server.h>

int main(int argc, char** argv)
{
    margo_instance_id mid = margo_init("na+sm", MARGO_SERVER_MODE, 0, 0);
    assert(mid);

    hg_addr_t my_address;
    margo_addr_self(mid, &my_address);
    char addr_str[128];
    size_t addr_str_size = 128;
    margo_addr_to_string(mid, addr_str, &addr_str_size, my_address);
    margo_addr_free(mid,my_address);

    margo_set_log_level(mid, MARGO_LOG_INFO);
    margo_info(mid, "Server running at address %s", addr_str);

    yk_return_t ret = yk_provider_register(mid, 42, NULL, YOKAN_PROVIDER_IGNORE);
    assert(ret == YOKAN_SUCCESS);

    margo_wait_for_finalize(mid);

    return 0;
}

Using the admin library to create a database

Now that we have our server running, let’s write some code using the admin library to create a database.

admin.c (show/hide)

#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <margo.h>
#include <yokan/admin.h>

int main(int argc, char** argv)
{
    if(argc != 3) {
        fprintf(stderr, "Usage: %s <address> <provider id>\n", argv[0]);
        exit(-1);
    }
    margo_instance_id mid = margo_init("na+sm", MARGO_CLIENT_MODE, 0, 0);
    assert(mid);

    uint16_t provider_id = atoi(argv[2]);
    hg_addr_t server_addr = HG_ADDR_NULL;
    hg_return_t hret = margo_addr_lookup(mid, argv[1], &server_addr);
    assert(hret == HG_SUCCESS);

    yk_return_t ret;
    yk_admin_t admin = YOKAN_ADMIN_NULL;
    yk_database_id_t db_id;

    ret = yk_admin_init(mid, &admin);
    assert(ret == YOKAN_SUCCESS);

    ret = yk_open_database(admin, server_addr, provider_id, NULL, "map", "{}", &db_id);
    assert(ret == YOKAN_SUCCESS);

    char db_id_str[37];
    yk_database_id_to_string(db_id, db_id_str);
    printf("Database id is %s (take note of it!)\n", db_id_str);

    ret = yk_admin_finalize(admin);
    assert(ret == YOKAN_SUCCESS);

    margo_finalize(mid);

    return 0;
}

After the typical Margo initialization, we lookup the address of the server from the string address provided in argv[1]. We then create a yk_admin_t object, which we initialize with yk_admin_init.

By using yk_open_database, the admin sends an RPC to the provider requesting it to open a database of type map, which is an in-memory key/value store implemented using C++ std::map. Since this database is in memory, this open operation will actually create it.

The yk_open_database function’s last parameter is a pointer to the returned yk_database_id_t referencing the database. These identifiers are unique and can be serialized into a 37-byte null-terminated string using yk_database_id_to_string. We print this identifier before calling yk_admin_finalize to finalize the admin.

This code, once compiled, can be called as follows (changing the address as needed).

$ ./admin na+sm://8972-0 42
Database id is 2e7ca988-9681-43f7-9ef3-ff4c941fbefd (take note of it!)

Interacting with the database via the client interface

Now let the fun start. We can use the client library to create a client object, create a database handle, and start interacting with our database.

client.c (show/hide)

#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <margo.h>
#include <yokan/client.h>
#include <yokan/database.h>

int main(int argc, char** argv)
{
    if(argc != 4) {
        fprintf(stderr, "Usage: %s <address> <provider id> <database id>\n", argv[0]);
        exit(-1);
    }
    margo_instance_id mid = margo_init("na+sm", MARGO_CLIENT_MODE, 0, 0);
    assert(mid);

    uint16_t provider_id = atoi(argv[2]);
    hg_addr_t server_addr = HG_ADDR_NULL;
    hg_return_t hret = margo_addr_lookup(mid, argv[1], &server_addr);
    assert(hret == HG_SUCCESS);

    yk_return_t ret;
    yk_client_t client = YOKAN_CLIENT_NULL;
    yk_database_id_t db_id;

    ret = yk_client_init(mid, &client);
    assert(ret == YOKAN_SUCCESS);

    yk_database_id_from_string(argv[3], &db_id);

    yk_database_handle_t db_handle = YOKAN_DATABASE_HANDLE_NULL;
    ret = yk_database_handle_create(
        client, server_addr, provider_id, db_id, &db_handle);
    assert(ret == YOKAN_SUCCESS);

    const char* key = "matthieu";
    const char* value_in = "dorier";

    ret = yk_put(db_handle, YOKAN_MODE_DEFAULT,
                 key, strlen(key), value_in, strlen(value_in));
    assert(ret == YOKAN_SUCCESS);

    char value_out[128];
    size_t value_out_size = 128;
    ret = yk_get(db_handle, YOKAN_MODE_DEFAULT,
                 key, strlen(key), value_out, &value_out_size);
    assert(ret == YOKAN_SUCCESS);

    assert(strcmp(value_in, value_out) == 0);

    ret = yk_database_handle_release(db_handle);
    assert(ret == YOKAN_SUCCESS);

    ret = yk_client_finalize(client);
    assert(ret == YOKAN_SUCCESS);

    margo_finalize(mid);

    return 0;
}

The client is created using yk_client_init. We then convert the string database id into a yk_database_id_t using yk_database_id_from_string, and create a yk_database_handle_t by passing it to yk_database_handle_create. This handle is the object that will let us interact with the database.

As an example of using the client API, we show the use of yk_put and yk_get to respectively put and get a key/value pair from the database. These functions will be detailed more in the next tutorial.

yk_database_handle_release should be called to destroy the database handle. yk_client_finalize is then used to finalize the client.

This program can be called as follows (changing the address and the database id as needed).

$ ./client na+sm://8972-0 42 2e7ca988-9681-43f7-9ef3-ff4c941fbefd