Using Argobots pools with Margo RPCs

In the previous tutorial, we saw that the alpha_provider_register function is taking an ABT_pool argument that is passed down to MARGO_REGISTER_PROVIDER.

Argobots pools are a good way to assign resources (typically cores) to particular providers. In the following example, we rewrite the server code in such a way that the Alpha provider gets its own execution stream.

server.c (show/hide)

#include <assert.h>
#include <stdio.h>
#include <margo.h>
#include <alpha-server.h>

static void finalize_xstream_cb(void* data);

int main(int argc, char** argv)
{
    margo_instance_id mid = margo_init("tcp", MARGO_SERVER_MODE, 0, 0);
    assert(mid);
    margo_set_log_level(mid, MARGO_LOG_INFO);

    hg_addr_t my_address;
    margo_addr_self(mid, &my_address);
    char addr_str[128];
    size_t addr_str_size = 128;
    margo_addr_to_string(mid, addr_str, &addr_str_size, my_address);
    margo_addr_free(mid,my_address);
    margo_info(mid, "Server running at address %s, with provider id 42", addr_str);

    ABT_pool pool;
    ABT_pool_create_basic(
            ABT_POOL_FIFO,
            ABT_POOL_ACCESS_SPSC,
            ABT_TRUE,
            &pool);

    ABT_xstream xstream;
    ABT_xstream_create_basic(
            ABT_SCHED_DEFAULT,
            1,
            &pool,
            ABT_SCHED_CONFIG_NULL,
            &xstream);

    alpha_provider_register(mid, 42, pool, ALPHA_PROVIDER_IGNORE);

    margo_push_finalize_callback(mid, finalize_xstream_cb, (void*)xstream);

    margo_wait_for_finalize(mid);

    return 0;
}

static void finalize_xstream_cb(void* data) {
    ABT_xstream xstream = (ABT_xstream)data;
    ABT_xstream_join(xstream);
    ABT_xstream_free(&xstream);
}

After initializing Margo (which initializes Argobots), we create an Argobots pool and an execution stream that will execute work (ULTs and tasklets) from this pool. We use ABT_POOL_ACCESS_MPSC as access type to indicate that there will be multiple producers of work units (in particular, the ES running the Mercury progress loop) and a single consumer of work units (the ES we are about to create).

ABT_xstream_create_basic is then used to create the ES. Because Margo is initializing and finalizing Argobots, we need a way to destroy this ES before Margo finalizes Argobots. Hence with use margo_push_finalize_callback to add a callback that will be called upon finalizing Margo. This callback joins the ES and destroys it.

We pass the newly created pool to the alpha_provider_register function, which will make the Alpha provider use this pool to execute its RPC handlers.

For more elaborate assignments of resources to providers, please see the Argobots section of this documentation.