Child instances with parent_mid

In some situations, you may need multiple Margo instances within the same process – for example to use multiple protocols. Normally, each Margo instance creates its own Argobots pools and execution streams (ES), which can be wasteful when the instances could share these resources.

The parent_mid field in struct margo_init_info solves this problem. By setting it to an existing Margo instance, the new (child) instance will reuse the parent’s Argobots environment – pools, execution streams, and schedulers – instead of creating its own. Each instance still gets its own Mercury class and context, so they can independently send and receive RPCs.

Basic usage

The simplest way to create a child instance is to set the parent_mid field and leave everything else at its defaults:

simple.c (show/hide)

#include <assert.h>
#include <stdio.h>
#include <margo.h>

int main(int argc, char** argv)
{
    /* Create a parent Margo instance with default configuration. */
    margo_instance_id parent_mid
        = margo_init("na+sm", MARGO_SERVER_MODE, 0, 0);
    assert(parent_mid);

    /* Create a child that shares the parent's Argobots environment.
     * The child gets its own Mercury class and context, so it can
     * independently send and receive RPCs. By default (no JSON config),
     * the child uses the same progress and RPC pools as the parent. */
    struct margo_init_info child_args = MARGO_INIT_INFO_INITIALIZER;
    child_args.parent_mid = parent_mid;

    margo_instance_id child_mid
        = margo_init_ext("na+sm", MARGO_CLIENT_MODE, &child_args);
    assert(child_mid);

    /* Both instances now share the same Argobots pools and ES. */

    /* Always finalize the child before the parent. */
    margo_finalize(child_mid);
    margo_finalize(parent_mid);

    return 0;
}

In this example, the child inherits the parent’s __primary__ pool for both its progress loop and its RPC handlers. Since no Argobots resources are duplicated, the child is very lightweight.

Selecting pools by name

When the parent has multiple pools, the child can select which ones to use for its progress loop and RPC handlers via the progress_pool and rpc_pool fields in the JSON configuration:

parent.c (show/hide)

#include <assert.h>
#include <stdio.h>
#include <margo.h>

int main(int argc, char** argv)
{
    /* Create a parent Margo instance (server) with custom pools. */
    const char* parent_config
        = "{"
          "\"argobots\": {"
          "\"pools\": ["
          "{\"name\":\"__primary__\",\"access\":\"mpmc\",\"kind\":\"fifo_wait\"},"
          "{\"name\":\"my_pool\",\"access\":\"mpmc\",\"kind\":\"fifo_wait\"}"
          "],"
          "\"xstreams\": ["
          "{\"name\":\"__primary__\","
          "\"scheduler\":{\"pools\":[\"__primary__\"],"
          "\"type\":\"basic_wait\"}"
          "},"
          "{\"name\":\"es1\","
          "\"scheduler\":{\"pools\":[\"my_pool\"],"
          "\"type\":\"basic_wait\"}"
          "}"
          "]"
          "}"
          "}";

    struct margo_init_info parent_args = MARGO_INIT_INFO_INITIALIZER;
    parent_args.json_config = parent_config;

    margo_instance_id parent_mid
        = margo_init_ext("na+sm", MARGO_SERVER_MODE, &parent_args);
    assert(parent_mid);
    margo_set_log_level(parent_mid, MARGO_LOG_INFO);

    hg_addr_t my_address;
    margo_addr_self(parent_mid, &my_address);
    char   addr_str[128];
    size_t addr_str_size = 128;
    margo_addr_to_string(parent_mid, addr_str, &addr_str_size, my_address);
    margo_addr_free(parent_mid, my_address);

    margo_info(parent_mid, "Parent running at address %s", addr_str);

    /* Create a child instance that reuses the parent's Argobots
     * environment. The child gets its own Mercury context but
     * references one of the parent's pools by name. */
    const char* child_config
        = "{"
          "\"progress_pool\":\"my_pool\","
          "\"rpc_pool\":\"my_pool\""
          "}";

    struct margo_init_info child_args = MARGO_INIT_INFO_INITIALIZER;
    child_args.parent_mid  = parent_mid;
    child_args.json_config = child_config;

    margo_instance_id child_mid
        = margo_init_ext("na+sm", MARGO_CLIENT_MODE, &child_args);
    assert(child_mid);

    margo_info(parent_mid, "Child instance created, sharing parent pools");

    /* Verify both instances share the same pools. */
    ABT_pool parent_pool, child_pool;
    margo_get_handler_pool(parent_mid, &parent_pool);
    margo_get_handler_pool(child_mid, &child_pool);
    /* Note: they may differ because they reference different pools
     * (parent uses __primary__, child uses my_pool). */

    /* The child can now be used for client-side RPCs while
     * the parent handles server-side RPCs, both sharing
     * the same Argobots execution streams and pools. */

    /* Always finalize the child before the parent. */
    margo_finalize(child_mid);
    margo_finalize(parent_mid);

    return 0;
}

Here the parent defines two pools (__primary__ and my_pool) with their own execution streams. The child references my_pool by name in its JSON configuration, so its progress loop and RPC handlers run on the ES associated with that pool.

Pool references in the child’s configuration can be specified either as a string (pool name) or as an integer (pool index in the parent’s pool array).

Important

The pool referenced by the child must have at least one execution stream associated with it in the parent’s configuration, otherwise ULTs pushed into that pool will never be executed.

Configuration restrictions

A child instance borrows its Argobots environment from the parent. As a consequence, the following fields are not allowed in the child’s JSON configuration and will cause initialization to fail:

  • "argobots" – the child cannot define its own pools or xstreams.

  • "use_progress_thread" – adding a progress thread would require creating a new ES, which is the parent’s responsibility.

  • "rpc_thread_count" – similarly, creating RPC threads would require new pools and xstreams.

The following fields are still accepted in the child’s configuration:

  • "mercury" – Mercury settings for the child’s own HG class.

  • "progress_pool" – pool name or index in the parent’s pool array.

  • "rpc_pool" – pool name or index in the parent’s pool array.

  • "progress_timeout_ub_msec", "progress_spindown_msec", "handle_cache_size" – per-instance tuning knobs.

Lifetime management

The child holds an internal reference to the parent instance. This means you should always follow these rules:

  1. Finalize the child before the parent. Calling margo_finalize on the parent while a child is still using its pools may lead to undefined behavior.

  2. The parent’s Argobots resources stay alive as long as any child (or the parent itself) is still running. The parent’s reference count is incremented when the child is created and decremented when the child is finalized or released.

  3. The reference-counting mechanism described in margo_13_lifetime applies to both parent and child instances.

Note

A child instance may also serve as the parent for another child, creating a chain. All instances in the chain share the root parent’s Argobots environment.

Warning about progress loop location

Some multi-instance setups can lead to undesirable behaviors. For instance if the two instances have their progress loop in the same pool, both progress loops will end up busy-spinning because they cannot simply block their ES while waiting for network activities.