Multiple Margo instances

By default, a Bedrock daemon initializes a single Margo instance (and therefore a single Mercury transport). In some cases, however, you may want a single process to listen on multiple networks or protocols simultaneously, for instance shared memory for intra-node communication and a fabric transport for inter-node communication, or a high-performance network inside a cluster and TCP for Bedrock to be reachable from outside. Bedrock supports this by allowing the margo field of the JSON configuration to be an array of Margo configurations instead of a single object.

Configuring multiple engines

When the margo field is an array, each element follows the same format as a regular Margo configuration object. Bedrock will create one Margo instance (engine) per element. The engines are numbered starting from 0 in the order they appear in the array. Engine 0 is the parent of all subsequent engines.

{
    "margo": [
        {
            "mercury": {
                "address": "na+sm"
            },
            "argobots": {
                "pools": [
                    { "name": "pool_sm", "kind": "fifo_wait", "access": "mpmc" }
                ],
                "xstreams": [
                    {
                        "name": "xstream_sm",
                        "scheduler": {
                            "type": "basic_wait",
                            "pools": ["pool_sm"]
                        }
                    }
                ]
            },
            "progress_pool": "pool_sm",
            "rpc_pool": "pool_sm"
        },
        {
            "mercury": {
                "address": "ofi+tcp"
            },
            "progress_pool": "pool_sm",
            "rpc_pool": "pool_sm"
        }
    ],
    "libraries": [
        "libmy-bedrock-module.so"
    ],
    "providers": [
        {
            "name": "my_provider_sm",
            "type": "my_module",
            "provider_id": 1,
            "engine": 0,
            "config": {},
            "dependencies": {}
        },
        {
            "name": "my_provider_tcp",
            "type": "my_module",
            "provider_id": 2,
            "engine": 1,
            "config": {},
            "dependencies": {}
        }
    ]
}

In the example above, engine 0 listens on shared memory (na+sm) and engine 1 listens on TCP (ofi+tcp). Engine 0 has its own pools, execution streams, and progress/RPC pool configuration. Engine 1 (a child instance) only specifies a Mercury address and pool/progress configuration, referencing the engine 0’s Argobots environment.

Important

Child instances (engines with index > 0) share the Argobots environment of the parent (engine 0). Their configuration must not contain the following fields:

argobots (pools and execution streams are inherited from the parent)
rpc_thread_count
use_progress_thread

Only engine 0 should define Argobots pools and execution streams. Child engines reference pools from the parent when configuring progress_pool and rpc_pool.

Note

When using multiple engines, the protocol for each engine is inferred from the mercury.address field in that engine’s configuration. The address command-line argument to the bedrock program becomes optional; if provided, it serves as a fallback for any engine whose configuration does not contain a mercury.address field.

Note

Bedrock automatically registers its management RPCs (configuration queries, pool/xstream management, etc.) on all engines, so a client can connect to any engine to query or reconfigure the service.

Assigning providers to engines

Each provider can be assigned to a specific engine using the engine field in its configuration. This integer field (defaulting to 0) specifies the index of the engine on which the provider will be registered.

{
    "name": "my_provider",
    "type": "my_module",
    "provider_id": 1,
    "engine": 1,
    "config": {},
    "dependencies": {
        "my_pool": "pool_tcp"
    }
}

In this example, the provider is registered on engine 1 (the TCP engine from the earlier configuration). All RPCs for this provider will be registered against that engine (i.e., they can be received only by sending the RPC to that engine’s address).

Important

When engine is omitted, the provider defaults to engine 0. The engine index must be less than the total number of engines defined in the margo array, otherwise Bedrock will report an error.

Depending on a Margo instance

Components can declare a dependency on a Margo instance itself. This is useful for components that need to create their own RPCs or endpoints on a specific network. A Margo dependency provides the component with a margo_instance_id that it can use directly.

To depend on a Margo instance, use the dependency type "margo" and reference the engine by its index:

{
    "name": "my_provider",
    "type": "my_module",
    "provider_id": 1,
    "engine": 0,
    "config": {},
    "dependencies": {
        "secondary_margo": 1
    }
}

In this example, the provider is registered on engine 0, but also receives a handle to engine 1’s Margo instance through its secondary_margo dependency. The Bedropck component can retrieve it using:

#include <margo.h>

static std::shared_ptr<bedrock::AbstractComponent>
Register(const bedrock::ComponentArgs& args) {
    // Retrieve the margo_instance_id from the dependency
    auto secondary_mid = args.dependencies["secondary_margo"][0]
                             ->getHandle<margo_instance_id>();
    // Use secondary_mid to register RPCs, create endpoints, etc.
    ...
}

When specifying a Margo dependency in the dependencies section, the value can be:

An integer: the engine index (e.g. 1);
A string containing an integer: the engine index as a string (e.g. "1" ).

Warning

The engine index must refer to a valid engine. Bedrock will raise an error if the index is out of range.

Output configuration

When querying the configuration of a Bedrock daemon running with multiple engines, the margo field in the output will be a JSON array (one element per engine). When running with a single engine, the margo field remains a JSON object for backward compatibility.

Providers in the output configuration will include an engine field indicating which engine they are registered on.