API Documentation

margo.h

(C) The University of Chicago

See COPYRIGHT in top-level directory.

Defines

MARGO_INSTANCE_NULL: Uninitialized margo_instance_id.

MARGO_REQUEST_NULL: Uninitialized margo_request.

MARGO_CLIENT_MODE: Client mode for margo_init.

MARGO_SERVER_MODE: Server mode for margo_init.

MARGO_DEFAULT_PROVIDER_ID

Default provider id. This is the provider id used by default in MARGO_REGISTER, margo_forward, and its variants that do not ask for a provider id.

Important: it is not recommended for users to use this provider id for actual providers. If you work with actual providers, use provider ids starting from 0. This default provider id is meant for clients and for RPC not associated with providers.

MARGO_MAX_PROVIDER_ID: Maximum allowed value for a provider id.

MARGO_INIT_INFO_INITIALIZER: Initializer for margo_init_info.

MARGO_REGISTER(__mid, __func_name, __in_t, __out_t, __handler)

Macro that registers a function as an RPC.

Parameters

__mid – Margo instance id.
__func_name – String name of the RPC.
__in_t – type of input.
__out_t – type of the output.
__handler – Name of the function to register.

Returns

The RPC id (hg_id_t).

MARGO_REGISTER_PROVIDER(__mid, __func_name, __in_t, __out_t, __handler, __provider_id, __pool)

Macro that registers a function as an RPC with a specific provider ID and Argobots pool.

Parameters

__mid – Margo instance id.
__func_name – String name of the RPC.
__in_t – type of input.
__out_t – type of the output.
__handler – Name of the function to register.
__provider_id – Provider id.
__pool – Argobots pool.

Returns

The RPC id (hg_id_t).

DEFINE_MARGO_RPC_HANDLER(__name)

Macro that defines a function to glue an RPC handler to a ULT handler.

Parameters

__name – [in] name of handler function.

DECLARE_MARGO_RPC_HANDLER(__name)

Macro that declares the prototype for a function to glue an RPC handler to a ULT.

Parameters

__name – [in] name of handler function.

Typedefs

typedef struct margo_instance *margo_instance_id: Margo instance id. Entry point to the margo runtime.

typedef struct margo_data *margo_data_ptr

typedef struct margo_request_struct *margo_request: Request for non-blocking operations.

typedef void (*margo_finalize_callback_t)(void*): Type of callback called during finalization or pre-finalization of Margo.

Enums

enum margo_request_type

Type of margo_request.

Values:

enumerator MARGO_RESPONSE_REQUEST

enumerator MARGO_FORWARD_REQUEST

enumerator MARGO_BULK_REQUEST

enumerator MARGO_INVALID_REQUEST

Functions

margo_instance_id margo_init(const char *addr_str, int mode, int use_progress_thread, int rpc_thread_count)

Initializes margo library.

Note

Servers (processes expecting to service incoming RPC requests) must specify non-zero values for use_progress_thread and rpc_thread_count or call margo_wait_for_finalize() after margo_init() to relinguish control to Margo.

Parameters

addr_str – [in] Mercury host address with port number
mode – [in] Mode to run Margo in (MARGO_CLIENT_MODE or MARGO_SERVER_MODE).
use_progress_thread – [in] Boolean flag to use a dedicated thread for running Mercury’s progress loop. If false, it will run in the caller’s thread context.
rpc_thread_count – [in] Number of threads to use for running RPC calls. A value of 0 directs Margo to execute RPCs in the caller’s thread context. Clients (i.e processes that will not service incoming RPCs) should use a value of 0. A value of -1 directs Margo to use the same execution context as that used for Mercury progress.

Returns

margo instance id on success, MARGO_INSTANCE_NULL upon error.

margo_instance_id margo_init_ext(const char *address, int mode, const struct margo_init_info *args)

Initializes a margo instance using a margo_init_info struct to provide extra arguments.

Example JSON configuration

{
  "mercury" : {
      "address" : "na+sm://",
      "listening" : false,
      "auto_sm" : true,
      "version" : "2.0.0",
      "stats" : false,
      "na_no_block" : false,
      "na_no_retry" : false,
      "na_request_mem_device" : false,
      "max_contexts" : 1,
      "ip_subnet" : "",
      "auth_key" : ""
  },
  "argobots" : {
      "abt_mem_max_num_stacks" : 8,
      "abt_thread_stacksize" : 2097152,
      "version" : "1.0.0",
      "pools" : [
          {
              "name" : "my_progress_pool",
              "kind" : "fifo_wait",
              "access" : "mpmc"
          },
          {
              "name" : "my_rpc_pool",
              "kind" : "fifo_wait",
              "access" : "mpmc"
          }
      ],
      "xstreams" : [
          {
              "name" : "my_progress_xstream",
              "cpubind" : 0,
              "affinity" : [ 0, 1 ],
              "scheduler" : {
                  "type" : "basic_wait",
                  "pools" : [ "my_progress_pool" ]
              }
          },
          {
              "name" : "my_rpc_xstream_0",
              "cpubind" : 2,
              "affinity" : [ 2, 3, 4, 5 ],
              "scheduler" : {
                  "type" : "basic_wait",
                  "pools" : [ "my_rpc_pool" ]
              }
          },
          {
              "name" : "my_rpc_xstream_1",
              "cpubind" : 6,
              "affinity" : [ 6, 7 ],
              "scheduler" : {
                  "type" : "basic_wait",
                  "pools" : [ "my_rpc_pool" ]
              }
          }
      ]
  },
  "handle_cache_size" : 32,
  "profile_sparkline_timeslice_msec" : 1000,
  "progress_timeout_ub_msec" : 100,
  "enable_profiling" : false,
  "enable_diagnostics" : false
}

The margo json configuration also supports the following convenience parameters at input time (the resulting runtime json will contain a fully resolved pool configuration):

use_progress_thread: bool (default false)
rpc_thread_count: integer (default 0)

Note

If you are configuring Argobots pools yourself before passing them into this function, please consider setting ABT_MEM_MAX_NUM_STACKS to a low value (like 8) either in your environment or programmatically with putenv() in your code before creating the pools to prevent excess memory consumption under load from producer/consumer patterns across execution streams that fail to utilize per-execution stream stack caches. See https://xgitlab.cels.anl.gov/sds/margo/issues/40 for details. The margo_init() function does this automatically.

Note

Note that supported kinds of pools are fifo_wait (default) fifo (for use with basic scheduler; will busy spin when idle) and prio_wait (custom pool implementation for Margo that favors existing ULTs over newly created ULTs when possible)

Parameters

address – Address or protocol.
mode – MARGO_CLIENT_MODE or MARGO_SERVER_MODE.
args – Extra arguments.

Returns

a margo_instance_id or MARGO_INSTANCE_NULL in case of failure.

int margo_set_environment(const char *optional_json_config)

Configures the runtime environment dependencies for use by Margo without initializing Margo.

The primary purpose of this function is to set preferred environment variables for Argobots (e.g., ULT stack size) if Argobots will be initialized before calling margo_init() or margo_init_ext().

Parameters: optional_json_config – [in] The json-formatted configuration parameters to be used by Margo when it is initialized later (if known). If not specified, then margo_set_environment() will use default values.
Returns: returns 0 on success, negative value on failure

margo_instance_id margo_init_opt (const char *addr_str, int mode, const struct hg_init_info *hg_init_info, int use_progress_thread, int rpc_thread_count) DEPRECATED("use margo_init_ext instead")

Initializes margo library with custom Mercury options.

Deprecated:

Note

Servers (processes expecting to service incoming RPC requests) must specify non-zero values for use_progress_thread and rpc_thread_count or call margo_wait_for_finalize() after margo_init() to relinguish control to Margo.

Parameters

addr_str – [in] Mercury host address with port number.
mode – [in] Mode to run Margo in (MARGO_CLIENT_MODE or MARGO_SERVER_MODE).
hg_init_info – [in] (Optional) Mercury init info, passed directly to Mercury.
use_progress_thread – [in] Boolean flag to use a dedicated thread for running Mercury’s progress loop. If false, it will run in the caller’s thread context.
rpc_thread_count – [in] Number of threads to use for running RPC calls. A value of 0 directs Margo to execute RPCs in the caller’s thread context. Clients (i.e processes that will not service incoming RPCs) should use a value of 0. A value of -1 directs Margo to use the same execution context as that used for Mercury progress.

Returns

margo instance id on success, MARGO_INSTANCE_NULL upon error

margo_instance_id margo_init_pool (ABT_pool progress_pool, ABT_pool handler_pool, hg_context_t *hg_context) DEPRECATED("use margo_init_ext instead")

Initializes the margo instance from the given argobots and Mercury instances.

Deprecated:

Note

If you are configuring Argobots pools yourself before passing them into this function, please consider setting ABT_MEM_MAX_NUM_STACKS to a low value (like 8) either in your environment or programmatically with putenv() in your code before creating the pools to prevent excess memory consumption under load from producer/consumer patterns across execution streams that fail to utilize per-execution stream stack caches. You may also look at margo_set_environment for this purpose. The margo_init() function does this automatically.

Parameters

progress_pool – [in] Argobots pool to drive communication.
handler_pool – [in] Argobots pool to service RPC handlers.
hg_context – [in] Mercury context.

Returns

margo instance id on success, MARGO_INSTANCE_NULL upon error.

void margo_finalize(margo_instance_id mid)

Requests the margo instance to shut down and free its resources.

Note

This function does not guarantee that the margo instance will be finalized immediately. When called in a server, where RPCs may be executing, for example, this call will only margo the margo instance as finalizing, preventing further RPCs from being scheduled, however the margo instance will effectively be destroyed only when all the RPCs have completed.

Note

This function is safe to use within an RPC handler.

Parameters: mid – [in] Margo instance.

void margo_wait_for_finalize(margo_instance_id mid)

Suspends the caller until some other entity (e.g. an RPC, thread, or signal handler) invokes margo_finalize() or margo_finalize_and_wait().

Note

This function informs Margo that the calling thread no longer needs to be scheduled for execution if it is sharing an Argobots pool with the progress engine.

Parameters: mid – [in] Margo instance.

void margo_finalize_and_wait(margo_instance_id mid)

Requests the margo instance to shut down and free its resources, and wait until it actually does.

Contrary to margo_finalize(), this function guarantees that the margo instance has been freed upon returning.

Note

This function should not be called from within an RPC handler.

Parameters: mid – [in] Margo instance.

hg_return_t margo_instance_ref_incr(margo_instance_id mid)

Increments the internal reference counter of the instance. By default a created margo_instance_id has a refcount of 0 and will be automatically freed when margo_finalize is called. Increasing the refcount will prevent this free operation. The instance will still be finalized by margo_finalize but its memory will remain valid until margo_instance_release decreases the refcount back to 0.

Parameters: mid – Margo instance ID
Returns: HG_SUCCESS or other Mercury error code.

hg_return_t margo_instance_release(margo_instance_id mid)

Decrements the internal refcount of the instance. If the refcount reaches 0, this will (1) call margo_finalize if the instance hasn’t been finalized yet, and (2) free the instance’s memory.

Parameters: mid – [in] Margo instance ID
Returns: HG_SUCCESS or other Mercury error code.

hg_return_t margo_instance_is_finalized(margo_instance_id mid, bool *flag)

Check whether the instance has been finalized. Note that this function is meant to be used by codes that rely on margo_instance_ref_incr and margo_instance_release to keep track of ownership of the instance.

Parameters: mid – [in] Margo instance ID
Returns: HG_SUCCESS or other Mercury error code.

hg_return_t margo_instance_ref_count(margo_instance_id mid, unsigned *refcount)

Get the internal refcount of the instance.

Parameters

mid – [in] Margo instance ID
refcount – [out] Refcount

Returns

HG_SUCCESS or other Mercury error code.

hg_bool_t margo_is_listening(margo_instance_id mid)

Checks whether a Margo instance we initialized is a server (i.e., listening for incoming RPC requests).

Parameters: mid – [in] Margo instance
Returns: true if listening or false if not, or not a valid margo instance.

void margo_push_prefinalize_callback(margo_instance_id mid, margo_finalize_callback_t cb, void *uargs)

Installs a callback to be called before the margo instance is finalized, and before the Mercury progress loop is terminated.

Callbacks installed will be called in reverse ordered than they have been pushed, and with the user-provider pointer as argument.

Note

Callbacks may not be called within margo_finalize. They are called when the margo instance is cleaned up, which may happen in margo_wait_for_finalize.

Note

Callbacks installed using this function are allowed to call RPCs or margo_thread_sleep. The process may itself receive and respond to RPCs in the mean time.

Parameters

mid – The margo instance.
cb – Callback to install.
uargs – User-provider argument to pass to the callback when called.

int margo_pop_prefinalize_callback(margo_instance_id mid)

Removes the last pre-finalize callback that was pushed into the margo instance without calling it. If a callback was removed, this function returns 1, otherwise it returns 0.

Parameters: mid – Margo instance.
Returns: Whether a callback was removed.

int margo_top_prefinalize_callback(margo_instance_id mid, margo_finalize_callback_t *cb, void **uargs)

Get the last pre-finalize callback that was pushed into the margo instance as well as its argument. If a callback is found, this function returns 1, otherwise it returns 0.

Parameters

mid – Margo instance.
cb – Returned callback.
uargs – Uargs.

Returns

1 is a callback was found, 0 otherwise.

void margo_provider_push_prefinalize_callback(margo_instance_id mid, const void *owner, margo_finalize_callback_t cb, void *uargs)

Installs a callback to be called before the margo instance is finalized, and before the Mercury progress loop is terminated. The owner pointer allows to identify callbacks installed by particular providers.

Note

One can install multiple callbacks with the same owner. If popped, they will be popped in reverse order of installation. If they are not popped, they will be called in reverse order of installation by the margo cleanup procedure.

Note

Callbacks installed using this function may call RPCs or margo_thread_sleep, but do not guarantee that the process isn’t itself going to receive RPCs in the mean time.

Parameters

mid – The margo instance.
owner – Owner of the callback (to be used when popping callbacks).
cb – Callback to install.
uargs – User-provider argument to pass to the callback when called.

int margo_provider_pop_prefinalize_callback(margo_instance_id mid, const void *owner)

Removes the last prefinalize callback that was pushed into the margo instance by the specified owner. If a callback is removed, this function returns 1, otherwise it returns 0.

Parameters

mid – Margo instance.
owner – Owner of the callback.

Returns

1 if a callback was removed, 0 otherwise.

int margo_provider_top_prefinalize_callback(margo_instance_id mid, const void *owner, margo_finalize_callback_t *cb, void **uargs)

Get the last prefinalize callback that was pushed into the margo instance by the specified owner. If a callback is found, this function returns 1, otherwise it returns 0.

Parameters

mid – Margo instance.
owner – Owner of the callback.
cb – Returned callback.
uargs – Returned user arguments.

Returns

1 if a callback was found, 0 otherwise.

void margo_push_finalize_callback(margo_instance_id mid, margo_finalize_callback_t cb, void *uargs)

Installs a callback to be called before the margo instance is finalized. Callbacks installed will be called in reverse ordered than they have been pushed, and with the user-provider pointer as argument.

Attention: Callbacks cannot make RPC calls nor use margo_thread_sleep. If you need to be able to make RPC calls or use margo_thread_sleep, you should use margo_push_prefinalize_callback instead.

Note

Callbacks may not be called within margo_finalize. They are called when the margo instance is cleaned up, which may happen in margo_wait_for_finalize.

Parameters

mid – The margo instance.
cb – Callback to install.
uargs – User-provider argument to pass to the callback when called.

int margo_pop_finalize_callback(margo_instance_id mid)

Removes the last finalize callback that was pushed into the margo instance without calling it.

Parameters: mid – Margo instance.
Returns: 1 if a callback was removed, 0 otherwise.

int margo_top_finalize_callback(margo_instance_id mid, margo_finalize_callback_t *cb, void **uargs)

Returns the last finalize callback that was pushed into the margo instance, along with its argument.

Parameters

mid – Margo instance.
cb – Returned callback.
uargs – Returned user arguments.

Returns

1 if successful, 0 otherwise.

void margo_provider_push_finalize_callback(margo_instance_id mid, const void *owner, margo_finalize_callback_t cb, void *uargs)

Installs a callback to be called before the margo instance is finalized. The owner pointer allows to identify callbacks installed by particular providers.

Attention: Callbacks cannot make RPC calls nor use margo_thread_sleep. If you need to be able to make RPC calls or use margo_thread_sleep, you should use margo_provider_push_prefinalize_callback instead.

Note

One can install multiple callbacks with the same owner. If popped, they will be popped in reverse order of installation. If they are not popped, they will be called in reverse order of installation by the margo cleanup procedure.

Parameters

mid – The margo instance.
owner – Owner of the callback (to be used when popping callbacks).
cb – Callback to install.
uargs – User-provider argument to pass to the callback when called.

int margo_provider_pop_finalize_callback(margo_instance_id mid, const void *owner)

Removes the last finalize callback that was pushed into the margo instance by the specified owner.

Parameters

mid – Margo instance.
owner – Owner of the callback.

Returns

1 if a callback was removed, 0 otherwise.

int margo_provider_top_finalize_callback(margo_instance_id mid, const void *owner, margo_finalize_callback_t *cb, void **uargs)

Gets the last finalize callback that was pushed into the margo instance by the specified owner.

Parameters

mid – Margo instance.
owner – Owner of the callback.
cb – Returned callback.
uargs – Returned user agument.

Returns

1 if a callback was found, 0 otherwise.

void margo_enable_remote_shutdown(margo_instance_id mid)

Allows the passed Margo instance to be shut down remotely using margo_shutdown_remote_instance().

Parameters: mid – Margo instance.

int margo_shutdown_remote_instance(margo_instance_id mid, hg_addr_t remote_addr)

Trigger the shutdown of the Margo instance running at remote_addr.

Parameters

mid – Local Margo instance.
remote_addr – Address of the Margo instance to shut down.

Returns

0 on success, -1 on failure.

hg_id_t margo_provider_register_name(margo_instance_id mid, const char *func_name, hg_proc_cb_t in_proc_cb, hg_proc_cb_t out_proc_cb, hg_rpc_cb_t rpc_cb, uint16_t provider_id, ABT_pool pool)

Registers an RPC with margo that is associated with a provider instance.

Note

It is generally more convenient to use the MARGO_REGISTER_PROVIDER macro instead of this function, as the macro version takes types as arguments instead of serialization functions (hg_proc_cb_t) for the RPC’s input and output.

Parameters

mid – [in] Margo instance.
func_name – [in] unique function name for RPC.
in_proc_cb – [in] pointer to input proc callback.
out_proc_cb – [in] pointer to output proc callback.
rpc_cb – [in] RPC callback.
provider_id – [in] provider identifier.
pool – [in] Argobots pool the handler will execute in.

Returns

A unique ID associated to the registered function.

static inline hg_id_t margo_register_name(margo_instance_id mid, const char *func_name, hg_proc_cb_t in_proc_cb, hg_proc_cb_t out_proc_cb, hg_rpc_cb_t rpc_cb)

Registers an RPC with margo.

Note

It is generally more convenient to use the MARGO_REGISTER macro instead of this function, as the macro version takes types as arguments instead of serialization functions (hg_proc_cb_t) for the RPC’s input and output.

Parameters

mid – [in] Margo instance.
func_name – [in] unique function name for RPC.
in_proc_cb – [in] pointer to input proc callback.
out_proc_cb – [in] pointer to output proc callback.
rpc_cb – [in] RPC callback.

Returns

A unique ID associated to the registered function.

hg_return_t margo_deregister(margo_instance_id mid, hg_id_t rpc_id)

Deregisters an RPC with margo.

Parameters

mid – [in] Margo instance.
rpc_id – [in] Id of the RPC to deregister.

Returns

HG_SUCCESS or corresponding error code.

hg_return_t margo_registered_name(margo_instance_id mid, const char *func_name, hg_id_t *id, hg_bool_t *flag)

Indicate whether margo_register_name() (or MARGO_REGISTER) has been called for the RPC specified by func_name.

Parameters

mid – [in] Margo instance.
func_name – [in] function name.
id – [out] registered RPC ID.
flag – [out] pointer to boolean.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_provider_registered_name(margo_instance_id mid, const char *func_name, uint16_t provider_id, hg_id_t *id, hg_bool_t *flag)

Indicate whether the given RPC name has been registered with the given provider id.

Parameters

mid – [in] Margo instance.
func_name – [in] function name.
provider_id – [in] provider id.
id – [out] registered RPC ID.
flag – [out] pointer to boolean.

Returns

HG_SUCCESS or corresponding HG error code

hg_return_t margo_register_data(margo_instance_id mid, hg_id_t id, void *data, void (*free_callback)(void*))

Associate user data to a registered RPC id. When HG_Finalize() is called free_callback (if defined) is called to free the registered data.

Parameters

mid – [in] Margo instance.
id – [in] registered function ID.
data – [in] pointer to data.
free_callback – [in] pointer to free function.

Returns

HG_SUCCESS or corresponding HG error code.

void *margo_registered_data(margo_instance_id mid, hg_id_t id)

Indicate whether margo_register_data() has been called and return associated data.

Parameters

mid – [in] Margo instance.
id – [in] Registered RPC ID.

Returns

Pointer to data or NULL.

hg_return_t margo_provider_register_identity(margo_instance_id mid, uint16_t provider_id, const char *identity)

Give a name for the provider with the specified provider ID. This will effectively register an internal RPC that can be invoked by using margo_provider_get_identity.

Parameters

mid – Margo instance.
provider_id – Provider ID.
identity – Identity (name) of the provider.

Returns

HG_SUCCESS or corresponding error code.

hg_return_t margo_provider_deregister_identity(margo_instance_id mid, uint16_t provider_id)

Deregisters the identity associated with the provider ID.

Parameters

mid – Margo instance.
provider_id – Provider ID.

Returns

HG_SUCCESS or corresponding error code.

const char *margo_provider_registered_identity(margo_instance_id mid, uint16_t provider_id)

Get the identity registered with a provider ID. NULL will be returned if no such ID exists.

Parameters

mid – Margo instance.
provider_id – Provider ID.

Returns

Identity.

hg_return_t margo_provider_get_identity(margo_instance_id mid, hg_addr_t address, uint16_t provider_id, char *buffer, size_t *bufsize)

Get the identity of a provider.

Note: if the buffer if too small, the function will return HG_NOMEM and the bufsize variable will be set to the required size.

Note: if there is no provider registered with this ID or if it has not specified an identity, this function will return HG_NOENTRY.

Parameters

mid – Margo instance.
address – Address.
provider_id – Provider ID.
buffer – Buffer in which to write the identity.
bufsize – Buffer capacity (in) / size of the identity (out).

Returns

HG_SUCCESS or corresponding error code.

const char *margo_rpc_get_name(margo_instance_id mid, hg_id_t id)

Get the name with which an RPC id was registered (NULL if the RPC id is invalid or wasn’t registered with a name).

Parameters

mid – [in] Margo instance.
id – [in] RPC id.

Returns

Registered name of the RPC.

hg_return_t margo_rpc_get_pool(margo_instance_id mid, hg_id_t id, ABT_pool *pool)

Get the pool in which RPCs of the provided id will execute their ULT.

Parameters

mid – [in] Margo instance.
id – [in] RPC id.
pool – [out] Pool.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_rpc_set_pool(margo_instance_id mid, hg_id_t id, ABT_pool pool)

Set the pool in which RPCs of the provided id will execute their ULTs. If ABT_POOL_NULL is passed, this function will set the pool to the margo instance’s default handler pool.

Parameters

mid – [in] Margo instance.
id – [in] RPC id.
pool – [in] Pool.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_registered_disable_response(margo_instance_id mid, hg_id_t id, int disable_flag)

Disable response for a given RPC ID.

Parameters

mid – [in] Margo instance.
id – [in] Registered RPC ID.
disable_flag – [in] flag to disable (1) or re-enable (0) responses.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_registered_disabled_response(margo_instance_id mid, hg_id_t id, int *disabled_flag)

Checks if response is disabled for a given RPC ID.

Parameters

mid – [in] Margo instance.
id – [in] Registered RPC ID
[ou] – disabled_flag flag indicating whether response is disabled (1) or not (0)

Returns

HG_SUCCESS or corresponding HG error code

hg_return_t margo_addr_lookup(margo_instance_id mid, const char *name, hg_addr_t *addr)

Lookup an addr from a peer address/name.

Parameters

name – [in] Lookup name.
addr – [out] Return address.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_addr_free(margo_instance_id mid, hg_addr_t addr)

Free the given Mercury addr.

Parameters

mid – [in] Margo instance.
addr – [in] Mercury address.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_addr_self(margo_instance_id mid, hg_addr_t *addr)

Access self address. Address must be freed with margo_addr_free.

Parameters

mid – [in] Margo instance.
addr – [in] pointer to abstract Mercury address.

Returns

HG_SUCCESS or corresponding HG error code.

hg_bool_t margo_addr_cmp(margo_instance_id mid, hg_addr_t addr1, hg_addr_t addr2)

Compare two addresses.

Parameters

mid – Margo instance.
addr1 – first address.
addr2 – second address.

Returns

HG_TRUE if addresses are determined to be equal, HG_FALSE otherwise.

hg_return_t margo_addr_set_remove(margo_instance_id mid, hg_addr_t addr)

Hint that the address is no longer valid. This may happen if the peer is no longer responding. This can be used to force removal of the peer address from the list of the peers, before freeing it and reclaim resources.

Parameters

mid – Margo instance.
addr – address.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_addr_dup(margo_instance_id mid, hg_addr_t addr, hg_addr_t *new_addr)

Duplicate an existing Mercury address.

Parameters

mid – [in] Margo instance.
addr – [in] Abstract Mercury address to duplicate.
new_addr – [out] Resulting Mercury address.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_addr_to_string(margo_instance_id mid, char *buf, hg_size_t *buf_size, hg_addr_t addr)

Convert a Mercury addr to a string (returned string includes the terminating null byte ‘\0’). If buf is NULL, the address is not converted and only the required size of the buffer is returned. If the input value passed through buf_size is too small, HG_SIZE_ERROR is returned and the buf_size output is set to the minimum size required.

Parameters

mid – [in] Margo instance.
[in/out] – buf Pointer to destination buffer.
[in/out] – buf_size Pointer to buffer size.
addr – [in] Abstract Mercury address.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_create(margo_instance_id mid, hg_addr_t addr, hg_id_t id, hg_handle_t *handle)

Initiate a new Mercury RPC using the specified RPC ID and the local/remote target defined by addr. The handle created can be used to query input and output, as well as issuing the RPC by calling HG_Forward(). After completion the handle must be freed using margo_destroy.

Parameters

mid – [in] Margo instance.
addr – [in] Abstract Mercury address of destination.
id – [in] Registered RPC ID.
handle – [out] Pointer to resulting HG handle.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_destroy(hg_handle_t handle)

Destroy Mercury handle.

Parameters: handle – [in] Mercury handle.
Returns: HG_SUCCESS or corresponding HG error code.

static inline hg_return_t margo_ref_incr(hg_handle_t handle)

Increment ref count on a Mercury handle.

Parameters: handle – [in] Mercury handle.
Returns: HG_SUCCESS or corresponding HG error code.

static inline const struct hg_info *margo_get_info(hg_handle_t handle)

Get info from handle.

Parameters: handle – [in] Mercury handle.
Returns: Pointer to info or NULL in case of failure.

hg_return_t margo_get_input(hg_handle_t handle, void *in_struct)

Get input from handle (requires registration of input proc to deserialize parameters). Input must be freed using margo_free_input.

Parameters

handle – [in] Mercury handle.
[in/out] – in_struct Pointer to input structure.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_free_input(hg_handle_t handle, void *in_struct)

Free resources allocated when deserializing the input.

Parameters

handle – [in] Mercury handle.
[in/out] – in_struct Pointer to input structure.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_get_output(hg_handle_t handle, void *out_struct)

Get output from handle (requires registration of output proc to deserialize parameters). Output must be freed using margo_free_output.

Parameters

handle – [in] Mercury handle.
[in/out] – out_struct Pointer to output structure.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_free_output(hg_handle_t handle, void *out_struct)

Free resources allocated when deserializing the output.

Parameters

handle – [in] Mercury handle.
[in/out] – out_struct Pointer to output structure.

Returns

HG_SUCCESS or corresponding HG error code.

const char *margo_handle_get_name(hg_handle_t handle)

Get the name of the RPC corresponding to this handle (NULL if the handle is invalid or the RPC wasn’t registered with a name).

Note: this function is more efficient than margo_rpc_get_name if you have a handle at hand, as opposed to calling margo_get_info to get the RPC id followed by margo_rpc_get_name to get the name from the id.

Parameters: handle – [in] Handle.
Returns: Registered name of the RPC.

hg_return_t margo_provider_forward(uint16_t provider_id, hg_handle_t handle, void *in_struct)

Forward an RPC request to a remote provider.

Parameters

provider_id – [in] Provider ID (may be MARGO_DEFAULT_PROVIDER_ID).
handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for RPC.

Returns

0 on success, hg_return_t values on error.

static inline hg_return_t margo_forward(hg_handle_t handle, void *in_struct)

Forward an RPC request to a remove host.

Parameters

handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for RPC.

Returns

0 on success, hg_return_t values on error.

hg_return_t margo_provider_iforward(uint16_t provider_id, hg_handle_t handle, void *in_struct, margo_request *req)

Forward (without blocking) an RPC request to a remote provider.

Parameters

provider_id – [in] Provider ID (may be MARGO_DEFAULT_PROVIDER_ID).
handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for the RPC.
req – [out] Request to wait on using margo_wait.

Returns

0 on success, hg_return_t values on error.

static inline hg_return_t margo_iforward(hg_handle_t handle, void *in_struct, margo_request *req)

Forward (without blocking) an RPC request to a remote host.

Parameters

handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for the RPC.
req – [out] Request to wait on using margo_wait.

Returns

0 on success, hg_return_t values on error.

hg_return_t margo_provider_cforward(uint16_t provider_id, hg_handle_t handle, void *in_struct, void (*on_comblete)(void*, hg_return_t), void *uargs)

Same as margo_provider_iforward, but will invoke a user-provided callback upon completion of the RPC.

Note: the callback will be invoked in the ULT running the progress loop, hence it should not be long-lived or it will prevent Mercury from making network progress. This function should be used for lightweight operations (e.g. setting an eventual, changing the value of variable, notifying an ABT_cond, etc.). For more complex operations, it is recommended to wrap a call to margo_provider_forward inside a ULT.

Parameters

provider_id – Provider id.
handle – Handle of the RPC.
in_struct – Input arguments.
on_complete – Completion callback.
uargs – Arguments for the callback.

Returns

0 on success, hg_return_t values on error.

static inline hg_return_t margo_cforward(hg_handle_t handle, void *in_struct, void (*on_comblete)(void*, hg_return_t), void *uargs)

Same as margo_iforward but will invoke a user-provided callback upon completion of the RPC.

Note: see note margo_provider_cforward comment regarding when to avoid using this function.

Parameters

handle – Handle of the RPC.
in_struct – Input arguments.
on_complete – Completion callback.
uargs – Arguments for the callback.

Returns

0 on success, hg_return_t values on error.

hg_return_t margo_provider_forward_timed(uint16_t provider_id, hg_handle_t handle, void *in_struct, double timeout_ms)

Forward an RPC request to a remote provider with a user-defined timeout.

Parameters

provider_id – [in] Provider id.
handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for RPC.
timeout_ms – [in] Timeout in milliseconds.

Returns

0 on success, hg_return_t values on error.

static inline hg_return_t margo_forward_timed(hg_handle_t handle, void *in_struct, double timeout_ms)

Forward an RPC request to a remote host with a user-defined timeout.

Parameters

handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for RPC.
timeout_ms – [in] Timeout in milliseconds.

Returns

0 on success, hg_return_t values on error.

hg_return_t margo_provider_iforward_timed(uint16_t provider_id, hg_handle_t handle, void *in_struct, double timeout_ms, margo_request *req)

Non-blocking version of margo_provider_forward_timed.

Parameters

provider_id – [in] Provider id.
handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for the RPC.
timeout_ms – [in] Timeout in milliseconds.

Returns

0 on success, hg_return_t values on error.

static inline hg_return_t margo_iforward_timed(hg_handle_t handle, void *in_struct, double timeout_ms, margo_request *req)

Non-blocking version of margo_forward_timed.

Parameters

handle – [in] Handle of the RPC to be sent.
in_struct – [in] Input argument struct for the RPC.
timeout_ms – [in] Timeout in milliseconds.
req – [out] Request.

Returns

0 on success, hg_return_t values on error

hg_return_t margo_provider_cforward_timed(uint16_t provider_id, hg_handle_t handle, void *in_struct, double timeout_ms, void (*on_comblete)(void*, hg_return_t), void *uargs)

Same as margo_provider_iforward_timed, but will invoke a user-provided callback upon completion of the RPC or timeout.

Note: see note margo_provider_cforward comment regarding when to avoid using this function.

Parameters

provider_id – Provider id.
handle – Handle of the RPC.
in_struct – Input arguments.
timeout_ms – Timeout.
on_complete – Completion callback.
uargs – Arguments for the callback.

Returns

0 on success, hg_return_t values on error.

static inline hg_return_t margo_cforward_timed(hg_handle_t handle, void *in_struct, double timeout_ms, void (*on_comblete)(void*, hg_return_t), void *uargs)

Same as margo_iforward_timed but will invoke a user-provided callback upon completion of the RPC or timeout.

Note: see note margo_provider_cforward comment regarding when to avoid using this function.

Parameters

handle – Handle of the RPC.
in_struct – Input arguments.
on_complete – Completion callback.
uargs – Arguments for the callback.

Returns

0 on success, hg_return_t values on error.

hg_return_t margo_wait(margo_request req)

Wait for an operation initiated by a non-blocking margo function (margo_iforward, margo_irespond, etc.).

Parameters: req – [in] Request to wait on.
Returns: 0 on success, hg_return_t values on error.

hg_return_t margo_wait_any(size_t count, margo_request *req, size_t *index)

Waits for any of the provided requests to complete.

Note

even if an error occures, the index will be set to the index of the request for which the error happened. This function will correctly ignore requests that are equal to MARGO_REQUEST_NULL. If all the requests are equal to MARGO_REQUEST_NULL, this function will return HG_SUCCESS and set index to count.

Parameters

req – Array of requests.
count – Number of requests.
index – index of the request that completed.

Returns

0 on success, hg_return_t values on error.

int margo_test(margo_request req, int *flag)

Test if an operation initiated by a non-blocking margo function (margo_iforward, margo_irespond, etc.) has completed.

Parameters

req – [in] Request created by the non-blocking call.
flag – [out] 1 if request is completed, 0 otherwise.

Returns

0 on success, ABT error code otherwise.

hg_handle_t margo_request_get_handle(margo_request req)

Retrieve the hg_handle_t (for iforward and irespond requests) that was used to create the request, or HG_HANDLE_NULL if the request is null or was created by a bulk transfer.

Parameters: req – Request
Returns: an hg_handle_t or HG_HANDLE_NULL.

margo_request_type margo_request_get_type(margo_request req)

Return the type of the request.

Parameters: req – Request
Returns: type of the request

margo_instance_id margo_request_get_instance(margo_request req)

Retrieve the margo_instance_id from a request.

Parameters: req – Request
Returns: margo instance id

hg_return_t margo_respond(hg_handle_t handle, void *out_struct)

Send an RPC response, waiting for completion before returning control to the calling ULT.

Parameters

handle – [in] Handle of the RPC for which a response is being sent.
out_struct – [in] Output argument struct for the response.

Returns

HG_SUCCESS on success, hg_return_t values on error.

hg_return_t margo_irespond(hg_handle_t handle, void *out_struct, margo_request *req)

Send an RPC response without blocking.

Parameters

handle – [in] Handle of the RPC for which a response is being sent.
out_struct – [in] Output argument struct for the response.
req – [out] Request on which to wait using margo_wait.

Returns

HG_SUCCESS on success, hg_return_t values on error.

hg_return_t margo_bulk_create(margo_instance_id mid, hg_uint32_t count, void **buf_ptrs, const hg_size_t *buf_sizes, hg_uint8_t flags, hg_bulk_t *handle)

Create an abstract bulk handle from specified memory segments. Memory allocated is then freed when margo_bulk_free() is called.

Remark

If NULL is passed to buf_ptrs, i.e.,

margo_bulk_create(mid, count, NULL, buf_sizes, flags, &handle)
*

The memory for the missing buf_ptrs array will be internally allocated.

Parameters

mid – [in] Margo instance.
count – [in] Number of segments.
buf_ptrs – [in] Array of pointers.
buf_sizes – [in] Array of sizes.
flags – [in] Permission flag:
- HG_BULK_READWRITE
- HG_BULK_READ_ONLY
- HG_BULK_WRITE_ONLY
handle – [out] Pointer to returned abstract bulk handle.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_bulk_free(hg_bulk_t handle)

Free bulk handle.

Parameters: [in/out] – handle Abstract bulk handle.
Returns: HG_SUCCESS or corresponding HG error code.

static inline hg_return_t margo_bulk_ref_incr(hg_bulk_t bulk)

Increment ref count on bulk handle.

Parameters: handle – Abstract bulk handle.
Returns: HG_SUCCESS or corresponding HG error code.

static inline hg_return_t margo_bulk_access(hg_bulk_t handle, hg_size_t offset, hg_size_t size, hg_uint8_t flags, hg_uint32_t max_count, void **buf_ptrs, hg_size_t *buf_sizes, hg_uint32_t *actual_count)

Access bulk handle to retrieve memory segments abstracted by handle.

Parameters

handle – [in] Abstract bulk handle.
offset – [in] Bulk offset.
size – [in] Bulk size.
flags – [in] Permission flag:
- HG_BULK_READWRITE
- HG_BULK_READ_ONLY
max_count – [in] Maximum number of segments to be returned.
[in/out] – buf_ptrs Array of buffer pointers.
[in/out] – buf_sizes Array of buffer sizes.
actual_count – [out] Actual number of segments returned.

Returns

HG_SUCCESS or corresponding HG error code.

static inline hg_size_t margo_bulk_get_size(hg_bulk_t bulk)

Get total size of data abstracted by bulk handle.

Parameters: Handle – [in] abstract bulk handle.
Returns: Non-negative value.

static inline hg_uint32_t margo_bulk_get_segment_count(hg_bulk_t bulk)

Get total number of segments abstracted by bulk handle.

Parameters: handle – [in] Abstract bulk handle.
Returns: Non-negative value.

static inline hg_size_t margo_bulk_get_serialize_size(hg_bulk_t bulk, unsigned long flags)

Get size required to serialize bulk handle.

Parameters

handle – [in] Abstract bulk handle.
flags – [in] option flags, valid flags are: HG_BULK_SM, HG_BULK_EAGER.

Returns

Non-negative value.

static inline hg_return_t margo_bulk_serialize(void *buf, hg_size_t buf_size, unsigned long flags, hg_bulk_t handle)

Serialize bulk handle into a buffer.

Parameters

[in/out] – buf Pointer to buffer.
buf_size – [in] Buffer size.
flags – [in] Option flags, valid flags are: HG_BULK_SM, HG_BULK_EAGER
handle – [in] Abstract bulk handle.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_bulk_deserialize(margo_instance_id mid, hg_bulk_t *handle, const void *buf, hg_size_t buf_size)

Deserialize bulk handle from an existing buffer.

Parameters

mid – [in] Margo instance.
handle – [out] Abstract bulk handle.
buf – [in] Pointer to buffer.
buf_size – [in] Buffer size.

Returns

HG_SUCCESS or corresponding HG error code.

hg_return_t margo_bulk_transfer(margo_instance_id mid, hg_bulk_op_t op, hg_addr_t origin_addr, hg_bulk_t origin_handle, size_t origin_offset, hg_bulk_t local_handle, size_t local_offset, size_t size)

Perform a bulk transfer.

Parameters

mid – [in] Margo instance.
op – [in] Type of operation to perform.
origin_addr – [in] Remote Mercury address.
origin_handle – [in] Remote Mercury bulk memory handle.
origin_offset – [in] Offset into remote bulk memory to access.
local_handle – [in] Local bulk memory handle.
local_offset – [in] Offset into local bulk memory to access.
size – [in] Size (in bytes) of transfer.

Returns

0 on success, hg_return_t values on error.

hg_return_t margo_bulk_itransfer(margo_instance_id mid, hg_bulk_op_t op, hg_addr_t origin_addr, hg_bulk_t origin_handle, size_t origin_offset, hg_bulk_t local_handle, size_t local_offset, size_t size, margo_request *req)

Asynchronously performs a bulk transfer.

Parameters

mid – [in] Margo instance.
op – [in] Type of operation to perform.
origin_addr – [in] Remote Mercury address.
origin_handle – [in] Remote Mercury bulk memory handle.
origin_offset – [in] Offset into remote bulk memory to access.
local_handle – [in] Local bulk memory handle.
local_offset – [in] Offset into local bulk memory to access.
size – [in] Size (in bytes) of transfer.
req – [out] Request to wait on using margo_wait.

Returns

0 on success, hg_return_t values on error.

void margo_thread_sleep(margo_instance_id mid, double timeout_ms)

Suspends the calling ULT for a specified time duration.

Parameters

mid – [in] Margo instance.
timeout_ms – [in] Timeout duration in milliseconds.

int margo_get_handler_pool(margo_instance_id mid, ABT_pool *pool)

Retrieve the abt_handler pool that was associated with the instance at initialization time.

Parameters

mid – [in] Margo instance.
pool – [out] Handler pool.

Returns

0 on success, error code on failure.

int margo_get_progress_pool(margo_instance_id mid, ABT_pool *pool)

Retrieves the progress pool that was associated with the instance at initialization time.

Parameters

mid – [in] Margo instance.
pool – [out] Progress pool.

Returns

0 on success, error code on failure.

ABT_pool margo_hg_handle_get_handler_pool(hg_handle_t h)

Retrieve the RPC handler pool that is associated with this handle.

Parameters: h – [in] Handle.
Returns: Argobots pool associated with the handler.

hg_context_t *margo_get_context(margo_instance_id mid)

Retrieve the Mercury context that was associated with this instance at initialization time.

Parameters: mid – [in] Margo instance.
Returns: The Mercury context internal to the margo instance.

hg_class_t *margo_get_class(margo_instance_id mid)

Retrieve the Mercury class that was associated with this instance at initialization time.

Parameters: mid – [in] Margo instance.
Returns: The Mercury class internal to the margo instance.

void *margo_get_data(hg_handle_t h)

Get the data that was associated with the handle using margo_set_data.

Parameters: h – Handle.
Returns: Data associated with the handle (NULL is no data attached).

hg_return_t margo_set_data(hg_handle_t h, void *data, void (*free_callback)(void*))

Attach data with the handle. Any previously-attached data will be freed using the previously-attached free_callback, before being replaced with the new data.

The free_callback with be called when the handle is destroyed.

Parameters

h – Handle.
data – Data to attach.
free_callback – Free callback.

Returns

HG_SUCCESS or HG error code.

margo_instance_id margo_hg_handle_get_instance(hg_handle_t h)

Get the margo_instance_id from a received RPC handle.

Parameters: h – [in] RPC handle.
Returns: Margo instance.

margo_instance_id margo_hg_info_get_instance (const struct hg_info *info) DEPRECATED("use margo_hg_handle_get_instance to get mid directly from handle")

Get the margo_instance_id from an hg_info struct.

Parameters: info – [in] hg_info struct.
Returns: Margo instance.

hg_return_t margo_set_current_rpc_id(margo_instance_id mid, hg_id_t id)

Inform Margo that the current ULT has been created as a consequence of the execution of a particular RPC (represented by its RPC id).

This information is used to track call paths in the monitoring subsystem. Calls to margo_forward or margo_bulk_transfer happening within an RPC handler will automatically detect that they are being called in the context of that RPC handler and set their callpath information accordingly. However, if an RPC handler spawns a ULT that calls margo_forward or margo_bulk_transfer, margo will not be able to detect the causality. Calling margo_set_current_rpc_id at the beginning of the ULT (or before margo_forward or margo_bulk_transfer are called) mitigates this problem.

Note that the user may use margo_get_info(handle)->id to get the RPC id of a handle within an RPC handler.

Parameters

mid – [in] Margo instance.
id – [in] Current RPC id.

Returns

HG_SUCCESS or other Mercury error code.

hg_return_t margo_get_current_rpc_id(margo_instance_id mid, hg_id_t *id)

Get the RPC id of the current RPC handler, if within a handler, or the RPC id set by margo_set_current_rpc_id if margo_set_current_rpc_id was previously called in the same ULT.

Note that outside of any handler and without having called margo_set_current_rpc_id, id will still be set to a valid value (equal to MARGO_DEFAULT_PROVIDER_ID, due to the way margo encodes its RPC ids and provider ids).

IMPORTANT: this function does not guarantee to return the real RPC id of the current RPC, it is a best-effort made by margo when access to the handle of the RPC is not possible. Users should use margo_get_info(handle)->id to make sure to get the id from the handle, when the handle is available.

Parameters

mid – [in] Margo instance.
id – [out] Current RPC id.

Returns

HG_SUCCESS or other Mercury error code.

int margo_start_abt_profiling(margo_instance_id mid, bool detailed)

Start Argobots profiling. Argobots must have been installed with the +tool variant.

Parameters

mid – Margo instance.
detailed – Whether profiling should be detailed or not.

Returns

0 on success, -1 on failure (e.g. if ABT profiling was already started or if Argobots was not compiled with tools).

int margo_stop_abt_profiling(margo_instance_id mid)

Stops Argobots profiling.

Parameters: mid – Margo instance.
Returns: 0 on success, -1 on failure (e.g. if ABT profiling was no started or if Argobots was not compiled with tools).

int margo_dump_abt_profiling(margo_instance_id mid, const char *file, int uniquify, char **resolved_file_name)

Dump ABT profiling data in a file.

Parameters

mid – [in] Margo instance.
file – [in] Output file (“-” for stdout). If string begins with ‘/’ character then it will be treated as an absolute path. Otherwise the file will be placed in the directory specified output_dir or MARGO_OUTPUT_DIR.
uniquify – [in] Flag indicating if file name should have additional information added to it to make output from different processes unique.
resolved_file_name – [out] (ignored if NULL) Pointer to char* that will be set to point to a string with the fully resolved path to the state file that was generated. Must be freed by caller.

Returns

0 on success, -1 on failure.

int margo_state_dump(margo_instance_id mid, const char *file, int uniquify, char **resolved_file_name)

Appends Margo state information (including Argobots stack information) to the specified file in text format.

Parameters

mid – [in] Margo instance.
file – [in] Output file (“-” for stdout).
uniquify – [in] flag indicating if file name should have additional information added to it to make output from different processes unique.
resolved_file_name – [out] (ignored if NULL) Pointer to char* that will be set to point to a string with the fully resolved path to the state file that was generated. Must be freed by caller.

int margo_set_progress_timeout_ub_msec(margo_instance_id mid, unsigned timeout)

Set the upper-bound of the timeout value used when calling HG_Progress internally.

Note that this value is an upper bound, margo may dynamically choose a smaller value if some other ULTs are sharing the same ES as the progress loop, or if some timers are installed (e.g. because the user called a margo_forward_timed) and would need to be triggered before the “normal” upper bound.

Parameters

mid – [in] Margo instance.
timeout – [in] Timeout value in milliseconds.

Returns

0 in case of success, -1 otherwise.

int margo_get_progress_timeout_ub_msec(margo_instance_id mid, unsigned *timeout)

Get the upper bound of the progress timeout value used when calling HG_Progress.

Parameters

mid – [in] Margo instance.
[ou] – timeout Timeout value in milliseconds.

Returns

0 in case of success, -1 otherwise.

int margo_set_param(margo_instance_id mid, const char *key, const char *value)

Sets configurable parameters/hints.

Parameters

mid – [in] Margo instance.
key – [in] Parameter name.
value – [in] Parameter value.

Returns

0 on success, -1 on failure.

struct margo_init_info

#include <margo.h>

The margo_init_info structure should be passed to margo_init_ext to finely configure Margo. The structure can be memset to 0 to have margo use default values (no progress thread, no rpc thread, default initialization of mercury, etc.). For any field that is not NULL, margo_init_ext will first look for a configuration in the json_config string. If no configuration is found or of json_config is NULL, margo will fall back to default.

Public Members

const char *json_config: JSON-formatted string

ABT_pool progress_pool: Progress pool

ABT_pool rpc_pool: RPC handler pool

hg_class_t *hg_class: Mercury class

hg_context_t *hg_context: Mercury context

struct hg_init_info *hg_init_info: Mercury init info

struct margo_logger *logger: User-provided logger

struct margo_monitor *monitor: User-provided monitor

margo-logging.h

See COPYRIGHT in top-level directory.

Typedefs

typedef struct margo_instance *margo_instance_id

typedef void (*margo_log_fn_t)(void *uargs, const char *message): Logging function type.

typedef enum margo_log_level margo_log_level

Log level.

trace: for entering/exiting function or providing additional detail about what code path was taken debug: reserve for when you want to actually debug some function; not meant to remain in the code after the bug is found info: information you expect users will want to know by default (e.g. a server address, etc.) warning: any warning (e.g. if a CPU affinity request in the configuration cannot be satisfied, but the code will execute anyway) error: something went wrong, but not wrong enough that the server should stop critical: right before you force the program to stop because of an unrecoverable problem

Enums

enum margo_log_level

Log level.

trace: for entering/exiting function or providing additional detail about what code path was taken debug: reserve for when you want to actually debug some function; not meant to remain in the code after the bug is found info: information you expect users will want to know by default (e.g. a server address, etc.) warning: any warning (e.g. if a CPU affinity request in the configuration cannot be satisfied, but the code will execute anyway) error: something went wrong, but not wrong enough that the server should stop critical: right before you force the program to stop because of an unrecoverable problem

Values:

enumerator MARGO_LOG_EXTERNAL

enumerator MARGO_LOG_TRACE

enumerator MARGO_LOG_DEBUG

enumerator MARGO_LOG_INFO

enumerator MARGO_LOG_WARNING

enumerator MARGO_LOG_ERROR

enumerator MARGO_LOG_CRITICAL

Functions

int margo_set_logger(margo_instance_id mid, const struct margo_logger *logger)

Set a logger structure for Margo to use. The structure will be internally copied and the user may free the input argument after the call. Passing NULL as logger will reset the logger to an internal logger that only prints errors and critical messages on stderr.

Parameters

mid – Margo instance
logger – Logger structure

Returns

0 in case of success, -1 otherwise

int margo_set_log_level(margo_instance_id mid, margo_log_level level)

Set the log level of the Margo instance.

Parameters

mid – Margo instance
level – Log level

Returns

0 in case of success, -1 otherwise

int margo_set_global_logger(const struct margo_logger *logger)

Set a logger structure for Margo to use in functions that don’t take a margo_instance_id (such as margo_init and its variants).

The structure will be internally copied and the user may free the input argument after the call. Passing NULL as logger will reset the global logger to a logger that only prints errors and critical messages on stderr.

Parameters: logger – Logger structure
Returns: 0 in case of success, -1 otherwise

int margo_set_global_log_level(margo_log_level level)

Set the log level of the logger used for global operations.

Parameters: level – Log level
Returns: 0 in case of success, -1 otherwise

void margo_trace(margo_instance_id mid, const char *fmt, ...)

Logging functions. These functions will use the logger registered with the margo instance, or the global logger if the margo instance is MARGO_INSTANCE_NULL.

Parameters

mid – Margo instance
fmt – Format string
... – Extra arguments

void margo_debug(margo_instance_id mid, const char *fmt, ...)

See also

margo_trace

void margo_info(margo_instance_id mid, const char *fmt, ...)

See also

margo_trace

void margo_warning(margo_instance_id mid, const char *fmt, ...)

See also

margo_trace

void margo_error(margo_instance_id mid, const char *fmt, ...)

See also

margo_trace

void margo_critical(margo_instance_id mid, const char *fmt, ...)

See also

margo_trace

struct margo_logger

#include <margo-logging.h>

Logging structure providing function pointers for different levels of logging.

Public Members

void *uargs

margo_log_fn_t trace

margo_log_fn_t debug

margo_log_fn_t info

margo_log_fn_t warning

margo_log_fn_t error

margo_log_fn_t critical

margo-timer.h

See COPYRIGHT in top-level directory.

Defines

MARGO_TIMER_NULL

Typedefs

typedef void (*margo_timer_callback_fn)(void*): Type of callback called when a timer expires.

typedef struct margo_timer *margo_timer_t: Margo timer type.

Functions

int margo_timer_create(margo_instance_id mid, margo_timer_callback_fn cb_fn, void *cb_dat, margo_timer_t *timer)

Creates a timer object. The callback will be submitted to the handler pool.

Parameters

mid – Margo instance
cb_fn – Callback to call when the timer finishes
cb_dat – Callback data
timer – Resulting timer

Returns

0 on success, negative value on failure

int margo_timer_create_with_pool(margo_instance_id mid, margo_timer_callback_fn cb_fn, void *cb_dat, ABT_pool pool, margo_timer_t *timer)

Creates a timer object and specifies the pool in which to run the callback.

Note

Passing ABT_POOL_NULL as the pool is allowed. In this case, the callback will be invoked directly within the ULT that runs the progress loop. This should generally be avoided unless the callback is very short, and does not call any margo_* or HG_* function. A typical example of a valid callback would be one that simply sets the value of an ABT_eventual, or one that submits a ULT and returns.

Parameters

mid – Margo instance
cb_fn – Callback to call when the timer finishes
cb_dat – Callback data
pool – Pool in which to run the callback
timer – Resulting timer

Returns

0 on success, negative value on failure

int margo_timer_start(margo_timer_t timer, double timeout_ms)

Start the timer with a provided timeout.

Parameters

margo_timer_t – Timer object to start
timeout_ms – Timeout

Returns

0 on success, negative value on failure

int margo_timer_cancel(margo_timer_t timer)

Cancel a started timer. If the timer’s callback has already been submitted as a ULT, this function will block until the ULT has executed. If the ULT hasn’t started yet when margo_timer_cancel is called, the ULT won’t run the callback and will simply return. If the ULT had started, it will run the callback to completion.

This function guarantees that there won’t be any invokation of the callback after margo_timer_cancel returns.

Parameters: timer – Timer to cancel
Returns: 0 on success, negative value on failure

int margo_timer_cancel_many(size_t n, margo_timer_t *timers)

Cancel n timers, blocking until it can ensure that no callback associated with any of the timers will be called.

This function is more efficient than calling margo_timer_cancel in a loop because it requests the cancelation of all the timers before blocking.

Warning

All the timers must be associated with the same margo instance.

Parameters

n – Number of timers
timer – Array of timers to cancel

Returns

0 on success, negative value on failure

int margo_timer_destroy(margo_timer_t timer)

Destroys a timer.

@important This function will not cancel the timer. If it was started, it will still fire, and the timer’s memory will be freed afterward.

Parameters: timer – Timer to destroy.
Returns: 0 on success, negative value on failure

margo-bulk-pool.h

See COPYRIGHT in top-level directory.

Defines

MARGO_BULK_POOL_NULL

MARGO_BULK_POOLSET_NULL

Typedefs

typedef struct margo_bulk_pool *margo_bulk_pool_t: Pool of bulk handles and buffers. All the buffers in the pool have the same size.

typedef struct margo_bulk_poolset *margo_bulk_poolset_t

Functions

hg_return_t margo_bulk_pool_create(margo_instance_id mid, hg_size_t count, hg_size_t size, hg_uint8_t flag, margo_bulk_pool_t *pool)

Creates a pool of buffers and bulk handles of a given size.

Parameters

mid – [in] Margo instance with which to create the bulk handles
count – [in] Number of bulk handles contained in the pool
size – [in] Size of each bulk buffer
flag – [in] HG_BULK_READ_ONLY, HG_BULK_WRITE_ONLY, or HG_BULK_READWRITE
pool – [out] Resulting pool of bulk handles

Returns

HG_SUCCESS in case of success, or HG error codes in case of failure.

int margo_bulk_pool_destroy(margo_bulk_pool_t pool)

Destroys a margo_bulk_pool_t object, clearing all bulk handles and and freeing the buffers. The pool should not be in use (i.e. none of the internal buffers should be in use) when this call happens.

Parameters: pool – margo_bulk_pool_t object to destroy.
Returns: 0 in case of success, -1 in case of failure.

int margo_bulk_pool_get(margo_bulk_pool_t pool, hg_bulk_t *bulk)

Gets a free hg_bulk_t object from the pool. This function will block until a bulk object is available.

Parameters

pool – [in] margo_bulk_pool_t object from which to take the hg_bulk_t.
bulk – [out] hg_bulk_t object (guaranteed not to be HG_BULK_NULL if the returned value is 0).

Returns

0 in case of success, -1 in case of failure.

int margo_bulk_pool_tryget(margo_bulk_pool_t pool, hg_bulk_t *bulk)

Gets a free hg_bulk_t object from the pool. This function will return HG_BULK_NULL if no hg_bulk_t object is free at the moment of the call.

Parameters

pool – margo_bulk_pool_t object from which to take the hg_bulk_t.
bulk – resulting bulk handle (can be HG_BULK_NULL).

Returns

0 in case of success, -1 in case of failure..

int margo_bulk_pool_release(margo_bulk_pool_t pool, hg_bulk_t bulk)

Puts a bulk handle back in the pool. Note that the function is expecting the bulk handle to have been taken from the pool in the first place. The function will return -1 if the bulk was not associated with this pool to begin with.

Parameters

pool – margo_bulk_pool_t object to which to return the bulk handle.
bulk – Bulk handle to release.

Returns

0 in case of success, -1 in case of failure.

hg_return_t margo_bulk_poolset_create(margo_instance_id mid, hg_size_t npools, hg_size_t nbufs, hg_size_t first_size, hg_size_t size_multiple, hg_uint8_t flag, margo_bulk_poolset_t *poolset)

Creates a poolset. A poolset is a set of pools with the same number of buffers in each pool, and buffer sizes increasing exponentially with the pool number.

Parameters

mid – [in] Margo instance
npools – [in] Number of pools in the poolset.
nbufs – [in] Number of buffers in each pool.
first_size – [in] Size of buffers in the first pool.
size_multiple – [in] Factor by which to multiply the size of the previous pool to get the size of the next.
flag – [in] HG_BULK_READ_ONLY, HG_BULK_WRITE_ONLY, or HG_BULK_READWRITE.
poolset – [out] Resulting poolset.

Returns

HG_SUCCESS of other HG error codes.

int margo_bulk_poolset_destroy(margo_bulk_poolset_t poolset)

Destroy a poolset. The poolset must not be in use when this function is called.

Parameters: poolset – Poolset to destroy.
Returns: 0 in case of success, -1 in case of failure.

void margo_bulk_poolset_get_max(margo_bulk_poolset_t poolset, hg_size_t *max_size)

Get maximum size supported by a pool set.

Parameters

poolset – Poolset
max_size – Maximum supported buffer size

int margo_bulk_poolset_get(margo_bulk_poolset_t poolset, hg_size_t size, hg_bulk_t *bulk)

Gets a bulk handle from the pool with the minimum size required to satisfy the provided size. May block until the pool has a bulk handle available.

Parameters

poolset – Poolset from which to get the bulk handle.
size – Size of the buffer needed.
bulk – Resulting bulk handle.

Returns

0 in case of success, -1 in case of failure.

int margo_bulk_poolset_tryget(margo_bulk_poolset_t poolset, hg_size_t size, hg_bool_t any_flag, hg_bulk_t *bulk)

Try getting a bulk handle from the poolset. If any_flag is HG_TRUE, this function will search in pools of increasingly larger buffers until it finds one (or return HG_BULK_NULL if it doesn’t). If any_flag is HG_FALSE, this function will only search in the pool with the minimum required size.

Parameters

poolset – Poolset in which to get a handle.
size – Size required.
any_flag – Whether to look in increasingly larger pools or not.
bulk – Resulting bulk handle.

Returns

0 in case of success (bulk = HG_BULK_NULL is also considered success), -1 in case of failure.

int margo_bulk_poolset_release(margo_bulk_poolset_t poolset, hg_bulk_t bulk)

Puts a bulk handle back in its pool.

Parameters

poolset – Poolset.
bulk – Bulk to release.

Returns

0 in case of success, -1 in case of success.

margo-bulk-util.h

See COPYRIGHT in top-level directory.

Functions

hg_return_t margo_bulk_parallel_transfer(margo_instance_id mid, hg_bulk_op_t op, hg_addr_t origin_addr, hg_bulk_t origin_handle, size_t origin_offset, hg_bulk_t local_handle, size_t local_offset, size_t size, size_t chunk_size)

Perform a bulk transfer by submitting multiple margo_bulk_transfer in parallel.

Parameters

mid – [in] Margo instance
op – [in] type of operation to perform
origin_addr – [in] remote Mercury address
origin_handle – [in] remote Mercury bulk memory handle
origin_offset – [in] offset into remote bulk memory to access
local_handle – [in] local bulk memory handle
local_offset – [in] offset into local bulk memory to access
size – [in] size (in bytes) of transfer
chunk_size – [in] size to by transferred by each operation

Returns

0 on success, hg_return_t values on error

margo-diag.h

Warning

doxygenfile: Cannot find file “margo-diag.h

margo-util.h

See COPYRIGHT in top-level directory.

Functions

int margo_set_abt_mem_max_num_stacks(unsigned max_num_stacks)

This function sets the ABT_MEM_MAX_NUM_STACKS environment variable for Argobots to use. It should be called before Argobots is initialized (hence before Margo is initialized).

Parameters: max_num_stacks –
Returns: 0 in case of success, -1 if Argobots is already initialized

int margo_set_abt_thread_stacksize(unsigned stacksize)

This function sets the ABT_THREAD_STACKSIZE environment variable for Argobots to use. It should be called before Argobots is initialized (hence before Margo is initialized).

Parameters: stacksize –
Returns: 0 in case of success, -1 if Argobots is already initialized

uint64_t margo_get_num_progress_calls(margo_instance_id mid)

Utility function to get the number of calls to HG_Progress that the margo instance did from the moment it was created.

Parameters: mid – Margo instance.
Returns: Number of calls to HG_Progress made.

uint64_t margo_get_num_trigger_calls(margo_instance_id mid)

Utility function to get the number of calls to HG_Trigger that the margo instance did from the moment it was created.

Parameters: mid – Margo instance.
Returns: Number of calls to HG_Trigger made.