Exercise 1: simple RPC and RDMA using Margo

Note

The instructions in these exercises have a checkbox that you can click on to help you keep track of your progress. These checkboxes are not connected to any action, they are just there for you to mark your progress.

The code for this exercise has been cloned in margo-tutorial-exercises in your development environment. In a terminal connected to your docker container, make sure you are in the appropriate directory.

cd margo-tutorial/margo-tutorial-exercises

The src directory provides a client.c client code, a server.c server code, a types.h header defining RPC types, and a phonebook.h file containing a (very naive) implementation of a phonebook.

In this exercise we will make the server manage a phonebook and service two kinds of RPCs: adding a new entry, and looking up a phone number associated with a name.

Let’s start by setting up the spack environment and building the existing code:

spack env create margo-tuto-env spack.yaml
spack env activate margo-tuto-env
spack install
mkdir build
cd build
cmake ..
make

This will create the client and server programs.

You can test your client and server programs by opening two terminals (make sure you have run spack env activate margo-tuto-env in them to activate your spack environment) and running the following from the build directory.

For the server:

src/server na+sm

This will start the server and print its address. na+sm (the shared memory transport) may be changed to tcp if you run this code on multiple machines connected via an Ethernet network.

For the client:

src/client <server-address>

Copy <server-address> from the standard output of the server command. The server is setup to run indefinitely. You may kill it with Ctrl-C.

Important

The server address will change whenever you restart it.

Note

If you used tcp as protocol, the server might print an address containing a semicolumn. If this is the case, place the address in quotes when passing it to the client, other your shell will interpret the semicolumn as the end of your command.

Now inspect the server.c file. One important data structure to take note of is the server_data struct. It contains the state of the example service. Anything added to this structure will be maintained across the lifetime of the service and can (optionally) be manipulated by via RPCs. This is where we will maintain the phonebook data structure for this hands-on exercise.

You an also see that individual RPCs are registered using the MARGO_REGISTER() macro, as in the “sum” example. Note that this example also calls the margo_register_data() function immediately after the RPC is registered. The purpose of margo_register_data() is to associate state (in this case the server_data struct instance) with RPCs so that RPC handlers can retrieve that pointer later without relying on a global variable. This convention makes it safe for a server daemon to run multiple copies of the same provider without interfering with each other. Any new RPCs we add that manipulate the phonebook state will similarly need to register that data pointer.

Look at the API in phonebook.h. This is a local API for manipulating a phonebook data structure. Your task now is to add new RPCs to the server that will allow remote clients to manipulate a phonebook as well. You will need to include phonebook.h in server.c so that the service has access to the phonebook API. Next you must initiate a single phonebook instance for the service to maintain. Edit server.c to add the creation of a phonebook object (i.e., a call to phonebook_new()) and its destruction (i.e., a call to phonebook_delete()) when the server terminates. This phonebook should be added as a field to the server_data structure and to the svr_data instance (see comments (1) to (3) in server.c).

Your next task is to add two new RPCs, which we will call “insert” and “lookup”. Begin by defining their input and output argument types. This is done using MERCURY_GEN_PROC() macros of the following form:

MERCURY_GEN_PROC(rpc_name,
   ((type)(arg1))\
   ((type)(arg2))\
   ...
   ((type)(argN)))

Edit the types.h file to add the necessary type definitions for these RPCs (insert_in_t, insert_out_t, lookup_in_t and lookup_out_t, see comment (4)). Do so using the Mercury macros, following the model of the sum_in_t and sum_out_t types. Recall that we will use a uint64_t type to represent phone numbers.

*Hint: Mercury represents null-terminated strings with the type hg_string_t and hg_const_string_t. These are defined in mercury_proc_string.h, so you will need to include that header in types.h in order to add string types to your RPC arguments. The only difference between the two is type checking; the latter expects to encode const string arguments. We recommend that you use the hg_const_string_t for insert in order to align with the client-side API.

Note

While the insertion operation does not technically return anything, it is still advised to make all RPCs return at least a uint32_t error code to inform the sender of the success (or failure) of the operation.

Note

If you only have half an hour to work on this problem, focus on the insert RPC first. You can come back and fill in the lookup RPC later as time permits. It may also be helpful to stub in the new RPC handlers to begin with such that they do nothing except call margo_info(), which is a logging function that you can use in a manner similar to printf(). This will enable you to validate that the RPC is being registered and executed as expected end-to-end from the client before filling in the phonebook logic.

Edit server.c to add the definitions and declarations of the handlers for our two RPCs (see comment (5) and (6)). Feel free to copy/paste and modify the existing sum RPC. Don’t forget to register your RPCs with the margo instance in main (comment (7)), and don’t forget to call margo_register_data to associate the server data with the RPC.

Edit client.c and use the existing code as an example to (1) register the two new RPCs here as well (comment (8)). Observe that the same MARGO_REGISTER() function is used on both the client and the server and that the name argument must match. The only difference is that the client sets the handler function (last argument) to NULL because this client will only issue RPCs, and never service them. Next define two insert and lookup convenience functions. Example prototypes are given in comment (9). These functions need to mimic the logic within the for loop that issues “sum” RPCs in the existing code. Rather than hardcoding these steps directly in main(), we want the insert and lookup functions to contain the logic to create an hg_handle_t, forward it to the server with the proper arguments, and receive the response. Note that you will need to change the input and output types to match your new RPCs (for example, insert_in_t and insert_out_t in place of sum_in_t and sum_out_t, with fields set accordingly). These client-side convenience functions will need to call margo_create to create the hg_handle_t handle for the RPC, margo_forward to forward it to the server, margo_get_output to retrieve the response from the server, margo_free_output to free this response, and margo_destroy to destroy the hg_handle_t handle.

Try out your code by calling insert and lookup a few times in main (comment (10)). Note that you can use the same svr_addr as was being used to issue the example “sum” RPCs, as long as you use it before it is destroyed with the margo_addr_free().

Bonus: using RDMA to transfer larger amounts of data

Do this bonus part only if you have time, or as an exercise after the tutorial. This part is less guided. You should now know how to add new RPCs to your code.

In this part, we will add a lookup_multi RPC that uses RDMA to send multiple names at once and return the array of associated phone numbers (in practice this would be too little data to call for the use of RDMA, but we will just pretend). For this, you may use the example on Transferring data over RDMA.

Here are some tips for this part:

On the client side, your lookup_multi function could take the number of names as a uint32_t and the list of names to look up as an array of null-terminated strings (const char* const*), as well as an output array of uint64_t. See comment (11) for a prototype.

The important functions to work with RDMA are the following: margo_bulk_create (create an hg_bulk_t to expose a list of local memory segments for RDMA), margo_bulk_transfer (push/pull data to/from a local bulk handle, to/from a remote bulk handle), and margo_bulk_free (free a local hg_bulk_t created by margo_bulk_create). Alongside the documentation on this website, the margo.h header provides the necessary information to work with these function.

You will need to create two bulk handles on the client and two on the server. On the client, the first will expose the names as read-only (remember that margo_bulk_create can take a list of non-contiguous segments, but you will need to use strlen(...)+1 as the size of each segment to keep the null terminator of each name), and the second will expose the output array as write only.

You will need to transfer the two bulk handles in the RPC arguments, and since names can have a varying size, you will have to also transfer the total size of the bulk handle wrapping names, so that the server knows how much memory to allocate for its local buffer. The hg_bulk_t type’s serialization routines are defined in mercury_proc_bulk.h.

On the server side, you will need to allocate two buffers; one to receive the names via a pull operation, the other to send the phone numbers via a push.

You will need to create two hg_bulk_t to expose these buffers.

After having transferred the names, they will be in the server’s buffer, which, contrary to the client’s memory, is contiguous. You can rely on the null-terminators to know where one name ends and the next starts.