Properly finalizing providers

In the previous tutorial, we have seen how to implement a provider class using Thallium. For convenience in the example, the provider was created on the stack and the call to wait_for_finalize was put inside its destructor. This ensured that the main function would block when the provider instance goes out of scope, and wait for the engine to be finalized.

This design works fine when the program runs only one provider, but it is flawed when we want to work with multiple providers, or when we want to be able to destroy providers before the engine is finalized (e.g. for services that dynamically spawn providers).

In this tutorial, we will look at another design that is more suitable to the general case of multiple providers per process.

Client

The client code is the same as in the previous tutorial, though we provide it here again for convenience. We have also added a call to shutdown_remote_engine from the client to shutdown the engine on the server and trigger the engine and providers finalization.

#include <iostream>
#include <thallium/serialization/stl/string.hpp>
#include <thallium.hpp>

namespace tl = thallium;

int main(int argc, char** argv) {
    if(argc != 3) {
        std::cerr << "Usage: " << argv[0] << " <address> <provider_id>" << std::endl;
        exit(0);
    }
    tl::engine myEngine("tcp", THALLIUM_CLIENT_MODE);
    tl::remote_procedure sum   = myEngine.define("sum");
    tl::remote_procedure prod  = myEngine.define("prod");
    tl::remote_procedure hello = myEngine.define("hello").disable_response();
    tl::remote_procedure print = myEngine.define("print").disable_response();
    tl::endpoint server = myEngine.lookup(argv[1]);
    uint16_t provider_id = atoi(argv[2]);
    tl::provider_handle ph(server, provider_id);
    int ret = sum.on(ph)(42,63);
    std::cout << "(sum) Server answered " << ret << std::endl;
    ret = prod.on(ph)(42,63);
    std::cout << "(prod) Server answered " << ret << std::endl;
    std::string name("Matthieu");
    hello.on(ph)(name);
    print.on(ph)(name);
    myEngine.shutdown_remote_engine(ph);
    return 0;
}

Server

The following code sample illustrates another version of our custom provider class, my_sum_provider.

#include <iostream>
#include <thallium.hpp>
#include <thallium/serialization/stl/string.hpp>

namespace tl = thallium;

class my_sum_provider : public tl::provider<my_sum_provider> {

    private:

    tl::remote_procedure m_prod;
    tl::remote_procedure m_sum;
    tl::remote_procedure m_hello;
    tl::remote_procedure m_print;

    void prod(const tl::request& req, int x, int y) {
        std::cout << "Computing " << x << "*" << y << std::endl;
        req.respond(x+y);
    }

    int sum(int x, int y) {
        std::cout << "Computing " << x << "+" << y << std::endl;
        return x+y;
    }

    void hello(const std::string& name) {
        std::cout << "Hello, " << name << std::endl;
    }

    int print(const std::string& word) {
        std::cout << "Printing " << word << std::endl;
        return word.size();
    }

    my_sum_provider(tl::engine& e, uint16_t provider_id=1)
    : tl::provider<my_sum_provider>(e, provider_id)
      // keep the RPCs in remote_procedure objects so we can deregister them.
    , m_prod(define("prod", &my_sum_provider::prod))
    , m_sum(define("sum", &my_sum_provider::sum))
    , m_hello(define("hello", &my_sum_provider::hello))
    , m_print(define("print", &my_sum_provider::print, tl::ignore_return_value()))
    {
        // setup a finalization callback for this provider, in case it is
        // still alive when the engine is finalized.
        get_engine().push_finalize_callback(this, [p=this]() { delete p; });
    }

    public:

    // this factory method and the private constructor prevent users
    // from putting an instance  of my_sum_provider on  the stack.
    static my_sum_provider* create(tl::engine& e, uint16_t provider_id=1) {
        return new my_sum_provider(e, provider_id);
    }

    ~my_sum_provider() {
        m_prod.deregister();
        m_sum.deregister();
        m_hello.deregister();
        m_print.deregister();
        // pop the finalize callback. If this destructor was called
        // from the finalization callback, there is nothing to pop
        get_engine().pop_finalize_callback(this);
    }
};

int main(int argc, char** argv) {

    tl::engine myEngine("tcp", THALLIUM_SERVER_MODE);
    myEngine.enable_remote_shutdown();
    std::cout << "Server running at address " << myEngine.self()
        << " with provider ids 22 and 23 " << std::endl;
    // create a pointer to the provider instance using the factory methods.
    my_sum_provider* myProvider22 = my_sum_provider::create(myEngine, 22);
    my_sum_provider* myProvider23 = my_sum_provider::create(myEngine, 23);

    myEngine.wait_for_finalize();
    // the finalization callbacks will ensure that providers are freed.

    return 0;
}

First, we are making the provider’s constructor private and force users to use the create factory method. This will ensure that any instance of the provider is created on the heap, not on the stack.

Then, we add a bunch of tl::remote_procedure fields to keep the registered RPCs. We use these fields to deregister the RPCs in the provider’s destructor.

The constructor of the provider also installs a finalization callback in the engine that will call delete on the provider’s pointer when the engine is finalized. Because we may want to delete the provider ourselves earlier than that, we don’t forget to add a call to pop_finalize_callback in the destructor.

Important

This design works fine if the provider pushes only one finalization callback, but is flawed if multiple callbacks are pushed by the provider. Suppose the provider pushes two callbacks f and g, in that order, and g calls delete. Upon finalizing, the engine will call g first, which itself will call pop_finalize_callback, which will pop f out. Ultimately, f will never be called by the engine.

The design presented in this tutorial is only an example of how to better handle the life time of provider objects. Obviously other designs could be envisioned (e.g. with pointers to implementation, or with smart pointers, etc.).

Important

Whatever the design you chose, it is important to rememver that all thallium resources (mutex, endpoints, etc.) should imperatively be destroyed before the engine itself finalizes.