Key/value and document filters

The yk_list_* and yk_doc_list_* functions accept a filter argument that can be used to only return key/value pairs or document satisfying a certain condition. This tutorial shows how to use such filters.

Key prefix and suffix filters

By default, any filter data provided to the yk_list_* functions will be interpreted as a prefix that the key must start with. If the YOKAN_MODE_SUFFIX mode is used, it will be interpreted as a suffix.

Lua key/value filters

When using YOKAN_MODE_LUA_FILTER is used, the content of the filter will be interpreted as Lua code, which will be executed against each key/value pair. Within such a code, the __key__ variable will be a string set to the current key. If YOKAN_MODE_FILTER_VALUE is specified in the mode, the __value__ variable will be set to the current value. The Lua script must return a boolean indicating whether the pair satisfies the user-provided condition.

Lua document filters

For yk_doc_list_* functions, Lua scripts will be provided with the __id__ and __doc__ variables. The former is an integer containing the document id. The latter is a string containing the document’s content.

Since document stores are often used to store JSON documents, Yokan already integrates the lua-cjson library in its code, making it possible to convert the __doc__ variable into a Lua hierarchy of tables:

data = cjson.decode(__doc__)

Dynamic library filters

The YOKAN_MODE_LIB_FILTER mode can use used to provide a filter implemented in a shared library. The code bellow shows a custom key/value filter and a custom document filter.

#include <yokan/filters.hpp>
#include <cstring>

struct CustomKeyValueFilter : public yokan::KeyValueFilter {

    std::string m_to_append;

    CustomKeyValueFilter(margo_instance_id mid,
                         int32_t mode,
                         const yokan::UserMem& data)
    : m_to_append(data.data, data.size) {
        (void)mid;
        (void)mode;
    }

    bool requiresValue() const override {
        return true;
    }

    bool check(const void* key, size_t ksize,
               const void* val, size_t vsize) const override {
        // This custom filter will check if the key size and
        // value size have the same parity
        (void)key;
        (void)val;
        return (vsize % 2) == (ksize % 2);
    }

    size_t keySizeFrom(const void* key, size_t ksize) const override {
        (void)key;
        return ksize;
    }

    size_t valSizeFrom(
        const void* val, size_t vsize) const override {
        (void)val;
        return vsize + m_to_append.size();
    }

    size_t keyCopy(
        void* dst, size_t max_dst_size,
        const void* key, size_t ksize) const override {
        // This custom copy function will reverse the key
        if(max_dst_size < ksize) return YOKAN_SIZE_TOO_SMALL;
        for(size_t i=0; i < ksize; i++) {
            ((char*)dst)[i] = ((const char*)key)[ksize-i-1];
        }
        return ksize;
    }

    size_t valCopy(
        void* dst, size_t max_dst_size,
        const void* val, size_t vsize) const override {
        // This custom copy function will append
        // the user-provided filter argument to the value
        if(max_dst_size < vsize + m_to_append.size()) return YOKAN_SIZE_TOO_SMALL;
        std::memcpy(dst, val, vsize);
        std::memcpy((char*)dst+vsize, m_to_append.data(), m_to_append.size());
        return vsize + m_to_append.size();
    }
};
YOKAN_REGISTER_KV_FILTER(custom_kv, CustomKeyValueFilter);

struct CustomDocFilter : public yokan::DocFilter {

    CustomDocFilter(margo_instance_id mid, int32_t mode,
                    const yokan::UserMem& data) {
        (void)mid;
        (void)mode;
        (void)data;
    }

    bool check(const char* collection, yk_id_t id, const void* doc, size_t docsize) const override {
        // This custom filter will only let through the document with an even id
        (void)collection;
        (void)doc;
        (void)docsize;
        return id % 2 == 0;
    }

    size_t docSizeFrom(const char* collection, const void* val, size_t vsize) const override {
        (void)collection;
        (void)val;
        return vsize;
    }

    /**
     * @brief Copy the document to the target destination. This copy may
     * be implemented differently depending on the mode, and may alter
     * the content of the document.
     * This function should return the size actually copied.
     */
    virtual size_t docCopy(
        const char* collection,
        void* dst, size_t max_dst_size,
        const void* val, size_t vsize) const {
        vsize = std::min(vsize, max_dst_size);
        std::memcpy(dst, val, vsize);
        return vsize;
    }
};
YOKAN_REGISTER_DOC_FILTER(custom_doc, CustomDocFilter);

A custom key/value filter must inherit from yokan::KeyValueFilter and provide the following member functions.

A constructor accepting a margo instance id, a mode, and filter data.
requiresValue should return whether the filter needs the value.
check runs the filter against a key/value pair.
keyCopy and valCopy are used to copy the keys and values into a destination buffer. These functions can be used to extract or modify keys and values on the fly when they are read back by the user. They should return YOKAN_SIZE_TOO_SMALL if the provided buffer size is too small for the data to copy.

The filter should be registered using the YOKAN_REGISTER_KV_FILTER macro, which takes the name of the filter, and the name of the class.

Once such a filter is provided, say in a library libmy_custom_filter.so, yk_list_* functions can be called with the following string as filter: "libmy_custom_filter.so:custom_kv:...". Anything after the second column will be interpreted as binary data and passed to the filter class constructor’s third argument, hence making it possible to provide arguments to a custom filter.

The code above also shows a custom document filter. Such a filter must provide the following member functions.

A constructor accepting a margo instance id, a mode, and filter data.
check runs the filter against the document.

Similarly, the YOKAN_REGISTER_DOC_FILTER macro should be used to register the filter.

There is currently no docCopy function in document filters, so documents cannot be modified on the fly when read.