Python API

This tutorial covers the Flock Python bindings, which provide a Pythonic interface to Flock’s group management capabilities. The Python API is built on top of the C++ implementation and integrates seamlessly with pymargo.

Installation

Install Flock with Python bindings using Spack:

spack install mochi-flock +python

Quickstart Example

Here’s a simple example showing the basics of Flock in Python:

"""
Quickstart example for Flock Python bindings.
Demonstrates basic server and client setup.
"""

import mochi.flock.server as server
import mochi.flock.client as client
from mochi.flock.view import GroupView
import mochi.margo
import json

# Create Margo engine
engine = mochi.margo.Engine("na+sm", mochi.margo.server)

# Create initial group view
initial_view = GroupView()
initial_view.members.add(str(engine.address), 42)

# Configure Flock provider with static backend
config = {
    "group": {
        "type": "static",
        "config": {}
    }
}

# Create Flock provider
provider = server.Provider(
    engine=engine,
    provider_id=42,
    config=json.dumps(config),
    initial_view=initial_view
)

print(f"Flock provider created at {engine.address} with provider_id 42")

# Create client
flock_client = client.Client(engine)

# Create group handle
group = flock_client.make_group_handle(
    address=str(engine.address),
    provider_id=42
)

print(f"Group handle created")
print(f"Group view has {len(group.view.members)} members")

# Access group view
for i, member in enumerate(group.view.members):
    print(f"  Member {i}: {member.address} (provider_id={member.provider_id})")

# Cleanup
engine.finalize()

print("\nQuickstart completed successfully!")

Key Points

Imports:

The main modules are: - mochi.flock.server - Provider (server-side) - mochi.flock.client - Client and GroupHandle - mochi.flock.view - GroupView class - mochi.margo - Margo engine

Creating a Provider:

provider = server.Provider(
    engine=engine,
    provider_id=42,
    config=json.dumps(config),
    initial_view=initial_view
)
The provider manages group membership. Configuration is passed as a JSON string.

Creating a Client:

flock_client = client.Client(engine)
Clients can connect to Flock providers to access group information.

Creating a Group Handle:

group = flock_client.make_group_handle(
    address=str(engine.address),
    provider_id=42
)

A GroupHandle provides access to the group view.

Server-Side: Provider API

Provider Class:

from mochi.flock.server import Provider

provider = Provider(
    engine: pymargo.core.Engine,
    provider_id: int,
    config: str,
    initial_view: GroupView
)

Parameters:

engine: Margo engine instance

provider_id: Unique provider identifier

config: JSON configuration string

initial_view: Initial group membership

Backend Configurations

Different backends suit different use cases:

"""
Examples of different Flock backend configurations.
"""

import mochi.flock.server as server
from mochi.flock.view import GroupView
import mochi.margo
import json

def static_backend_example():
    """Static backend: membership is fixed at initialization."""
    print("=== Static Backend Example ===")

    engine = mochi.margo.Engine("na+sm", mochi.margo.server)

    # Create initial view with multiple members
    initial_view = GroupView()
    initial_view.members.add(str(engine.address), 42)
    print(f"Initial view size: {len(initial_view.members)}")
    # Could add more members if we had multiple processes

    config = {
        "group": {
            "type": "static",
            "config": {}
        }
    }

    provider = server.Provider(
        engine=engine,
        provider_id=42,
        config=json.dumps(config),
        initial_view=initial_view
    )

    print(f"Static backend provider created")

    engine.finalize()


def centralized_backend_example():
    """Centralized backend: one server manages membership."""
    print("\n=== Centralized Backend Example ===")

    engine = mochi.margo.Engine("na+sm", mochi.margo.server)

    initial_view = GroupView()
    initial_view.members.add(str(engine.address), 43)

    config = {
        "group": {
            "type": "centralized",
            "config": {
            }
        }
    }

    provider = server.Provider(
        engine=engine,
        provider_id=43,
        config=json.dumps(config),
        initial_view=initial_view
    )

    print(f"Centralized backend provider created")
    print("Centralized backend allows dynamic membership changes")

    engine.finalize()


if __name__ == "__main__":
    static_backend_example()
    centralized_backend_example()
    print("\nBackend examples completed!")

Static Backend:

Fixed membership set at initialization

Lightweight, no coordination overhead

Use for: Fixed-size groups, static deployments

Centralized Backend:

Dynamic membership management

One provider coordinates membership

Use for: Dynamic groups, join/leave scenarios

Client-Side: Client API

Client Class:

from mochi.flock.client import Client

# Initialize from address
client = Client("ofi+tcp")

# Or from existing engine
client = Client(engine)

Creating Group Handles:

Three ways to create a GroupHandle:

From address: .. code-block:: python

group = client.make_group_handle(
address=”…”, provider_id=42

)
From file: .. code-block:: python

group = client.make_group_handle_from_file(“group.json”)

From serialized string:

group = client.make_group_handle_from_serialized(serialized_data)

GroupHandle Operations

Updating Group View:

group.update()  # Fetch latest membership from server

Accessing View:

view = group.view  # Get current GroupView

Properties:

group.client  # Get associated Client object

Working with GroupView

GroupView represents group membership and metadata:

"""
Working with GroupView objects.
"""

from mochi.flock.view import GroupView

print("=== GroupView Operations ===\n")

# Create empty group view
view = GroupView()
print(f"Created a view with {len(view.members)} members and {len(view.metadata)} metadata")

# Add members
view.members.add("tcp://127.0.0.1:1234", 1)
view.members.add("tcp://127.0.0.1:1235", 2)
view.members.add("tcp://127.0.0.1:1236", 3)

print(f"\nAfter adding 3 members, size: {len(view.members)}")

# Iterate over members
print("\nMembers in view:")
for i, member in enumerate(view.members):
    print(f"  {i}: {member.address} (provider_id={member.provider_id})")

# Access specific member by index
print(f"\nFirst member: {view.members[0].address}")
print(f"Second member: {view.members[1].address}")

# Digest (hash) of view
digest = view.digest
print(f"\nView digest (hash): {digest}")

# Metadata
view.metadata.add("application", "my_app")
view.metadata.add("version", "1.0")

# Access metadata
for i, m in enumerate(view.metadata):
    print(f" - metadata {i}: {m.key} => {m.value}")

print("\nGroupView operations completed!")

File Operations and Serialization

Flock supports file-based bootstrapping and serialization:

"""
File-based bootstrapping and serialization examples.
"""

import mochi.flock.server as server
import mochi.flock.client as client
from mochi.flock.view import GroupView
import mochi.margo
import json
import tempfile
import os

print("=== File-Based Operations ===\n")

# Create temporary file for this example
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.json')
group_file = temp_file.name
temp_file.close()

try:
    # Setup server
    engine = mochi.margo.Engine("na+sm", mochi.margo.server)

    initial_view = GroupView()
    initial_view.members.add(str(engine.address), 42)

    config = {
        "group": {
            "type": "static",
            "config": {}
        },
        "file": group_file # Flock will write group info to this file
    }

    provider = server.Provider(
        engine=engine,
        provider_id=42,
        config=json.dumps(config),
        initial_view=initial_view
    )

    print(f"Provider created, group info written to: {group_file}")

    # Client can bootstrap from file
    flock_client = client.Client(engine)

    # Method 1: Create group handle from file
    print("\nMethod 1: Loading from file...")
    group = flock_client.make_group_handle_from_file(group_file)
    print(f"Loaded group with {len(group.view.members)} members")

    # Cleanup
    engine.finalize()

finally:
    # Clean up temp file
    if os.path.exists(group_file):
        os.unlink(group_file)
        print(f"\nCleaned up temporary file: {group_file}")

print("\nFile operations completed!")

File-Based Configuration:

Include "file" in config to write group info to a file:

config = {
    "group": {
        "type": "static"
    }
    "file": "/path/to/group.json"
}

Loading from File:

group = client.make_group_handle_from_file(filename)

This is useful for:

Passing group info between processes

Storing group configuration

Bootstrapping without a file

Group Updates

With centralized backend, you can update group membership:

"""
Updating group membership (for centralized backend).
"""

import mochi.flock.server as server
import mochi.flock.client as client
from mochi.flock.view import GroupView
import mochi.margo
import json

print("=== Group Updates Example ===\n")

# Setup server with centralized backend (allows updates)
engine = mochi.margo.Engine("na+sm", mochi.margo.server)

initial_view = GroupView()
initial_view.members.add(str(engine.address), 42)

config = {
    "group": {
        "type": "centralized",
        "config": {}
    }
}

provider = server.Provider(
    engine=engine,
    provider_id=42,
    config=json.dumps(config),
    initial_view=initial_view
)

print(f"Centralized backend provider created")

# Create client and group handle
flock_client = client.Client(engine)
group = flock_client.make_group_handle(
    address=str(engine.address),
    provider_id=42
)

print(f"Initial group size: {len(group.view.members)}")
for i, member in enumerate(group.view.members):
    print(f"  Member {i}: {member.address}")

# Update group view (fetch latest from server)
print("\nCalling update() to refresh view...")
group.update()

print(f"After update, group size: {len(group.view.members)}")

# Note: In a real distributed scenario, other processes would join/leave
# and update() would reflect those changes

# Access view properties
print(f"\nGroup view digest: {group.view.digest}")
print(f"View has metadata: {len(group.view.metadata)} entries")

# Cleanup
engine.finalize()

print("\nGroup updates example completed!")

Updating:

group.update()  # Refresh view from server

This fetches the latest membership from the provider, reflecting any joins/leaves.