Python bindings

Bedrock provides Python bindings that allow you to start Bedrock services, connect to them, query their configuration, and manipulate them at runtime, all from Python.

Installing Bedrock with Python support

To use Bedrock’s Python bindings, you need to install Bedrock with Python support enabled using Spack:

spack install mochi-bedrock+python

This will build Bedrock with Python bindings and install the mochi.bedrock Python package.

Starting a Bedrock service from Python

The mochi.bedrock module provides a Server class that can be used to start a Bedrock service directly from Python.

#!/usr/bin/env python
"""
Example of starting a Bedrock service from Python.
"""
from mochi.bedrock.server import Server
import time

# Configuration as a Python dictionary
config = {
    "margo": {
        "argobots": {
            "pools": [
                {"name": "my_pool", "kind": "fifo_wait", "access": "mpmc"}
            ]
        }
    },
    "libraries": [
        "libyokan-bedrock-module.so",
        "libflock-bedrock-module.so"
    ],
    "providers": [
        {
            "name": "my_database",
            "type": "yokan",
            "provider_id": 42,
            "config": {
                "database": {
                    "type": "map"
                }
            },
            "dependencies": {
                "pool": "__primary__"
            }
        },
        {
            "name": "my_group",
            "type": "flock",
            "provider_id": 33,
            "config": {
                "bootstrap": "self",
                "group": {
                    "type": "static",
                    "config": {}
                },
                "file": "mygroup.flock"
            },
            "dependencies": {
                "pool": "__primary__"
            }
        }
    ]
}

# Start the Bedrock server
server = Server("na+sm", config=config)

print(f"Bedrock server started at {server.margo.engine.address}")

server.wait_for_finalize()
print("Server finalized")

The Server constructor takes a Mercury address (protocol) and an optional configuration. The configuration can be provided as:

  • A Python dictionary

  • A JSON string

  • A ProcSpec object (see below)

  • A file path (if using Server.from_config_file())

The server runs in its own thread and can be finalized using server.finalize().

Connecting to a Bedrock service

The Client class allows you to connect to a running Bedrock service and interact with it.

#!/usr/bin/env python
"""
Example of connecting to a Bedrock service from Python.
"""
from mochi.bedrock.client import Client
import sys

if len(sys.argv) != 2:
    print(f"Usage: {sys.argv[0]} <server_address>")
    sys.exit(1)

server_address = sys.argv[1]

# Create a Bedrock client
client = Client("na+sm")

# Create a handle to the service
service = client.make_service_handle(server_address, provider_id=0)

print(f"Connected to Bedrock service at {service.address}")
print(f"Provider ID: {service.provider_id}")

# Get the configuration
config = service.config
print("\nService configuration:")
print(f"  Number of providers: {len(config.get('providers', []))}")

# List all providers
if 'providers' in config:
    print("\nProviders:")
    for provider in config['providers']:
        print(f"  - {provider['name']} (type={provider['type']}, id={provider['provider_id']})")

The ServiceHandle object returned by make_service_handle() provides methods to query and manipulate the service’s configuration at runtime.

Querying configuration

You can retrieve the complete configuration of a Bedrock service:

#!/usr/bin/env python
"""
Example of querying a Bedrock service configuration.
"""
from mochi.bedrock.client import Client
import sys
import json

if len(sys.argv) != 2:
    print(f"Usage: {sys.argv[0]} <server_address>")
    sys.exit(1)

server_address = sys.argv[1]

# Create client and service handle
client = Client("na+sm")
service = client.make_service_handle(server_address, provider_id=0)

# Get complete configuration
config = service.config
print("=== Complete Configuration ===")
print(json.dumps(config, indent=2))

# Use a Jx9 query to get only provider names
jx9_script = """
$result = [];
foreach($__config__['providers'] as $provider) {
    $result[] = $provider['name'];
}
return $result;
"""

provider_names = service.query(jx9_script)
print("\n=== Provider Names (via Jx9) ===")
print(provider_names)

# Query for pools
jx9_pools = """
$result = [];
if(array_key_exists('margo', $__config__) &&
   array_key_exists('argobots', $__config__['margo']) &&
   array_key_exists('pools', $__config__['margo']['argobots'])) {
    foreach($__config__['margo']['argobots']['pools'] as $pool) {
        $result[] = $pool['name'];
    }
}
return $result;
"""

pools = service.query(jx9_pools)
print("\n=== Argobots Pools ===")
print(pools)

The config property returns a Python dictionary representing the current configuration of the service. You can also use Jx9 scripts to query specific parts of the configuration.

Runtime configuration manipulation

The ServiceHandle class provides methods to modify a running service’s configuration dynamically.

Loading modules at runtime

# Load a new module
service.load_module("/path/to/libmy-module.so")

Adding pools and execution streams

# Add a new Argobots pool
pool_config = {
    "name": "new_pool",
    "kind": "fifo_wait",
    "access": "mpmc"
}
service.add_pool(pool_config)

# Add a new execution stream
xstream_config = {
    "name": "new_xstream",
    "scheduler": {
        "type": "basic_wait",
        "pools": ["new_pool"]
    }
}
service.add_xstream(xstream_config)

# Remove an xstream
service.remove_xstream("new_xstream")

# Remove a pool (must not be in use)
service.remove_pool("new_pool")

Adding providers at runtime

# Add a new provider
provider_config = {
    "name": "my_new_provider",
    "type": "yokan",
    "provider_id": 99,
    "dependencies": {
        "pool": "__primary__"
    },
    "config": {
        "database": {
            "type": "map"
        }
    }
}
provider_id = service.add_provider(provider_config)
print(f"Created provider with ID: {provider_id}")

Complete example

Here’s a complete example showing runtime manipulation:

#!/usr/bin/env python
"""
Example of runtime configuration manipulation with Bedrock Python API.
"""
from mochi.bedrock.client import Client
import time
import json
import sys

if len(sys.argv) != 2:
    print(f"Usage: {sys.argv[0]} <server_address>")
    sys.exit(1)

address = sys.argv[1]

# Connect as a client
print("\nConnecting to server...")
client = Client("na+sm")
service = client.make_service_handle(address, provider_id=0)

# Show initial configuration
print("\n=== Initial Configuration ===")
config = service.config
print(f"Providers: {len(config.get('providers', []))}")

# Load a module
print("\n=== Loading Module ===")
service.load_module("libyokan-bedrock-module.so")

# Add a new pool
print("\n=== Adding Pool ===")
pool_config = {
    "name": "dynamic_pool",
    "kind": "fifo_wait",
    "access": "mpmc"
}
service.add_pool(pool_config)
print("Added pool: dynamic_pool")

# Add a new execution stream
print("\n=== Adding Execution Stream ===")
xstream_config = {
    "name": "dynamic_xstream",
    "scheduler": {
        "type": "basic_wait",
        "pools": ["dynamic_pool"]
    }
}
service.add_xstream(xstream_config)
print("Added xstream: dynamic_xstream")

# Add a new provider
print("\n=== Adding Provider ===")
provider_config = {
    "name": "runtime_database",
    "type": "yokan",
    "provider_id": 100,
    "dependencies": {
        "pool": "dynamic_pool",
    },
    "config": {
        "database": {
            "type": "map"
        }
    }
}
new_provider_id = service.add_provider(provider_config)
print(f"Added provider with ID: {new_provider_id}")

# Query updated configuration
print("\n=== Updated Configuration ===")
config = service.config
print(f"Providers: {len(config.get('providers', []))}")
print("\nProvider list:")
for provider in config.get('providers', []):
    print(f"  - {provider['name']} (type={provider['type']}, id={provider['provider_id']})")

Working with service groups

When multiple Bedrock services are organized in a Flock group, you can interact with them collectively using ServiceGroupHandle:

#!/usr/bin/env python
"""
Example of working with Bedrock service groups using Flock.
"""
from mochi.bedrock.client import Client
import sys

if len(sys.argv) != 2:
    print(f"Usage: {sys.argv[0]} <flock_group_file>")
    sys.exit(1)

group_file = sys.argv[1]

# Create a Bedrock client
client = Client("na+sm")

# Create a handle to the service group
# This assumes all members have Bedrock provider at ID 0
group = client.make_service_group_handle_from_flock(group_file, provider_id=0)

# You can also use a list of addresses if Flock is not used
# group = client.make_service_group_handle([addr1, addr2, ...])

print(f"Connected to service group")
print(f"Group size: {group.size}")

# Refresh group membership (in case it changed)
group.refresh()

# Access individual members
print("\n=== Service Members ===")
for i in range(group.size):
    service = group[i]
    config = service.config
    num_providers = len(config.get('providers', []))
    print(f"Member {i}:")
    print(f"  Address: {service.address}")
    print(f"  Providers: {num_providers}")

# Query all members for their provider names
print("\n=== Provider Names from All Members ===")
jx9_script = """
$result = [];
foreach($__config__['providers'] as $provider) {
    $result[] = $provider['name'];
}
return $result;
"""

for i in range(group.size):
    service = group[i]
    provider_names = service.query(jx9_script)
    print(f"Member {i}: {provider_names}")

The server code previously shown has a Flock provider that created a mygroup.flock file; try querying the server using this file.

The ServiceGroupHandle allows you to:

  • Query all members of the group

  • Access individual service handles by index

  • Refresh the group membership

  • Broadcast operations to all members

Using ProcSpec for configuration

The mochi.bedrock.spec module provides Python classes for building configurations programmatically:

from mochi.bedrock.spec import (
        ProcSpec,
        PoolSpec,
        XstreamSpec,
        ProviderSpec,
        SchedulerSpec,
        MargoSpec
)
from mochi.bedrock.server import Server

# Create pool specifications
pool1 = PoolSpec(name="pool1", kind="fifo_wait", access="mpmc")
pool2 = PoolSpec(name="pool2", kind="fifo_wait", access="mpmc")

# Create xstream specifications
xstream1 = XstreamSpec(
    name="xstream1",
    scheduler=SchedulerSpec(
        type="basic_wait",
        pools=[pool1, pool2]
    )
)

# Create provider specification
provider = ProviderSpec(
    name="my_provider",
    type="yokan",
    provider_id=42,
    dependencies={
        "pool": "pool1"
    },
    config={"database": {"type": "map"}}
)

# Build complete process specification
spec = ProcSpec(margo="na+sm")

# Convert to dictionary or JSON
config_dict = spec.to_dict()
config_json = spec.to_json()

# Start server with this configuration
server = Server("na+sm", config=spec)
server.wait_for_finalize()

The advantage of using the mochi.bedrock.spec package to build such configuration is that Python will check the configuration for consistency. It can also be used to program parameter space exploration of Bedrock configurations.

Integration with PyMargo

The Bedrock Python bindings integrate with PyMargo. You can extract the underlying Margo instance:

from mochi.bedrock.server import Server

# Start Bedrock server
server = Server("na+sm", config={})
# Access the underlying engine
engine = server.engine