Self Operations and Yielding

Self operations give ULTs direct control over their execution: yielding to the scheduler, suspending/resuming, and explicit control flow transfers. These are advanced techniques critical for implementing custom execution patterns and integrating with external event loops.

Key Concepts

Self Operations

Operations that a ULT performs on itself: - ABT_self_yield() or ABT_thread_yield(): Voluntarily return control to scheduler - ABT_self_suspend(): Suspend self, requiring external resume - ABT_self_exit(): Terminate self immediately

Cooperative Scheduling

Unlike preemptive scheduling (OS threads), Argobots ULTs run until they voluntarily yield or block. This gives predictable execution but requires cooperation.

Progress Polling

Common pattern in asynchronous systems: repeatedly check for completion while yielding to let other work progress. Essential in Margo for network progress.

Yield-Based Synchronization

Yielding enables fair cooperative scheduling:

 1/*
 2 * Yield-based synchronization: Cooperative multitasking
 3 */
 4
 5#include <stdio.h>
 6#include <abt.h>
 7
 8#define NUM_WORKERS 3
 9#define ITERATIONS 5
10
11typedef struct {
12    int worker_id;
13    int *shared_counter;
14    ABT_mutex mutex;
15} worker_arg_t;
16
17void cooperative_worker(void *arg)
18{
19    worker_arg_t *worker = (worker_arg_t *)arg;
20
21    for (int i = 0; i < ITERATIONS; i++) {
22        /* Try to acquire lock */
23        ABT_mutex_lock(worker->mutex);
24
25        (*worker->shared_counter)++;
26        printf("Worker %d: iteration %d, counter = %d\n",
27               worker->worker_id, i, *worker->shared_counter);
28
29        ABT_mutex_unlock(worker->mutex);
30
31        /* Voluntarily yield to let other workers run */
32        printf("  Worker %d yielding...\n", worker->worker_id);
33        ABT_self_yield();
34    }
35
36    printf("Worker %d completed\n", worker->worker_id);
37}
38
39int main(int argc, char **argv)
40{
41    ABT_xstream xstream;
42    ABT_pool pool;
43    ABT_thread workers[NUM_WORKERS];
44    ABT_mutex mutex;
45    worker_arg_t worker_args[NUM_WORKERS];
46    int shared_counter = 0;
47
48    ABT_init(argc, argv);
49
50    printf("=== Yield-Based Cooperative Scheduling ===\n\n");
51
52    ABT_mutex_create(&mutex);
53    ABT_xstream_self(&xstream);
54    ABT_xstream_get_main_pools(xstream, 1, &pool);
55
56    /* Create workers */
57    for (int i = 0; i < NUM_WORKERS; i++) {
58        worker_args[i].worker_id = i;
59        worker_args[i].shared_counter = &shared_counter;
60        worker_args[i].mutex = mutex;
61
62        ABT_thread_create(pool, cooperative_worker, &worker_args[i],
63                          ABT_THREAD_ATTR_NULL, &workers[i]);
64    }
65
66    /* Wait for all */
67    for (int i = 0; i < NUM_WORKERS; i++) {
68        ABT_thread_free(&workers[i]);
69    }
70
71    ABT_mutex_free(&mutex);
72
73    printf("\nYielding enabled fair cooperative scheduling\n");
74    printf("Final counter: %d\n", shared_counter);
75
76    ABT_finalize();
77    return 0;
78}
Key Points:

  • ABT_self_yield() (line 31): Voluntarily gives up CPU

  • Scheduler selects next ULT to run

  • Yielding ULT remains runnable, will be rescheduled

  • Enables fair sharing without preemption

Use Cases:

  • Long-running computations that should share CPU

  • Cooperative critical sections

  • Polling loops

Progress Polling Pattern

Critical pattern for Margo/Mochi network progress:

  1/*
  2 * Progress polling pattern: Non-blocking checks with yield
  3 * Common in Margo/Mochi for network progress
  4 */
  5
  6#include <stdio.h>
  7#include <abt.h>
  8
  9#define NUM_REQUESTS 5
 10
 11typedef struct {
 12    int request_id;
 13    int completed;
 14} async_request_t;
 15
 16typedef struct {
 17    async_request_t *requests;
 18    int num_requests;
 19} poller_arg_t;
 20
 21void request_simulator(void *arg)
 22{
 23    async_request_t *requests = (async_request_t *)arg;
 24
 25    /* Simulate async completion of requests */
 26    for (int i = 0; i < NUM_REQUESTS; i++) {
 27        /* Simulate delay */
 28        for (int j = 0; j < 100000 * (i + 1); j++);
 29
 30        requests[i].completed = 1;
 31        printf("  Background: Request %d completed\n", requests[i].request_id);
 32        ABT_self_yield();
 33    }
 34}
 35
 36void progress_poller(void *arg)
 37{
 38    poller_arg_t *poller = (poller_arg_t *)arg;
 39    int completed_count = 0;
 40
 41    printf("Poller: Starting to poll for completions\n\n");
 42
 43    while (completed_count < poller->num_requests) {
 44        /* Poll for completed requests (non-blocking) */
 45        for (int i = 0; i < poller->num_requests; i++) {
 46            if (poller->requests[i].completed &&
 47                poller->requests[i].request_id >= 0) {
 48                printf("Poller: Detected completion of request %d\n",
 49                       poller->requests[i].request_id);
 50                poller->requests[i].request_id = -1;  /* Mark as processed */
 51                completed_count++;
 52            }
 53        }
 54
 55        /* Yield to let other work progress */
 56        ABT_self_yield();
 57    }
 58
 59    printf("\nPoller: All requests completed\n");
 60}
 61
 62int main(int argc, char **argv)
 63{
 64    ABT_xstream xstream;
 65    ABT_pool pool;
 66    ABT_thread simulator;
 67    ABT_thread poller;
 68    async_request_t requests[NUM_REQUESTS];
 69    poller_arg_t poller_arg;
 70
 71    ABT_init(argc, argv);
 72
 73    printf("=== Progress Polling Pattern ===\n");
 74    printf("Simulates async I/O or network progress polling\n\n");
 75
 76    /* Initialize requests */
 77    for (int i = 0; i < NUM_REQUESTS; i++) {
 78        requests[i].request_id = i;
 79        requests[i].completed = 0;
 80    }
 81
 82    poller_arg.requests = requests;
 83    poller_arg.num_requests = NUM_REQUESTS;
 84
 85    ABT_xstream_self(&xstream);
 86    ABT_xstream_get_main_pools(xstream, 1, &pool);
 87
 88    /* Create poller thread */
 89    ABT_thread_create(pool, progress_poller, &poller_arg,
 90                      ABT_THREAD_ATTR_NULL, &poller);
 91
 92    /* Create simulator thread */
 93    ABT_thread_create(pool, request_simulator, requests,
 94                      ABT_THREAD_ATTR_NULL, &simulator);
 95
 96    /* Wait for both */
 97    ABT_thread_free(&poller);
 98    ABT_thread_free(&simulator);
 99
100    printf("\nProgress polling with yield: common pattern in Margo\n");
101
102    ABT_finalize();
103    return 0;
104}
Key Points:

  • Poll for completion in loop

  • Yield after each check

  • Non-blocking: doesn’t wait, just checks

  • Lets other ULTs (like background workers) make progress

Example Usage:
/* Typical progress loop pattern */
while (!request_completed) {
    make_progress(...);  /* Non-blocking progress */
    ABT_self_yield();    /* Let other work run */
}

Other Self Operations

Suspend and Resume:
/* ULT A */
ABT_self_suspend();  /* Suspends self, requires external resume */

/* ULT B or external thread */
ABT_thread_resume(thread_handle_of_A);  /* Resume A */

Use for event-driven execution where external events trigger continuation.

Exit:
ABT_self_exit();  /* Terminate self immediately */

Cleaner than returning from function for early termination.

Yield To:
ABT_self_yield_to(target_thread);  /* Yield and schedule target next */

Direct control flow transfer. Use sparingly; breaks scheduler abstraction.

Performance Implications

Yielding Overhead:

  • ABT_self_yield() has context switch overhead

  • Don’t yield in tight loops

  • Balance between fairness and overhead

Progress Polling:

  • Good: Allows concurrent progress

  • Bad: Burns CPU if polling too frequently

  • Best practice: Yield after each poll

Common Pitfalls

Infinite Yield Loop:
/* Bad: Yields forever with no progress */
while (!condition) {
    ABT_self_yield();
}

Ensure some ULT can make progress towards the condition!

Forgetting to Yield in Polling:
/* Bad: Busy-wait, starves other ULTs */
while (!ready) {
    /* No yield - monopolizes CPU */
}

Always yield in polling loops.

Suspending Without Resume Path:
ABT_self_suspend();  /* Who will resume us? */

Ensure there’s a clear path for resume, or ULT hangs forever.

API Reference

Self Operations:

  • int ABT_self_yield()

    Voluntarily yield to scheduler. Returns when rescheduled.

  • int ABT_self_yield_to(ABT_thread thread)

    Yield and request specific ULT be scheduled next.

  • int ABT_self_suspend()

    Suspend self. Requires ABT_thread_resume() from another ULT.

  • int ABT_self_exit()

    Terminate self immediately.

  • int ABT_self_schedule(ABT_thread thread, ABT_pool pool)

    Schedule another ULT to a pool while running.

Related Operations:

  • int ABT_thread_resume(ABT_thread thread)

    Resume a suspended ULT.