Skip to content

Mantle 2.0 Guide: Building a DAG

This guide describes how to structure Directed Acyclic Graphs (DAGs) for Mantle Flow. It covers DAG structure, task groups, execution order, conditional execution, variable injection, parameterization, and invoking automation assets.

Create a DAG with DAG Builder

DAG Builder provides a visual editor for assembling playbook and script tasks, conditions, parallel paths, and task groups. You can also upload a custom Python DAG using the authoring reference below.

Configure the workflow

  1. Open Assets and select Open DAG Builder.
  2. In DAG summary, enter DAG name. Choose Group (optional) if the published DAG should be assigned to a group. Leave Schedule as Manual for operator-initiated Flows.
  3. Expand DAG options to review DAG ID, Required variables, and Default extra variables. The DAG ID follows the name when Use name automatically is enabled; use a unique ID for a new DAG.
  4. Drag tasks from Task library into the workflow, or use a task's add control. Select a step in the summary to configure it. Playbook and script tasks need their corresponding installed asset and any required inputs.
  5. Use Condition to choose a path based on an earlier task result, Parallel execution for independent paths, and Task Group to organize related steps.
  6. Review Builder, Graph, or DAG JSON as needed. Resolve entries marked Needs attention before publishing.

DAG Builder with an example workflow and its configuration

Required variables is a JSON mapping from input keys to display labels, for example {"site_name": "Site name"}. Operators supply these values when deploying a Flow. A task can also consume a compatible result from an earlier task.

Keep Mantle wrap up tasks enabled when the workflow should aggregate results, post workflow logs, and check for failures at the end.

Save a draft and publish

  1. Select Save draft to preserve unfinished work on the appliance.
  2. Reopen it from Assets → Drafts, using Open in DAG Builder. These drafts are separate from Plan drafts on the Dashboard.
  3. Select Publish DAG when the configuration is complete.
  4. Wait for asset installation and Airflow discovery to finish before selecting the DAG in a Flow. Publishing creates or updates an asset; it does not execute the workflow.

An installation failure leaves the saved draft available for correction. Successful installation removes the published draft snapshot; edits saved while installation is in progress remain available.

Edit an existing DAG

Use Edit DAG in the builder's library, or Actions → Edit in Flow Builder on a builder-created DAG asset. After editing, select Publish DAG, then choose Publish changes to update the asset or Publish as new DAG with a different DAG ID.

Custom Python uploads without saved builder JSON cannot be opened for visual editing. Update those files through asset upload instead. Import DAG JSON restores a configuration exported from the builder; it does not import arbitrary Python source.

Builder-created DAGs record their referenced playbooks and scripts automatically. For custom Python DAGs, declare or select dependencies as described in DAG required assets.

1. DAG Structure

Directed Acyclic Graph (DAG) overview

A Directed Acyclic Graph (DAG) is the core structural component of a Mantle Flow. The DAG defines the workflow's tasks, their execution order, and the dependencies between them.

By organizing automation logic into discrete steps, the DAG controls how those steps are sequenced and executed.

Within Mantle Flow, the DAG acts as the bridge between workflow logic and platform functionality by invoking asset execution, such as playbooks, scripts, and other supported automation components.

DAG file structure

A Mantle Flow DAG is organized into two major sections:

  1. Variable definition
  2. DAG definition block

Variable definition

This section defines workflow variables that are consumed during execution.

REQUIRED_VARS

A dictionary supplied by the DAG author that defines the variables required for workflow execution.

Format:

REQUIRED_VARS = {
    "variable_name": "display label",
}

TASK_LABELS

A dictionary supplied by the DAG author that defines human-readable labels for tasks within the DAG.

If a task is not included in this dictionary, Mantle Flow displays the task ID as the label.

Format:

TASK_LABELS = {
    "full_task_id": "task label",
}

LOG_LABELS

A dictionary supplied by the DAG author that defines human-readable labels for log output included in the Flow Summary.

DAG authors should only add labels for tasks that are configured to generate output for the Flow Summary.

If a task is not included in this dictionary, Mantle Flow uses the task ID as the log label.

Format:

LOG_LABELS = {
    "full_task_id": "log label",
}

The following example shows the three dictionaries together:

Example DAG variable and label definitions

DAG definition block

The DAG definition block is the primary section where DAG authors define workflow behavior.

Within this block, authors:

  • Define tasks
  • Create task groups
  • Establish execution order
  • Configure task dependencies

This section represents the main workflow logic.

DAG metadata

When defining a new DAG, the author must provide metadata used by Apache Airflow to configure workflow behavior.

Example parameters include:

dag_id

A unique identifier for the DAG within the Airflow instance. This value must be unique and cannot contain whitespace.

start_date

Defines the earliest possible execution date for the DAG. Mantle Flow does not rely on scheduled DAG execution, so this value is typically set to:

days_ago(1)

schedule_interval

Mantle Flow is designed for manual, run-based workflow execution and does not currently support scheduled DAG runs. This value should therefore be left unset or configured appropriately for manual execution.

catchup

Controls whether Airflow attempts to run historical DAG executions. Since Mantle Flow does not use scheduled runs, this should be set to:

False

default_args

Arguments passed to Airflow that define task behavior and execution characteristics.

Typical configuration:

default_args = {
    "owner": "airflow",
    "retries": 0,
    "depends_on_past": False,
    "trigger_rule": TriggerRule.ALL_SUCCESS,
}

trigger_rule

This parameter defines the default execution behavior for downstream tasks. It determines how task dependencies are evaluated before a task executes.

This value may be modified by the DAG author if alternative dependency behavior is required. See Task execution order for additional details.

tags

A list of tags used to categorize the DAG within Airflow and Mantle Flow.

render_template_as_native_obj

This should always be set to True because it is required to use script assets.

The following example shows the metadata block:

Example DAG metadata

Task definition

Tasks within a DAG are defined as Airflow operators, which represent individual units of work executed during workflow execution.

To define a task, the DAG author calls the constructor for the appropriate operator and supplies the required parameters.

One common example is the PythonOperator, which allows a task to execute a Python function defined within the DAG.

First define the function outside of the DAG definition block:

def helloWorld():
    print('Hello World!')

Then define the task within the DAG definition block:

hello_world = PythonOperator(
    task_id="hello_world",
    python_callable=helloWorld,
)

The resulting task object can then be used to define task dependencies and execution order. See Task execution order for additional details.

DAG discovery in Mantle Flow

When a DAG is uploaded to Mantle Flow, the file is processed and placed in a directory monitored by Apache Airflow for DAG discovery.

Airflow periodically scans this directory to identify new or updated DAG files. The polling interval for this process is approximately 30 seconds.

Once Airflow discovers and processes the DAG, the workflow becomes available as a Flow option in the Mantle Flow Deployment Wizard for any groups to which the DAG has been assigned.

2. Task Groups

Task Group overview

Task Groups are an optional feature that allows DAG authors to logically organize related tasks within a Flow. By grouping tasks together, complex Flows can be structured in a more readable and maintainable format.

Grouping tasks is particularly useful for workflows that contain many steps or multiple phases of automation, such as validation, configuration, and verification stages.

Task Groups do not change execution behavior by themselves. Instead, they provide a way to visually and logically organize Flow components while preserving the dependencies defined between individual tasks. At runtime, Airflow does not factor task groups into task flow evaluation. When evaluation occurs between tasks belonging to separate groups, the execution policies of those specific tasks are enforced.

Logical grouping of tasks

Tasks that perform related functions can be grouped together within a Task Group. Examples of potential grouping categories include, but are not limited to:

  • Flow setup tasks
  • Configuration tasks
  • Validation or health check tasks
  • Cleanup and/or reporting tasks

By organizing tasks into groups, DAG authors can separate different Flow stages and make the Flow easier to understand and maintain.

Improving DAG readability

Large Flows may contain dozens or hundreds of tasks. Without organization, these Flows can become difficult to interpret.

Task Groups help improve readability by allowing related tasks to be visually collapsed or expanded when viewing the DAG structure. This allows DAG authors and operators to quickly understand the high-level structure of the automation without needing to examine every individual task.

Creating Task Groups

Task Groups can be defined using the with TaskGroup functionality provided by Airflow. The DAG author must supply the group name and a tooltip value.

# # # ------- HelloWorld Task Group ---------------------------------------
with TaskGroup("hello_group", tooltip="Workflow Setup") as hello_group:
    # Create the first hello_group task using the example provided above
    hello_world = PythonOperator(
        task_id="hello_world",
        python_callable=helloWorld,
    )

    # Create a second duplicate task to demonstrate flow definition
    # Note: task_id values must be unique across the entire DAG
    hello_world_again = PythonOperator(
        task_id="hello_world_again",
        python_callable=helloWorld,
    )

    # Define serial task flow. Execute hello_world, if successful then execute
    # hello_world_again. See Task execution order for more details.
    hello_world >> hello_world_again

Mantle Flow representation

Within Mantle Flow, Task Groups improve visibility when monitoring a Flow's progress using the Flow Diagram. Groups are visibly separated and can be collapsed for a simplified view of a Flow's progress.

Task Groups in a Flow Diagram

3. Task Execution Order

Execution order overview

Task execution order within a DAG is determined by the dependencies defined between tasks. These dependencies specify which tasks must complete before other tasks are allowed to begin.

By defining upstream and downstream relationships, DAG authors can control the sequence in which automation steps are executed.

Upstream and downstream tasks

Each task within a DAG may have one or more upstream or downstream dependency relationships.

  • Upstream tasks must complete before the current task can execute.
  • Downstream tasks depend on the current task and will execute afterward.

These relationships form the directed structure of the DAG and define the workflow's execution path.

Dependency operators

In Airflow, task dependencies are typically defined using dependency operators.

Common operators include:

task_a >> task_b

This indicates that task_a must complete before task_b executes.

Similarly:

task_b << task_a

This defines the same dependency using reverse syntax.

These operators allow DAG authors to quickly establish execution order between tasks.

Parallel execution

If multiple tasks share the same upstream dependency but do not depend on each other, they may execute in parallel.

Example:

task_a >> [task_b, task_c]

In this scenario:

  • task_b and task_c both depend on task_a
  • Once task_a completes, both tasks may execute concurrently

Parallel execution allows workflows to complete more efficiently by performing independent operations simultaneously.

4. Conditional Execution

Conditional execution overview

Conditional execution allows workflows to dynamically determine which tasks should run based on runtime conditions.

Instead of executing every task sequentially, conditional logic can direct the workflow down different execution paths depending on workflow inputs, task results, or environment conditions.

This capability enables workflows to support multiple operational scenarios within a single DAG definition.

Branching tasks

Branching tasks are used to determine which execution path a workflow should follow.

A branching task evaluates a condition and selects one or more downstream tasks to execute. Tasks that are not selected are automatically marked as Skipped.

Branching allows Flows to support conditional operations such as:

  • Executing different automation paths based on environment type
  • Performing validation checks before configuration steps

Conditional paths

Conditional paths define the possible execution routes that a Flow may follow.

For example, a Flow may branch based on the value of a runtime variable:

  • Production configuration path
  • Development configuration path

Or on the status of an upstream task:

  • Success
  • Failure

Each path may contain a different set of tasks appropriate for the selected condition.

Skipped tasks

When branching occurs, tasks that are not part of the selected execution path are marked as Skipped.

Skipped tasks do not execute but remain visible in the workflow execution record. This allows users to see which portions of the Flow were bypassed during execution.

This status must be taken into account when defining downstream task trigger rules because a default trigger rule of ALL_SUCCESS will not match Skipped.

Implementing branching tasks in Mantle Flow

Mantle Flow leverages a custom Airflow plugin to provide a helper function that allows DAG authors to easily evaluate the status of upstream tasks and make routing decisions accordingly.

The BranchPythonOperator can be used to execute this helper function and is structured similarly to PythonOperator with two primary deviations.

op_kwargs

The choose_branch helper function requires three arguments:

  • upstream_task_id | String
  • success_task_ids | List
  • failure_task_ids | List

These arguments must be the full task IDs, including any Task Groups of which they are members.

trigger_rule

If the upstream task is not configured to generate a graceful failure, the trigger rule of the BranchPythonOperator task must be updated to a value that will allow execution, such as all_done.

BranchPythonOperator based on the previous example:

branch_on_hello_world_result = BranchPythonOperator(
    task_id="branch_on_hello_world_result",
    python_callable=choose_branch,
    op_kwargs={
        'upstream_task_id': hello_group.hello_world',
        'success_task_ids': [hello_group.skip_hello_again'],
        'failure_task_ids': ['hello_group.hello_world_again']
    },
    provide_context=True,
    trigger_rule="all_done"
)

This operator evaluates the status of the hello_world task. If the task failed, hello_world_again is executed; otherwise, the replay is skipped using a dummy task.

5. Variable Injection

Variable injection overview

Variable injection refers to the process by which runtime values are provided to a Flow when it is executed.

These variables allow Flows to receive input parameters that influence automation behavior without requiring modifications to the DAG itself.

Within Mantle Flow, variables are collected during Flow deployment and injected into the runtime configuration for use by any associated automation assets.

Sources of runtime variables

Runtime variables may originate from several sources, including:

  • Variables entered manually during Flow deployment
  • Uploaded JSON variable files
  • Configuration objects supplied during deployment, such as when rebuilding a historical Flow

These inputs allow users to provide the values required for Flow execution.

Runtime variable input form

Mantle Flow variable handling

During Flow execution, Mantle Flow processes user-provided variables and injects them into an Ansible inventory file that is accessible within the Mantle Flow runtime environment. For non-playbook asset types, variables are injected into the scripts' runtime environment as environmental variables.

Currently, variables are not directly accessible within the context of the DAG. However, they can bubble up to the DAG level when explicitly passed through playbook output.

Example sample-inv.yml:

{
  "all": {
    "vars": {
      "ansible_user": "administrator",
      "ansible_password": "password123",
      "subnet_mask": "255.255.255.0",
      "domain_user": "admin@mantle.lab",
      "domain_pass": "password123",
      "domain_name": "MANTLE.LAB",
      "default_gateway": "192.168.0.1",
      "vcenter_server": "svcsa.mantle.lab",
      "vcenter_username": "administrator@vsphere.local",
      "vcenter_password": "password123"
    },
    "children": {
      "esxi": {
        "hosts": {
          "esxi1": {
            "ansible_host": "192.168.1.2",
            "ansible_user": "root",
            "ansible_password": "password123",
            "ansible_become_password": "password123"
          },
          "esxi2": {
            "ansible_host": "192.168.1.3",
            "ansible_user": "root",
            "ansible_password": "password123",
            "ansible_become_password": "password123"
          }
        },
        "vars": {
          "ansible_python_interpreter": "/usr/bin/python3",
          "ansible_connection": "ssh"
        }
      },
      "vcenter": {
        "hosts": {
          "svcsa": {
            "ansible_host": "192.168.1.1",
            "ansible_user": "administrator@vsphere.local",
            "ansible_password": "password123"
          }
        },
        "vars": {
          "vcenter_hostname": "svcsa",
          "vcenter_ip": "192.168.1.1",
          "vcenter_username": "administrator@vsphere.local",
          "vcenter_password": "password123"
        }
      },
      "lab_dc": {
        "hosts": {
          "DC-1": {
            "ansible_host": "192.168.0.10",
            "target_cpu": 2,
            "target_memory": 4,
            "target_disks": [90]
          }
        }
      },
      "WEB-1": {
        "hosts": {
          "WEB": {
            "ansible_host": "192.168.0.12",
            "target_cpu": 2,
            "target_memory": 4,
            "target_disks": [90]
          }
        }
      }
    }
  }
}

Mantle configuration object

Mantle utilizes a standardized configuration model tailored to the deployment of tactical kits. This configuration model is heavily utilized by the Mantle Build capability during deployment of an environment. Mantle Flow can utilize and expand upon the configuration generated by a Mantle Build to perform additional, post-deployment tasking. This allows for a seamless transfer of variables between Mantle components.

When not utilizing an externally generated configuration, Mantle Flow uses a dedicated object within the configuration model to pass through any variables required by a Flow.

6. Parameterization

Parameterization overview

Parameterization in Mantle Flow allows Flow tasks to dynamically consume and pass data during execution. Rather than relying solely on static input variables, tasks may exchange data at runtime to influence downstream behavior.

This enables Flows to adapt based on execution results, allowing for more flexible and context-aware automation.

Inter-task data sharing

Within Apache Airflow, inter-task communication is handled through a mechanism known as XCom (cross-communication).

XCom provides a way for one task to store data during execution and make that data available to downstream tasks within the same Flow.

In Mantle Flow, XCom is used as the primary mechanism for passing data between tasks.

Automation asset output handling

When an automation asset such as an Ansible playbook is executed, its output is captured by the Mantle Flow backend.

Relevant output data is then returned to the DAG and stored within XCom. This allows the results of one task to be programmatically accessed by subsequent tasks within the workflow.

Custom Mantle Flow Airflow plugins perform a similar function for script assets.

Generating output using Ansible playbooks

Once an Ansible playbook completes execution, Mantle Flow scans the playbook output for a designated return object. This object is extracted and passed back to the DAG for use in downstream tasks.

Within Mantle Flow, playbooks must define and return a variable named playbook_return_object, which must be serialized to JSON. DAG authors are responsible for defining the contents of this object, converting it to JSON format, and printing it to standard output using a debug task.

If a workflow does not require output data, an empty object ({}) may be returned to satisfy this requirement.

Although Mantle Flow expects a single return object, it is standard practice to structure this object with two child components:

playbook_return_object

  • Intended for passing data between workflow tasks
  • Consumed by downstream DAG tasks via XCom
  • Has no enforced schema; authors may structure this data as needed

log_return_object

  • Intended for use in the Flow Summary
  • Provides a simplified or human-readable summary of playbook execution
  • May include key results, environment details, or other relevant output
  • May be left empty if not required

Implementation example:

---
- name: Hello World
  hosts: localhost
  gather_facts: no
  tasks:
    - name: Print hello world
      debug:
        msg: "{{ hello_message }}"

    - name: Log
      set_fact:
        playbook_return_object:
          playbook_return_object:
            hello_message: "{{ hello_message }}"
          log_return_object: "Your hello message was: {{ hello_message }}"

    - name: Convert to json
      set_fact:
        playbook_return_object: "{{ playbook_return_object | to_json }}"

    - name: Print
      debug:
        var: playbook_return_object

Storing output in XCom

A Mantle Flow Airflow plugin includes an HttpSensor helper function. When invoked as a task, this HttpSensor periodically polls the Mantle Flow backend for Ansible playbook status updates.

Once a playbook reports completion, Mantle Flow extracts the return object and returns it to the HttpSensor under the key json_stdout. The HttpSensor task extracts this key and pushes playbook_return_object and log_return_object to XCom.

The DAG author is not required to take any action to ensure these objects are stored.

Accessing stored data

To pass data between tasks, DAG authors must retrieve values stored in XCom at runtime by using the Mantle-provided task_output helper function, which accepts a task object as a parameter.

This is accomplished by accessing the task instance (ti) object and calling the xcom_pull method with the appropriate task ID and key. The retrieved data can then be passed to downstream tasks, such as Ansible playbooks, through parameters like extra_vars.

The xcom_pull method is used to access data produced by a previous task. This requires:

  • The task ID of the upstream task
  • The key under which the data was stored

If no data is found, a default value, such as an empty object, may be provided to prevent runtime errors.

DAG authors do not need to concern themselves with this process and can instead use the task_output helper.

Passing data to a playbook

Once retrieved, the data can be injected into a downstream task as part of its runtime configuration.

Example:

hello_bash = BashScriptOperator(
    task_id='hello_bash',
    filename='hello_world_xcom.sh',
    vars={"hello_var": task_output(wait_hello_world)},
)

Passing XCom data to a task

Example (legacy):

parse_hello_output = create_trigger_task(
    "parse_output.yml",
    "parse_hello_output",
    extra_vars={
        "hello_output": "{{ ti.xcom_pull(task_ids='hello_group.wait_hello_world', key='playbook_return_object') or {} }}"
    },
)

Legacy XCom data example

In these examples:

  • The task hello_group.wait_hello_world produces a return object.
  • The sensor task (wait_hello_world) stores the return object, not the trigger task.
  • The value stored under the key playbook_return_object is retrieved from XCom.
  • The result is assigned to the variable hello_output or hello_var.
  • This variable is passed to the playbook as part of extra_vars or vars, depending on the asset type.

7. Invoking Automation Assets

Overview

Executing automation assets within a Mantle Flow workflow requires interfacing with the Mantle runtime environment. Rather than invoking playbooks or scripts directly, DAG authors use helper functions provided by the Mantle Flow Airflow plugins to initiate and monitor asset execution.

These helper functions abstract the underlying execution process and ensure that automation assets are executed consistently within the Mantle platform.

Helper functions

The Mantle Flow DAG template provides two primary helper functions for invoking automation assets:

create_trigger_task

  • Initiates execution of an automation asset (currently only Ansible playbooks)
  • Submits the task to the Mantle execution environment
  • Passes any required runtime parameters, such as extra_vars

create_sensor_task

  • Monitors the status of a previously triggered automation task
  • Periodically queries Mantle Flow for execution status
  • Waits until the task completes before allowing downstream execution

These helper functions work together to manage the full lifecycle of an Ansible automation task.

Execution pattern

Automation asset invocation follows a two-step pattern:

  1. Trigger task: Initiates execution of the automation asset.
  2. Sensor task: Waits for execution to complete and retrieves the result.

By separating execution and monitoring into two tasks, Mantle Flow enables better visibility into workflow state and allows task status to be tracked independently.

The sensor task is typically named using the convention:

wait_<trigger_task_name>

This naming pattern helps maintain consistency and readability within the DAG.

Example:

hello_world = create_trigger_task("hello_world.yml", "hello_world")
wait_hello_world = create_sensor_task("hello_world")

Automation asset trigger and sensor example

In this example:

  • The trigger task (hello_world) initiates execution of the playbook.
  • The sensor task (wait_hello_world) monitors execution until completion.

Dependency considerations

When defining task execution order, downstream tasks should almost always depend on the sensor task rather than the trigger task.

This ensures that:

  • The automation asset has fully completed before subsequent tasks execute
  • Output data is available for retrieval, such as through XCom
  • Task status accurately reflects execution results

Example:

hello_world >> wait_hello_world >> parse_hello_output >> wait_parse_hello_output

Recommended trigger and sensor dependency chain

Since parse_hello_output relies on the output of wait_hello_world, it should be executed after wait_hello_world has completed execution.

8. Mantle Flow Airflow Plugin Reference

Mantle Flow provides two plugin modules for DAG authors:

  • mantle_operators: operators for running supported automation assets and communicating with the Mantle backend
  • mantle_functions: helper functions for creating Mantle tasks, passing task output, branching, compiling summaries, and sending notifications

These plugins were created to standardize the execution of automation assets in the Mantle ecosystem and simplify the DAG authorship process. The operators and functions below describe the shared author-facing interface in the current Mantle Flow plugin assets. If the template bundled with a specific deployment uses a different signature, follow that template.

Importing the plugins

Most DAGs import script operators directly and use helper functions for playbook execution:

from mantle_functions import (
    aggregate_results,
    choose_branch,
    create_sensor_task,
    create_trigger_task,
    fail_if_upstream_failed,
    post_logs,
    send_notification,
    task_output,
    write_log,
)
from mantle_operators import (
    AnsibleOperator,
    BashScriptOperator,
    PowerShellScriptOperator,
    PythonScriptOperator,
)

The plugin modules are already available in the Mantle Flow runtime. Authors should use the provided operators and functions rather than recreating the backend request and polling logic in each DAG.

Automation asset operators

These operators run uploaded assets in the Mantle execution environment. Each operator creates an Airflow task and accepts a filename and task_id. The vars argument supplies task inputs.

Operator Use
AnsibleOperator Run an uploaded Ansible playbook
BashScriptOperator Run an uploaded Bash script
PowerShellScriptOperator Run an uploaded PowerShell script
PythonScriptOperator Run an uploaded Python script

For scripts, values supplied through vars are exposed to the script at runtime. The operator also produces a task output that can be passed to a downstream task with task_output.

Example:

hello_bash = BashScriptOperator(
    task_id="hello_bash",
    filename="hello_world.sh",
    vars={"hello_var": "Hello from Mantle Flow"},
)

hello_powershell = PowerShellScriptOperator(
    task_id="hello_powershell",
    filename="hello_world.ps1",
    vars={"hello_var": task_output(hello_bash)},
)

Script return objects

Script operators do not infer a return object from arbitrary standard output. A Bash, PowerShell, or Python asset must set the Mantle return environment variables during execution.

The vars argument passed to an operator supplies inputs to the script. The variables below are outputs produced by the script:

Environment variable Purpose Airflow result
MANTLE_RETURN_OBJECT Machine-readable data for downstream tasks Published as the script task's primary output under docker_return_object
MANTLE_LOG_OBJECT Human-readable data for the Flow Summary Published under log_return_object for aggregate_results

Return-object lifecycle

For Bash, PowerShell, and Python operators:

  1. The script builds its result values.
  2. The script serializes those values to strings and assigns them to MANTLE_RETURN_OBJECT and, when needed, MANTLE_LOG_OBJECT.
  3. The Mantle wrapper captures the variables when the script exits.
  4. Mantle Flow publishes the return object to XCom and makes the log object available to Flow Summary aggregation.

Use JSON for structured results. A plain string is also accepted, but downstream tasks will receive it as text rather than as a dictionary or list. Standard output is still useful for logging, but printing a value does not make it the task return object.

Bash example

MANTLE_RETURN_OBJECT='{"status":"ok","items":["host-1","host-2"]}'
MANTLE_LOG_OBJECT='{"summary":"Validation complete","host_count":2}'

PowerShell example

$returnObject = @{
    status = "ok"
    items = @("host-1", "host-2")
}

$logObject = @{
    summary = "Validation complete"
    host_count = 2
}

$env:MANTLE_RETURN_OBJECT = $returnObject | ConvertTo-Json -Compress
$env:MANTLE_LOG_OBJECT = $logObject | ConvertTo-Json -Compress

Python example

import json
import os

return_object = {
    "status": "ok",
    "items": ["host-1", "host-2"],
}

log_object = {
    "summary": "Validation complete",
    "host_count": 2,
}

os.environ["MANTLE_RETURN_OBJECT"] = json.dumps(return_object)
os.environ["MANTLE_LOG_OBJECT"] = json.dumps(log_object)

Passing a script result to another task

Because the script operator publishes its primary result under docker_return_object, downstream tasks can use task_output without manually pulling XCom values:

validate_hosts = PythonScriptOperator(
    task_id="validate_hosts",
    filename="validate_hosts.py",
)

configure_hosts = BashScriptOperator(
    task_id="configure_hosts",
    filename="configure_hosts.sh",
    vars={"validation": task_output(validate_hosts)},
)

validate_hosts >> configure_hosts

The MANTLE_LOG_OBJECT value is published separately under log_return_object. Use aggregate_results to collect those values for a Flow Summary.

BaseDockerOperator is the shared implementation behind these operators. DAG authors normally use the asset-specific subclasses and do not need to instantiate the base class directly.

Playbook trigger and sensor operators

Mantle Flow also provides backend-aware operator classes:

  • AnsiblePlaybookOperator submits a playbook run to the Mantle backend.
  • AnsibleSensor polls the backend for the result of a previously submitted playbook.
  • PostLogsOperator submits the compiled workflow log data for the Flow Summary.
  • SendNotificationOperator submits a notification to a Mantle group.

DAG authors generally create these tasks through the helper functions described below. This keeps task IDs, backend request fields, authentication, polling, and XCom behavior consistent.

Standard Airflow operators used with Mantle plugins

Mantle Flow plugins are used alongside standard Airflow operators:

Airflow operator Typical Mantle Flow use
PythonOperator Run DAG-local Python logic, such as preparing variables or compiling results
BranchPythonOperator Select a success, failure, or remediation path
EmptyOperator Represent a branch endpoint or merge point
TaskGroup Organize related tasks in the Flow Diagram
TriggerRule Control execution when upstream tasks succeed, fail, or are skipped

Helper function reference

task_output

Use task_output(task, key=None) to pass a task's XCom output to another operator. The helper returns an Airflow reference that is resolved when the DAG runs.

By default, the helper uses the operator's primary return-object key. Use key when the task publishes a named XCom value.

hello_bash = BashScriptOperator(
    task_id="hello_bash",
    filename="hello_world.sh",
)

hello_python = PythonScriptOperator(
    task_id="hello_python",
    filename="hello_world.py",
    vars={"hello_var": task_output(hello_bash)},
)

custom_value = PythonOperator(
    task_id="custom_value",
    python_callable=produce_value,
)

use_custom_value = PythonScriptOperator(
    task_id="use_custom_value",
    filename="consume_value.py",
    vars={"value": task_output(custom_value, key="result")},
)

For Mantle playbook sensors, task_output(wait_task) resolves the playbook return object. For script operators, it resolves the script return object.

create_trigger_task

create_trigger_task(playbook, task_id, extra_vars={})

Creates the trigger task for an Ansible playbook.

  • playbook: the uploaded playbook filename
  • task_id: the unique Airflow task ID for the trigger
  • extra_vars: optional playbook inputs; values may include templated values or XCom references

The helper submits the playbook to Mantle Flow and returns an AnsiblePlaybookOperator. It does not wait for the playbook to finish.

configure = create_trigger_task(
    "configure_hosts.yml",
    "configure_hosts",
    extra_vars={"host_config": task_output(prepare_hosts)},
)

create_sensor_task

create_sensor_task(task_id, strict=True)

Creates the sensor that waits for a playbook started by create_trigger_task.

The sensor task is named wait_<task_id>. It polls Mantle Flow until the backend reports success or failure, then stores the playbook status and returned objects in XCom:

  • status
  • playbook_return_object
  • log_return_object

When strict=True, a failed playbook causes the sensor task to fail. Set strict=False when the DAG must inspect the failure and branch or remediate explicitly.

Always place downstream dependencies after the sensor task:

trigger = create_trigger_task("configure_hosts.yml", "configure_hosts")
wait_for_configure = create_sensor_task("configure_hosts")

trigger >> wait_for_configure

choose_branch

choose_branch(upstream_task_id, success_task_ids, failure_task_ids, **kwargs)

Use this callable with BranchPythonOperator to route the Flow according to a sensor's status XCom value.

  • upstream_task_id: the full task ID of the sensor being evaluated
  • success_task_ids: task ID or task IDs to run when the sensor reports success
  • failure_task_ids: task ID or task IDs to run when the sensor reports failure

Task IDs must include their Task Group prefix when the task is inside a group. The branch task should usually use trigger_rule="all_done" so it can run after a non-strict sensor reports a failed backend task.

wait_for_validation = create_sensor_task("validate_hosts", strict=False)

branch = BranchPythonOperator(
    task_id="branch_on_validation",
    python_callable=choose_branch,
    op_kwargs={
        "upstream_task_id": "wait_validate_hosts",
        "success_task_ids": ["configure_hosts"],
        "failure_task_ids": ["remediate_hosts"],
    },
    trigger_rule="all_done",
)

wait_for_validation >> branch

post_logs

post_logs(task_id, logs={})

Creates a task that sends compiled workflow log data to Mantle for the Flow Summary.

Use post_logs after a task that gathers log_return_object values, commonly a PythonOperator whose callable is aggregate_results.

aggregate_results

aggregate_results(LOG_LABELS, **kwargs)

Collects log_return_object values from completed tasks and assembles them into the structure used by the Flow Summary.

LOG_LABELS is a mapping from full task IDs to human-readable labels. Tasks that do not produce a log return object are omitted. When tasks are inside a Task Group, use the full ID in the form group_id.task_id.

Example:

LOG_LABELS = {
    "validation.wait_validate_hosts": "Validation Results",
}

compile_logs = PythonOperator(
    task_id="compile_logs",
    python_callable=aggregate_results,
    op_kwargs={"LOG_LABELS": LOG_LABELS},
    trigger_rule="all_done",
)

post_workflow_logs = post_logs(
    "post_workflow_logs",
    logs="{{ ti.xcom_pull(task_ids='compile_logs') or {} }}",
)

compile_logs >> post_workflow_logs

send_notification

send_notification(task_id, subject="", msg="", group="group_demo")

Creates a task that sends a notification to a Mantle group.

  • task_id: unique Airflow task ID
  • subject: notification subject
  • msg: notification body
  • group: Mantle group ID that receives the notification

Notifications can be placed at the start, end, or failure path of a Flow.

notify_completion = send_notification(
    "notify_completion",
    subject="Validation complete",
    msg="The validation Flow completed. Review Track Flow for details.",
    group="group_demo",
)

Utility functions

The plugin also includes smaller utilities that can be used in DAG-local tasks:

Function Purpose
fail_if_upstream_failed Raises an Airflow failure so a final check task fails when one or more upstream tasks failed
write_log Writes a message to the Mantle Flow log at the plugin's notice level
dump_all_xcoms Prints XCom values for the current DAG run while troubleshooting

These functions do not create an operator by themselves. Pass them to a PythonOperator when a task is needed.

Common plugin patterns

Use the following patterns when building a Mantle Flow DAG:

  1. Pair every create_trigger_task call with a create_sensor_task call.
  2. Make downstream work depend on the sensor, not only the trigger.
  3. Use strict=False only when the DAG has an intentional failure or remediation path.
  4. Pass task results with task_output instead of hard-coding values or manually constructing XCom pulls.
  5. Use full Task Group prefixes in choose_branch and LOG_LABELS.
  6. Use aggregate_results and post_logs when the Flow should expose a consolidated Flow Summary.