Tasks, Actions & Agents

Tasks

Tasks are workflows that preprocess and queue up data for annotation or review and start the annotation server. They are created from a recipe, a Python function describing the workflow and its available settings. You can also think of tasks as an instance of a given recipe with a specific configuration. Tasks can be created in the UI or via the CLI and once started, annotators will be able to connect to them and create annotations.

To create a new task, you can navigate to TasksNew task in the UI or use tasks create on the command line. The available settings are automatically converted into a web form and CLI arguments.

To start a task, you can run tasks start with the name or ID of the created task:

This will start the annotation server and will allow human annotators and annotation agents to connect and create data. Each time an annotator connects to the interface, a new session is created and the annotations and sessions can be viewed in the app. Annotations are saved into the provided dataset, which can then be used by other tasks and actions, e.g. to review the data or train a model, or exported to a file.

Actions

Actions are workflows that execute any logic and exit. They’re created from a recipe, a Python function describing the workflow and its available settings. You can also think of actions as an instance of a given recipe with a specific configuration, similar to jobs running in a CI system like GitHub Actions. Actions can be created in the UI or via the CLI and have access to all resources on your cluster, so they can be used to train a model from your data, perform data analysis, trigger deployment or integrate Ellf with other tools and services.

To create a new action, you can navigate to ActionsNew action in the UI or use actions create on the command line. The available settings are automatically converted into a web form and CLI arguments.

To start an action, you can run actions start with the name or ID of the created action:

This will create a new execution, run the code and save all output to the logs. Actions can have many executions over time – for instance, for regular training runs, end-to-end testing, deployment or data import and export.

Agents

Agents are autonomous workers and annotators that can be assigned to tasks. They’re created from a recipe, a Python function describing the agent’s intended behavior, its available settings and the type of work it’s specialized for. Agents are typically powered by LLMs and can use models running on the cluster or access them via APIs. When assigned to an annotation task, an agent will run in the background and automatically create annotated data in its own dataset.

To create a new agent, you can navigate to AgentsNew agent in the UI or use agents create on the command line. The available settings are automatically converted into a web form and CLI arguments. If API keys are required, you can use Ellf’s built-in secrets feature.

To start an agent, you can run agents start with the name or ID of the created agent:

Once started, the agent will automatically discover its assigned task servers, fetch tasks, annotate them using the logic defined in the recipe, and submit the results. Each agent runs in its own session and creates annotations in the task’s dataset, just like a human annotator would. You can assign agents to tasks via TasksAssign agent in the UI or using agents assign on the CLI.

Agents support two optional settings that control their runtime behavior: poll_interval sets how frequently the agent checks for new tasks (in seconds), and idle_shutdown_seconds configures how long the agent will wait without tasks before shutting down automatically. These are defined as parameters in the agent recipe and can be configured when creating the agent. For details on building your own agent recipes, see the recipe development guide and the @agent_recipe API reference.

Workers

Task, actions and agents can have different cloud hardware requirements and may require a high-performance CPU or GPU to run or train machine learning models. When starting a task, action or agent, you can customize the worker class to be used so that the cluster can provision the required workers. For example, you can change the worker used by an action by clicking the arrow on the Start action button or by setting the --worker-class argument on the CLI.

By default, the cluster is set up to provide the workers base, small, medium and gpu. See the cluster documentation for more details on their specs and how to define custom workers.

Annotation sessions

Once a task is running, annotators with access to it can navigate to it in their dashboard and open the annotation UI. You’ll be able to see their work and monitor the activity as it happens. You can also click the Annotate now button on the task page to annotate yourself. A session counts every time a user opens the annotation UI, until they navigate away from it. This means that the same user can start multiple sessions over time and create annotations during those sessions.

The annotations tab lists the annotated examples with an inline preview. On click, it shows the example and its underlying JSON data, which depends on the data type and interface used. Editing an example will create a copy, save it from your user account and mark the edited example as disabled, so it’s excluded from any further processes. Once an example is added to the database, it can never be changed in place. This ensures that no data points are lost and you’ll always be able to compute accurate metrics like inter-annotator agreement.

Monitoring and observability

The monitoring module lets you check in on the status of your running tasks, actions and agents. It can report scores and potential problems for actions like model training, task state, progress and activity for annotation tasks and annotation agents.