
workflow.py
and place it in your project directory.
<workfspace-fqn>
with the workspace fqn which you can find on the UI.
- square: This is a simple task that squares a given integer.
- map_task: The map_task function is used to apply the square task to each element of the numbers list.
- my_map_workflow: This workflow demonstrates how to use the map task to process a list of numbers in parallel.
Map task on ui
Examples of using map tasks with different types of inputs
Map task with a single input
This is the most basic type of map task where you want to do an operation with just a single input for example in this task we are just returning the square of the input numberMap task with multiple inputs
Now lets say you want to pass multiple inputs which are constants and a list as an input to map tasks, so to accomplish this you can use a partial function from functools library, lets see it in the example.Note
It is important to remember that you cannot use list as an input to partial taskWarning
Please not that the length of all the lists should be same while passing muliple list as input in map function.Setting up the concurrency limit
You can set the concurrency limit for the map tasks, so that it limits the number of map tasks that can run in parallel to the specified batch size, for example if you set concurrency as3
, then only 3
map task will run at a given interval of time and the next 3
map task will run only when first 3 task are completed, so it runs in a batch of n
numbers of mapped tasks.
If the concurrency is not set then the workflow runs as many task concurrently as possible.You can set the concurrency using the below syntax:
3
, so a batch of 3
map task will run at a given period of time.