Skip to main content

Workflows

The Workflows menu allows you to schedule various tasks and execute them at a specific date and time. This allows the execution of different Extractors and Transformers, so that they are tightly chained together.

Creating a Workflow

To create a new Workflow, go to 'Data Pipelines' > 'Workflows', then click the green button labeled 'Create Workflow'. Then create one or more New Schedule intervals:

When choosing the start time and timezone you have to be aware of the following things:

  • There are certain timezones that have a Daylight Saving Time (DST)
  • If the timezone selected by you has a DST, the workflow frequency execution might be affected. Here is an example:

If the timezone doesn't use daylight saving time (i.e. (UTC) Coordinated Universal Time, (UTC+01:00) West Central Africa etc.), no changes are applied to the next run of the workflow. Otherwise, if the timezone uses daylight saving time, we check that the start time and the next run calculated at the previous step have the same offset to UTC. Consider the following scenario:

  • In winter, 00:01 UTC means 01:01 in Berlin
  • In summer, 00:01 UTC means 02:01 in Berlin

Imagine you have a setup with a workflow starting at 01:01 Berlin winter time. Because of a different offset, our Workflow engine applies a correction to set the next run at the same local time as what was in Berlin when the start time is set. Since the start time is in winter and represents 01:01 local time, we subtract one hour from the calculated next run in summer to start it also at 01:01 local time. And that means the next run is in UTC the previous day at 23:01.

In the above scenario, to avoid any extractors/transformers being run a day earlier than your expected execution, we recommend aligning your server time to UTC. If your server time is set to the Berlin timezone, then you are likely to experience that type of unalignment.

Then add one or more steps that should be executed:

  • Provide a meaningful Name for the Workflow
  • Optionally you can provide a detailed Description
  • Set the Type of the step using the drop-down menu
  • Provide the option that goes with the selected Type. This can be your Extractor, Transformer, Report or another name
  • Depending on the selected Step Type, you can provide an offset date. This value is used during the execution of that step. Typically this would be used for a From date offset (i.e. -1 for yesterday)
  • A To date offset can be provided for some step types (i.e. 0 for today)
  • In case multiple steps need to be executed in parallel, you may choose to remove the Wait checkbox.
  • Additional arguments can be sent to the step. This applies only to some Extractors and when executing a custom Command

You can delete a step using the bin icon in the right upper corner. To execute a workflow click the Run Now button:

Bear in mind that the selected date will be influenced by date offsets defined in any given step. To view historical Workflow results, click the Status tab.

Special Workflow Steps

Apart from adding Extractor, Transformer and Report steps, there are a few other special Workflow Step types:

Publish Report

This step type will trigger a Publish Report event, which can be consumed by the notification engine.

Evaluate Budget

This step type will evaluate all budgets, which can be consumed by the notification engine.

Execute

The Execute step enables you to execute an external command, like a script:

Call external commands or scripts

As an example: you could run a Powershell script to obtain some data from a special data source that Exivity Extractors are not able or allowed to connect to. This script could be executed in the following manner:

powershell.exe "D:\script\special.ps1" ((get-date).addDays(-1)).ToString("""yyyyMMdd""")

The above command calls the Powershell executable to run the special.ps1 script, with a dynamically generated parameter that is evaluated at run time. This particular example always provides yesterday's date in yyyyMMdd format as a parameter to the special.ps1 script. Many other variations and scripting languages are possible. Feel free to experiment.