API Reference

Invocation

This page provides details on invocation of data flows and monitoring.

Data flows are a collection of steps organized into a decision tree specifying which steps should be executed and under what conditions. A starting step is always defined within the data flow, and the data flow map contains the decision tree outlining subsequent steps, and which is executed in what order when a success, failure, or exception result is encountered in the previous (or first) step.

Example

To invoke a data flow, invoke its trigger. For example, if you have a trigger defined as follows:

{
    "GUID": "9c331d90-fe5e-41ac-9fac-91de6da2544b",
    "TenantGUID": "1e39e439-3242-44ea-802c-93b33fb0b6a8",
    "TriggerType": "HTTP",
    "Name": "My processing pipeline trigger",
    "HttpMethod": "POST",
    "HttpUrlPrefix": "/processor",
    "CreatedUtc": "2024-07-10T05:10:14.000000Z"
}

And a data flow defined using this specific trigger by GUID (e.g. processor):

{
    "GUID": "cb68c2de-05a3-4938-84c4-bd9badd77f63",
    "TenantGUID": "1e39e439-3242-44ea-802c-93b33fb0b6a8",
    "TriggerGUID": "9c331d90-fe5e-41ac-9fac-91de6da2544b",
    "StepGUID": "a1846a1f-e47c-4f0b-804e-68fcfc2c0059",
    "Name": "My processing pipeline data flow",
    "LogRetentionDays": 28,
    "CreatedUtc": "2024-07-10T05:10:14.000000Z"
}

Calling POST /processor against View Orchestrator (which by default listens on port 8501) would invoke the data flow, and pass the request body onto the first step, which is defined by GUID a1846a1f-e47c-4f0b-804e-68fcfc2c0059:

{
    "GUID": "a1846a1f-e47c-4f0b-804e-68fcfc2c0059",
    "TenantGUID": "1e39e439-3242-44ea-802c-93b33fb0b6a8",
    "Name": "Data processing pipeline step",
    "Runtime": "Dotnet8",
    "StepArchiveFilename": "./Steps/Csharp/View.ProcessingPipeline.zip",
    "StepEntrypointFilename": "View.ProcessingPipeline.dll",
    "StepEntrypointType": "View.ProcessingPipeline.ViewProcessingPipeline",
    "MD5Hash": "2E6A7043429B2B4B3C01954323133B29",
    "SHA1Hash": "8FDAAFC02AE4B1C1838250C7E3772EA8327114E2",
    "SHA256Hash": "927C9960B645422F5BC9DE177C394F889439DA05FECEE8B583E74B397D4A66D8",
    "DebugAssemblyLoad": true,
    "DebugWrapperScript": false,
    "DebugRequestData": false,
    "DebugResponseData": false,
    "ConsoleLogging": true,
    "ReferenceCount": 0,
    "CreatedUtc": "2024-07-10T05:10:14.000000Z",
    "TimestampFormat": "yyyy-MM-ddTHH:mm:ss.ffffffZ"
}

Within the data flow Map property, additional steps can be defined based on the resultant Status from the invocation of the first step, e.g. Success, Failure, or Exception.

{
    "GUID": "cb68c2de-05a3-4938-84c4-bd9badd77f63",
    "TenantGUID": "1e39e439-3242-44ea-802c-93b33fb0b6a8",
    "TriggerGUID": "9c331d90-fe5e-41ac-9fac-91de6da2544b",
    "StepGUID": "a1846a1f-e47c-4f0b-804e-68fcfc2c0059",
    "Name": "My processing pipeline data flow",
    "LogRetentionDays": 28,
    "Map": {
        "StepGUID": "a1846a1f-e47c-4f0b-804e-68fcfc2c0059",
        "Success": {
            "StepGUID": "0c63b524-5fc7-4cfb-8371-d1c238ba40b8",
            "Success": {
                "StepGUID": "fd9e8370-786e-464e-a420-00e3715dac85",
                "Success": null,
                "Failure": {
                    "StepGUID": "2fcf4f69-8d03-4c18-80fb-90c37550f369",
                    "Success": null,
                    "Failure": null,
                    "Exception": null
                },
                "Exception": null
            },
            "Failure": null,
            "Exception": null
        },
        "Failure": null,
        "Exception": null
    }
}

In this example, a call to POST /processor would invoke step a1846a1f-e47c-4f0b-804e-68fcfc2c0059. If no Map were defined, the result of this step would return directly to the caller.

In this case, however, there is a data flow map present. This map specifies that step a1846a1f-e47c-4f0b-804e-68fcfc2c0059, upon success, will invoke step 0c63b524-5fc7-4cfb-8371-d1c238ba40b8. Should this step succeed, it will invoke step fd9e8370-786e-464e-a420-00e3715dac85. Should this step succeed, it will simply return the result to the caller, as indicated by the null in the Success property. Should this step fail, step 2fcf4f69-8d03-4c18-80fb-90c37550f369 will be invoked, which will always return directly to the caller, as all three properties Success, Failure, and Exception are null.

Important if a step within a map has a null value in any of its Success, Failure, or Exception properties, when those conditions are encountered, the value is returned to the caller, and no further processing is performed.

Monitoring

When a data flow is invoked by trigger, a GUID is assigned to that individual invocation of the entire data flow, whether it is comprised of a single step or an entire decision tree of steps. The response returned to the caller will always include the following headers:

  • x-data-flow indicating the GUID of the data flow that was invoked
  • x-trigger indicating the GUID of the trigger that was matched
  • x-total-runtime indicating the total runtime in milliseconds, e.g. 494.2ms
  • x-request indicating the GUID of the data flow invocation

It is important to note the request GUID for the data flow invocation. Logs can be retrieved from the data flow request by calling GET /v1.0/tenants/[tenant-guid]/dataflows/[dataflow-guid]/logfile?request=[request-guid]. This logfile is a text file, in human-readable form. An example is shown below.


Data flow request log

| Start UTC               : 2024-10-29 14:54:45.000000Z UTC
| Data flow GUID and name : csharppublicip My Csharp public IP data flow
| Tenant GUID             : default
| Trigger details         : csharppublicip GET /test/csharp/publicip
| Request GUID            : f94167ef-36e2-4f89-b592-03d616dcdc7b
| Content type and length : (null) 0 bytes
| Headers                 : 7
  | Host: viewdemo
  | X-Real-IP: 172.29.0.13
  | Authorization: Bearer default
  | User-Agent: PostmanRuntime/7.42.0
  | Accept: */*
  | Postman-Token: 86eb43d6-0c5f-4a78-bd30-84346e7361a6
  | Accept-Encoding: gzip, deflate, br


Step execution event

| Time UTC                : 2024-10-29 14:54:45.220389Z UTC
| Data flow               : csharppublicip My Csharp public IP data flow
| Step GUID and name      : csharppublicip Csharp Public IP step
| Runtime                 : Dotnet8
| Archive filename        : ./Steps/Csharp/PublicIpStep.zip
| Entrypoint filename     : PublicIpStep.dll
| Entrypoint type         : PublicIpStep.PublicIpStep
| Binary directory        : /app/binaries/csharppublicip/


Dependency load event

| Time UTC                : 2024-10-29 14:54:45.238280Z UTC
| Requested assembly      : RestWrapper, Version=3.0.21.0, Culture=neutral, PublicKeyToken=null
| Requested version       : 3.0.21.0
| Assembly filename       : /app/binaries/csharppublicip/RestWrapper.dll

csharppublicip | 2024-10-29 14:54:45.273395Z [DEBUG] processing public IP request
csharppublicip | 2024-10-29 14:54:45.540045Z [DEBUG] success status 200 from http://ifconfig.me

Step response log

| End UTC                 : 2024-10-29 14:54:45.549184Z UTC
| Runtime                 : 328.83ms
| Result and status code  : Success 200
| Content type and length : text/plain 14 bytes
| Headers                 : 4
  | Date: Tue, 29 Oct 2024 14:54:44 GMT
  | Access-Control-Allow-Origin: *
  | Via: 1.1 google
  | Connection: close


Data flow response log

| End UTC                 : 2024-10-29 14:54:45.549619Z UTC
| Runtime                 : 379.92ms
| Result and status code  : Success 200
| Content type and length : text/plain 14 bytes
| Headers                 : 4
  | Date: Tue, 29 Oct 2024 14:54:44 GMT
  | Access-Control-Allow-Origin: *
  | Via: 1.1 google
  | Connection: close