Overview

The View Crawl Operation management system provides comprehensive monitoring and tracking of crawl plan executions. Crawl operations serve as metadata containers that capture detailed information about each invocation of a crawl plan, including execution status, performance metrics, object counts, and timing information for complete visibility into data ingestion processes.

Key Features

Execution Tracking: Complete monitoring of crawl plan invocations and execution status
Performance Metrics: Detailed statistics on objects processed, bytes transferred, and processing times
Status Management: Real-time tracking of operation states including enumeration, retrieval, and completion
Timing Information: Comprehensive timestamps for all operation phases and milestones
Error Monitoring: Tracking of failed objects and error conditions during crawl execution
Resource Utilization: Monitoring of processing endpoints and cleanup operations
Historical Data: Retention of operation metadata for analysis and troubleshooting
Integration Support: Seamless integration with crawl plans, schedules, and data repositories

Supported Operations

Read: Retrieve individual crawl operation metadata and execution details
Enumerate: List all crawl operations with pagination support
Read All: Retrieve all crawl operations in the tenant
Retrieve Enumeration: Access detailed enumeration data for specific operations
Stop: Terminate running crawl operations
Delete: Remove crawl operation records and associated metadata
Existence Check: Verify crawl operation presence without retrieving details

API Endpoints

Crawl operations are managed via the Crawler server API at [http|https]://[hostname]:[port]/v1.0/tenants/[tenant-guid]/crawloperations

Supported HTTP Methods: GET, HEAD, DELETE

Important: All crawl operation operations require appropriate authentication tokens.

Crawl Operation Object Structure

Crawl operation objects contain comprehensive metadata about crawl plan executions. Here's the complete structure:

{
    "GUID": "9ced1af3-2e19-4cd6-81ce-a8c90a9ed32d",
    "TenantGUID": "default",
    "CrawlPlanGUID": "e9a7d61e-7cbd-46e4-9956-533e22008978",
    "CrawlScheduleGUID": "oneminute",
    "CrawlFilterGUID": "default",
    "DataRepositoryGUID": "1a56c067-9e6d-4f7b-85bf-eb6b04aeda3f",
    "MetadataRuleGUID": "example-metadata-rule",
    "EmbeddingsRuleGUID": "crawler-embeddings-rule",
    "ProcessingEndpoint": "http://nginx-orchestrator:8501/processor",
    "CleanupEndpoint": "http://nginx-orchestrator:8501/processor/cleanup",
    "Name": "Alienware CIFS (started 2024-10-25T22:29:31 UTC)",
    "ObjectsEnumerated": 123,
    "BytesEnumerated": 61052490,
    "ObjectsAdded": 0,
    "BytesAdded": 0,
    "ObjectsUpdated": 0,
    "BytesUpdated": 0,
    "ObjectsDeleted": 0,
    "BytesDeleted": 0,
    "ObjectsSuccess": 0,
    "BytesSuccess": 0,
    "ObjectsFailed": 2,
    "BytesFailed": 16384,
    "State": "Success",
    "CreatedUtc": "2024-10-25T22:29:31.000000Z",
    "StartUtc": "2024-10-25T22:29:31.000000Z",
    "StartEnumerationUtc": "2024-10-25T22:29:31.000000Z",
    "StartRetrievalUtc": "2024-10-25T22:29:39.000000Z",
    "FinishEnumerationUtc": "2024-10-25T22:29:39.000000Z",
    "FinishRetrievalUtc": "2024-10-25T22:29:39.000000Z",
    "FinishUtc": "2024-10-25T22:29:39.000000Z",
    "AdditionalData": "No objects detected during enumeration"
}

Field Descriptions

GUID (GUID): Globally unique identifier for the crawl operation object
TenantGUID (GUID): Globally unique identifier for the tenant
CrawlPlanGUID (GUID): Globally unique identifier for the associated crawl plan
CrawlScheduleGUID (GUID): Globally unique identifier for the crawl schedule
CrawlFilterGUID (GUID): Globally unique identifier for the crawl filter
DataRepositoryGUID (GUID): Globally unique identifier for the data repository
MetadataRuleGUID (GUID): Globally unique identifier for the metadata rule
EmbeddingsRuleGUID (GUID): Globally unique identifier for the embeddings rule
ProcessingEndpoint (string): URL endpoint for processing new and changed objects
CleanupEndpoint (string): URL endpoint for processing deleted objects
Name (string): Display name for the crawl operation
ObjectsEnumerated (integer): Total number of objects enumerated during the operation
BytesEnumerated (integer): Total number of bytes enumerated during the operation
ObjectsAdded (integer): Number of objects added since the latest enumeration
BytesAdded (integer): Number of bytes added since the latest enumeration
ObjectsUpdated (integer): Number of objects updated since the latest enumeration
BytesUpdated (integer): Number of bytes updated since the latest enumeration
ObjectsDeleted (integer): Number of objects deleted from the latest enumeration
BytesDeleted (integer): Number of bytes deleted from the latest enumeration
ObjectsSuccess (integer): Number of objects successfully processed
BytesSuccess (integer): Number of bytes successfully processed
ObjectsFailed (integer): Number of objects that failed processing
BytesFailed (integer): Number of bytes that failed processing
State (enum): Current state of the crawl operation (NotStarted, Starting, Stopped, Canceled, Enumerating, Retrieving, Deleting, Success, Failed)
CreatedUtc (datetime): UTC timestamp when the crawl operation was created
StartUtc (datetime): UTC timestamp when the crawl operation started
StartEnumerationUtc (datetime): UTC timestamp when enumeration phase began
StartRetrievalUtc (datetime): UTC timestamp when retrieval phase began
FinishEnumerationUtc (datetime): UTC timestamp when enumeration phase completed
FinishRetrievalUtc (datetime): UTC timestamp when retrieval phase completed
FinishUtc (datetime): UTC timestamp when the crawl operation completed
AdditionalData (string): Additional information or notes about the crawl operation execution

Important Notes

Read-Only Objects: Crawl operations are automatically created and managed by the system
Execution Tracking: Operations provide complete visibility into crawl plan execution
Performance Metrics: Detailed statistics help monitor and optimize crawl performance
State Management: Real-time status tracking enables proactive operation management

Enumerate Crawl Operations

Retrieves a paginated list of all crawl operation objects in the tenant using GET /v2.0/tenants/[tenant-guid]/crawloperations. This endpoint provides comprehensive enumeration with pagination support for monitoring multiple crawl operation executions.

Request Parameters

No additional parameters required beyond authentication.

curl --location 'http://localhost:8000/v2.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/' \
--header 'Authorization: ••••••'

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const enumerateCrawlOperations = async () => {
  try {
    const response = await api.CrawlOperation.enumerate();
    console.log(response, "Crawl operations fetched successfully");
  } catch (err) {
    console.log("Error fetching Crawl operations:", err);
  }
};

enumerateCrawlOperations();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)

def enumerateCrawlOperations():
    crawlOperations = crawler.CrawlOperation.enumerate()
    print(crawlOperations)

enumerateCrawlOperations()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
EnumerationResult<CrawlOperation> response = await sdk.CrawlOperation.Enumerate();

Response Structure

The enumeration response includes pagination metadata and crawl operation objects with complete execution details:

Response

Returns a paginated list of crawl operation objects:

{
    "Success": true,
    "Timestamp": {
        "Start": "2024-10-21T02:36:37.677751Z",
        "TotalMs": 23.58,
        "Messages": {}
    },
    "MaxResults": 10,
    "IterationsRequired": 1,
    "EndOfResults": true,
    "RecordsRemaining": 0,
    "Objects": [
        {
            "GUID": "9ced1af3-2e19-4cd6-81ce-a8c90a9ed32d",
            "TenantGUID": "default",
            "CrawlPlanGUID": "e9a7d61e-7cbd-46e4-9956-533e22008978",
            "CrawlScheduleGUID": "oneminute",
            "CrawlFilterGUID": "default",
            "DataRepositoryGUID": "1a56c067-9e6d-4f7b-85bf-eb6b04aeda3f",
            "MetadataRuleGUID": "example-metadata-rule",
            "EmbeddingsRuleGUID": "crawler-embeddings-rule",
            "ProcessingEndpoint": "http://nginx-orchestrator:8501/processor",
            "CleanupEndpoint": "http://nginx-orchestrator:8501/processor/cleanup",
            "Name": "Alienware CIFS (started 2024-10-25T22:29:31 UTC)",
            "ObjectsEnumerated": 123,
            "BytesEnumerated": 61052490,
            "ObjectsAdded": 0,
            "BytesAdded": 0,
            "ObjectsUpdated": 0,
            "BytesUpdated": 0,
            "ObjectsDeleted": 0,
            "BytesDeleted": 0,
            "ObjectsSuccess": 0,
            "BytesSuccess": 0,
            "ObjectsFailed": 2,
            "BytesFailed": 16384,
            "State": "Success",
            "CreatedUtc": "2024-10-25T22:29:31.000000Z",
            "StartUtc": "2024-10-25T22:29:31.000000Z",
            "StartEnumerationUtc": "2024-10-25T22:29:31.000000Z",
            "StartRetrievalUtc": "2024-10-25T22:29:39.000000Z",
            "FinishEnumerationUtc": "2024-10-25T22:29:39.000000Z",
            "FinishRetrievalUtc": "2024-10-25T22:29:39.000000Z",
            "FinishUtc": "2024-10-25T22:29:39.000000Z",
            "AdditionalData": "No objects detected during enumeration"
        }
    ],
    "ContinuationToken": null
}

Read Crawl Operation

Retrieves crawl operation metadata and execution details by GUID using GET /v1.0/tenants/[tenant-guid]/crawloperations/[crawloperation-guid]. Returns the complete crawl operation information including execution status, performance metrics, and timing details. If the operation doesn't exist, a 404 error is returned.

Request Parameters

crawloperation-guid (string, Path, Required): GUID of the crawl operation object to retrieve

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const readCrawlOperation = async () => {
  try {
    const response = await api.CrawlOperation.read(
      "<crawloperation-guid>"
    );
    console.log(response, "Crawl operation fetched successfully");
  } catch (err) {
    console.log("Error fetching Crawl operation:", err);
  }
};

readCrawlOperation();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)
def readCrawlOperation():
    crawlOperation = crawler.CrawlOperation.retrieve("<crawloperation-guid>")
    print(crawlOperation)

readCrawlOperation()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
CrawlOperation response = await sdk.CrawlOperation.Retrieve(Guid.Parse("<crawloperation-guid>"));

Response

Returns the complete crawl operation metadata:

{
    "GUID": "9ced1af3-2e19-4cd6-81ce-a8c90a9ed32d",
    "TenantGUID": "default",
    "CrawlPlanGUID": "e9a7d61e-7cbd-46e4-9956-533e22008978",
    "CrawlScheduleGUID": "oneminute",
    "CrawlFilterGUID": "default",
    "DataRepositoryGUID": "1a56c067-9e6d-4f7b-85bf-eb6b04aeda3f",
    "MetadataRuleGUID": "example-metadata-rule",
    "EmbeddingsRuleGUID": "crawler-embeddings-rule",
    "ProcessingEndpoint": "http://nginx-orchestrator:8501/processor",
    "CleanupEndpoint": "http://nginx-orchestrator:8501/processor/cleanup",
    "Name": "Alienware CIFS (started 2024-10-25T22:29:31 UTC)",
    "ObjectsEnumerated": 123,
    "BytesEnumerated": 61052490,
    "ObjectsAdded": 0,
    "BytesAdded": 0,
    "ObjectsUpdated": 0,
    "BytesUpdated": 0,
    "ObjectsDeleted": 0,
    "BytesDeleted": 0,
    "ObjectsSuccess": 0,
    "BytesSuccess": 0,
    "ObjectsFailed": 2,
    "BytesFailed": 16384,
    "State": "Success",
    "CreatedUtc": "2024-10-25T22:29:31.000000Z",
    "StartUtc": "2024-10-25T22:29:31.000000Z",
    "StartEnumerationUtc": "2024-10-25T22:29:31.000000Z",
    "StartRetrievalUtc": "2024-10-25T22:29:39.000000Z",
    "FinishEnumerationUtc": "2024-10-25T22:29:39.000000Z",
    "FinishRetrievalUtc": "2024-10-25T22:29:39.000000Z",
    "FinishUtc": "2024-10-25T22:29:39.000000Z",
    "AdditionalData": "No objects detected during enumeration"
}

Note: the HEAD method can be used as an alternative to get to simply check the existence of the object. HEAD requests return either a 200/OK in the event the object exists, or a 404/Not Found if not. No response body is returned with a HEAD request.

Retrieve Crawl Operation Enumeration

Retrieves detailed enumeration data for a specific crawl operation using GET /v1.0/tenants/[tenant-guid]/crawloperations/[crawloperation-guid]/enumeration. This endpoint provides access to the complete enumeration results and object details discovered during the crawl operation.

Request Parameters

crawloperation-guid (string, Path, Required): GUID of the crawl operation object to retrieve enumeration data for

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/78d881af-ad82-48bc-8097-1fccd2787624/enumeration' \
--header 'Authorization: ••••••'

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const retrieveEnumerationCrawlOperations = async () => {
  try {
    const response = await api.CrawlOperation.readEnumeration(
      "<crawloperation-guid>"
    );
    console.log(response, "Crawl operation enumeration fetched successfully");
  } catch (err) {
    console.log("Error fetching Crawl operation enumeration:", err);
  }
};

retrieveEnumerationCrawlOperations();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)

def enumerateCrawlOperation():
    enumeration = crawler.CrawlOperation.enumerateCrawlOperation("<crawloperation-guid>")
    print(enumeration)

enumerateCrawlOperation()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
CrawlEnumeration response = await sdk.CrawlOperation.RetrieveEnumeration(Guid.Parse("<crawloperation-guid>"));

Response

Returns the detailed enumeration data for the crawl operation:

{
    "GUID": "9ced1af3-2e19-4cd6-81ce-a8c90a9ed32d",
    "TenantGUID": "default",
    "CrawlPlanGUID": "e9a7d61e-7cbd-46e4-9956-533e22008978",
    "DataRepositoryGUID": "1a56c067-9e6d-4f7b-85bf-eb6b04aeda3f",
    "Name": "Alienware CIFS (started 2024-10-25T22:29:31 UTC)",
    "ObjectsEnumerated": 123,
    "BytesEnumerated": 61052490,
    "EnumerationDirectory": "./enumerations/",
    "EnumerationsToRetain": 16,
    "MaxDrainTasks": 4,
    "ProcessAdditions": true,
    "ProcessDeletions": true,
    "ProcessUpdates": true,
    "CreatedUtc": "2024-10-25T22:29:31.000000Z",
    "StartUtc": "2024-10-25T22:29:31.000000Z",
    "FinishUtc": "2024-10-25T22:29:39.000000Z"
}

Read All Crawl Operations

Retrieves all crawl operation objects in the tenant using GET /v1.0/tenants/[tenant-guid]/crawloperations/. Returns an array of crawl operation objects with complete execution details for all operations in the tenant.

Request Parameters

No additional parameters required beyond authentication.

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/' \
--header 'Authorization: ••••••'

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const readAllCrawlOperations = async () => {
  try {
    const response = await api.CrawlOperation.readAll();
    console.log(response, "All crawl operations fetched successfully");
  } catch (err) {
    console.log("Error fetching All crawl operations:", err);
  }
};

readAllCrawlOperations();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)

def readAllCrawlOperations():
    crawlOperations = crawler.CrawlOperation.retrieve_all()
    print(crawlOperations)

readAllCrawlOperations()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
List<CrawlOperation> response = await sdk.CrawlOperation.RetrieveAll();

Response

Returns an array of all crawl operation objects:

[
    {
        "GUID": "9ced1af3-2e19-4cd6-81ce-a8c90a9ed32d",
        "TenantGUID": "default",
        "CrawlPlanGUID": "e9a7d61e-7cbd-46e4-9956-533e22008978",
        "CrawlScheduleGUID": "oneminute",
        "CrawlFilterGUID": "default",
        "DataRepositoryGUID": "1a56c067-9e6d-4f7b-85bf-eb6b04aeda3f",
        "MetadataRuleGUID": "example-metadata-rule",
        "EmbeddingsRuleGUID": "crawler-embeddings-rule",
        "ProcessingEndpoint": "http://nginx-orchestrator:8501/processor",
        "CleanupEndpoint": "http://nginx-orchestrator:8501/processor/cleanup",
        "Name": "Alienware CIFS (started 2024-10-25T22:29:31 UTC)",
        "ObjectsEnumerated": 123,
        "BytesEnumerated": 61052490,
        "ObjectsAdded": 0,
        "BytesAdded": 0,
        "ObjectsUpdated": 0,
        "BytesUpdated": 0,
        "ObjectsDeleted": 0,
        "BytesDeleted": 0,
        "ObjectsSuccess": 0,
        "BytesSuccess": 0,
        "ObjectsFailed": 2,
        "BytesFailed": 16384,
        "State": "Success",
        "CreatedUtc": "2024-10-25T22:29:31.000000Z",
        "StartUtc": "2024-10-25T22:29:31.000000Z",
        "StartEnumerationUtc": "2024-10-25T22:29:31.000000Z",
        "StartRetrievalUtc": "2024-10-25T22:29:39.000000Z",
        "FinishEnumerationUtc": "2024-10-25T22:29:39.000000Z",
        "FinishRetrievalUtc": "2024-10-25T22:29:39.000000Z",
        "FinishUtc": "2024-10-25T22:29:39.000000Z",
        "AdditionalData": "No objects detected during enumeration"
    }
]

Stop Crawl Operation

Terminates a running crawl operation by GUID using DELETE /v1.0/tenants/[tenant-guid]/crawloperations/[crawloperation-guid]/stop. This operation gracefully stops the crawl execution and updates the operation state to indicate termination.

Request Parameters

crawloperation-guid (string, Path, Required): GUID of the crawl operation object to stop

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/00000000-0000-0000-0000-000000000000/stop' \
--header 'content-type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "Name": "My tenant"
}'

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const stopCrawlOperation = async () => {
  try {
    const response = await api.CrawlOperation.stop(
      "<crawloperation-guid>",
      {
        Name: "My crawl operation [ASH]",
      }
    );
    console.log(response, "Crawl operation stopped successfully");
  } catch (err) {
    console.log("Error stopping Crawl operation:", err);
  }
};
stopCrawlOperation();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)

def stopCrawlOperation():
    crawlOperation = crawler.CrawlOperation.stop("<crawloperation-guid>",Name="My crawl operation")
    print(crawlOperation)

stopCrawlOperation()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
CrawlOperation crawlRequest = new CrawlOperationRequest
{
    GUID = "<crawloperation-guid>",
    Name = "My First Crawl Operation"
};
CrawlOperation response = await sdk.CrawlOperation.Stop(operation);

Response

Returns the updated crawl operation object with the stopped state:

{
    "GUID": "9ced1af3-2e19-4cd6-81ce-a8c90a9ed32d",
    "TenantGUID": "default",
    "CrawlPlanGUID": "e9a7d61e-7cbd-46e4-9956-533e22008978",
    "CrawlScheduleGUID": "oneminute",
    "CrawlFilterGUID": "default",
    "DataRepositoryGUID": "1a56c067-9e6d-4f7b-85bf-eb6b04aeda3f",
    "MetadataRuleGUID": "example-metadata-rule",
    "EmbeddingsRuleGUID": "crawler-embeddings-rule",
    "ProcessingEndpoint": "http://nginx-orchestrator:8501/processor",
    "CleanupEndpoint": "http://nginx-orchestrator:8501/processor/cleanup",
    "Name": "My crawl operation [ASH]",
    "ObjectsEnumerated": 123,
    "BytesEnumerated": 61052490,
    "ObjectsAdded": 0,
    "BytesAdded": 0,
    "ObjectsUpdated": 0,
    "BytesUpdated": 0,
    "ObjectsDeleted": 0,
    "BytesDeleted": 0,
    "ObjectsSuccess": 0,
    "BytesSuccess": 0,
    "ObjectsFailed": 2,
    "BytesFailed": 16384,
    "State": "Stopped",
    "CreatedUtc": "2024-10-25T22:29:31.000000Z",
    "StartUtc": "2024-10-25T22:29:31.000000Z",
    "StartEnumerationUtc": "2024-10-25T22:29:31.000000Z",
    "StartRetrievalUtc": "2024-10-25T22:29:39.000000Z",
    "FinishEnumerationUtc": "2024-10-25T22:29:39.000000Z",
    "FinishRetrievalUtc": "2024-10-25T22:29:39.000000Z",
    "FinishUtc": "2024-10-25T22:29:39.000000Z",
    "AdditionalData": "No objects detected during enumeration"
}

Delete Crawl Operation

Deletes a crawl operation object by GUID using DELETE /v1.0/tenants/[tenant-guid]/crawloperations/[crawloperation-guid]. This operation permanently removes the crawl operation record and associated metadata from the system.

Request Parameters

crawloperation-guid (string, Path, Required): GUID of the crawl operation object to delete

curl --location --request DELETE 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••' \
--data ''

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const deleteCrawlOperation = async () => {
  try {
    const response = await api.CrawlOperation.delete(
      "<crawloperation-guid>"
    );
    console.log(response, "Crawl operation deleted successfully");
  } catch (err) {
    console.log("Error deleting Crawl operation:", err);
  }
};

deleteCrawlOperation();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)

def deleteCrawlOperation():
    crawlOperation = crawler.CrawlOperation.delete("<crawloperation-guid>")
    print(crawlOperation)

deleteCrawlOperation()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
bool deleted = await sdk.CrawlOperation.Delete(Guid.Parse("<crawloperation-guid>"));

Response

Returns 200 No Content on successful deletion. No response body is returned.

Check Crawl Operation Existence

Verifies if a crawl operation object exists without retrieving its details using HEAD /v1.0/tenants/[tenant-guid]/crawloperations/[crawloperation-guid]. This is an efficient way to check operation presence before performing operations.

Request Parameters

crawloperation-guid (string, Path, Required): GUID of the crawl operation object to check

curl --location --head 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/crawloperations/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'

import { ViewCrawlerSdk } from "view-sdk";

const api = new ViewCrawlerSdk(
  "http://localhost:8000/", //endpoint
  "default", //tenant Id
  "default" //access key
);

const existsCrawlOperation = async () => {
  try {
    const response = await api.CrawlOperation.exists(
      "<crawloperation-guid>"
    );
    console.log(response, "Crawl operation exists");
  } catch (err) {
    console.log("Error checking Crawl operation:", err);
  }
};

existsCrawlOperation();

import view_sdk
from view_sdk import crawler
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.CRAWLER: 8000},
)

def existsCrawlOperation():
    crawlOperation = crawler.CrawlOperation.exists("<crawloperation-guid>")
    print(crawlOperation)

existsCrawlOperation()

using View.Sdk;
using View.Crawler;

ViewCrawlerSdk sdk = new ViewCrawlerSdk(Guid.Parse("00000000-0000-0000-0000-000000000000"), 
                                        "default", 
                                        "http://localhost:8000/");
bool exists = await sdk.CrawlOperation.Exists(Guid.Parse("<crawloperation-guid>"));

Response

200 No Content: Crawl operation exists
404 Not Found: Crawl operation does not exist
No response body: Only HTTP status code is returned

Note: HEAD requests do not return a response body, only the HTTP status code indicating whether the crawl operation exists.

Best Practices

When managing crawl operations in the View platform, consider the following recommendations for optimal operation monitoring and management:

Regular Monitoring: Monitor crawl operation status and performance metrics to ensure optimal execution
Error Analysis: Review failed objects and error conditions to identify and resolve processing issues
Performance Optimization: Use timing information and object counts to optimize crawl plan configurations
Resource Management: Monitor processing endpoints and cleanup operations for efficient resource utilization
Historical Analysis: Retain operation metadata for trend analysis and performance improvement

Next Steps

After successfully monitoring crawl operations, you can:

Crawl Plans: Create and configure crawl plans to define automated data ingestion workflows
Data Repositories: Set up data repositories to define source data locations for crawling
Crawl Schedules: Configure crawl schedules to define when and how frequently crawling occurs
Performance Tuning: Optimize crawl performance based on operation metrics and execution patterns
Integration: Integrate crawl operations with other View platform services for comprehensive data processing