API Reference
Start typing to search…

Ingest Queue Management

Comprehensive guide to managing ingest queue operations in the Lexi metadata database.

Overview

The Ingest Queue manages document processing operations within the Lexi metadata database. It tracks the status, progress, and performance metrics of document ingestion processes, providing visibility into data processing workflows and enabling monitoring of document processing operations.

Ingest Queue entries are managed via the Lexi server API and provide comprehensive tracking of document processing status, including processing times, success/failure rates, and detailed performance metrics for optimization and troubleshooting.

Ingest Queue Object Structure

Ingest Queue entries have the following structure:

{
    "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
    "DocumentType": "Json",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentLength": 1345,
    "Message": "Ingestion complete",
    "StartUtc": "2025-09-22T12:32:44.491633Z",
    "SuccessUtc": "2025-09-22T12:32:44.597593Z",
    "TotalMs": 115.42000000,
    "TermProcessingMs": 32.66000000,
    "SchemaProcessingMs": 57.32000000,
    "CreatedUtc": "2025-09-22T12:32:43.662480Z"
}

Field Descriptions

  • GUID (GUID): Globally unique identifier for the ingest queue entry
  • TenantGUID (GUID): Globally unique identifier for the tenant
  • CollectionGUID (GUID): Globally unique identifier for the target collection
  • SourceDocumentGUID (GUID): Globally unique identifier for the source document being processed
  • DocumentType (string): Type of document being processed (e.g., "Json", "Pdf", "Text")
  • ObjectKey (string): Key identifier for the object being processed
  • ObjectVersion (string): Version identifier for the object
  • ContentLength (integer): Length of the content being processed in bytes
  • Message (string): Status message describing the processing result
  • StartUtc (datetime): Timestamp when processing began
  • SuccessUtc (datetime): Timestamp when processing completed successfully
  • TotalMs (decimal): Total processing time in milliseconds
  • TermProcessingMs (decimal): Time spent on term processing in milliseconds
  • SchemaProcessingMs (decimal): Time spent on schema processing in milliseconds
  • CreatedUtc (datetime): Timestamp when the queue entry was created

Read Ingest Queue Entry

Retrieves a specific ingest queue entry by GUID using GET /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]. Returns the complete queue entry with processing status, timestamps, and performance metrics.

Request Parameters

  • ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to retrieve
curl --location 'http://view.homedns.org:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'
import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.read(
      "<ingestqueue-guid>",
    );
    console.log(response, "IngestQueue fetched successfully");
  } catch (err) {
    console.log("Error fetching IngestQueue:", err);
  }
};

retrieveIngestQueue();
import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readIngestQueue():
    queue = lexi.IngestQueue.retrieve("<ingestqueue-guid>")
    print(queue)

readIngestQueue()
using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<ingestqueue-guid>"),"default", "http://localhost:8000/");
            
IngestionQueueEntry IngestionQueueEntry = await sdk.IngestQueue.Retrieve(Guid.Parse("<ingestqueue-guid>"));

Response Structure

Returns the ingest queue entry object with all processing details if found, or a 404 Not Found error if the entry doesn't exist.

Note: The HEAD method can be used as an alternative to simply check the existence of the object. HEAD requests return either a 200 OK if the object exists, or a 404 Not Found if not. No response body is returned with a HEAD request.

Response

Returns the complete ingest queue entry with processing details:

{
    "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
    "DocumentType": "Json",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentLength": 1345,
    "Message": "Ingestion complete",
    "StartUtc": "2025-09-22T12:32:44.491633Z",
    "SuccessUtc": "2025-09-22T12:32:44.597593Z",
    "TotalMs": 115.42000000,
    "TermProcessingMs": 32.66000000,
    "SchemaProcessingMs": 57.32000000,
    "CreatedUtc": "2025-09-22T12:32:43.662480Z"
}

Read Ingest Queue Statistics

Retrieves comprehensive statistics for an ingest queue entry using GET /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]?stats. Provides detailed performance metrics, processing times, and status information for monitoring and analysis.

Request Parameters

  • ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to analyze
  • stats (string, Query, Required): Must be included to retrieve statistics
curl --location 'http://view.homedns.org:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000?stats' \
--header 'Authorization: ••••••'
import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveIngestQueueStats = async () => {
  try {
    const response = await api.ingestQueueSdk.readStatistics(
      "<ingestqueue-guid>"
    );
    console.log(response, "IngestQueue stats retrieved successfully");
  } catch (err) {
    console.log("Error retrieving IngestQueue stats:", err);
  }
};

retrieveIngestQueueStats();
import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readIngestQueue():
    queue = lexi.IngestQueue.retrieve_statistics("<ingestqueue-guid>")
    print(queue)

readIngestQueue()
using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<ingestqueue-guid>"),"default", "http://localhost:8000/");
            
IngestionQueueEntry response = await sdk.IngestQueue.Retrieve(Guid.Parse("<ingestqueue-guid>"), includeStats: true);

Response Structure

Returns a JSON object containing the ingest queue entry with comprehensive statistics including processing metrics, performance data, and status information, or a 404 Not Found error if the entry doesn't exist.

Response

Returns the ingest queue entry with comprehensive statistics:

{
    "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
    "DocumentType": "Json",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentLength": 1345,
    "Message": "Ingestion complete",
    "StartUtc": "2025-09-22T12:32:44.491633Z",
    "SuccessUtc": "2025-09-22T12:32:44.597593Z",
    "TotalMs": 115.42000000,
    "TermProcessingMs": 32.66000000,
    "SchemaProcessingMs": 57.32000000,
    "CreatedUtc": "2025-09-22T12:32:43.662480Z",
    "Statistics": {
        "TotalProcessingTime": 115.42,
        "AverageProcessingTime": 105.50,
        "SuccessRate": 0.95,
        "FailureRate": 0.05,
        "RetryRate": 0.02
    }
}

Read All Ingest Queue Entries

Retrieves all ingest queue entries for a tenant using GET /v1.0/tenants/[tenant-guid]/ingestqueue. Returns a JSON array containing all queue entries with their processing status and metrics.

Request Parameters

No additional parameters required beyond authentication.

curl --location 'http://view.homedns.org:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue' \
--header 'Authorization: ••••••'
import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveAllIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.readAll();
    console.log(response, "IngestQueue retrieved successfully");
  } catch (err) {
    console.log("Error retrieving IngestQueue:", err);
  }
};

retrieveAllIngestQueue();
import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readAllIngestQueue():
    queues = lexi.IngestQueue.retrieve_all()
    print(queues)

readAllIngestQueue()
using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");

 List<IngestionQueueEntry> response = await sdk.IngestQueue.RetrieveMany();

Response Structure

Returns an array of all ingest queue entries for the tenant, or a 404 Not Found error if no entries exist.

Response

Returns an array of all ingest queue entries:

[
    {
        "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
        "TenantGUID": "00000000-0000-0000-0000-000000000000",
        "CollectionGUID": "00000000-0000-0000-0000-000000000000",
        "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
        "DocumentType": "Json",
        "ObjectKey": "blake.json",
        "ObjectVersion": "1",
        "ContentLength": 1345,
        "Message": "Ingestion complete",
        "StartUtc": "2025-09-22T12:32:44.491633Z",
        "SuccessUtc": "2025-09-22T12:32:44.597593Z",
        "TotalMs": 115.42000000,
        "TermProcessingMs": 32.66000000,
        "SchemaProcessingMs": 57.32000000,
        "CreatedUtc": "2025-09-22T12:32:43.662480Z"
    },
    {
        "GUID": "22592233-7181-4b6e-b43d-a13fce138c0b",
        "TenantGUID": "00000000-0000-0000-0000-000000000000",
        "CollectionGUID": "00000000-0000-0000-0000-000000000000",
        "SourceDocumentGUID": "ee264360-a302-4a35-a6c6-a3fb8c989240",
        "DocumentType": "Pdf",
        "ObjectKey": "document.pdf",
        "ObjectVersion": "2",
        "ContentLength": 2567,
        "Message": "Processing failed: Invalid format",
        "StartUtc": "2025-09-22T12:33:44.491633Z",
        "SuccessUtc": null,
        "TotalMs": 0,
        "TermProcessingMs": 0,
        "SchemaProcessingMs": 0,
        "CreatedUtc": "2025-09-22T12:33:43.662480Z"
    }
]

Check Ingest Queue Entry Existence

Checks whether an ingest queue entry exists using HEAD /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]. Returns only HTTP status codes without response body for efficient existence verification.

Request Parameters

  • ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to check
curl --location --head 'http://view.homedns.org:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'
import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const existIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.exists(
      "<ingestqueue-guid>"
    );
    console.log(response, "IngestQueue existence checked successfully");
  } catch (err) {
    console.log("Error checking IngestQueue existence:", err);
  }
};

existIngestQueue();
import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def existsIngestQueue():
    exists = lexi.IngestQueue.exists("<ingestqueue-guid>")
    print(exists)

existsIngestQueue()
using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
Guid IngestionQueueEntryGuid = Guid.Parse("<ingestqueue-guid>");
        
bool exists = await sdk.IngestQueue.Exists(IngestionQueueEntryGuid);

Response

  • 200 OK: Ingest queue entry exists
  • 404 Not Found: Ingest queue entry does not exist
  • No response body: Only HTTP status code is returned

Note: HEAD requests do not return a response body, only the HTTP status code indicating whether the ingest queue entry exists.

Response

  • 200 No Content: Ingest queue entry exists
  • 404 Not Found: Ingest queue entry does not exist
  • No response body: Only HTTP status code is returned

Note: HEAD requests do not return a response body, only the HTTP status code indicating whether the ingest queue entry exists.

Delete Ingest Queue Entry

Deletes an ingest queue entry by GUID using DELETE /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]. Removes the queue entry from the system, typically used for cleaning up completed or failed processing entries.

Request Parameters

  • ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to delete
curl --location --request DELETE 'http://view.homedns.org:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' 
import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const deleteIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.delete(
      "<ingestqueue-guid>"
    );
    console.log(response, "IngestQueue deleted successfully");
  } catch (err) {
    console.log("Error deleting IngestQueue:", err);
  }
};

deleteIngestQueue();
import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def deleteIngestQueue():
    response = lexi.IngestQueue.delete("<ingestqueue-guid>")
    print(response)

deleteIngestQueue()
using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
Guid IngestionQueueEntryGuid = Guid.Parse("<ingestqueue-guid>");
        
bool deleted = await sdk.IngestQueue.Delete(IngestionQueueEntryGuid);

Response

  • 200 OK: Ingest queue entry deleted successfully
  • 404 Not Found: Ingest queue entry does not exist

Note: Deleting an ingest queue entry removes it from the system permanently. This is typically done for completed or failed entries that no longer need to be tracked.

Response

Returns 204 No Content on successful deletion. No response body is returned.

Best Practices

When managing ingest queue operations in the Lexi metadata database, consider the following recommendations for optimal processing monitoring, performance tracking, and system maintenance:

  • Queue Monitoring: Regularly monitor ingest queue entries to track processing status and identify bottlenecks or failures
  • Performance Analysis: Use statistics and metrics to analyze processing times and optimize document ingestion workflows
  • Error Handling: Implement proper error handling and retry mechanisms based on queue entry status and error messages
  • Cleanup Operations: Periodically clean up completed or failed queue entries to maintain system performance and storage efficiency
  • Status Tracking: Monitor queue entry status changes to ensure proper document processing workflow completion

Next Steps

After successfully managing ingest queue operations, you can:

  • Document Processing: Monitor and manage document processing workflows through queue entry tracking
  • Performance Optimization: Analyze processing metrics to optimize document ingestion performance and identify improvement opportunities
  • Error Resolution: Use queue entry information to troubleshoot and resolve document processing issues
  • System Monitoring: Implement monitoring dashboards using queue statistics for real-time processing visibility
  • Workflow Integration: Integrate queue monitoring with other View platform services for comprehensive data processing management