Overview

The Ingest Queue manages document processing operations within the Lexi metadata database. It tracks the status, progress, and performance metrics of document ingestion processes, providing visibility into data processing workflows and enabling monitoring of document processing operations.

Ingest Queue entries are managed via the Lexi server API and provide comprehensive tracking of document processing status, including processing times, success/failure rates, and detailed performance metrics for optimization and troubleshooting.

Ingest Queue Object Structure

Ingest Queue entries have the following structure:

{
    "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
    "DocumentType": "Json",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentLength": 1345,
    "Message": "Ingestion complete",
    "StartUtc": "2025-09-22T12:32:44.491633Z",
    "SuccessUtc": "2025-09-22T12:32:44.597593Z",
    "TotalMs": 115.42000000,
    "TermProcessingMs": 32.66000000,
    "SchemaProcessingMs": 57.32000000,
    "CreatedUtc": "2025-09-22T12:32:43.662480Z"
}

Field Descriptions

GUID (GUID): Globally unique identifier for the ingest queue entry
TenantGUID (GUID): Globally unique identifier for the tenant
CollectionGUID (GUID): Globally unique identifier for the target collection
SourceDocumentGUID (GUID): Globally unique identifier for the source document being processed
DocumentType (string): Type of document being processed (e.g., "Json", "Pdf", "Text")
ObjectKey (string): Key identifier for the object being processed
ObjectVersion (string): Version identifier for the object
ContentLength (integer): Length of the content being processed in bytes
Message (string): Status message describing the processing result
StartUtc (datetime): Timestamp when processing began
SuccessUtc (datetime): Timestamp when processing completed successfully
TotalMs (decimal): Total processing time in milliseconds
TermProcessingMs (decimal): Time spent on term processing in milliseconds
SchemaProcessingMs (decimal): Time spent on schema processing in milliseconds
CreatedUtc (datetime): Timestamp when the queue entry was created

Read Ingest Queue Entry

Retrieves a specific ingest queue entry by GUID using GET /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]. Returns the complete queue entry with processing status, timestamps, and performance metrics.

Request Parameters

ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to retrieve

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.read(
      "<ingestqueue-guid>",
    );
    console.log(response, "IngestQueue fetched successfully");
  } catch (err) {
    console.log("Error fetching IngestQueue:", err);
  }
};

retrieveIngestQueue();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readIngestQueue():
    queue = lexi.IngestQueue.retrieve("<ingestqueue-guid>")
    print(queue)

readIngestQueue()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<ingestqueue-guid>"),"default", "http://localhost:8000/");
            
IngestionQueueEntry IngestionQueueEntry = await sdk.IngestQueue.Retrieve(Guid.Parse("<ingestqueue-guid>"));

Response Structure

Returns the ingest queue entry object with all processing details if found, or a 404 Not Found error if the entry doesn't exist.

Note: The HEAD method can be used as an alternative to simply check the existence of the object. HEAD requests return either a 200 OK if the object exists, or a 404 Not Found if not. No response body is returned with a HEAD request.

Response

Returns the complete ingest queue entry with processing details:

{
    "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
    "DocumentType": "Json",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentLength": 1345,
    "Message": "Ingestion complete",
    "StartUtc": "2025-09-22T12:32:44.491633Z",
    "SuccessUtc": "2025-09-22T12:32:44.597593Z",
    "TotalMs": 115.42000000,
    "TermProcessingMs": 32.66000000,
    "SchemaProcessingMs": 57.32000000,
    "CreatedUtc": "2025-09-22T12:32:43.662480Z"
}

Read Ingest Queue Statistics

Retrieves comprehensive statistics for an ingest queue entry using GET /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]?stats. Provides detailed performance metrics, processing times, and status information for monitoring and analysis.

Request Parameters

ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to analyze
stats (string, Query, Required): Must be included to retrieve statistics

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000?stats' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveIngestQueueStats = async () => {
  try {
    const response = await api.ingestQueueSdk.readStatistics(
      "<ingestqueue-guid>"
    );
    console.log(response, "IngestQueue stats retrieved successfully");
  } catch (err) {
    console.log("Error retrieving IngestQueue stats:", err);
  }
};

retrieveIngestQueueStats();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readIngestQueue():
    queue = lexi.IngestQueue.retrieve_statistics("<ingestqueue-guid>")
    print(queue)

readIngestQueue()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<ingestqueue-guid>"),"default", "http://localhost:8000/");
            
IngestionQueueEntry response = await sdk.IngestQueue.Retrieve(Guid.Parse("<ingestqueue-guid>"), includeStats: true);

Response Structure

Returns a JSON object containing the ingest queue entry with comprehensive statistics including processing metrics, performance data, and status information, or a 404 Not Found error if the entry doesn't exist.

Response

Returns the ingest queue entry with comprehensive statistics:

{
    "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
    "DocumentType": "Json",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentLength": 1345,
    "Message": "Ingestion complete",
    "StartUtc": "2025-09-22T12:32:44.491633Z",
    "SuccessUtc": "2025-09-22T12:32:44.597593Z",
    "TotalMs": 115.42000000,
    "TermProcessingMs": 32.66000000,
    "SchemaProcessingMs": 57.32000000,
    "CreatedUtc": "2025-09-22T12:32:43.662480Z",
    "Statistics": {
        "TotalProcessingTime": 115.42,
        "AverageProcessingTime": 105.50,
        "SuccessRate": 0.95,
        "FailureRate": 0.05,
        "RetryRate": 0.02
    }
}

Read All Ingest Queue Entries

Retrieves all ingest queue entries for a tenant using GET /v1.0/tenants/[tenant-guid]/ingestqueue. Returns a JSON array containing all queue entries with their processing status and metrics.

Request Parameters

No additional parameters required beyond authentication.

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveAllIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.readAll();
    console.log(response, "IngestQueue retrieved successfully");
  } catch (err) {
    console.log("Error retrieving IngestQueue:", err);
  }
};

retrieveAllIngestQueue();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readAllIngestQueue():
    queues = lexi.IngestQueue.retrieve_all()
    print(queues)

readAllIngestQueue()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");

 List<IngestionQueueEntry> response = await sdk.IngestQueue.RetrieveMany();

Response Structure

Returns an array of all ingest queue entries for the tenant, or a 404 Not Found error if no entries exist.

Response

Returns an array of all ingest queue entries:

[
    {
        "GUID": "12592233-7181-4b6e-b43d-a13fce138c0b",
        "TenantGUID": "00000000-0000-0000-0000-000000000000",
        "CollectionGUID": "00000000-0000-0000-0000-000000000000",
        "SourceDocumentGUID": "de264360-a302-4a35-a6c6-a3fb8c989240",
        "DocumentType": "Json",
        "ObjectKey": "blake.json",
        "ObjectVersion": "1",
        "ContentLength": 1345,
        "Message": "Ingestion complete",
        "StartUtc": "2025-09-22T12:32:44.491633Z",
        "SuccessUtc": "2025-09-22T12:32:44.597593Z",
        "TotalMs": 115.42000000,
        "TermProcessingMs": 32.66000000,
        "SchemaProcessingMs": 57.32000000,
        "CreatedUtc": "2025-09-22T12:32:43.662480Z"
    },
    {
        "GUID": "22592233-7181-4b6e-b43d-a13fce138c0b",
        "TenantGUID": "00000000-0000-0000-0000-000000000000",
        "CollectionGUID": "00000000-0000-0000-0000-000000000000",
        "SourceDocumentGUID": "ee264360-a302-4a35-a6c6-a3fb8c989240",
        "DocumentType": "Pdf",
        "ObjectKey": "document.pdf",
        "ObjectVersion": "2",
        "ContentLength": 2567,
        "Message": "Processing failed: Invalid format",
        "StartUtc": "2025-09-22T12:33:44.491633Z",
        "SuccessUtc": null,
        "TotalMs": 0,
        "TermProcessingMs": 0,
        "SchemaProcessingMs": 0,
        "CreatedUtc": "2025-09-22T12:33:43.662480Z"
    }
]

Check Ingest Queue Entry Existence

Checks whether an ingest queue entry exists using HEAD /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]. Returns only HTTP status codes without response body for efficient existence verification.

Request Parameters

ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to check

curl --location --head 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const existIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.exists(
      "<ingestqueue-guid>"
    );
    console.log(response, "IngestQueue existence checked successfully");
  } catch (err) {
    console.log("Error checking IngestQueue existence:", err);
  }
};

existIngestQueue();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def existsIngestQueue():
    exists = lexi.IngestQueue.exists("<ingestqueue-guid>")
    print(exists)

existsIngestQueue()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
Guid IngestionQueueEntryGuid = Guid.Parse("<ingestqueue-guid>");
        
bool exists = await sdk.IngestQueue.Exists(IngestionQueueEntryGuid);

Response

200 OK: Ingest queue entry exists
404 Not Found: Ingest queue entry does not exist
No response body: Only HTTP status code is returned

Note: HEAD requests do not return a response body, only the HTTP status code indicating whether the ingest queue entry exists.

Response

200 No Content: Ingest queue entry exists
404 Not Found: Ingest queue entry does not exist
No response body: Only HTTP status code is returned

Note: HEAD requests do not return a response body, only the HTTP status code indicating whether the ingest queue entry exists.

Delete Ingest Queue Entry

Deletes an ingest queue entry by GUID using DELETE /v1.0/tenants/[tenant-guid]/ingestqueue/[ingestqueue-guid]. Removes the queue entry from the system, typically used for cleaning up completed or failed processing entries.

Request Parameters

ingestqueue-guid (string, Path, Required): GUID of the ingest queue entry to delete

curl --location --request DELETE 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/ingestqueue/00000000-0000-0000-0000-000000000000' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/",//endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const deleteIngestQueue = async () => {
  try {
    const response = await api.ingestQueueSdk.delete(
      "<ingestqueue-guid>"
    );
    console.log(response, "IngestQueue deleted successfully");
  } catch (err) {
    console.log("Error deleting IngestQueue:", err);
  }
};

deleteIngestQueue();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def deleteIngestQueue():
    response = lexi.IngestQueue.delete("<ingestqueue-guid>")
    print(response)

deleteIngestQueue()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
Guid IngestionQueueEntryGuid = Guid.Parse("<ingestqueue-guid>");
        
bool deleted = await sdk.IngestQueue.Delete(IngestionQueueEntryGuid);

Response

200 OK: Ingest queue entry deleted successfully
404 Not Found: Ingest queue entry does not exist

Note: Deleting an ingest queue entry removes it from the system permanently. This is typically done for completed or failed entries that no longer need to be tracked.

Response

Returns 204 No Content on successful deletion. No response body is returned.

Best Practices

When managing ingest queue operations in the Lexi metadata database, consider the following recommendations for optimal processing monitoring, performance tracking, and system maintenance:

Queue Monitoring: Regularly monitor ingest queue entries to track processing status and identify bottlenecks or failures
Performance Analysis: Use statistics and metrics to analyze processing times and optimize document ingestion workflows
Error Handling: Implement proper error handling and retry mechanisms based on queue entry status and error messages
Cleanup Operations: Periodically clean up completed or failed queue entries to maintain system performance and storage efficiency
Status Tracking: Monitor queue entry status changes to ensure proper document processing workflow completion

Next Steps

After successfully managing ingest queue operations, you can:

Document Processing: Monitor and manage document processing workflows through queue entry tracking
Performance Optimization: Analyze processing metrics to optimize document ingestion performance and identify improvement opportunities
Error Resolution: Use queue entry information to troubleshoot and resolve document processing issues
System Monitoring: Implement monitoring dashboards using queue statistics for real-time processing visibility
Workflow Integration: Integrate queue monitoring with other View platform services for comprehensive data processing management