Overview

Source documents contain comprehensive metadata about processed objects and are stored within collections in the Lexi metadata database. They serve as the foundation for document search, content analysis, and metadata management, providing detailed information about document content, structure, terms, and semantic data.

Source documents are managed via the Lexi server API at [http|https]://[hostname]:[port]/v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents and support comprehensive operations including document upload, metadata retrieval, content analysis, and search functionality.

Source Document Object Structure

Source documents have the following structure:

{
    "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "CollectionGUID": "default",
    "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
    "ObjectKey": "5.pdf",
    "ObjectVersion": "1",
    "ContentType": "application/pdf",
    "DocumentType": "Pdf",
    "SourceUrl": "http://dcc249eaaf06:8001/v1.0/tenants/default/buckets/example-data-bucket/objects/5.pdf",
    "ContentLength": 31811,
    "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
    "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
    "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
    "CreatedUtc": "2024-10-25T14:14:22.000000Z"
}

Field Descriptions

GUID (GUID): Globally unique identifier for the source document object
TenantGUID (GUID): Globally unique identifier for the tenant that owns this document
BucketGUID (GUID): Globally unique identifier for the bucket where the source object is stored
DataRepositoryGUID (GUID): Globally unique identifier for the data repository where the object is stored
CollectionGUID (GUID): Globally unique identifier for the collection containing this document
ObjectGUID (GUID): Globally unique identifier for the source object
ObjectKey (string): Key/name of the source object
ObjectVersion (string): Version identifier of the source object
ContentType (string): MIME content type of the source document
DocumentType (enum): Type of document (e.g., "Pdf", "Text", "Json", "Html")
SourceUrl (string): URL from which the source object can be retrieved
ContentLength (long): Length of the source document content in bytes
MD5Hash (string): MD5 hash of the document content as a hexadecimal string
SHA1Hash (string): SHA1 hash of the document content as a hexadecimal string
SHA256Hash (string): SHA256 hash of the document content as a hexadecimal string
CreatedUtc (datetime): Timestamp indicating when the source document was created, in UTC time

Create Source Document

Uploads a source document to a collection using PUT /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents. Requires a fully-populated source document with comprehensive metadata and UDR (Unified Document Representation) data for content analysis and search functionality.

Request Parameters

Required Parameters

TenantGUID (GUID, Body, Required): GUID of the tenant
CollectionGUID (GUID, Body, Required): GUID of the target collection
ObjectKey (string, Body, Required): Key/name of the source object
ObjectVersion (string, Body, Required): Version identifier of the source object
ObjectGUID (GUID, Body, Required): GUID of the source object
ContentType (string, Body, Required): MIME content type of the document
DocumentType (enum, Body, Required): Type of document (e.g., "JSON", "Pdf", "Text")
SourceUrl (string, Body, Required): URL from which the source object can be retrieved
UdrDocument (object, Body, Required): Unified Document Representation containing processed content, terms, and metadata

Optional Parameters

BucketGUID (GUID, Body, Optional): GUID of the bucket where the source object is stored
DataRepositoryGUID (GUID, Body, Optional): GUID of the data repository
ContentLength (long, Body, Optional): Length of the document content in bytes
MD5Hash (string, Body, Optional): MD5 hash of the document content
SHA1Hash (string, Body, Optional): SHA1 hash of the document content
SHA256Hash (string, Body, Optional): SHA256 hash of the document content

Response

Returns the created source document object with all metadata and processing information:

{
    "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "CollectionGUID": "default",
    "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentType": "application/json",
    "DocumentType": "JSON",
    "SourceUrl": "http://localhost:9000/tenants/default/buckets/data/objects/sample.json",
    "ContentLength": 1024,
    "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
    "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
    "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
    "CreatedUtc": "2024-10-25T14:14:22.000000Z"
}

Read Source Document

Retrieves a specific source document by GUID using GET /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/[document-guid]. Returns the complete source document with metadata, content information, and processing details.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
document-guid (string, Path, Required): GUID of the source document to retrieve

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/0000000000000/documents/fd937de1-480a-4db8-9025-c7ac0bd8d66c' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveSourceDocument = async () => {
  try {
    const response = await api.sourceDocumentSdk.read(
      "<collection-guid>",
      "<sourcedocument-guid>"
    );
    console.log(response, "SourceDocument fetched successfully");
  } catch (err) {
    console.log("Error fetching SourceDocument:", err);
  }
};

retrieveSourceDocument();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readSourceDocument():
    document = lexi.SourceDocument.retrieve("<collection-guid>", "<sourcedocument-guid>")
    print(document)

readSourceDocument()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
SourceDocument sourceDocument = await sdk.SourceDocument.Retrieve(Guid.Parse("<collection-guid>"),Guid.Parse("<sourcedocument-guid>"));

Response Structure

Returns the source document object with all metadata and content information if found, or a 404 Not Found error if the document doesn't exist.

Note: The HEAD method can be used as an alternative to simply check the existence of the object. HEAD requests return either a 200 OK if the object exists, or a 404 Not Found if not. No response body is returned with a HEAD request.

Response

Returns the complete source document with metadata:

{
    "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "CollectionGUID": "default",
    "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
    "ObjectKey": "5.pdf",
    "ObjectVersion": "1",
    "ContentType": "application/pdf",
    "DocumentType": "Pdf",
    "SourceUrl": "http://dcc249eaaf06:8001/v1.0/tenants/default/buckets/example-data-bucket/objects/5.pdf",
    "ContentLength": 31811,
    "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
    "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
    "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
    "CreatedUtc": "2024-10-25T14:14:22.000000Z"
}

Read Source Document with Data

Retrieves a source document including its processed content data using GET /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/[document-guid]?incldata=null. Returns the complete source document with UDR content and processed data.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
document-guid (string, Path, Required): GUID of the source document to retrieve
incldata (string, Query, Required): Must be set to "null" to include processed content data

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents/fd937de1-480a-4db8-9025-c7ac0bd8d66c?incldata=null' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveSourceDocumentWithData = async () => {
  try {
    const response = await api.sourceDocumentSdk.read(
      "<collection-guid>",
      "<sourcedocument-guid>",
      true
    );
    console.log(response, "SourceDocument fetched successfully");
  } catch (err) {
    console.log("Error fetching SourceDocument:", err);
  }
};

retrieveSourceDocumentWithData();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readSourceDocument():
    document = lexi.SourceDocument.retrieve("<collection-guid>", "<sourcedocument-guid>",True)
    print(document)

readSourceDocument()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
SourceDocument response = await sdk.SourceDocument.Retrieve(Guid.Parse("<collection-guid>"),Guid.Parse("<sourcedocument-guid>"), includeData: true);

Response Structure

Returns the source document object with complete metadata and processed content data, or a 404 Not Found error if the document doesn't exist.

Response

Returns the source document with complete metadata and processed content data:

{
    "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "CollectionGUID": "default",
    "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
    "ObjectKey": "5.pdf",
    "ObjectVersion": "1",
    "ContentType": "application/pdf",
    "DocumentType": "Pdf",
    "SourceUrl": "http://dcc249eaaf06:8001/v1.0/tenants/default/buckets/example-data-bucket/objects/5.pdf",
    "ContentLength": 31811,
    "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
    "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
    "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
    "CreatedUtc": "2024-10-25T14:14:22.000000Z",
    "UdrDocument": {
        "Success": true,
        "AdditionalData": "Parsed successfully",
        "Key": "5.pdf:1",
        "Type": "Pdf",
        "Metadata": {
            "Author": "John Doe",
            "Title": "Sample Document"
        },
        "Terms": ["document", "content", "analysis", "text"],
        "TopTerms": {
            "document": 15,
            "content": 12,
            "analysis": 8,
            "text": 6
        },
        "Postings": [
            {
                "Term": "document",
                "Count": 15,
                "AbsolutePositions": [0, 10, 20],
                "RelativePositions": [0, 1, 2]
            }
        ],
        "Schema": {
            "Type": "Pdf",
            "MaxDepth": 2,
            "NumObjects": 5,
            "NumArrays": 0,
            "NumKeyValues": 10
        }
    }
}

Read Source Document by Key and Version

Read a source document by object key and version using GET /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/?key=blake.json&versionId=1. Provides an alternative method for reading document when GUID is not available.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
key (string, Query, Required): Object key/name of the source document to delete
versionId (string, Query, Required): Version identifier of the source document to delete

Response

200 OK: Source document deleted successfully
404 Not Found: Source document with specified key and version does not exist

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents?key=blake.json&versionId=1' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const readSourceDocumentByKey = async () => {
  try {
    const response = await api.sourceDocument.readByKey('00000000-0000-0000-0000-000000000000', '35.pdf', '1');
    console.log(response);
  } catch (error) {
    console.error('Error reading source document:', error);
  }
};

readSourceDocumentByKey();

Read Top Terms

Retrieves the most frequently occurring terms within a specific source document using GET /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/[document-guid]/topterms?max-keys=10. Useful for content analysis and term frequency analysis of individual documents.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
document-guid (string, Path, Required): GUID of the source document to analyze
max-keys (integer, Query, Optional): Maximum number of top terms to return (default: 10)

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents/fd937de1-480a-4db8-9025-c7ac0bd8d66c/topterms?max-keys=10' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveSourceDocumentTopTerms = async () => {
  try {
    const response = await api.sourceDocumentSdk.readTopTerms(
      "<collection-guid>",
      "<sourcedocument-guid>"
    );
    console.log(response, "SourceDocument top terms fetched successfully");
  } catch (err) {
    console.log("Error fethcing SourceDocument top terms:", err);
  }
};

retrieveSourceDocumentTopTerms();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readTopTerms():
    terms = lexi.SourceDocument.retrieve_top_terms("<collection-guid>", "<sourcedocument-guid>")
    print(terms)

readTopTerms()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
CollectionTopTerms response = await sdk.SourceDocument.RetrieveTopTerms(Guid.Parse("<collection-guid>"),Guid.Parse("<sourcedocument-guid>"), maxKeys: 5);

Response Structure

Returns a JSON object with terms as keys and their frequency counts as values, or a 404 Not Found error if the document doesn't exist.

Response

Returns the most frequently occurring terms within the specific document:

{
  "document": 15,
  "content": 12,
  "analysis": 8,
  "text": 6,
  "processing": 4
}

Read Statistics

Retrieves comprehensive statistics for a specific source document using GET /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/[document-guid]?stats=null. Provides detailed metrics about document content, processing results, and analytical data.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
document-guid (string, Path, Required): GUID of the source document to analyze
stats (string, Query, Required): Must be set to "null" to retrieve statistics

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents/fd937de1-480a-4db8-9025-c7ac0bd8d66c?stats=null' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveSourceDocumentStatistics = async () => {
  try {
    const response = await api.sourceDocumentSdk.readStatistics(
      "<collection-guid>",
      "<sourcedocument-guid>"
    );
    console.log(response, "SourceDocument stats fetched successfully");
  } catch (err) {
    console.log("Error fethcing SourceDocument stats:", err);
  }
};

retrieveSourceDocumentStatistics();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readSourceDocumentStatistics():
    statistics = lexi.SourceDocument.retrieve_statistics("<collection-guid>", "<sourcedocument-guid>")
    print(statistics)

readSourceDocumentStatistics()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http:/localhost:8000/");
            
SourceDocumentStatistics sourceDocumentStatistics = await sdk.SourceDocument.RetrieveTopTerms(Guid.Parse("<collection-guid>"),Guid.Parse("<sourcedocument-guid>"));

Response

Returns a JSON object containing comprehensive document statistics including content metrics, processing information, and analytical data, or a 404 Not Found error if the document doesn't exist.

Response

Returns comprehensive document statistics:

{
    "Document": {
        "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
        "TenantGUID": "default",
        "BucketGUID": "example-data-bucket",
        "CollectionGUID": "default",
        "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
        "ObjectKey": "5.pdf",
        "ObjectVersion": "1",
        "ContentType": "application/pdf",
        "DocumentType": "Pdf",
        "SourceUrl": "http://dcc249eaaf06:8001/v1.0/tenants/default/buckets/example-data-bucket/objects/5.pdf",
        "ContentLength": 31811,
        "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
        "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
        "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
        "CreatedUtc": "2024-10-25T14:14:22.000000Z"
    },
    "TermCount": 45,
    "UniqueTerms": 32,
    "AverageTermLength": 6.2,
    "ProcessingTimeMs": 1250,
    "SchemaComplexity": 0.75
}

Read All Source Documents

Retrieves all source documents within a collection using GET /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents. Returns a JSON array containing all source documents with their metadata and content information.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection to retrieve documents from

Response

Returns an array of all source documents in the collection, or a 404 Not Found error if no documents exist.

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const retrieveSourceDocuments = async () => {
  try {
    const response = await api.sourceDocumentSdk.readAll(
      "<collection-guid>"
    );
    console.log(response, "SourceDocuments fetched successfully");
  } catch (err) {
    console.log("Error fetching SourceDocuments:", err);
  }
};

retrieveSourceDocuments();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def readAllSourceDocuments():
    documents = lexi.SourceDocument.retrieve_all("<collection-guid>")
    print(documents)

readAllSourceDocuments()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
List<SourceDocument> sourceDocuments = await sdk.SourceDocument.RetrieveMany(Guid.Parse("<collection-guid>"));

Response

Returns an array of all source documents in the collection:

[
    {
        "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
        "TenantGUID": "default",
        "BucketGUID": "example-data-bucket",
        "CollectionGUID": "default",
        "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
        "ObjectKey": "5.pdf",
        "ObjectVersion": "1",
        "ContentType": "application/pdf",
        "DocumentType": "Pdf",
        "SourceUrl": "http://dcc249eaaf06:8001/v1.0/tenants/default/buckets/example-data-bucket/objects/5.pdf",
        "ContentLength": 31811,
        "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
        "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
        "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
        "CreatedUtc": "2024-10-25T14:14:22.000000Z"
    },
    {
        "GUID": "another-document-guid",
        "TenantGUID": "default",
        "BucketGUID": "example-data-bucket",
        "CollectionGUID": "default",
        "ObjectGUID": "another-object-guid",
        "ObjectKey": "document.docx",
        "ObjectVersion": "1",
        "ContentType": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
        "DocumentType": "Word",
        "SourceUrl": "http://dcc249eaaf06:8001/v1.0/tenants/default/buckets/example-data-bucket/objects/document.docx",
        "ContentLength": 25600,
        "MD5Hash": "A1B2C3D4E5F6789012345678901234567890ABCD",
        "SHA1Hash": "1234567890ABCDEF1234567890ABCDEF12345678",
        "SHA256Hash": "ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890",
        "CreatedUtc": "2024-10-25T15:30:45.123456Z"
    }
]

Upload Source Document

Uploads a source document to a collection using PUT /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents. Creates a new source document entry with comprehensive metadata and UDR processing information for search and analysis capabilities.

Request Parameters

Required Parameters

TenantGUID (GUID, Body, Required): GUID of the tenant
CollectionGUID (GUID, Body, Required): GUID of the target collection
ObjectKey (string, Body, Required): Key/name of the source object
ObjectVersion (string, Body, Required): Version identifier of the source object
ObjectGUID (GUID, Body, Required): GUID of the source object
ContentType (string, Body, Required): MIME content type of the document
DocumentType (enum, Body, Required): Type of document (e.g., "JSON", "Pdf", "Text")
SourceUrl (string, Body, Required): URL from which the source object can be retrieved
UdrDocument (object, Body, Required): Unified Document Representation containing processed content, terms, and metadata

Response

Returns the uploaded source document object with all metadata and processing information:

{
    "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
    "TenantGUID": "00000000-0000-0000-0000-000000000000",
    "BucketGUID": "example-data-bucket",
    "CollectionGUID": "00000000-0000-0000-0000-000000000000",
    "ObjectGUID": "00000000-0000-0000-0000-000000000000",
    "ObjectKey": "blake.json",
    "ObjectVersion": "1",
    "ContentType": "application/json",
    "DocumentType": "JSON",
    "SourceUrl": "http://localhost:9000/tenants/default/buckets/data/objects/sample.json",
    "ContentLength": 1024,
    "MD5Hash": "DC477A85FF3882BBFDEB03D7B79ECC9E",
    "SHA1Hash": "CC5D85073F193A578F97D46B8A6E4CE946270B5F",
    "SHA256Hash": "E5285C6023A46E4E8917C67CCB56B91FED2E578A7AA3129680012C029868B321",
    "CreatedUtc": "2024-10-25T14:14:22.000000Z"
}

curl --location --request PUT 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "TenantGUID": "00000000-0000-0000-0000-000000000000",
  "CollectionGUID": "00000000-0000-0000-0000-000000000000",
  "ObjectKey": "blake.json",
  "ObjectVersion": "1",
  "ObjectGUID": "00000000-0000-0000-0000-000000000000",
  "ContentType": "application/json",
  "DocumentType": "JSON",
  "SourceUrl": "http://localhost:9000/tenants/default/buckets/data/objects/sample.json",
  "UdrDocument": {
    "Success": true,
    "AdditionalData": "My additional data",
    "Metadata": {
      "Foo": "Bar"
    },
    "Key": "sample.json",
    "TypeResult": {
        "MimeType": "application/json",
        "Extension": "json",
        "Type": "Json"
    },
    "Terms": [
        "foo",
        "bar",
        "baz"
    ],
    "TopTerms": {
      "foo": 1,
      "bar": 1,
      "baz": 1
    },
    "Postings": [
      {
        "Term": "baz",
        "Count": 2,
        "AbsolutePositions": [
          0
        ],
        "RelativePositions": [
          0
        ]
      },
      {
        "Term": "foo",
        "Count": 2,
        "AbsolutePositions": [
          1
        ],
        "RelativePositions": [
          1
        ]
      },
      {
        "Term": "bar",
        "Count": 2,
        "AbsolutePositions": [
          2
        ],
        "RelativePositions": [
          2
        ]
      }
    ],
    "Schema": {
      "Type": "Json",
      "MaxDepth": 1,
      "NumObjects": 1,
      "NumArrays": 0,
      "NumKeyValues": 1,
      "Schema": {
        "root": "Object",
        "root.Message": "String"
      },
      "Metadata": {
        "Foo": "Bar"
      },
      "Flattened": [
        {
          "Key": "root",
          "Type": "Object"
        },
        {
          "Key": "root.Message",
          "Type": "String",
          "Data": "Your foo is bar baz!"
        }
      ]
    }
  }
}
'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const uploadSourceDocument = async () => {
  try {
    const response = await api.sourceDocumentSdk.upload({
      TenantGUID: "<tenant-guid>",
      CollectionGUID: "<collection-guid>",
      ObjectKey: "blake.json",
      ObjectVersion: "1",
      ObjectGUID: "<object-guid>",
      ContentType: "application/json",
      DocumentType: "JSON",
      SourceUrl:
        "http://localhost:9000/tenants/default/buckets/data/objects/sample.json",
      UdrDocument: {
        Success: true,
        AdditionalData: "My additional data",
        Metadata: {
          Foo: "Bar",
        },
        Key: "sample.json",
        TypeResult: {
          MimeType: "application/json",
          Extension: "json",
          Type: "Json",
        },
        Terms: ["foo", "bar", "baz"],
        TopTerms: {
          foo: 1,
          bar: 1,
          baz: 1,
        },
        Postings: [
          {
            Term: "baz",
            Count: 2,
            AbsolutePositions: [0],
            RelativePositions: [0],
          },
          {
            Term: "foo",
            Count: 2,
            AbsolutePositions: [1],
            RelativePositions: [1],
          },
          {
            Term: "bar",
            Count: 2,
            AbsolutePositions: [2],
            RelativePositions: [2],
          },
        ],
        Schema: {
          Type: "Json",
          MaxDepth: 1,
          NumObjects: 1,
          NumArrays: 0,
          NumKeyValues: 1,
          Schema: {
            root: "Object",
            "root.Message": "String",
          },
          Metadata: {
            Foo: "Bar",
          },
          Flattened: [
            {
              Key: "root",
              Type: "Object",
            },
            {
              Key: "root.Message",
              Type: "String",
              Data: "Your foo is bar baz!",
            },
          ],
        },
      },
    });
    console.log(response, "SourceDocument uploaded successfully");
  } catch (err) {
    console.log("Error uploading SourceDocument:", err);
  }
};

uploadSourceDocument();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def uploadSourceDocument():
    document = lexi.SourceDocument.upload("<collection-guid>", "<object-guid>", "foo.txt")
    print(document)

uploadSourceDocument()

using View.Sdk;
using View.Sdk.Lexi;
using View.Sdk.Semantic;
using System.Collections.Generic;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
            
        SourceDocument sourceDocument = new SourceDocument
        {
            GUID = Guid.Parse("<sourcedocument-guid>"),
            TenantGUID = Guid.Parse("<tenant-guid>"),
            BucketGUID = Guid.Parse("<bucket-guid>"),
            CollectionGUID = Guid.Parse("<collection-guid>"),
            ObjectGUID = Guid.Parse("<object-guid>"),
            GraphRepositoryGUID = null,
            GraphNodeIdentifier = null,
            DataRepositoryGUID = null,
            DataFlowRequestGUID = null,
            DataFlowSuccess = null,
            ObjectKey = "10.pdf",
            ObjectVersion = "1",
            ContentType = "application/pdf",
            DocumentType = DocumentTypeEnum.Unknown,
            SourceUrl = "http://localhost:9000/tenants/default/buckets/sample/objects/10.pdf",
            ContentLength = 29096038,
            MD5Hash = "******************************0FCA",
            SHA1Hash = "*******************************************50F",
            SHA256Hash = "*******************************************************D3D",
            CreatedUtc = DateTime.UtcNow,
            ExpirationUtc = null,
            Score = new DocumentScore
            {
                Score = 0.85m,
                TermsScore = 0.9m,
                FiltersScore = 0.75m
            },
            UdrDocument = new UdrDocument
             {
                GUID = Guid.NewGuid(),
                Success = true,
                AdditionalData = "Parsed successfully.",
                Key = "sample.pdf:1",
                Type = DocumentTypeEnum.Pdf,
                Metadata = new Dictionary<string, object>
                {
                    { "Author", "John Doe" },
                    { "Category", "Medical" }
                },
                Terms = new List<string>
                {
                    "botox", "treatment", "dose", "patient",
                    "botox", "injection", "dose", "patient",
                    "botox"
                },
                Postings = new List<Posting>
                {
                    new Posting
                    {
                        Term = "botox",
                        AbsolutePositions = new List<long> { 5, 15, 25 },
                        RelativePositions = new List<long> { 0, 1, 2 }
                    },
                    new Posting
                    {
                        Term = "dose",
                        AbsolutePositions = new List<long> { 35, 55 },
                        RelativePositions = new List<long> { 3, 4 }
                    }
                },
                Schema = new SchemaResult
                {
                    Type = DocumentTypeEnum.Pdf,
                    Schema = new Dictionary<string, DataTypeEnum>
                    {
                        { "root", DataTypeEnum.Object },
                        { "root.Message", DataTypeEnum.String }
                    },

                    Metadata = new Dictionary<string, object>
                    {
                        { "KeyCount", 2 }
                    },
                },
                SemanticCells = new List<SemanticCell>
                {
                    new SemanticCell
                    {
                        GUID = Guid.NewGuid(),
                        CellType = SemanticCellTypeEnum.Text,
                        Position = 0,
                        Chunks = new List<SemanticChunk>
                        {
                            new SemanticChunk
                            {
                                GUID = Guid.NewGuid(),
                                Position = 0,
                                Start = 0,
                                End = 127,
                                Length = 128,
                                Content = "Botox treatment is effective." 
                            }
                        }
                    }
                }
            }
        };

SourceDocument result = await sdk.SourceDocument.Upload(sourceDocument);

Response

Returns the uploaded source document with complete metadata and UDR processing:

{
    "GUID": "1fdbe0c8-8b85-4b0e-ac42-dd4757684a9f",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "CollectionGUID": "default",
    "ObjectGUID": "f615ac92-d1d1-4b46-8cc5-acf721131067",
    "ObjectKey": "10.pdf",
    "ObjectVersion": "1",
    "ContentType": "application/pdf",
    "DocumentType": "Pdf",
    "SourceUrl": "http://localhost:9000/tenants/default/buckets/sample/objects/10.pdf",
    "ContentLength": 29096038,
    "MD5Hash": "******************************0FCA",
    "SHA1Hash": "*******************************************50F",
    "SHA256Hash": "*******************************************************D3D",
    "CreatedUtc": "2024-10-25T14:14:22.000000Z",
    "Score": {
        "Score": 0.85,
        "TermsScore": 0.9,
        "FiltersScore": 0.75
    },
    "UdrDocument": {
        "GUID": "udr-document-guid",
        "Success": true,
        "AdditionalData": "Parsed successfully.",
        "Key": "sample.pdf:1",
        "Type": "Pdf",
        "Metadata": {
            "Author": "John Doe",
            "Category": "Medical"
        },
        "Terms": ["botox", "treatment", "dose", "patient"],
        "TopTerms": {
            "botox": 15,
            "treatment": 12,
            "dose": 8,
            "patient": 6
        },
        "Postings": [
            {
                "Term": "botox",
                "Count": 15,
                "AbsolutePositions": [5, 15, 25],
                "RelativePositions": [0, 1, 2]
            }
        ],
        "Schema": {
            "Type": "Pdf",
            "MaxDepth": 2,
            "NumObjects": 5,
            "NumArrays": 0,
            "NumKeyValues": 10
        }
    }
}

Delete Source Document

Deletes a source document by GUID using DELETE /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/[document-guid]. Removes the document from the collection and all associated metadata.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
document-guid (string, Path, Required): GUID of the source document to delete

Response

200 OK: Source document deleted successfully
404 Not Found: Source document does not exist

Note: Deleting a source document removes it permanently from the collection and all associated search indexes.

curl --location --request DELETE 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents/00000000-0000-0000-0000-000000000000' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const deleteSourceDocument = async () => {
  try {
    const response = await api.sourceDocumentSdk.delete(
      "<collection-guid>",
      "<sourcedocument-guid>"
    );
    console.log(response, "SourceDocument deleted successfully");
  } catch (err) {
    console.log("Error deleting SourceDocument:", err);
  }
};

deleteSourceDocument();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def deleteSourceDocument():
    response = lexi.SourceDocument.delete("document-guid","collection-guid")
    print(response)


deleteCollection()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
        
Guid CollectionGuid = Guid.Parse("<collection-guid>");
        
Guid SourceDocumentGuid = Guid.Parse("<sourcedocument-guid>");
        
bool deleted = await sdk.SourceDocument.Delete(CollectionGuid, SourceDocumentGuid);

Response

Returns 200 No Content on successful deletion. No response body is returned.

Delete Source Document by Key and Version

Deletes a source document by object key and version using DELETE /v1.0/tenants/[tenant-guid]/collections/[collection-guid]/documents/?key=blake.json&versionId=1. Provides an alternative method for document deletion when GUID is not available.

Request Parameters

collection-guid (string, Path, Required): GUID of the collection containing the document
key (string, Query, Required): Object key/name of the source document to delete
versionId (string, Query, Required): Version identifier of the source document to delete

Response

200 OK: Source document deleted successfully
404 Not Found: Source document with specified key and version does not exist

Note: This method provides an alternative way to delete documents when you have the object key and version but not the document GUID.

curl --location --request DELETE 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/collections/00000000-0000-0000-0000-000000000000/documents?key=blake.json&versionId=1' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••'

import { ViewLexiSdk } from "view-sdk";

const api = new ViewLexiSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const deleteSourceDocumentFromKey = async () => {
  try {
    const response = await api.sourceDocumentSdk.deleteFromKey(
      "<collection-guid>",
      "https://www.traegerforum.com/forums/traeger-recipes.27/", //key
      "1" //version
    );
    console.log(response, "SourceDocument deleted successfully");
  } catch (err) {
    console.log("Error deleting SourceDocument:", err);
  }
};

deleteSourceDocumentFromKey();

import view_sdk
from view_sdk import lexi
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="tenant-guid",
    service_ports={Service.LEXI:8000},
)

def uploadSourceDocument():
    document = lexi.SourceDocument.create("00000000-0000-0000-0000-000000000000",
        TenantGUID="00000000-0000-0000-0000-000000000000",
        CollectionGUID="00000000-0000-0000-0000-000000000000",
        ObjectKey="blake.json",
        ObjectVersion="1",
        ObjectGUID="00000000-0000-0000-0000-000000000000",
        ContentType="application/json",
        DocumentType="JSON",
        SourceUrl="http://localhost:9000/tenants/default/buckets/data/objects/sample.json",
        UdrDocument={}
    )
    print(document)

uploadSourceDocument()

using View.Sdk;
using View.Sdk.Lexi;

ViewLexiSdk sdk = new ViewLexiSdk(Guid.Parse("<tenant-guid>"),"default", "http://localhost:8000/");
        
bool deleted = await sdk.SourceDocument.Delete(Guid.Parse("<collection-guid>"), "10.pdf", "1");

Response

Returns 200 No Content on successful deletion. No response body is returned.

Best Practices

When managing source documents in the Lexi metadata database, consider the following recommendations for optimal document processing, search performance, and data management:

Document Organization: Organize documents within logical collections based on content type, source, or business domain for better search and management
Metadata Quality: Ensure comprehensive and accurate metadata is provided during document upload for optimal search and analysis capabilities
Content Processing: Use proper UDR document structure with complete term extraction and semantic analysis for enhanced search functionality
Version Management: Implement proper versioning strategies for document updates and maintain document history for audit and rollback capabilities
Performance Optimization: Monitor document statistics and processing metrics to optimize content analysis and search performance

Next Steps

After successfully managing source documents, you can:

Search Operations: Implement advanced search functionality using document content, metadata, and term analysis capabilities
Content Analysis: Analyze document statistics and top terms to gain insights into content patterns and trends
Collection Management: Organize and manage document collections for better content discovery and organization
Integration: Integrate source document management with other View platform services for comprehensive data processing workflows
Document Processing: Set up automated document processing pipelines for continuous content ingestion and analysis