API Reference

Embeddings Rules

This page covers configuration and management of View embeddings rule objects.

Object Overview

Embeddings rules define how embeddings are generated and where resultant data is stored within View.

Endpoint, URL, and Supported Methods

Objects are managed via the configuration server API at [http|https]://[hostname]:[port]/v1.0/tenants/[tenant-guid]/embeddingsrules

By default, the configuration server is accessible on port 8601.

Supported methods include: GET HEAD PUT DELETE

Structure

Objects have the following structure:

{
    "GUID": "embeddings-rule",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "OwnerGUID": "default",
    "Name": "My embeddings rule",
    "ContentType": "*",
    "TargetBucketGUID": "example-embeddings-bucket",
    "GraphRepositoryGUID": "example-graph-repository",
    "VectorRepositoryGUID": "example-vector-repository",
    "DataFlowEndpoint": "http://nginx-orchestrator:8501/processor",
    "EmbeddingsGenerator": "LCProxy",
    "GeneratorUrl": "http://nginx-embeddings:8301/",
    "GeneratorApiKey": "",
    "BatchSize": 16,
    "MaxGeneratorTasks": 16,
    "MaxRetries": 3,
    "MaxFailures": 3,
    "VectorStoreUrl": "http://nginx-vector:8311/",
    "MaxContentLength": 16777216,
    "CreatedUtc": "2024-07-10T05:09:32.000000Z"
}

Properties:

  • GUID string globally unique identifier for the object
  • TenantGUID string globally unique identifier for the tenant
  • BucketGUID string GUID of the bucket to which this embeddings rule should be associated
  • OwnerGUID string GUID of the user to which this rule should be attributed
  • Name string name of the rule
  • ContentType string content-type on which this rule should match
  • TargetBucketGUID string the GUID of the bucket to which results should be emitted
  • GraphRepositoryGUID string the GUID of the graph repository to which embeddings should be emitted
  • VectorRepositoryGUID string the GUID of the vector repository to which embeddings should be emitted
  • DataFlowEndpoint string the endpoint URL of the processing endpoint
  • EmbeddingsGenerator enum the embeddings generator to use (currently only LCProxy
  • GeneratorUrl string the URL of the embeddings generator
  • GeneratorApiKey string the HuggingFace API key used to interact with models on HuggingFace
  • BatchSize int maximum number of chunks to process per batch
  • MaxGeneratorTasks int maximum number of parallel tasks to invoke for embeddings generation
  • MaxRetries int maximum number of times to attempt embeddings generation against a given batch before failing
  • MaxFailure int maximum number of failures to accept before declaring that a job has failed
  • VectorStoreUrl string URL to vector server where embeddings should be stored
  • MaxContentLength int maximum content length of data to process
  • CreatedUtc datetime timestamp from creation, in UTC time

Create

To create, call PUT /v1.0/tenants/[tenant-guid]/embeddingsrules with the following properties using the configuration server: BucketGUID OwnerGUID Name ContentType TargetBucketGUID EmbeddingsGenerator GeneratorUrl GeneratorApiKey VectorRepositoryGUID

curl -X PUT http://localhost:8601/v1.0/tenants/[tenant-guid]/metadatarules \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [accesskey]" \
     -d '
{
    "BucketGUID": "example-data-bucket",
    "OwnerGUID": "default",
    "Name": "Embeddings rule",
    "ContentType": "*",
    "TargetBucketGUID": "example-embeddings-bucket",
    "EmbeddingsGenerator": "LCProxy",
    "GeneratorUrl": "http://localhost:8301/",
    "GeneratorApiKey": "[huggingface API key]",
    "VectorRepositoryGUID": "example-vector-repository"
}'

Enumerate

Refer to the Enumeration page in REST API for details about the use of enumeration APIs.

Enumerate objects by using GET /v2.0/tenants/[tenant-guid]/embeddingsrules. The resultant object will appear as:

{
    "Success": true,
    "Timestamp": {
        "Start": "2024-10-21T02:36:37.677751Z",
        "TotalMs": 23.58,
        "Messages": {}
    },
    "MaxResults": 10,
    "IterationsRequired": 1,
    "EndOfResults": true,
    "RecordsRemaining": 16,
    "Objects": [
        {
            "GUID": "example-embeddingsrule",
            ... embeddingsrule details ...
        },
        { ... }
    ],
    "ContinuationToken": "[continuation-token]"
}

Read

To read an object by GUID, call GET /v1.0/tenants/[tenant-guid]/embeddingsrule/[embeddingsrule-guid]. If the object exists, it will be returned as a JSON object in the response body. If it does not exist, a 404 will be returned with a NotFound error response.

{
    "GUID": "59f7c18d-7342-4aef-9308-86083459dd81",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "OwnerGUID": "default",
    "Name": "Embeddings rule",
    "ContentType": "*",
    "TargetBucketGUID": "example-embeddings-bucket",
    "VectorRepositoryGUID": "example-vector-repository",
    "DataFlowEndpoint": "http://localhost:8501/processor",
    "EmbeddingsGenerator": "LCProxy",
    "GeneratorUrl": "http://localhost:8301/",
    "GeneratorApiKey": "[huggingface API key]",
    "BatchSize": 16,
    "MaxGeneratorTasks": 16,
    "MaxRetries": 3,
    "MaxFailures": 3,
    "VectorStoreUrl": "http://localhost:8311/",
    "MaxContentLength": 16777216,
    "CreatedUtc": "2024-10-21T15:19:09.000000Z"
}

Note: the HEAD method can be used as an alternative to get to simply check the existence of the object. HEAD requests return either a 200/OK in the event the object exists, or a 404/Not Found if not. No response body is returned with a HEAD request.

Update

To update an object by GUID, call PUT /v1.0/tenants/[tenant-guid]/embeddingsrule/[embeddingsrule-guid] with a fully populated object in the request body. The updated object will be returned to you.

Note: certain fields cannot be modified and will be preserved across updates.

Request body:

{
    "BucketGUID": "example-data-bucket",
    "OwnerGUID": "default",
    "Name": "My updated embeddings rule",
    "ContentType": "*",
    "TargetBucketGUID": "example-embeddings-bucket",
    "VectorRepositoryGUID": "example-vector-repository",
    "DataFlowEndpoint": "http://localhost:8501/processor",
    "EmbeddingsGenerator": "LCProxy",
    "GeneratorUrl": "http://localhost:8301/",
    "GeneratorApiKey": "[huggingface API key]",
    "BatchSize": 16,
    "MaxGeneratorTasks": 16,
    "MaxRetries": 3,
    "MaxFailures": 3,
    "VectorStoreUrl": "http://localhost:8311/",
    "MaxContentLength": 16777216
}

Response body:

{
    "GUID": "59f7c18d-7342-4aef-9308-86083459dd81",
    "TenantGUID": "default",
    "BucketGUID": "example-data-bucket",
    "OwnerGUID": "default",
    "Name": "My updated embeddings rule",
    "ContentType": "*",
    "TargetBucketGUID": "example-embeddings-bucket",
    "VectorRepositoryGUID": "example-vector-repository",
    "DataFlowEndpoint": "http://localhost:8501/processor",
    "EmbeddingsGenerator": "LCProxy",
    "GeneratorUrl": "http://localhost:8301/",
    "GeneratorApiKey": "[huggingface API key]",
    "BatchSize": 16,
    "MaxGeneratorTasks": 16,
    "MaxRetries": 3,
    "MaxFailures": 3,
    "VectorStoreUrl": "http://localhost:8311/",
    "MaxContentLength": 16777216,
    "CreatedUtc": "2024-10-21T15:19:09.000000Z"
}

Delete

To delete an object by GUID, call DELETE /v1.0/tenants/[tenant-guid]/embeddingsrule/[embeddingsrule-guid].