Generating Embeddings - View Platform

Overview

Embeddings generation provides comprehensive AI-powered content analysis and vector generation capabilities within the View platform. It enables automated generation of vector embeddings from text content, semantic cells, and data chunks for enhanced search, similarity matching, and AI-powered content understanding.

Embeddings generation is accessible via the View Embeddings server at [http|https]://[hostname]:[port]/v1.0/tenants/[tenant-guid]/embeddings/ and supports comprehensive vector generation and embeddings management.

API Endpoints

POST /v1.0/tenants/[tenant-guid]/embeddings/ - Generate embeddings for content

Generate Embeddings

Generates vector embeddings for content using POST /v1.0/tenants/[tenant-guid]/embeddings/. Creates high-dimensional vector representations of text content for AI-powered analysis, search, and similarity matching.

Request Parameters

Required Parameters

EmbeddingsRule (object, Body, Required): Embeddings rule configuration for vector generation
Model (string, Body, Required): Model name for embeddings generation
Contents (array, Body, Required): Array of content strings for embeddings generation

Optional Parameters

ApiKey (string, Body, Optional): HuggingFace API key for models requiring authentication
EmbeddingsRule Configuration:
- EmbeddingsGenerator (string, Body, Required): Name of the embeddings generator service (e.g., "LCProxy")
- EmbeddingsGeneratorUrl (string, Body, Required): URL of the embeddings generator endpoint
- EmbeddingsGeneratorApiKey (string, Body, Required): API key for embeddings generator authentication
- BatchSize (number, Body, Required): Number of items to process in a single batch request
- MaxGeneratorTasks (number, Body, Required): Maximum number of concurrent generator tasks allowed
- MaxRetries (number, Body, Required): Number of times to retry a failed generation task
- MaxFailures (number, Body, Required): Number of allowed failures before halting further processing

{
    "EmbeddingsRule": {
        "EmbeddingsGenerator": "LCProxy",
        "EmbeddingsGeneratorUrl": "http://nginx-lcproxy:8000/",
        "EmbeddingsGeneratorApiKey": "default",
        "BatchSize": 16,
        "MaxGeneratorTasks": 16,
        "MaxRetries": 3,
        "MaxFailures": 3
    },
    "Model": "all-MiniLM-L6-v2",
    "ApiKey": "",
    "Contents": [
        "This is a sample chunk of text, hello!",
    ]
}

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/embeddings/' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "EmbeddingsRule": {
        "EmbeddingsGenerator": "LCProxy",
        "EmbeddingsGeneratorUrl": "http://nginx-lcproxy:8000/",
        "EmbeddingsGeneratorApiKey": "default",
        "BatchSize": 16,
        "MaxGeneratorTasks": 16,
        "MaxRetries": 3,
        "MaxFailures": 3
    },
    "Model": "all-MiniLM-L6-v2",
    "ApiKey": "",
    "Contents": [
        "This is a sample chunk of text, hello!",
    ]
}'

import { ViewEmbeddingsSdk } from "view-sdk";

  const api = new ViewEmbeddingsSdk(
  "http://localhost:8000/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access token
);

const generateEmbeddings = async () => {
  try {
    const response = await api.generateEmbeddings({
      EmbeddingsRule: {
        EmbeddingsGenerator: "LCProxy",
        EmbeddingsGeneratorUrl: "http://nginx-lcproxy:8000/",
        EmbeddingsGeneratorApiKey: "default",
        BatchSize: 16,
        MaxGeneratorTasks: 16,
        MaxRetries: 3,
        MaxFailures: 3,
      },
      Model: "all-MiniLM-L6-v2",
      ApiKey: "",
      Contents: ["This is a sample chunk of text, hello!"],
    });
    console.log(response, "Generate embeddings response");
  } catch (err) {
    console.log("Error", err);
  }
};

generateEmbeddings();

import view_sdk
from view_sdk import director
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost", 
    tenant_guid="default",
    service_ports={Service.DIRECTOR: 8000},
)

def generateEmbeddings():
    result = director.GenerateEmbeddings.create(
        EmbeddingsRule={
        "EmbeddingsGenerator": "LCProxy",
        "EmbeddingsGeneratorUrl": "http://nginx-lcproxy:8000/",
        "EmbeddingsGeneratorApiKey": "default",
        "BatchSize": 16,
        "MaxGeneratorTasks": 16,
        "MaxRetries": 3,
        "MaxFailures": 3
        },
        Model="all-MiniLM-L6-v2",
        ApiKey="",
        Contents=[
            "This is a sample.",
        ],
    )
    print(result)

using View.Sdk;
using View.Sdk.Embeddings;
using View.Sdk.Embeddings;
using View.Sdk.Embeddings.Providers;
using View.Sdk.Embeddings.Providers.Ollama; 


Guid _TenantGuid = Guid.Parse("00000000-0000-0000-0000-000000000000");
string _AccessKey = "default";
string _Endpoint = "http://localhost:8000/";
EmbeddingsProviderSdkBase sdk = new ViewOllamaSdk(_TenantGuid, baseUrl: _Endpoint, apiKey: _AccessKey);         
GenerateEmbeddingsRequest request = new GenerateEmbeddingsRequest
{
    EmbeddingsRule = new EmbeddingsRule
    {
        EmbeddingsGenerator = EmbeddingsGeneratorEnum.LCProxy,
        EmbeddingsGeneratorUrl = "http://nginx-lcproxy:8000/",
        EmbeddingsGeneratorApiKey = "default",
        BatchSize = 16,
        MaxGeneratorTasks = 16,
        MaxRetries = 3,
        MaxFailures = 3
    },
    Model = "all-MiniLM-L6-v2",
    Contents = new List<string>
    {
        "This is a sample chunk of text, hello!",
        "Oh wow, here's another chunk of text"
    }
};
GenerateEmbeddingsResult response = await sdk.GenerateEmbeddings(request);

Response

Returns embeddings generation results with success status, batch count, and generated embeddings. The contents of the Embeddings array are ordered to match the input content.

{
    "Success": true,
    "StatusCode": 200,
    "BatchCount": 1,
    "SemanticCells": [],
    "ContentEmbeddings": [
        {
            "Content": "This is a sample chunk of text, hello!",
            "Embeddings": [
                -0.044824813,
                0.08454175,
                0.043137725,
                ...
            ]
        }
    ]
}

Best Practices

When generating embeddings in the View platform, consider the following recommendations for optimal vector generation, AI-powered analysis, and embeddings management:

Model Selection: Choose appropriate embedding models based on your content types, language requirements, and performance needs
Batch Processing: Configure appropriate batch sizes and concurrent tasks to optimize processing performance and resource utilization
Content Preparation: Ensure content is properly formatted and preprocessed before embeddings generation for optimal vector quality
Error Handling: Implement robust error handling and retry mechanisms to manage generation failures and ensure reliability
Performance Monitoring: Monitor embeddings generation performance and optimize parameters for large-scale content processing

Next Steps

After successfully generating embeddings, you can:

Vector Search: Implement advanced vector search operations using generated embeddings for AI-powered content discovery
Similarity Matching: Use vector embeddings for similarity matching and content recommendation systems
AI-Powered Analysis: Leverage embeddings for advanced content analysis, classification, and understanding
Search Integration: Integrate generated embeddings with search capabilities to enable AI-powered content discovery
Content Processing: Continue processing additional content and documents to build comprehensive embeddings databases