Overview

The View Storage Pool management system provides comprehensive configuration for physical data storage infrastructure. Storage pools define where physical data written to object storage buckets resides, offering flexible storage backend options including local disk, cloud storage, and hybrid configurations for optimal data placement and performance.

Key Features

Physical Storage Management: Define and configure physical storage backends for data persistence
Multi-Provider Support: Support for local disk, Amazon S3, and Azure Blob storage backends
Performance Optimization: Configurable compression, caching, and write modes for optimal performance
Storage Allocation: Flexible allocation of storage resources across different pools
Data Placement: Strategic data placement for performance, cost, and compliance requirements
Write Mode Configuration: Support for GUID-based and key-based write modes
Compression Control: Optional compression for storage optimization
Caching Management: Configurable read caching for improved access performance
SSL Configuration: Secure communication options for cloud storage backends

Supported Storage Providers

Disk: Local filesystem storage for on-premises deployments
Amazon S3: Cloud object storage integration with AWS S3
Azure Blob: Microsoft Azure Blob Storage integration
CIFS:Windows file share or SAMBA
NFS:Linux export

Supported Operations

Create: Create new storage pool configurations with provider-specific settings
Read: Retrieve storage pool configuration and metadata
Read All: List all storage pools in the tenant
Update: Modify storage pool configurations and settings
Delete: Remove storage pool configurations
Existence Check: Verify storage pool presence without retrieving details

API Endpoints

Storage pools are managed via the Storage server API at [http|https]://[hostname]:[port]/v1.0/tenants/[tenant-guid]/pools

Default Ports:

View REST API: Port 8001
S3-Compatible API: Port 8002 (Note: Storage pools are not manageable via the S3 API)

Supported HTTP Methods: GET, HEAD, PUT, DELETE

Important: All storage pool operations require appropriate authentication tokens.

Storage Pool Object Structure

Storage pool objects contain comprehensive configuration for physical storage backends. The structure varies based on the storage provider type. Here's the complete structure for a local disk storage pool:

{
    "GUID": "default",
    "TenantGUID": "default",
    "Name": "default",
    "Provider": "Disk",
    "WriteMode": "GUID",
    "UseSsl": false,
    "DiskDirectory": "./disk/",
    "Compress": "None",
    "EnableReadCaching": false,
    "CreatedUtc": "2024-07-10T05:09:32.000000Z"
}

Field Descriptions

GUID (GUID): Globally unique identifier for the storage pool object
TenantGUID (GUID): Globally unique identifier for the tenant
Name (string): Display name for the storage pool
Provider (enum): Storage provider type (Disk, AwsS3, Azure,SMB,NFS)
WriteMode (enum): Write mode for object storage (GUID, Key)
UseSsl (boolean): Whether to use SSL for cloud storage connections
DiskDirectory (string): Directory path for local disk storage
Compress (enum): Compression type to apply (None, Gzip, etc.)
EnableReadCaching (boolean): Whether to enable read caching for performance
CreatedUtc (datetime): UTC timestamp when the storage pool was created

Provider-Specific Fields

Disk Storage Pools

DiskDirectory: Local filesystem path for data storage

Amazon S3 Storage Pools

S3EndpointUrl: S3 endpoint URL
S3AccessKey: S3 access key for authentication
S3SecretKey: S3 secret key for authentication
S3BucketName: S3 bucket name
S3Region: AWS region

Azure Blob Storage Pools

AzureEndpointUrl: Azure Blob endpoint URL
AzureAccountName: Azure storage account name
AzureContainerName: Azure container name
AzureAccessKey: Azure storage access key

Important Notes

Physical Storage: Storage pools define the physical location where data is stored
Provider Configuration: Each provider type has specific configuration requirements
Performance Tuning: Compression and caching settings can be optimized for performance
Security: SSL configuration ensures secure communication with cloud storage providers

Create Storage Pool

Creates a new storage pool configuration using PUT /v1.0/tenants/[tenant-guid]/pools. This endpoint allows you to define physical storage backends for data persistence with provider-specific configurations.

Request Parameters

Required Parameters

Name (string, Body, Required): Display name for the storage pool
Provider (enum, Body, Required): Storage provider type (Disk, AmazonS3, AzureBlob)
WriteMode (enum, Body, Required): Write mode for object storage (GUID, Key)

Optional Parameters

UseSsl (boolean, Body, Optional): Whether to use SSL for cloud storage connections
DiskDirectory (string, Body, Optional): Directory path for local disk storage
Compress (enum, Body, Optional): Compression type to apply (defaults to "None")
EnableReadCaching (boolean, Body, Optional): Whether to enable read caching (defaults to false)

Important Notes

Immutable Configuration: Once created, storage pool configurations cannot be updated
Provider Dependencies: Ensure proper configuration for the selected storage provider
Performance Considerations: Configure compression and caching based on performance requirements
Security: Use SSL for secure communication with cloud storage providers

curl -X PUT http://localhost:8001/v1.0/tenants/[tenant-guid]/pools \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer [accesskey]" \
     -d '
{
    "Name": "My disk storage pool",
    "Provider": "Disk",
    "WriteMode": "GUID",
    "UseSsl": false,
    "DiskDirectory": "./disk/",
    "Compress": "None",
    "EnableReadCaching": false
}'

import { ViewStorageSdk } from "view-sdk";

const api = new ViewStorageSdk(
  "http://localhost:8001/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const createStoragePool = async () => {
  try {
    const response = await api.storagePool.create({
      Name: "My disk storage pool",
      Provider: "Disk",
      WriteMode: "GUID",
      UseSsl: false,
      DiskDirectory: "./disk/",
      Compress: "None",
      EnableReadCaching: false,
    });
    console.log(response, "Storage pool created successfully");
  } catch (err) {
    console.log("Error creating Storage pool:", err);
  }
};

createStoragePool();

import view_sdk
from view_sdk import storage
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost",
    tenant_guid="tenant-guid",
    service_ports={Service.STORAGE: 8001},
)
def createStoragePool():
    storagePool = storage.Pool.create(
      Name="My disk storage pool",
      Provider="Disk",
      WriteMode="GUID",
      UseSsl=False,
      DiskDirectory="./disk/",
      Compress="None",
      EnableReadCaching=False,
    )
    print(storagePool)  

createStoragePool()

Response

Returns the created storage pool object with all configuration details:

{
    "GUID": "default",
    "TenantGUID": "default",
    "Name": "My disk storage pool",
    "Provider": "Disk",
    "WriteMode": "GUID",
    "UseSsl": false,
    "DiskDirectory": "./disk/",
    "Compress": "None",
    "EnableReadCaching": false,
    "CreatedUtc": "2024-07-10T05:09:32.000000Z"
}

Read Storage Pool

Retrieves storage pool configuration and metadata by GUID using GET /v1.0/tenants/[tenant-guid]/pools/[pool-guid]. Returns the complete storage pool configuration including provider settings and performance options. If the pool doesn't exist, a 404 error is returned.

Request Parameters

pool-guid (string, Path, Required): GUID of the storage pool object to retrieve

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/pools/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'

import { ViewStorageSdk } from "view-sdk";

const api = new ViewStorageSdk(
  "http://localhost:8001/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const readStoragePool = async () => {
  try {
    const response = await api.storagePool.read(
      "<pool-guid>"
    );
    console.log(response, "Storage pool fetched successfully");
  } catch (err) {
    console.log("Error fetching Storage pool:", err);
  }
};

readStoragePool();

import view_sdk
from view_sdk import storage
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost",
    tenant_guid="tenant-guid",
    service_ports={Service.STORAGE: 8001},
)

def readStoragePool():
    storagePool = storage.Pool.retrieve("<pool-guid>")
    print(storagePool)

readStoragePool()

Response

Returns the complete storage pool configuration:

{
    "GUID": "default",
    "TenantGUID": "default",
    "Name": "default",
    "Provider": "Disk",
    "WriteMode": "GUID",
    "UseSsl": false,
    "DiskDirectory": "./disk/",
    "Compress": "None",
    "EnableReadCaching": false,
    "CreatedUtc": "2024-07-10T05:09:32.000000Z"
}

Read All Storage Pools

Retrieves all storage pool objects in the tenant using GET /v1.0/tenants/[tenant-guid]/pools/. Returns an array of storage pool objects with complete configuration details for all pools in the tenant.

Request Parameters

No additional parameters required beyond authentication.

curl --location 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/pools/' \
--header 'Authorization: ••••••'

import { ViewStorageSdk } from "view-sdk";

const api = new ViewStorageSdk(
  "http://localhost:8001/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const readAllStoragePools = async () => {
  try {
    const response = await api.storagePool.readAll();
    console.log(response, "Storage pools fetched successfully");
  } catch (err) {
    console.log("Error fetching Storage pools:", err);
  }
};

readAllStoragePools();

import view_sdk
from view_sdk import storage
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost",
    tenant_guid="tenant-guid",
    service_ports={Service.STORAGE: 8001},
)

def readAllStoragePools():
    storagePools = storage.Pool.retrieve_all()
    print(storagePools)

readAllStoragePools()

Response

Returns an array of all storage pool objects:

[
    {
        "GUID": "default",
        "TenantGUID": "default",
        "Name": "default",
        "Provider": "Disk",
        "WriteMode": "GUID",
        "UseSsl": false,
        "DiskDirectory": "./disk/",
        "Compress": "None",
        "EnableReadCaching": false,
        "CreatedUtc": "2024-07-10T05:09:32.000000Z"
    },
    {
        "GUID": "s3-pool",
        "TenantGUID": "default",
        "Name": "S3 Storage Pool",
        "Provider": "AmazonS3",
        "WriteMode": "Key",
        "UseSsl": true,
        "S3EndpointUrl": "https://s3.amazonaws.com",
        "S3AccessKey": "***key",
        "S3SecretKey": "***key",
        "S3BucketName": "my-bucket",
        "S3Region": "us-west-1",
        "Compress": "Gzip",
        "EnableReadCaching": true,
        "CreatedUtc": "2024-07-11T10:15:30.123456Z"
    }
]

Update Storage Pool

Updates an existing storage pool configuration using PUT /v1.0/tenants/[tenant-guid]/pools/[pool-guid]. This endpoint allows you to modify storage pool parameters while preserving certain immutable fields.

Request Parameters

pool-guid (string, Path, Required): GUID of the storage pool object to update

Updateable Fields

All configuration parameters can be updated except for:

GUID: Immutable identifier
TenantGUID: Immutable tenant association
CreatedUtc: Immutable creation timestamp

Important Notes

Field Preservation: Certain fields cannot be modified and will be preserved across updates
Complete Object: Provide a fully populated object in the request body
Configuration Validation: All updated parameters will be validated before applying changes
Impact Assessment: Consider the impact of pool changes on existing buckets and objects

curl --location --request PUT 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/encryptionkeys/00000000-0000-0000-0000-000000000000' \
--header 'content-type: application/json' \
--header 'Authorization: ••••••' \
--data '{
     "GUID": "4a931b31-a8a7-4b65-ab35-80e647a46ffd",
      "Name": "My disk storage pool [updated]",
      "Provider": "Disk",
      "WriteMode": "GUID",
      "UseSsl": false,
      "DiskDirectory": "./disk/",
      "Compress": "None",
      "EnableReadCaching": false,
    }'

import { ViewStorageSdk } from "view-sdk";

const api = new ViewStorageSdk(
  "http://localhost:8001/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const updateStoragePool = async () => {
  try {
    const response = await api.storagePool.update({
      GUID: "<pool-guid>",
      Name: "My disk storage pool [updated]",
      Provider: "Disk",
      WriteMode: "GUID",
      UseSsl: false,
      DiskDirectory: "./disk/",
      Compress: "None",
      EnableReadCaching: false,
    });
    console.log(response, "Storage pool updated successfully");
  } catch (err) {
    console.log("Error updating Storage pool:", err);
  }
};

updateStoragePool();

import view_sdk
from view_sdk import storage
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost",
    tenant_guid="tenant-guid",
    service_ports={Service.STORAGE: 8001},
)

def updateStoragePool():
    storagePool = storage.Pool.update(
        "<pool-guid>",
        Name="My disk storage pool [updated]",     
        Provider="Disk",  
        WriteMode="GUID",
        UseSsl=False,
        DiskDirectory="./disk/",
        Compress="None",
        EnableReadCaching=False,  
    )
    print(storagePool)

updateStoragePool()

Response

Returns the updated storage pool object with all configuration details:

{
    "GUID": "4a931b31-a8a7-4b65-ab35-80e647a46ffd",
    "TenantGUID": "default",
    "Name": "My disk storage pool [updated]",
    "Provider": "Disk",
    "WriteMode": "GUID",
    "UseSsl": false,
    "DiskDirectory": "./disk/",
    "Compress": "None",
    "EnableReadCaching": false,
    "CreatedUtc": "2024-07-10T05:09:32.000000Z"
}

Delete Storage Pool

Deletes a storage pool object by GUID using DELETE /v1.0/tenants/[tenant-guid]/pools/[pool-guid]. This operation permanently removes the storage pool configuration from the system. Use with caution as this action cannot be undone.

Important Note: Ensure no active buckets are using this storage pool before deletion, as this will break bucket operations.

Request Parameters

pool-guid (string, Path, Required): GUID of the storage pool object to delete

curl --location --request DELETE 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/pools/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••' \

import { ViewStorageSdk } from "view-sdk";

const api = new ViewStorageSdk(
  "http://localhost:8001/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const deleteStoragePool = async () => {
  try {
    const response = await api.storagePool.delete(
      "<pool-guid>"
    );
    console.log(response, "Storage pool deleted successfully");
  } catch (err) {
    console.log("Error deleting Storage pool:", err);
  }
};

deleteStoragePool();

import view_sdk
from view_sdk import storage
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost",
    tenant_guid="tenant-guid",
    service_ports={Service.STORAGE: 8001},
)

def deleteStoragePool():
    storagePool = storage.Pool.delete("<pool-guid>")
    print(storagePool)

deleteStoragePool()

Response

Returns 204 No Content on successful deletion. No response body is returned.

Check Storage Pool Existence

Verifies if a storage pool object exists without retrieving its configuration using HEAD /v1.0/tenants/[tenant-guid]/pools/[pool-guid]. This is an efficient way to check pool presence before performing operations.

Request Parameters

pool-guid (string, Path, Required): GUID of the storage pool object to check

curl --location --head 'http://localhost:8000/v1.0/tenants/00000000-0000-0000-0000-000000000000/pools/00000000-0000-0000-0000-000000000000' \
--header 'Authorization: ••••••'

import { ViewStorageSdk } from "view-sdk";

const api = new ViewStorageSdk(
  "http://localhost:8001/", //endpoint
  "<tenant-guid>", //tenant Id
  "default" //access key
);

const existsStoragePool = async () => {
  try {
    const response = await api.storagePool.exists(
      "<pool-guid>"
    );
    console.log(response, "Storage pool exists");
  } catch (err) {
    console.log("Error checking Storage pool:", err);
  }
};

existsStoragePool();

import view_sdk
from view_sdk import storage
from view_sdk.sdk_configuration import Service

sdk = view_sdk.configure(
    access_key="default",
    base_url="localhost",
    tenant_guid="tenant-guid",
    service_ports={Service.STORAGE: 8001},
)

def existsStoragePool():
    storagePool = storage.Pool.exists("<pool-guid>")
    print(storagePool)

existsStoragePool()

Response

200 No Content: Storage pool exists
404 Not Found: Storage pool does not exist
No response body: Only HTTP status code is returned

Note: HEAD requests do not return a response body, only the HTTP status code indicating whether the storage pool exists.

Best Practices

When managing storage pools in the View platform, consider the following recommendations for optimal data placement, performance, and cost efficiency:

Provider Selection: Choose appropriate storage providers based on performance, cost, and compliance requirements
Performance Optimization: Configure compression and caching settings based on data access patterns and performance needs
Security Configuration: Use SSL for secure communication with cloud storage providers
Capacity Planning: Plan storage capacity and performance requirements before creating pools
Data Placement Strategy: Use multiple pools for different data types and access patterns

Next Steps

After successfully configuring storage pools, you can:

Buckets: Create buckets that utilize your configured storage pools for data organization
Objects: Upload and manage objects within buckets that use your storage pools
Multipart Uploads: Use multipart upload functionality for efficient large file handling
Performance Monitoring: Monitor storage pool performance and optimize configurations as needed
Integration: Integrate storage pools with other View platform services for comprehensive data processing workflows