Unstructured Indexes

Creating an Unstructured Index (resource)

terraform {
  required_providers {
    marqo = {
      source  = "marqo-ai/marqo"
      version = "1.2.4"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "example" {
  index_name = "example_index_3"
  settings = {
    type                              = "unstructured"
    vector_numeric_type               = "float"
    treat_urls_and_pointers_as_images = true
    treat_urls_and_pointers_as_media  = true
    model                             = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    normalize_embeddings              = true
    inference_type                    = "marqo.CPU.large"
    number_of_inferences              = 1
    number_of_replicas                = 0
    number_of_shards                  = 1
    storage_class                     = "marqo.basic"
    text_preprocessing = {
      split_length  = 2
      split_method  = "sentence"
      split_overlap = 0
    }
    ann_parameters = {
      space_type = "prenormalized-angular"
      parameters = {
        ef_construction = 512
        m               = 16
      }
    }
    filter_string_max_length = 20
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

Minimal Unstructured Index Example

This example shows the minimal configuration needed to create an unstructured index:

terraform {
  required_providers {
    marqo = {
      source  = "marqo-ai/marqo"
      version = "1.2.4"
    }
  }
}

provider "marqo" {
  host    = "https://controller.marqo-staging.com/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "example" {
  index_name = "minimal"
  settings = {
    type                 = "unstructured"
    model                = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    inference_type       = "marqo.CPU.large"
    number_of_inferences = 1
    number_of_replicas   = 0
    number_of_shards     = 1
    storage_class        = "marqo.balanced"
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

This configuration includes only the essential settings required for an unstructured index. For more advanced configurations with additional settings, see the examples above and below.

Creating an Unstructured Index with Custom Models (resource)

terraform {
  required_providers {
    marqo = {
      source  = "marqo-ai/marqo"
      version = "1.2.4"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "example" {
  index_name = "example_index_custom_model_13"
  settings = {
    type                              = "unstructured"
    vector_numeric_type               = "float"
    treat_urls_and_pointers_as_images = true
    treat_urls_and_pointers_as_media  = true
    model                             = "your-own-sentence-transformers-mode"
    model_properties = {
      url               = "https://marqo-ecs-50-audio-test-dataset.s3.us-east-1.amazonaws.com/test-hf.zip"
      dimensions        = 384
      type              = "hf"
      trust_remote_code = false
    }
    normalize_embeddings = false
    inference_type       = "marqo.CPU.large"
    number_of_inferences = 1
    number_of_replicas   = 0
    number_of_shards     = 1
    storage_class        = "marqo.basic"

    text_preprocessing = {
      split_length  = 2
      split_method  = "sentence"
      split_overlap = 0
    }
    ann_parameters = {
      space_type = "prenormalized-angular"
      parameters = {
        ef_construction = 512
        m               = 16
      }
    }
    filter_string_max_length = 20
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

Importing an Existing Index

You can import an existing Marqo index into your Terraform state using the terraform import command. This is useful when you want to start managing an existing index with Terraform.

To import an index, use the following command:

terraform import marqo_index.example <index_name>

For example, to import an index named "my_existing_index":

terraform import marqo_index.example my_existing_index

After importing, Terraform will automatically detect and manage the index's configuration, including: - Index type (structured/unstructured) - Model configuration - Storage settings - Inference settings - Preprocessing configurations - ANN parameters - And other index-specific settings

Note: Some settings like timeouts are not imported from the existing index and will use default values. You can specify these in your Terraform configuration after import.

Example Configuration After Import

After importing an index, you should add a corresponding resource block to your Terraform configuration that matches the imported index's settings. For example:

resource "marqo_index" "example" {
  index_name = "my_existing_index"
  settings = {
    type                              = "unstructured"
    vector_numeric_type               = "float"
    treat_urls_and_pointers_as_images = true
    treat_urls_and_pointers_as_media  = true
    model                             = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    normalize_embeddings              = true
    inference_type                    = "marqo.CPU.large"
    number_of_inferences             = 1
    number_of_replicas               = 0
    number_of_shards                 = 1
    storage_class                    = "marqo.basic"
    # Add other settings as needed to match your existing index
  }
}

You can use the Marqo API or Cloud Console to view your existing index's settings and replicate them in your Terraform configuration.