Structured Indexes

Creating a Structured Index (resource)

terraform {
  required_providers {
    marqo = {
      source  = "marqo-ai/marqo"
      version = "1.2.4"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "example" {
  index_name = "example_index_dependent_2"
  settings = {
    type                = "structured"
    vector_numeric_type = "float"
    all_fields = [
      { "name" : "textField", "type" : "text", "features" : ["lexical_search"] },
      { "name" : "imageField", "type" : "image_pointer" },
      {
        "name" : "multimodalField",
        "type" : "multimodal_combination",
        "dependent_fields" : {
          "imageField" : 0.8,
          "textField" : 0.1
        },
      },
    ],
    number_of_inferences = 1
    storage_class        = "marqo.basic"
    number_of_replicas   = 0
    number_of_shards     = 2
    tensor_fields        = ["multimodalField"],
    model                = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    normalize_embeddings = true
    inference_type       = "marqo.CPU.large"
    text_preprocessing = {
      split_length  = 2
      split_method  = "sentence"
      split_overlap = 0
    }
    image_preprocessing = {
      patch_method = null
    }
    ann_parameters = {
      space_type = "prenormalized-angular"
      parameters = {
        ef_construction = 512
        m               = 16
      }
    }
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

Creating a Structured audio-video Index using Languagebind (resource)

terraform {
  required_providers {
    marqo = {
      source  = "marqo-ai/marqo"
      version = "1.2.4"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "example" {
  index_name = "example_index_languagebind"
  settings = {
    type                = "structured"
    vector_numeric_type = "float"
    all_fields = [
      { "name" : "textField", "type" : "text", "features" : ["lexical_search"] },
      { "name" : "videoField", "type" : "video_pointer" },
      { "name" : "audioField", "type" : "audio_pointer" },
      { "name" : "imageField", "type" : "image_pointer" },
      {
        "name" : "multimodal_field",
        "type" : "multimodal_combination",
        "dependent_fields" : {
          "imageField" : 0.8,
          "textField" : 0.1,
          "videoField" : 0.1,
          "audioField" : 0.1
        },
      },
    ],
    number_of_inferences = 1
    storage_class        = "marqo.basic"
    number_of_replicas   = 0
    number_of_shards     = 1
    tensor_fields        = ["multimodal_field", "textField", "videoField", "audioField", "imageField"],
    model                = "LanguageBind/Video_V1.5_FT_Audio_FT_Image"
    normalize_embeddings = true
    inference_type       = "marqo.GPU"
    text_preprocessing = {
      split_length  = 2
      split_method  = "sentence"
      split_overlap = 0
    }
    image_preprocessing = {
      patch_method = null
    }
    video_preprocessing = {
      split_length  = 5
      split_overlap = 1
    }
    audio_preprocessing = {
      split_length  = 5
      split_overlap = 1
    }
    ann_parameters = {
      space_type = "prenormalized-angular"
      parameters = {
        ef_construction = 512
        m               = 16
      }
    }
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

Importing an Existing Index

You can import an existing Marqo index into your Terraform state using the terraform import command. This is useful when you want to start managing an existing index with Terraform.

To import an index, use the following command:

terraform import marqo_index.example <index_name>

For example, to import an index named "my_existing_structured_index":

terraform import marqo_index.example my_existing_structured_index

After importing, Terraform will automatically detect and manage the index's configuration, including: - Index type (structured/unstructured) - All fields configuration - Tensor fields - Model configuration - Storage settings - Inference settings - Preprocessing configurations - ANN parameters - And other index-specific settings

Note: Some settings like timeouts are not imported from the existing index and will use default values. You can specify these in your Terraform configuration after import.

Example Configuration After Import

After importing an index, you should add a corresponding resource block to your Terraform configuration that matches the imported index's settings. For example:

resource "marqo_index" "example" {
  index_name = "my_existing_structured_index"
  settings = {
    type                = "structured"
    vector_numeric_type = "float"
    all_fields = [
      { "name" : "textField", "type" : "text", "features" : ["lexical_search"] },
      { "name" : "imageField", "type" : "image_pointer" },
      {
        "name" : "multimodalField",
        "type" : "multimodal_combination",
        "dependent_fields" : {
          "imageField" : 0.8,
          "textField" : 0.1
        },
      },
    ]
    number_of_inferences = 1
    storage_class        = "marqo.basic"
    number_of_replicas   = 0
    number_of_shards     = 1
    tensor_fields        = ["multimodalField"]
    model                = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    normalize_embeddings = true
    inference_type       = "marqo.CPU.large"
    # Add other settings as needed to match your existing index
  }
}

You can use the Marqo API or Cloud Console to view your existing index's settings and replicate them in your Terraform configuration.