Skip to content

Using Terraform or OpenTofu to Manage Your Cloud Resources

This Terraform/OpenTofu provider allows you to manage your Marqo Cloud resources using infrastructure as code through either Terraform or OpenTofu. Both platforms enable you to safely and predictably create, change, and delete cloud resources.

OpenTofu is an open-source fork of Terraform, and this provider is compatible with both platforms. For more information on opentofu and terraform, please visit the following links: OpenTofu and Terraform.

Provider Registries

Platform Choice

The provider works identically on both platforms, with only minor differences in setup:

Terraform:

  • Use provider source marqo-ai/marqo, as in: 

    terraform {
  required_providers {
    marqo = {
      source = "registry.terraform.io/marqo/marqo"
      # version = # If this field is not set, the latest version will be used
    }
  }
}

  • Use terraform commands

OpenTofu:

  • Use provider source registry.opentofu.org/marqo-ai/marqo, as in: 

    terraform {
  required_providers {
    marqo = {
      source = "registry.opentofu.org/marqo-ai/marqo"
      # version = # If this field is not set, the latest version will be used
    }
  }
}

  • Use tofu commands

Getting Started

Installation

  1. Install either Terraform or OpenTofu
  2. Create a new directory for your Marqo configuration(s)
  3. Create a .tf file with your configuration (see examples below)
  4. Create a terraform.tfvars file with your API key as follows: 
    marqo_api_key = "<KEY>"
  5. Initialize your configuration using terraform or tofu init command
  6. Plan your configuration using terraform or tofu plan command
  7. Apply your configuration using terraform or tofu apply command

Some Basic Commands

Terraform OpenTofu Description
terraform init tofu init Initialize working directory
terraform refresh tofu refresh Update state with remote changes
terraform plan tofu plan Preview changes before applying
terraform apply tofu apply Apply the planned changes
terraform destroy tofu destroy Remove all resources

See the Opentofu documentation for more information on how to use Opentofu or the Terraform documentation for more information on how to use Terraform.

Example directory structure for multiple indexes

If you only have one index, you can simply create a single marqo_index.tf file with the index configuration, resource, and output. Otherwise, you can follow the following structure:

my-marqo-config/
├── main.tf
├── terraform.tfvars
├── marqo_index_unstructured.tf
├── marqo_index_video.tf
├── marqo_index_structured.tf
├── output.tf

Where main.tf contains the provider and data sources/resources, as in: 

terraform {
  required_providers {
    marqo = {
      source = "registry.opentofu.org/marqo-ai/marqo"
      version = "1.0.1"
    }
  }
}
terraform.tfvars contains the API key, and marqo_index_unstructured.tf contains the unstructured index configuration as in:

resource "marqo_index" "index_unstructured" {
  index_name = "marqo_index_unstructured"
  ...
}
and the output.tf file contains the output, as in:

output "index_unstructured" {
  value = marqo_index.index_unstructured
}
output "index_video" {
  value = marqo_index.index_video
}
output "index_structured" {
  value = marqo_index.index_structured
}

Overview of Features

The marqo opentofu provider supports the following:

  • A datasource called marqo_read_indices that allows you to read all of your marqo indexes in your account.
  • A resource called marqo_index that allows you to create and manage a marqo index.

Sample Configuration for Unstructured Index Creation

Make sure that the host is set to "https://api.marqo.ai/api/v2"

terraform {
  required_providers {
    marqo = {
      source = "registry.terraform.io/marqo/marqo"
    }
  }
}

provider "marqo" {
  host    = "https://api.marqo.ai/api/v2"
  api_key = var.marqo_api_key
}

resource "marqo_index" "my_unstructured_index" {
  index_name = "my_unstructured_index"
  settings = {
    type                              = "unstructured"
    vector_numeric_type               = "float"
    treat_urls_and_pointers_as_images = true
    model                             = "open_clip/ViT-L-14/laion2b_s32b_b82k"
    normalize_embeddings              = true
    inference_type                    = "marqo.CPU.large"
    all_fields                        = []
    number_of_inferences              = 1
    number_of_replicas                = 0
    number_of_shards                  = 1
    storage_class                     = "marqo.basic"
    text_preprocessing = {
      split_length  = 2
      split_method  = "sentence"
      split_overlap = 0
    }
    image_preprocessing = {}
    ann_parameters = {
      space_type = "prenormalized-angular"
      parameters = {
        ef_construction = 512
        m               = 16
      }
    }
    filter_string_max_length = 20
  }
}

output "created_index" {
  value = marqo_index.example
}

variable "marqo_api_key" {
  type        = string
  description = "Marqo API key"
}

For more example configurations, please see the following:

Configuration Options

For more information on the configuration options, please see the Configuration Options page.