Multimodal Unstructured Indexes
For documentation on unstructured indexes see here.
Unstructured indexes have the multimodal combination fields defined during indexing. Multimodal combination field mappings can be different for each document though for most use cases it is best to keep them consistent.
In this section we will go over how to create a simple unstructured index with a multimodal field to vectorise.
Minimal example of creating an untructured multimodal index
For Marqo to know that it needs to identify image URLs and download the images we need to set the treatUrlsAndPointersAsImages
setting to True
. Marqo will automatically detect image URLs, fetch the image, and vectorise it with the CLIP model.
import marqo
settings = {
"model": "open_clip/ViT-L-14/laion2b_s32b_b82k",
"treatUrlsAndPointersAsImages": True,
}
mq = marqo.Client(url="http://localhost:8882")
mq.create_index("my-mm-unstructured-index", settings_dict=settings)
Index Settings: model
The model
field is used to specify the model to use for the vectorisation. To do multimodal search this must be a CLIP model.
Example Add Documents Usage
Because we are using an unstructured index we need to specify the tensor fields or the multimodal mappings in add documents.
documents = [
{
"_id": "1",
"text_field": "New York",
"image_field": "https://example.com/image.jpg",
},
{
"_id": "2",
"text_field": "Los Angeles",
"image_field": "https://example.com/image2.jpg",
},
]
mq.index("my-mm-unstructured-index").add_documents(
documents,
mappings={
"multimodal_field": {
"type": "multimodal_combination",
"weights": {"text_field": 0.1, "image_field": 0.9},
}
},
tensor_fields=["multimodal_field"],
)
Example Search Usage
This index will allow us to search the multimodal_field
with the TENSOR
search method and the text_field
field using the LEXICAL
search method.
Tensor Search (search_method="TENSOR"
is the default):
results = mq.index("my-mm-unstructured-index").search(q="New York")
Lexical Search:
results = mq.index("my-mm-unstructured-index").search(
q="New York", search_method="LEXICAL"
)