Embed

Vectorise a piece of content (string or weighted dictionary) or list of content and return the corresponding embeddings.

POST /indexes/{index_name}/embed

Path parameters

Name	Type	Description
`index_name`	String	name of the requested index

Body

The body parameters below would be used for HTTP requests (if you were using cURL, for example). Python client users should use the pythonic snakecase equivalents (for example, image_download_headers rather than imageDownloadHeaders).

Search Parameter	Type	Default value	Description
`content`	String OR Dict OR List of (String OR Dict)	`null`	Content string to embed, weighted content strings (if Dict), or list of either of them. List or Dict cannot be empty.
`imageDownloadHeaders`	Dict	`{}`	Headers for the image download. Can be used to authenticate the images for download.
`modelAuth`	Dict	`null`	Authorisation details used by Marqo to download non-publicly available models. Check here for examples.
`content_type`	String	`query`	The type of prefix to be added to all text content in the embed request. Can be either `"query"` (to use the default text query prefix), `"document"` (to use the default document text chunk prefix), or `None` (to have no prefix entirely). If the user would like a custom prefix, the prefix would have to be hardcoded into the document in the `content` field with `content_type` set to `None`.

Query parameters

Search Parameter	Type	Default value	Description
`device`	String	`null`	The device used to embed. If `device` is not specified and CUDA devices are available to Marqo (see here for more info), Marqo will speed up embed by using an available CUDA device. Otherwise, the CPU will be used. Options include `cpu` and `cuda`, `cuda1`, `cuda2` etc. The `cuda` option tells Marqo to use any available cuda devices.
`telemetry`	Boolean	`False`	If true, the `telemetry` object is returned in the embed response body. This includes information like latency metrics. This is set at client instantiation time in the Python client: `mq = marqo.Client(return_telemetry=True)`

Response

Name	Type	Description
`embeddings`	Array of Array of floats	List of embeddings generated from the `content` (in the same order). If `content` was a single item, `embeddings` will contain a single element.
`content`	String OR Dict OR List of (String OR Dict)	Content that was input for the request
`processingTimeMs`	Number	Processing time of the query

Example

cURLPython

curl -XPOST 'http://localhost:8882/indexes/my-first-index/embed' -H 'Content-type:application/json' -d '
{
    "content": [
        "Men shoes brown", 
        {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
    ],
    "content_type": None
}'

mq.index("my-first-index").embed(
    content = [
        "Men shoes brown", 
        {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
    ],
    content_type=None
)

Response: `200 Ok`

{
  "embeddings": [
    [-0.02587328478693962, -0.010530706495046616, 0.005109857302159071, ...],
    [0.013989399092603298, 0.015067596229836486, -0.02084693302954349, ...]
  ],
  "content": ["Men shoes brown", {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}],
  "processingTimeMs": 48
}

Content

Parameter: content

Expected value: Content string, a dictionary of weighted content strings, or a list of a combination of the previous types.

Content strings are text or a url to an image, if the index has treatUrlsAndPointersAsImages set to True.

If content is weighted, each weight acts as a (possibly negative) multiplier for that text, relative to the other text.

Default value: null

Examples

# content string: 
content = "How do I keep my plant alive?"

# a dictionary of weighted content strings
content = {
    # a weighting of 1 gives this query a neutral effect:
    "Which dogs are the best pets": 1.0,
    # we give this a weighting of 2 because we really want results similar to this:
    "https://image_of_a_golden_retriever.png": 2.0,
    # we give this a negative weighting to make it less likely to appear: 
    "Poodle": -1
}

# a list of content strings and weighted content strings
content = [
    "How do I keep my plant alive?",
    {
        "Which dogs are the best pets": 1.0,
        "https://image_of_a_golden_retriever.png": 2.0,
        "Poodle": -1
    }
]

Model Auth

Parameter: modelAuth

Expected value: Dictionary with either an s3 or an hf model store authorisation object.

Default value: null

The ModelAuth object allows embedding on indexes that use OpenCLIP and CLIP models from private Hugging Face and AWS S3 stores.

The modelAuth object contains either an s3 or an hf model store authorisation object. The model store authorisation object contains credentials needed to access the index's non publicly accessible model. See the example for details.

The index's settings must specify the non publicly accessible model's location in the setting's modelProperties object.

ModelAuth is used to initially download the model. After downloading, Marqo caches the model so that it doesn't need to be redownloaded.

Example: AWS s3

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during embedding process to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_s3_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B/32",
            "dimensions": 512,
            "modelLocation": {
                "s3": {
                    "Bucket": "<SOME BUCKET>",
                    "Key": "<KEY TO IDENTIFY MODEL>",
                },
                "authRequired": True
            },
            "type": "open_clip",
        }
    }
)

# Specify the authorisation needed to access the private model during embed:
# We recommend setting up the credential's AWS user so that it has minimal 
# accesses needed to retrieve the model
mq.index("my-cool-index").embed(
    content="Chocolate chip cookies",
    model_auth={
        's3': {
            "aws_access_key_id": "<SOME ACCESS KEY ID>",
            "aws_secret_access_key": "<SOME SECRET ACCESS KEY>"
        }
    }
)

Example: Hugging Face (HF)

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during embedding process to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_hf_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B/32",
            "dimensions": 512,
            "modelLocation": {
                "hf": {
                    "repoId": "<SOME HF REPO NAME>",
                    "filename": "<THE FILENAME TO DOWNLOAD>",
                },
                "authRequired": True
            },
            "type": "open_clip",
        }
    }
)

# specify the authorisation needed to access the private model during embedding process:
mq.index("my-cool-index").embed(
    content="Chocolate chip cookies",
    model_auth={
        'hf': {
            "token": "<SOME HF TOKEN>",
        }
    }
)

Prefixes in Embed

Parameters: content_type

Expected value: "query", "document", or None

Default value: ""

This field instructs Marqo to use the query or text chunk prefixes defined in the index settings when embedding text content.

To use a custom prefix, users must hardcode the prefix into the content field and explicitly set content_type=None.

setting content_type="query" or content_type="document" will add the prefixes defined in the index settings or the model's defaults to the content. setting content_type to any other string will raise an error.

Example: Embedding without prefixes

Embedding with content_type = None does not add any prefixes to the content.

cURLPython

curl -XPOST 'http://localhost:8882/indexes/{index_name}/embed' \
-H 'Content-type:application/json' -d '
{
    "content": [
        "Men shoes brown", 
        {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
    ],
    "content_type": None
}'

mq.index("{index_name}").embed(
    content=[
        "Men shoes brown",
        {
            "Large grey hat": 0.7,
            "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3,
        },
    ],
    content_type=None,
)

Example: Embedding with default prefix

When embedding without specifying a content type, Marqo uses the index setting's query prefixes.

cURLPython

curl -XPOST 'http://localhost:8882/indexes/{index_name}/embed' \
-H 'Content-type:application/json' -d '
{
    "content": ["Dogs are great"]  # The embedded content will be "query: Dogs are great" for an e5 model 
}'

mq.index("{index_name}").embed(
    content=[
        "Dogs are great"
    ]  # The embedded content will be "query: Dogs are great" for an e5 model
)

Embed

Path parameters

Body

Query parameters

Response

Example

Response: 200 Ok

Content

Model Auth

Example: AWS s3

Example: Hugging Face (HF)

Prefixes in Embed

Example: Embedding without prefixes

Example: Embedding with default prefix

Subscribe to our mailing list

Response: `200 Ok`