Embed
Vectorise a piece of content (string or weighted dictionary) or list of content and return the corresponding embeddings.
POST /indexes/{index_name}/embed
Path parameters
Name | Type | Description |
---|---|---|
index_name |
String | name of the requested index |
Body
The body parameters below would be used for HTTP requests (if you were using cURL, for example). Python client users
should use the pythonic snakecase equivalents (for example, media_download_headers
rather than mediaDownloadHeaders
).
Search Parameter | Type | Default value | Description |
---|---|---|---|
content |
String OR Dict OR List of (String OR Dict) | null |
Content string to embed, weighted content strings (if Dict), or list of either of them. List or Dict cannot be empty. |
imageDownloadHeaders (deprecated) |
Dict | {} |
Headers for the image download. Can be used to authenticate the images for download. |
mediaDownloadHeaders |
Dict | {} |
Headers for the media download. Can be used to authenciate the all types of media for download. |
modelAuth |
Dict | null |
Authorisation details used by Marqo to download non-publicly available models. Check here for examples. |
content_type |
String | query |
The type of prefix to be added to all text content in the embed request. Can be either "query" (to use the default text query prefix), "document" (to use the default document text chunk prefix), or None (to have no prefix entirely). If the user would like a custom prefix, the prefix would have to be hardcoded into the document in the content field with content_type set to None . |
Query parameters
Search Parameter | Type | Default value | Description |
---|---|---|---|
device |
String | null |
The device used to embed. If device is not specified and CUDA devices are available to Marqo (see here for more info), Marqo will speed up embed by using an available CUDA device. Otherwise, the CPU will be used. Options include cpu and cuda , cuda1 , cuda2 etc. The cuda option tells Marqo to use any available cuda devices. |
telemetry |
Boolean | False |
If true, the telemetry object is returned in the embed response body. This includes information like latency metrics. This is set at client instantiation time in the Python client: mq = marqo.Client(return_telemetry=True) |
Response
Name | Type | Description |
---|---|---|
embeddings |
Array of Array of floats | List of embeddings generated from the content (in the same order). If content was a single item, embeddings will contain a single element. |
content |
String OR Dict OR List of (String OR Dict) | Content that was input for the request |
processingTimeMs |
Number | Processing time of the query |
Example
curl -XPOST 'http://localhost:8882/indexes/my-first-index/embed' \
-H 'Content-type:application/json' -d '
{
"content": [
"Men shoes brown",
{"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
],
"content_type": null
}'
mq.index("my-first-index").embed(
content = [
"Men shoes brown",
{"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
],
content_type=None
)
For Marqo Cloud, you will need to access the endpoint of your index and replace your_endpoint
with this. To do this, visit Find Your Endpoint. You will also need your API Key. To obtain this key visit Find Your API Key.
curl -XPOST 'your_endpoint/indexes/my-first-index/embed' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"content": [
"Men shoes brown",
{"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
],
"content_type": null
}'
mq.index("my-first-index").embed(
content = [
"Men shoes brown",
{"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
],
content_type=None
)
Response: 200 Ok
{
"embeddings": [
[-0.02587328478693962, -0.010530706495046616, 0.005109857302159071, ...],
[0.013989399092603298, 0.015067596229836486, -0.02084693302954349, ...]
],
"content": ["Men shoes brown", {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}],
"processingTimeMs": 48
}
Content
Parameter: content
Expected value: Content string, a dictionary of weighted content strings, or a list of a combination of the previous types.
Content strings are text or a url to an image, if the index has treatUrlsAndPointersAsImages
set to True.
If content is weighted, each weight acts as a (possibly negative) multiplier for that text, relative to the other text.
Default value: null
Examples
# content string:
content = "How do I keep my plant alive?"
# a dictionary of weighted content strings
content = {
# a weighting of 1 gives this query a neutral effect:
"Which dogs are the best pets": 1.0,
# we give this a weighting of 2 because we really want results similar to this:
"https://image_of_a_golden_retriever.png": 2.0,
# we give this a negative weighting to make it less likely to appear:
"Poodle": -1
}
# a list of content strings and weighted content strings
content = [
"How do I keep my plant alive?",
{
"Which dogs are the best pets": 1.0,
"https://image_of_a_golden_retriever.png": 2.0,
"Poodle": -1
}
]
Model Auth
Parameter: modelAuth
Expected value: Dictionary with either an s3
or an hf
model store authorisation object.
Default value: null
The ModelAuth
object allows embedding on indexes that use OpenCLIP and CLIP models from private Hugging Face and AWS
S3 stores.
The modelAuth
object contains either an s3
or an hf
model store authorisation object. The model store
authorisation object contains credentials needed to access the index's non publicly accessible model. See the example
for details.
The index's settings must specify the non publicly accessible model's location in the setting's modelProperties
object.
ModelAuth
is used to initially download the model. After downloading, Marqo caches the model so that it doesn't need
to be redownloaded.
Example: AWS s3
# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during embedding process to download the model
mq.create_index(
index_name="my-cool-index",
settings_dict={
"treatUrlsAndPointersAsImages": True,
"model": 'my_s3_model',
"normalizeEmbeddings": True,
"modelProperties": {
"name": "ViT-B-32",
"dimensions": 512,
"model_location": {
"s3": {
"Bucket": "<SOME BUCKET>",
"Key": "<KEY TO IDENTIFY MODEL>",
},
"auth_required": True
},
"type": "open_clip",
}
}
)
# Specify the authorisation needed to access the private model during embed:
# We recommend setting up the credential's AWS user so that it has minimal
# accesses needed to retrieve the model
mq.index("my-cool-index").embed(
content="Chocolate chip cookies",
model_auth={
's3': {
"aws_access_key_id": "<SOME ACCESS KEY ID>",
"aws_secret_access_key": "<SOME SECRET ACCESS KEY>"
}
}
)
Example: Hugging Face (HF)
# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during embedding process to download the model
mq.create_index(
index_name="my-cool-index",
settings_dict={
"treatUrlsAndPointersAsImages": True,
"model": 'my_hf_model',
"normalizeEmbeddings": True,
"modelProperties": {
"name": "ViT-B-32",
"dimensions": 512,
"model_location": {
"hf": {
"repo_id": "<SOME HF REPO NAME>",
"filename": "<THE FILENAME TO DOWNLOAD>",
},
"auth_required": True
},
"type": "open_clip",
}
}
)
# specify the authorisation needed to access the private model during embedding process:
mq.index("my-cool-index").embed(
content="Chocolate chip cookies",
model_auth={
'hf': {
"token": "<SOME HF TOKEN>",
}
}
)
Prefixes in Embed
Parameters: content_type
Expected value: "query"
, "document"
, or None
Default value: ""
This field instructs Marqo to use the query or text chunk prefixes defined in the index settings when embedding text content.
To use a custom prefix, users must hardcode the prefix into the content field and explicitly set content_type=None
.
setting content_type="query"
or content_type="document"
will add the prefixes defined in the index settings or the model's defaults to the content.
setting content_type
to any other string will raise an error.
Example: Embedding without prefixes
Embedding with content_type = None
does not add any prefixes to the content.
curl -XPOST 'http://localhost:8882/indexes/{index_name}/embed' \
-H 'Content-type:application/json' -d '
{
"content": [
"Men shoes brown",
{"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
],
"content_type": null
}'
mq.index("{index_name}").embed(
content=[
"Men shoes brown",
{
"Large grey hat": 0.7,
"https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3,
},
],
content_type=None,
)
For Marqo Cloud, you will need to access the endpoint of your index and replace your_endpoint
with this. To do this, visit Find Your Endpoint. You will also need your API Key. To obtain this key visit Find Your API Key.
curl -XPOST 'your_endpoint/indexes/{index_name}/embed' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"content": [
"Men shoes brown",
{"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
],
"content_type": null
}'
mq.index("{index_name}").embed(
content=[
"Men shoes brown",
{
"Large grey hat": 0.7,
"https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3,
},
],
content_type=None,
)
Example: Embedding with default prefix
When embedding without specifying a content type, Marqo uses the index setting's query prefixes.
curl -XPOST 'http://localhost:8882/indexes/{index_name}/embed' \
-H 'Content-type:application/json' -d '
{
"content": ["Dogs are great"]
}'
mq.index("{index_name}").embed(
content=[
"Dogs are great"
] # The embedded content will be "query: Dogs are great" for an e5 model
)
For Marqo Cloud, you will need to access the endpoint of your index and replace your_endpoint
with this. To do this, visit Find Your Endpoint. You will also need your API Key. To obtain this key visit Find Your API Key.
curl -XPOST 'your_endpoint/indexes/my-first-index/embed' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"content": ["Dogs are great"]
}'
mq.index("{index_name}").embed(
content=[
"Dogs are great"
] # The embedded content will be "query: Dogs are great" for an e5 model
)