Text Search with Marqo
First, select your platform:
This guide will walk you through using Marqo to index and search textual documents (movie descriptions).
Full code:
If you have any questions or need help, visit our Community and ask in the get-help channel.
Step 1: Obtain Marqo Cloud API Key
We need to obtain our Marqo Cloud API Key. For more information on how you can obtain this, visit our article. Once you have obtained this, replace your_api_key with your actual API Key:
api_key = "your_api_key"
Let's now dive into the code.
Step 2: Create a Marqo Index
from marqo import Client
# Replace this with your API Key
api_key = "your_api_key"
# Set up the Client
mq = Client("https://api.marqo.ai", api_key=api_key)
# Name your index
index_name = "text-search"
# Index settings
index_settings = {"model": "hf/e5-base-v2", "inferenceType": "marqo.CPU.large"}
# Create the movie index
mq.create_index(index_name, settings_dict=index_settings)
curl -X POST 'https://api.marqo.ai/api/v2/indexes/text-search' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' \
-d '
{
"model": "hf/e5-base-v2",
"inferenceType": "marqo.CPU.large"
}'
Replace the API Key with the one you obtained earlier.
Step 3: Add Documents to Index
# Add documents (movie descriptions) to the index
mq.index(index_name).add_documents(
[
{
"Title": "Inception",
"Description": "A mind-bending thriller about dream invasion and manipulation.",
},
{
"Title": "Shrek",
"Description": "An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.",
},
{
"Title": "Interstellar",
"Description": "A team of explorers travel through a wormhole in space to ensure humanity's survival.",
},
{
"Title": "The Martian",
"Description": "An astronaut becomes stranded on Mars and must find a way to survive.",
},
],
tensor_fields=["Description"],
)
curl -XPOST 'your_endpoint/indexes/text-search/documents' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"documents": [
{
"Title": "Inception",
"Description": "A mind-bending thriller about dream invasion and manipulation"
},
{
"Title": "Shrek",
"Description": "An ogres peaceful life is disrupted by a horde of fairy tale characters who need his help"
},
{
"Title": "Interstellar",
"Description": "A team of explorers travel through a wormhole in space to ensure humanitys survival."
},
{
"Title": "The Martian",
"Description": "An astronaut becomes stranded on Mars and must find a way to survive."
}
],
"tensorFields": ["Description"]
}'
your_endpoint with your actual endpoint. To find your endpoint, visit Find Your Endpoint.
Note also that if your description contains an apostrophe ('), shell will throw an error.
Step 4: Search with Marqo
# Perform a search query on the index
results = mq.index(index_name).search(q="Which movie is about space exploration?")
# Print the search results
for result in results["hits"]:
print(
f"Title: {result['Title']}, Description: {result['Description']}. Score: {result['_score']}"
)
curl -XPOST 'your_endpoint/indexes/text-search/search' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"q": "Which movie is about space exploration?"
}'
This returns:
Title: Interstellar, Description: A team of explorers travel through a wormhole in space to ensure humanity's survival.. Score: 0.8173517436600624
Title: The Martian, Description: An astronaut becomes stranded on Mars and must find a way to survive.. Score: 0.8081475581626953
Title: Inception, Description: A mind-bending thriller about dream invasion and manipulation.. Score: 0.7978701791216605
Title: Shrek, Description: An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.. Score: 0.7619883916893311
Full Code
quick_text_search_cloud.py
#####################################################
### STEP 1. Setting UP
#####################################################
# 1. Sign Up to Marqo Cloud: https://cloud.marqo.ai/authentication/register/
# 2. Get a Marqo API Key: https://www.marqo.ai/blog/finding-my-marqo-api-key
#####################################################
### STEP 2. Create Marqo Client
#####################################################
from marqo import Client
# Replace this with your API Key
api_key = "your_api_key"
# Set up the Client
mq = Client(
"https://api.marqo.ai",
api_key=api_key
)
#####################################################
### STEP 3. Create a Marqo Index & Add Documents
#####################################################
# Name your index
index_name = 'text-search'
# Index settings
index_settings = {
"model":"hf/e5-base-v2",
"inferenceType":"marqo.CPU.large"
}
# Create the movie index
mq.create_index(index_name, settings_dict=index_settings)
# Add documents (movie descriptions) to the index
mq.index(index_name).add_documents(
[
{
"Title": "Inception",
"Description": "A mind-bending thriller about dream invasion and manipulation.",
},
{
"Title": "Shrek",
"Description": "An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.",
},
{
"Title": "Interstellar",
"Description": "A team of explorers travel through a wormhole in space to ensure humanity's survival.",
},
{
"Title": "The Martian",
"Description": "An astronaut becomes stranded on Mars and must find a way to survive.",
},
],
tensor_fields=["Description"],
)
#####################################################
### STEP 4. Search with Marqo
#####################################################
# Perform a search query on the index
results = mq.index(index_name).search(
q="Which movie is about space exploration?"
)
# Print the search results
for result in results['hits']:
print(f"Title: {result['Title']}, Description: {result['Description']}. Score: {result['_score']}")
This guide will walk you through using Marqo to index and search textual documents (movie descriptions).
GitHub Code: Text Search with Marqo Open Source Code
If you have any questions or need help, visit our Community and ask in the get-help channel.
Step 1: Run Marqo
Next, we need to get Marqo up and running. You can do this by executing the following command in your terminal:
docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
For more detailed instructions, check the Installation Guide.
Step 2: Create a Marqo Index
from marqo import Client
# Create marqo client
mq = Client("http://localhost:8882")
# Name your index
index_name = "text-search"
# Create the movie index
mq.create_index(index_name, model="hf/e5-base-v2")
curl -X POST 'http://localhost:8882/indexes/text-search' \
-H 'Content-type:application/json' \
-d '
{
"model": "hf/e5-base-v2"
}'
Step 3: Add Documents to Index
# Add documents (movie descriptions) to the index
mq.index(index_name).add_documents(
[
{
"Title": "Inception",
"Description": "A mind-bending thriller about dream invasion and manipulation.",
},
{
"Title": "Shrek",
"Description": "An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.",
},
{
"Title": "Interstellar",
"Description": "A team of explorers travel through a wormhole in space to ensure humanity's survival.",
},
{
"Title": "The Martian",
"Description": "An astronaut becomes stranded on Mars and must find a way to survive.",
},
],
tensor_fields=["Description"],
)
curl -XPOST 'http://localhost:8882/indexes/text-search/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [
{
"Title": "Inception",
"Description": "A mind-bending thriller about dream invasion and manipulation"
},
{
"Title": "Shrek",
"Description": "An ogres peaceful life is disrupted by a horde of fairy tale characters who need his help"
},
{
"Title": "Interstellar",
"Description": "A team of explorers travel through a wormhole in space to ensure humanitys survival."
},
{
"Title": "The Martian",
"Description": "An astronaut becomes stranded on Mars and must find a way to survive."
}
],
"tensorFields": ["Description"]
}'
Step 4: Search with Marqo
# Perform a search query on the index
results = mq.index(index_name).search(q="Which movie is about space exploration?")
# Print the search results
for result in results["hits"]:
print(
f"Title: {result['Title']}, Description: {result['Description']}. Score: {result['_score']}"
)
curl -XPOST 'http://localhost:8882/indexes/text-search/search' \
-H 'Content-type:application/json' -d '
{
"q": "Which movie is about space exploration?"
}'
This returns:
Title: Interstellar, Description: A team of explorers travel through a wormhole in space to ensure humanity's survival.. Score: 0.8173517436600624
Title: The Martian, Description: An astronaut becomes stranded on Mars and must find a way to survive.. Score: 0.8081475581626953
Title: Inception, Description: A mind-bending thriller about dream invasion and manipulation.. Score: 0.7978701791216605
Title: Shrek, Description: An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.. Score: 0.7619883916893311
Full Code
GitHub Code: Text Search with Marqo Open Source Code
text_search_open_source.py
#####################################################
### STEP 1. Start Marqo
#####################################################
# 1. Marqo requires Docker. To install Docker go to Docker
# Docs and install for your operating system.
# 2. Once Docker is installed, you can use it to run Marqo.
# First, open the Docker application and then head to your
# terminal and enter the following:
"""
docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
"""
#####################################################
### STEP 2. Create Marqo Client
#####################################################
import marqo
# Create a Marqo client
mq = marqo.Client(url="http://localhost:8882")
#####################################################
### STEP 3. Create a Marqo Index & Add Documents
#####################################################
# Name your index
index_name = "text-search"
# Delete the movie index if it already exists (housekeeping)
try:
mq.index(index_name).delete()
except:
pass
# Create the movie index
mq.create_index(index_name, model="hf/e5-base-v2")
# Add documents (movie descriptions) to the index
mq.index(index_name).add_documents(
[
{
"Title": "Inception",
"Description": "A mind-bending thriller about dream invasion and manipulation.",
},
{
"Title": "Shrek",
"Description": "An ogre's peaceful life is disrupted by a horde of fairy tale characters who need his help.",
},
{
"Title": "Interstellar",
"Description": "A team of explorers travel through a wormhole in space to ensure humanity's survival.",
},
{
"Title": "The Martian",
"Description": "An astronaut becomes stranded on Mars and must find a way to survive.",
},
],
tensor_fields=["Description"],
)
#####################################################
### STEP 4. Search with Marqo
#####################################################
# Perform a search query on the index
results = mq.index(index_name).search(
q="Which movie is about space exploration?"
)
# Print the search results
for result in results['hits']:
print(f"Title: {result['Title']}, Description: {result['Description']}. Score: {result['_score']}")