Efficient Document Embedding Management with ChromaDB: Deleting, Resetting, and More | by Anay Dongre | May, 2024

ChromaDB, a robust and environment friendly vector database, provides a complete resolution for dealing with these embeddings. Nevertheless, as your dataset grows, chances are you’ll encounter conditions the place it is advisable delete particular paperwork, collections, and even reset your entire database. This text will information you thru the method of deleting doc embeddings, collections, supply information, and resetting the database utilizing ChromaDB.

Supply Code

The next Python code demonstrates delete paperwork and collections utilizing ChromaDB:

import chromadb
import os
from chromadb.config import Settings
# Perform to delete paperwork by IDs
def delete_documents(assortment, ids):
if ids:
# Delete the paperwork by IDs
assortment.delete(ids=ids)
print("Paperwork have been deleted from the gathering.")
else:
print("No paperwork discovered with the given filename.")
# Perform to immediate consumer for deleting the gathering
def delete_collection(consumer, assortment):
affirm = enter("Do you wish to delete the gathering? (y/n): ")
if affirm.decrease() == 'y':
collection_name = assortment.title
consumer.delete_collection(title=collection_name)
print(f"Assortment '{collection_name}' has been deleted.")
else:
print("Assortment deletion cancelled.")
# Persist listing for storing information
persist_directory = "./testing"
if not os.path.exists(persist_directory):
os.makedirs(persist_directory)
# Get the ChromaDB object
chroma_db = chromadb.PersistentClient(path=persist_directory, settings=Settings(allow_reset=True))
assortment = chroma_db.get_collection(title="docs_store_v2")
# Get all paperwork within the assortment
db_data = assortment.get()
# Extract metadata
metadatas = db_data['metadatas']
ids = db_data['ids']
# Show all supply file names current within the assortment
print("Supply file names current inside the gathering:")
source_file_names = set(metadata.get('supply') for metadata in metadatas)
for source_file_name in source_file_names:
print("- " + source_file_name)
# Get the filename from the consumer
filename = enter("nEnter the filename you wish to delete (e.g., 'instance.txt'): ")
# Discover doc IDs with matching filename
ids_to_delete = [id for id, metadata in zip(ids, metadatas) if metadata.get('source') == filename]
# Delete the paperwork with matching IDs
delete_documents(assortment, ids_to_delete)
# Ask the consumer in the event that they wish to delete the gathering
delete_collection(chroma_db, assortment)

Resetting the Database

Resetting the ChromaDB database is one other frequent operation. The next code snippet demonstrates reset the database:

# Perform to immediate consumer for resetting the database
def reset_database(consumer):
affirm = enter("Do you wish to reset the database? (y/n): ")
if affirm.decrease() == 'y':
consumer.reset()
print("Database has been reset.")
else:
print("Database reset cancelled.")
# Reset the database if requested
reset_database(chroma_db)

Understanding the Code

Let’s break it down and clarify every part:

1. Importing Dependencies and Organising ChromaDB: The script begins by importing the mandatory modules, together with chromadb, os, and Settings from chromadb.config. It then creates a persistent ChromaDB consumer occasion utilizing the PersistentClient class, specifying the listing path for storing information and enabling the allow_reset possibility.
2. Deleting Paperwork by IDs: The delete_documents operate takes a assortment and a listing of ids as enter. If the ids listing just isn’t empty, it deletes the corresponding paperwork from the gathering utilizing the delete methodology. This operate is useful while you wish to take away particular paperwork based mostly on their distinctive identifiers.
3. Deleting a Assortment: The delete_collection operate prompts the consumer to substantiate whether or not they wish to delete the desired assortment. If the consumer confirms, it retrieves the gathering title and deletes your entire assortment utilizing the delete_collection methodology of the ChromaDB consumer.
4. Resetting the Database: The reset_database operate asks the consumer to substantiate whether or not they wish to reset your entire database. If the consumer confirms, it calls the reset methodology of the ChromaDB consumer, successfully clearing all information saved within the database.
5. Displaying Supply File Names and Prompting for Deletion: The script retrieves all paperwork from the desired assortment and extracts their metadata and IDs. It then shows the listing of distinctive supply file names current within the assortment. The consumer is prompted to enter the filename they wish to delete.
6. Deleting Paperwork by Filename: Based mostly on the consumer’s enter filename, the script filters out the doc IDs related to that filename. It then calls the delete_documents operate with the filtered listing of IDs, successfully eradicating the paperwork associated to the desired filename from the gathering.
7. Updating the Record of Recordsdata: After deleting the paperwork, the script retrieves the up to date listing of information current within the assortment and shows them to the consumer.
8. Prompting for Assortment Deletion and Database Reset: Lastly, the script prompts the consumer to substantiate whether or not they wish to delete your entire assortment and reset the database utilizing the delete_collection and reset_database features, respectively.

Hope you realized/loved studying my article and completely satisfied coding!