Skip to content

huggingface/tei-gaudi

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Text Embeddings Inference on Habana Gaudi

To use 🤗 text-embeddings-inference on Habana Gaudi/Gaudi2, follow these steps:

  1. Pull the official Docker image with:
    docker pull ghcr.io/huggingface/tei-gaudi:latest

Note

Alternatively, you can build the Docker image using Dockerfile-hpu located in this folder with:

docker build -f Dockerfile-hpu -t tei_gaudi .
  1. Launch a local server instance on 1 Gaudi card:
    model=BAAI/bge-large-en-v1.5
    volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
    
    docker run -p 8080:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e MAX_WARMUP_SEQUENCE_LENGTH=512 --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tei-gaudi:latest --model-id $model --pooling cls
    For models within the Transformers library that need remote code to run customized implementations, please set the environment variable -e TRUST_REMOTE_CODE=TRUE within docker run command line. Here is an example:
    model="Alibaba-NLP/gte-large-en-v1.5"
    volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
    
    docker run -p 8080:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e MAX_WARMUP_SEQUENCE_LENGTH=512 -e TRUST_REMOTE_CODE=TRUE --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tei-gaudi:latest --model-id $model --pooling cls
    
  2. You can then send a request:
     curl 127.0.0.1:8080/embed \
         -X POST \
         -d '{"inputs":"What is Deep Learning?"}' \
         -H 'Content-Type: application/json'

For more information and documentation about Text Embeddings Inference, checkout the README of the original repo.

Not all features of TEI are currently supported as this is still a work in progress.

Validated Models

Architecture Model Type Models
BERT Embedding
  • BAAI/bge-large-en-v1.5
  • sentence-transformers/all-MiniLM-L6-v2
  • sentence-transformers/all-MiniLM-L12-v2
  • sentence-transformers/multi-qa-MiniLM-L6-cos-v1
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • sentence-transformers/paraphrase-MiniLM-L3-v2
  • MPNet Embedding
  • sentence-transformers/all-mpnet-base-v2
  • sentence-transformers/paraphrase-multilingual-mpnet-base-v2
  • sentence-transformers/multi-qa-mpnet-base-dot-v1
  • ALBERT Embedding
  • sentence-transformers/paraphrase-albert-small-v2
  • Mistral Embedding
  • intfloat/e5-mistral-7b-instruct
  • Salesforce/SFR-Embedding-2_R
  • GTE Embedding
  • Alibaba-NLP/gte-large-en-v1.5
  • JinaBERT Embedding
  • jinaai/jina-embeddings-v2-base-en
  • Roberta Sequence Classification
  • SamLowe/roberta-base-go_emotions
  • The license to use TEI on Habana Gaudi is the one of TEI: https://github.com/huggingface/text-embeddings-inference/blob/main/LICENSE

    Please reach out to [email protected] if you have any question.

    Packages

     
     
     

    Languages

    • Rust 92.2%
    • Python 4.8%
    • JavaScript 1.9%
    • Other 1.1%