Move Wav2Lip model download commands from Dockerfiles to entrypoint.sh (#811)

ctao456 · hteeyeoh · XinyuYe-Intel · web-flow · commit aea7364d4e1d · 2024-10-22T10:32:43.000+08:00
* Update audioqna gateway to print text, in gateway.py Signed-off-by: Chun Tao <chun.tao@intel.com> * updates needed for demo Signed-off-by: Chun Tao <chun.tao@intel.com> * original pr content Signed-off-by: Chun Tao <chun.tao@intel.com> * Revert "updates needed for demo" This reverts commit f0c7a02. Signed-off-by: Chun Tao <chun.tao@intel.com> * remove improper images Signed-off-by: Chun Tao <chun.tao@intel.com> * Addressed some comments on previous pr Signed-off-by: Chun Tao <chun.tao@intel.com> * Add Dockerfile for cpu support Signed-off-by: Chun Tao <chun.tao@intel.com> * CODEOWNER: Update comp CODEOWNER (#757) Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com> Signed-off-by: Chun Tao <chun.tao@intel.com> * Add stable diffusion microservice (#729) * add stable diffusion microservice. Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> * added test. Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> * changed output to images bytes data Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * unified inference and wrapper into one microservice. Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test. Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> --------- Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Chun Tao <chun.tao@intel.com> * Compatible with different platforms. (#766) * Compatible with different platforms. Signed-off-by: ZePan110 <ze.pan@intel.com> * Fix issue. Signed-off-by: ZePan110 <ze.pan@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue Signed-off-by: ZePan110 <ze.pan@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ZePan110 <ze.pan@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Chun Tao <chun.tao@intel.com> * Optimize path and link validity check. (#745) Signed-off-by: ZePan110 <ze.pan@intel.com> Signed-off-by: Chun Tao <chun.tao@intel.com> * Add timeout for ut test (#773) Signed-off-by: chensuyue <suyue.chen@intel.com> Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Chun Tao <chun.tao@intel.com> * test hyperlink Signed-off-by: Chun Tao <chun.tao@intel.com> * test hyperlink Signed-off-by: Chun Tao <chun.tao@intel.com> * test hyperlink issue Signed-off-by: Chun Tao <chun.tao@intel.com> * test hyperlink issue Signed-off-by: Chun Tao <chun.tao@intel.com> * put back hyperlinks in readme Signed-off-by: Chun Tao <chun.tao@intel.com> * remove possible error hyperlink Signed-off-by: Chun Tao <chun.tao@intel.com> * put hyperlink back Signed-off-by: Chun Tao <chun.tao@intel.com> * major update to use FastAPI for wav2lip, and structure component format Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add dockerfiles in animation-compose-cd.yaml Signed-off-by: Chun Tao <chun.tao@intel.com> * Fix end of file issue in animation-compose-cd.yaml Signed-off-by: Chun Tao <chun.tao@intel.com> * Fix Docker deployment on Xeon Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add versioning for all pip packages Signed-off-by: Chun Tao <chun.tao@intel.com> * e2e test script for animation Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update e2e test script Signed-off-by: Chun Tao <chun.tao@intel.com> * update e2e test script Signed-off-by: Chun Tao <chun.tao@intel.com> * update readme Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update Signed-off-by: Chun Tao <chun.tao@intel.com> * update Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update gateway Signed-off-by: Chun Tao <chun.tao@intel.com> * udpate gateway Signed-off-by: Chun Tao <chun.tao@intel.com> * Fix AVATAR_CHATBOT Signed-off-by: Chun Tao <chun.tao@intel.com> * update gateway Signed-off-by: Chun Tao <chun.tao@intel.com> * update Signed-off-by: Chun Tao <chun.tao@intel.com> * test Signed-off-by: Chun Tao <chun.tao@intel.com> * update Signed-off-by: Chun Tao <chun.tao@intel.com> * update Signed-off-by: Chun Tao <chun.tao@intel.com> * update gateway Signed-off-by: Chun Tao <chun.tao@intel.com> * fix max_tokens in AvatarChatbot gateway Signed-off-by: Chun Tao <chun.tao@intel.com> * test Signed-off-by: Chun Tao <chun.tao@intel.com> * update Signed-off-by: Chun Tao <chun.tao@intel.com> * update Signed-off-by: Chun Tao <chun.tao@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Model download script moved from Dockerfiles to Docker entrypoint Signed-off-by: Chun Tao <chun.tao@intel.com> * update paths Signed-off-by: Chun Tao <chun.tao@intel.com> * Correct paths in readme Signed-off-by: Chun Tao <chun.tao@intel.com> * revert changes to audioqna gateway Signed-off-by: Chun Tao <chun.tao@intel.com> * longer wait time after docker run Signed-off-by: Chun Tao <chun.tao@intel.com> * add mount volume in test scripts Signed-off-by: Chun Tao <chun.tao@intel.com> * add volume mount in test scripts Signed-off-by: Chun Tao <chun.tao@intel.com> * udpate test script Signed-off-by: Chun Tao <chun.tao@intel.com> --------- Signed-off-by: Chun Tao <chun.tao@intel.com> Signed-off-by: Yeoh, Hoong Tee <hoong.tee.yeoh@intel.com> Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com> Signed-off-by: ZePan110 <ze.pan@intel.com> Signed-off-by: chensuyue <suyue.chen@intel.com> Co-authored-by: Hoong Tee, Yeoh <hoong.tee.yeoh@intel.com> Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ZePan110 <ze.pan@intel.com> Co-authored-by: chen, suyue <suyue.chen@intel.com>
diff --git a/comps/animation/wav2lip/README.md b/comps/animation/wav2lip/README.md
@@ -42,11 +42,11 @@ export WAV2LIP_PORT=7860
 export ANIMATION_PORT=9066
 export INFERENCE_MODE='wav2lip+gfpgan'
 export CHECKPOINT_PATH='/usr/local/lib/python3.11/site-packages/Wav2Lip/checkpoints/wav2lip_gan.pth'
-export FACE="comps/animation/wav2lip/assets/img/avatar1.jpg"
+export FACE="assets/img/avatar1.jpg"
 # export AUDIO='assets/audio/eg3_ref.wav' # audio file path is optional, will use base64str in the post request as input if is 'None'
 export AUDIO='None'
 export FACESIZE=96
-export OUTFILE="comps/animation/wav2lip/assets/outputs/result.mp4"
+export OUTFILE="assets/outputs/result.mp4"
 export GFPGAN_MODEL_VERSION=1.4 # latest version, can roll back to v1.3 if needed
 export UPSCALE_FACTOR=1
 export FPS=10
@@ -61,11 +61,11 @@ export WAV2LIP_PORT=7860
 export ANIMATION_PORT=9066
 export INFERENCE_MODE='wav2lip+gfpgan'
 export CHECKPOINT_PATH='/usr/local/lib/python3.10/dist-packages/Wav2Lip/checkpoints/wav2lip_gan.pth'
-export FACE="comps/animation/wav2lip/assets/img/avatar1.jpg"
+export FACE="assets/img/avatar1.jpg"
 # export AUDIO='assets/audio/eg3_ref.wav' # audio file path is optional, will use base64str in the post request as input if is 'None'
 export AUDIO='None'
 export FACESIZE=96
-export OUTFILE="comps/animation/wav2lip/assets/outputs/result.mp4"
+export OUTFILE="assets/outputs/result.mp4"
 export GFPGAN_MODEL_VERSION=1.4 # latest version, can roll back to v1.3 if needed
 export UPSCALE_FACTOR=1
 export FPS=10
@@ -78,13 +78,13 @@ export FPS=10
 - Xeon CPU
 
 ```bash
-docker run --privileged -d --name "wav2lip-service" -p 7860:7860 --ipc=host -w /home/user/comps/animation/wav2lip -e PYTHON=/usr/bin/python3.11 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip:latest
+docker run --privileged -d --name "wav2lip-service" -p 7860:7860 --ipc=host -w /home/user/comps/animation/wav2lip -e PYTHON=/usr/bin/python3.11 -v $(pwd)/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip:latest
 ```
 
 - Gaudi2 HPU
 
 ```bash
-docker run --privileged -d --name "wav2lip-gaudi-service" -p 7860:7860 --runtime=habana --cap-add=sys_nice --net=host --ipc=host -w /home/user/comps/animation/wav2lip -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PYTHON=/usr/bin/python3.10 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip-gaudi:latest
+docker run --privileged -d --name "wav2lip-gaudi-service" -p 7860:7860 --runtime=habana --cap-add=sys_nice --net=host --ipc=host -w /home/user/comps/animation/wav2lip -v $(pwd)/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PYTHON=/usr/bin/python3.10 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip-gaudi:latest
 ```
 
 ## 2.2 Run Animation Microservice
@@ -100,8 +100,8 @@ Once microservice starts, user can use below script to validate the running micr
 ## 3.1 Validate Wav2Lip service
 
 ```bash
-cd GenAIComps/comps/animation/wav2lip
-python3 dependency/check_wav2lip_server.py
+cd GenAIComps
+python3 comps/animation/wav2lip/dependency/check_wav2lip_server.py
 ```
 
 ## 3.2 Validate Animation service
@@ -115,14 +115,14 @@ curl http://${ip_address}:9066/v1/animation -X POST -H "Content-Type: applicatio
 or
 
 ```bash
-cd GenAIComps/comps/animation/wav2lip
-python3 check_animation_server.py
+cd GenAIComps
+python3 comps/animation/wav2lip/dependency/check_animation_server.py
 ```
 
 The expected output is a message similar to the following:
 
 ```bash
-{'wav2lip_result': '.../GenAIComps/comps/animation/wav2lip/assets/outputs/result.mp4'}
+{'wav2lip_result': '....../GenAIComps/comps/animation/wav2lip/assets/outputs/result.mp4'}
 ```
 
 Please find "comps/animation/wav2lip/assets/outputs/result.mp4" as a reference generated video.
diff --git a/comps/animation/wav2lip/assets/outputs/result.mp4 b/comps/animation/wav2lip/assets/outputs/result.mp4
diff --git a/comps/animation/wav2lip/check_animation_server.py b/comps/animation/wav2lip/check_animation_server.py
@@ -11,7 +11,7 @@
 outfile = os.environ.get("OUTFILE")
 
 # Read the JSON file
-with open("assets/audio/sample_question.json", "r") as file:
+with open("comps/animation/wav2lip/assets/audio/sample_question.json", "r") as file:
     data = json.load(file)
 
 response = requests.post(url=endpoint, json=data, headers={"Content-Type": "application/json"}, proxies={"http": None})
diff --git a/comps/animation/wav2lip/dependency/Dockerfile b/comps/animation/wav2lip/dependency/Dockerfile
@@ -51,8 +51,6 @@ ENV PYTHONPATH="$PYTHONPATH:/usr/local/lib/python3.11/site-packages/gfpgan"
 
 # Download pre-trained models
 WORKDIR /usr/local/lib/python3.11/site-packages
-RUN chmod +x /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
-RUN /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
 
 # Install pip dependencies
 RUN pip install -r /home/user/comps/animation/wav2lip/requirements.txt
diff --git a/comps/animation/wav2lip/dependency/Dockerfile.intel_hpu b/comps/animation/wav2lip/dependency/Dockerfile.intel_hpu
@@ -45,8 +45,6 @@ ENV PYTHONPATH="$PYTHONPATH:/usr/local/lib/python3.10/dist-packages/gfpgan"
 
 # Download pre-trained models
 WORKDIR /usr/local/lib/python3.10/dist-packages
-RUN chmod +x /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
-RUN /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
 
 # Install pip dependencies
 RUN pip install -r /home/user/comps/animation/wav2lip/requirements.txt
diff --git a/comps/animation/wav2lip/dependency/check_wav2lip_server.py b/comps/animation/wav2lip/dependency/check_wav2lip_server.py
@@ -10,7 +10,7 @@
 outfile = os.environ.get("OUTFILE")
 
 # Read the JSON file
-with open("assets/audio/sample_question.json", "r") as file:
+with open("comps/animation/wav2lip/assets/audio/sample_question.json", "r") as file:
     data = json.load(file)
 
 inputs = {"audio": data["byte_str"]}
diff --git a/comps/animation/wav2lip/dependency/entrypoint.sh b/comps/animation/wav2lip/dependency/entrypoint.sh
@@ -1,14 +1,29 @@
 #!/bin/sh
-cd /home/user/comps/animation/wav2lip/ || exit
 
 # Copyright (C) 2024 Intel Corporation
 # SPDX-License-Identifier: Apache-2.0
 
+# Check the value of $DEVICE and cd to the download path accordingly
+if [ "$DEVICE" = "hpu" ]; then
+    cd /usr/local/lib/python3.10/dist-packages
+else
+    cd /usr/local/lib/python3.11/site-packages
+fi
+
+# Download model weights
+wget https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth -O Wav2Lip/face_detection/detection/sfd/s3fd.pth
+mkdir -p Wav2Lip/checkpoints
+wget "https://iiitaphyd-my.sharepoint.com/:f:/g/personal/radrabha_m_research_iiit_ac_in/Eb3LEzbfuKlJiR600lQWRxgBIY27JZg80f7V9jtMfbNDaQ?download=1" -O Wav2Lip/checkpoints/wav2lip.pth
+wget "https://iiitaphyd-my.sharepoint.com/:f:/g/personal/radrabha_m_research_iiit_ac_in/EdjI7bZlgApMqsVoEUUXpLsBxqXbn5z8VTmoxp55YNDcIA?download=1" -O Wav2Lip/checkpoints/wav2lip_gan.pth
+wget https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth -P gfpgan/experiments/pretrained_models
+echo "Face Detector, Wav2Lip, GFPGAN weights downloaded."
+
 # Environment variables
 export PT_HPU_LAZY_MODE=0
 export PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES=1
 
 # Wav2Lip, GFPGAN
+cd /home/user/comps/animation/wav2lip/ || exit
 python3 dependency/wav2lip_server.py \
 --device $DEVICE \
 --port $((WAV2LIP_PORT)) \
diff --git a/tests/animation/test_animation_wav2lip.sh b/tests/animation/test_animation_wav2lip.sh
@@ -17,7 +17,7 @@ function build_docker_images() {
     else
         echo "opea/wav2lip built successful"
     fi
-    docker build -t opea/animation:comps -f comps/animation/wav2lip/Dockerfile .
+    docker build --no-cache -t opea/animation:comps -f comps/animation/wav2lip/Dockerfile .
     if [ $? -ne 0 ]; then
         echo "opea/animation built fail"
         exit 1
@@ -43,12 +43,13 @@ function start_service() {
     export UPSCALE_FACTOR=1
     export FPS=10
 
-    docker run -d --name="test-comps-animation-wav2lip" -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT -p 7860:7860 --ipc=host opea/wav2lip:comps
-    docker run -d --name="test-comps-animation" -e WAV2LIP_ENDPOINT=http://$ip_address:7860 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9066:9066 --ipc=host opea/animation:comps
-    sleep 1m
+    docker run -d --name="test-comps-animation-wav2lip" -v $WORKPATH/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT -p 7860:7860 --ipc=host opea/wav2lip:comps
+    docker run -d --name="test-comps-animation" -v $WORKPATH/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e WAV2LIP_ENDPOINT=http://$ip_address:7860 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9066:9066 --ipc=host opea/animation:comps
+    sleep 3m
 }
 
 function validate_microservice() {
+    cd $WORKPATH
     result=$(http_proxy="" curl http://localhost:9066/v1/animation -X POST -H "Content-Type: application/json" -d @comps/animation/wav2lip/assets/audio/sample_question.json)
     if [[ $result == *"result.mp4"* ]]; then
         echo "Result correct."
