Add support for Audio and Video summarization to Docsum

[MSCetin37]

Description

Extend the current Document Summarization Application by incorporating video and audio summary features. This enhancement will enable the application to summarize video and audio content in addition to text documents, thereby broadening its utility and applicability.

Issues

https://github.com/opea-project/docs/blob/main/community/rfcs/24-06-21-OPEA-001-DocSum_Video_Audio.md

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Tests

Whisper Service

Run the following command to validate the Whisper Service:

python comps/asr/whisper/dependency/check_whisper_server.py 

Expected output:

{'asr_result': 'who is pat gelsinger'}

Audio2Text Service

Run the following command to validate the Audio2Text Service:

python comps/dataprep/multimedia2text/audio2text/check_a2t_server.py

Expected output:

{'downstream_black_list': [], 'id': '21b0459477abea6d85d20f4b5ddcb714', 'query': 'who is pat gelsinger'}

Note: The id value will be different.

Video2Audio Service

Run the following command to validate the Video2Audio Service:

python comps/dataprep/multimedia2text/video2audio/check_v2a_microserver.py

Expected output:

========= Audio file saved as ======
comps/dataprep/multimedia2text/video2audio/converted_audio.wav
====================================

Multimedia2Text Service

Run the following command to validate the Multimedia2Text Service:

python comps/dataprep/multimedia2text/check_multimedia2text.py 

Expected output:

Running test: Whisper service
>>> Whisper service Test Passed ... 

Running test: Audio2Text service
>>> Audio2Text service Test Passed ... 

Running test: Video2Text service
>>> Video2Text service Test Passed ... 

Running test: Multimedia2text service
>>> Multimedia2text service test for text data type passed ... 
>>> Multimedia2text service test for audio data type passed ... 
>>> Multimedia2text service test for video data type passed ... 

cd GenAIComps/tests
bash tests/dataprep/test_dataprep_multimedia.sh

[codecov]

Codecov Report

Attention: Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
comps/cores/mega/gateway.py	40.00%	3 Missing ⚠️

Files with missing lines	Coverage Δ
comps/cores/proto/api_protocol.py	96.06% <100.00%> (+0.05%)	⬆️
comps/cores/proto/docarray.py	99.42% <100.00%> (+0.02%)	⬆️
comps/cores/mega/gateway.py	29.82% <40.00%> (+0.80%)	⬆️

... and 1 file with indirect coverage changes

[ashahba]

LGTM!

[mkbhanda]

LGTM

[lvliang-intel]

@MSCetin37,
please update comps/dataprep/multimedia2text/audio2text/Dockerfile to .github/workflows/docker/compose/dataprep-compose.yaml. The yaml is used for release images build.

[ashahba]

@MSCetin37, please update comps/dataprep/multimedia2text/audio2text/Dockerfile to .github/workflows/docker/compose/dataprep-compose.yaml. The yaml is used for release images build.

@lvliang-intel the 3 Dockerfiles in the PR are already part of the build in that file so we are good.

[lianhao]

The audio2text service is almost the same as the asr service except the returned data format. Why can't we combine these 2 services into one?