Implement Indexer and Retriever functionalities for given environment clearance data

[KaranrajM]

Description

Strategize a suitable chunking technique to index the given environment clearance data, where each file contains a list of projects and their details. Additionally, implement a retriever that can perform the following actions:

• Search and look up a project with given location details (latitude, longitude, or State and City)
• Query for specific details only within the project

Goal

To develop an information retrieval system specific to environment clearance data.

Expected Outcome

• Efficient indexer and retriever services that enable users to:
  • Lookup and search for a list of projects in a particular location
  • Query specific details within a project

Acceptance Criteria

An information retrieval system specific to environment clearance data with high accuracy.

Implementation Details

1. Implement a suitable and efficient chunking technique for the given dataset.
2. Build a retriever that can:
   • Search and look up a project using location details (latitude, longitude, or State and City).
   • Search for specific details only within the project.
3. Sample parsed and cleaned EC data can be found here. Their respective bare data can be found here.

Mockups/Wireframes

NOT APPLICABLE

Product Name

Jugalbandi

Organisation Name

OpenNyAI

Domain

Legal

Tech Skills Needed

• Python

Requisites

• OpenAI API key for Indexing and RAG

Complexity

Medium

Category

Backend

[SudoSu-bham]

I am not able to understand, whether the environmental clearance data is stored in a spreadsheet or is it stored in text form in some files or you already have some database of environmental clearance data?

where each file contains a list of projects and their details
May you please elaborate?

[KaranrajM]

Hi @SudoSu-bham I have updated the issue with the drive link to the data. Basically both the issues in the repo are linked. The parsed data from the previous issue will be used to build and train RAG systems here. However I have given some already parsed EC data as an example for getting started. The link to their respective bare data files (for your reference) are also updated. Let me know if you have any other questions.

[SGCODEX]

Greetings @KaranrajM Sir, I have been selected for this issue. Please assign me this issue sir. And how can we connect sir to get started? Thank you sir.

[DevvStrange]

Hello, Thanks for taking out time for your contribution @SGCODEX . Excited to connect with you soon!

[Lekhanrao]

@SGCODEX, Assigned this ticket to you.
@KaranrajM and @DevvStrange will help you with Technical stuff.

[SGCODEX]

Thank you @Lekhanrao Sir and @DevvStrange Sir. Where can I reach out sir for connecting with you all?

[Lekhanrao]

@SGCODEX Arun will reach out to you via Discord or email and arrange for an Intro meeting this week.

[SGCODEX]

@Lekhanrao Sure sir, Thank you