Support pinecone-client to 3.0+ #37158

Activity

1. Taragolis

   added

   area:providers

   good first issue

   kind:metaHigh-level information important to the communityHigh-level information important to the community

   provider:pinecone

   on Feb 3, 2024
2. rawwar commented on Feb 7, 2024

   

   rawwar

   on Feb 7, 2024

   Contributor

   More actions

   I will be working on this. Thanks

   👍React with 👍1Reacted by Elad Kalif
3. eladkal

   assigned

   rawwar

   on Feb 7, 2024
4. rawwar commented on Feb 15, 2024

   

   rawwar

   on Feb 15, 2024

   Last edited by rawwar

   Contributor

   More actions

   UPDATE: I will be going with approach 2 and give examples in the documentation. If you do have feedback, please comment. Thanks!

   In the 3.x client, creating index signature has changed. Now, they use podSpec or serverlessSpec classes to pass configurations w.r.t infra. I've been thinking how to rewrite the hook and am split between two ways and need help in finalizing which one to go with.

   Approach 1:

   will have two methods in PineconeHook namely create_pod_index and create_serverless_index which will take all possible parameters w.r.t index and the spec and internally will create the Spec object and pass it to the create_index method

   method will be as follows for the pod based index creation

   def create_pod_index(
           self,
           index_name: str,
           dimension: int,
           environment: str,
           metric: str | None = "cosine",
           replicas: int | None = 1,
           shards: int | None = 1,
           pods: int | None = 1,
           pod_type: str | None = "p1.x1",
           metadata_config: dict[str, str] | None = None,
           source_collection: str | None = "",
           timeout: int | None = None,
       ) -> None:
           pod_spec = PodSpec(
               environment=environment,
               replicas=replicas,
               shards=shards,
               pods=pods,
               pod_type=pod_type,
               metadata_config=metadata_config,
               source_collection=source_collection,
           )
           
           self.conn.create_index(
               name=index_name,
               dimension=dimension,
               spec=pod_spec
               metric=metric,
               timeout=timeout,
           )
   

   Approach 2:

   Will have helper methods to create PodSpec and ServerlessSpec objects and then have the create_index method follow the pinecone signature and accept only one of these two types.

   def get_pod_spec_obj(self,environment, replicas, shards, pods, pod_type, metadata_config, source_collection):
           return PodSpec(
               environment=environment,
               replicas=replicas,
               shards=shards,
               pods=pods,
               pod_type=pod_type,
               metadata_config=metadata_config,
               source_collection=source_collection,
           )
   
   def create_index(self, name ,dimension,spec,metric,timeout):
       return self.conn.create_index(name=name,dimension=dimension,spec=spec,metric=metric,timeout=timeout)
   
   
5. Taragolis commented on Feb 17, 2024

   

   Taragolis

   on Feb 17, 2024

   ContributorAuthor

   More actions

   @rawwar difficult to say which approach are better. I can't find any usage into the operators, so assume this might only use into the custom operators or taskflow operators, so maybe create method create_index_from_spec?

   👍React with 👍1Reacted by Kalyan R
6. rawwar commented on Feb 22, 2024

   

   rawwar

   on Feb 22, 2024 · Hidden as outdated

   show commentMore actions
7. rawwar

   removed their assignment

   on Feb 22, 2024
8. rawwar commented on Apr 12, 2024

   

   rawwar

   on Apr 12, 2024

   Contributor

   More actions

   I am picking this back up.
9. eladkal

   assigned

   rawwar

   on Apr 13, 2024
10. eladkal

    added

    kind:taskA task that needs to be completed as part of a larger issueA task that needs to be completed as part of a larger issue

    and removed

    kind:metaHigh-level information important to the communityHigh-level information important to the community

    on Apr 13, 2024
11. Lee-W

    closed this as completedin #37307on Apr 30, 2024