Feature: Support dataframes in lui() calls via Arrow upload

[lmeyerov]

Feature Request

Description

Enable passing dataframes directly to lui() function calls using Arrow format for efficient data transfer.

Proposed API

from typing import Union, Dict, Optional
import pandas as pd
import pyarrow as pa

DfLike = Union[pd.DataFrame, pa.Table]

# Single dataframe
lui('analyze this data', df=sales_df)

# Multiple named dataframes  
lui('compare these datasets', dfs={'sales': sales_df, 'inventory': inventory_df})

# Optional parameters
def lui(prompt: str, df: Optional[DfLike] = None, dfs: Optional[Dict[str, DfLike]] = None):
    ...

Use Case

Users working with dataframes currently need to convert them to text or manually serialize. Direct dataframe support would:

• Eliminate manual serialization steps
• Preserve data types and schemas
• Enable efficient transfer of large datasets
• Work seamlessly with existing pandas/pyarrow workflows

Implementation Details

• Accept pd.DataFrame and pa.Table as DfLike types
• Automatically convert pandas DataFrames to Arrow format internally
• Use Arrow IPC format for efficient binary serialization
• Support both single dataframe (df param) and multiple named dataframes (dfs param)
• Handle upload behind the scenes transparently

Example Usage

import pandas as pd
from louie import lui

# Single dataframe
sales_df = pd.DataFrame({'product': ['A', 'B'], 'revenue': [1000, 1500]})
response = lui('What are the top selling products?', df=sales_df)

# Multiple dataframes
inventory_df = pd.DataFrame({'product': ['A', 'B'], 'stock': [50, 30]})
response = lui(
    'Which products need restocking based on sales velocity?',
    dfs={'sales': sales_df, 'inventory': inventory_df}
)

# With pyarrow.Table
import pyarrow as pa
arrow_table = pa.Table.from_pandas(sales_df)
response = lui('Analyze this Arrow table', df=arrow_table)

Benefits

• Clean, intuitive API matching existing lui() signature
• Zero friction for data scientists already using pandas/pyarrow
• Type safety with Union types
• Efficient binary format reduces latency
• Preserves nulls, timestamps, and other complex types correctly

Technical Considerations

• Arrow IPC format for wire protocol
• Automatic chunking for large dataframes
• Memory-efficient streaming where possible
• Graceful fallback for unsupported column types

🤖 Generated with Claude Code

[onestardao]

nice proposal — arrow definitely helps with bulk transfer and dtype clarity.

just wanted to flag a pattern we’ve seen before when piping dataframes directly into logic chains or semantic functions like lui(). once the data hits downstream models, things can get tricky if column names are too abstract or symbolic, especially when their semantic weight isn’t preserved across the pipeline. this can lead to broken embeddings or logic freezes even when the dataframe looks technically fine.

we’ve encountered this in systems where tabular inputs silently degrade reasoning quality due to mismatched field semantics. if your downstream expects any sort of semantic routing or inference, it might be worth considering lightweight field descriptors or ΔS-based heuristics to avoid symbolic collapse.

happy to share what we tried if that becomes relevant later.

[lmeyerov]

Thanks @onestardao --Curious if you have recs?

The 80% case is probably doing with least user friction:

• Analyst has a CSV file or a DB query result
• They want to ask the AI a question about it

(Maybe also txt, pdf, or binary -- but I think we special case semistructured data for this issue)

Pandas, and esp Arrow/parquet, is pretty good at encoding data representation schema, so we get that.

Agreed that default not great at semantic / ontology. User might have good col names for that, or maybe use metadata field to add more. Maybe that's enough here for the typical case, passing that to the AI system? Or should something more be passed?

[onestardao]

yeah... you just described one of the core failures we tracked in open-source RAG setups:
semantic collapse not because of missing data — but because the wrong chunk API was used, or internal chunking logic wasn't properly surfaced upstream.

we documented this as Problem No.1: "Mismatch in Chunk API Call", and it's way more common than folks realize.
breakdown here (with solution ideas):
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

hope it saves you 2 weeks of tracing a bug that was never in the data itself.
(been there.)

[lmeyerov]

Hi @onestardao thank you, maybe you can help clarify -

Afaict, txt os has powerful RAG reasoning methods, and as part of that, careful data prep methods that compute useful data representations. Reductively, it sounds like we should imagine extending the df type signature to support passing in RAG indexes that agents with custom retrievers can take advantage of, and maybe an opinionated 'common case' recommendation.

For example, in graph rag, the summary index structures are different representations for capturing vector & text search indexes, an edge table mapping them to one another, and a source citation index. As another, txtos has its own data representations, though I don't have a clear view of that internalized.

A key part of lui is keeping things serializable and composable. I can imagine two paths to supporting that here:

1. Lean into DFs, and allow passing around uninterpreted augmenting DFs. Pandas supports custom indexes, pluggable dtypes, etc, so this can go far -- I'm unsure for arrow support of same, but maybe.

Ex: For GraphRAG, Imagine vector_df, edges_df, etc. that have different shapes and pluggable types vs original df. We would leave their handling uninterpreted and up to a specialized agent. This feels close to about something like langchain would do wrt tools & agents.

txt_semantic_index: List[pd.DataFrame] = txtos(df)
df.metadata = mark_txtos(df) # + "This is a txtos df, see associated df's"

lui(
  'some tricky question',
  df=(df,txt_semantic_index)
  agent='txtos'
)

2. Marry a cross-cutting notion of a retriever interface so that different agents can take advantage of these smarter retrieval methods without being aware of how they actually work inside. This feels close to systems like llmindex.

retriever = txtos(df, ...)

# df methods + new .search(txt, k, thresh), .ask(txt, out_type=txt|df), ...
assert isinstance(retriever, Retriever)


lui('some question', retriever=retriever)

Note that by supporting a richer structure than a dataframe, every agent can benefit now as long as it sticks to retrieval methods like search and ask, In addition to usual DF methods.

Unfortunately this introduces mobile code for defining custom retrievers, so opens up questions I don't have clear answers to. But I can imagine somebody having a loader script to import these custom types depending on the environment...

---

Maybe you have thoughts here, here or can point to the specific type signatures that would be easy to understand?

[onestardao]

great points — and yes, you’re absolutely right that aligning reasoning methods with structured data access (like dataframes or typed APIs) introduces sharp tradeoffs.

there are two main architectural frictions we’ve seen in systems like this:

1. dual semantics between runtime data vs retrieval logic — sometimes the data has all the labels, but the retriever or agent layer discards or remaps structure due to unscoped summarization, especially if text→text→tool chains aren’t tightly type-bound.
2. agent-oblivious encoding paths — where embeddings or intermediate calls lose context of structure (like dataframe row-grouping or schema breaks) even if the final agent has powerful inspection abilities.

this shows up in many real-world RAG setups when chunk metadata or index logic goes async — or when retrieval is optimized separately from agent alignment.

we wrote up some of this tension in our “rag architecture and recovery” notes, especially the parts on type flattening and post-hoc patching:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md

if useful, happy to share how we’ve worked around this with dual-mode agents or structure-preserving context bridges — let me know what setup you're working with!

[lmeyerov]

I'm not sure how to make progress on your thinking. Maybe we should have a new issue about adding llmindex support, or a custom retriever type, as a way to support additional data types with rich & reusable retriever semantics? Would that solve your use case?

[onestardao]

yeah, i think you're spot on

adding a new retriever class that encodes type signatures + bridges to structured semantics is definitely one clean way to go.

happy to jam on the interface design or show how we’ve been doing type‑preserving reasoning across agent hops if it helps — no pressure though, sounds like you're already on a great track

[lmeyerov]

Awesome! Yeah I'll file a follow-up issue to explore this