Issue with IsIn Predicate Formatting in Iceberg-Go

[rameshkanna3]

Apache Iceberg version

v0.1.0

Please describe the bug 🐞

When using the IsIn filter in Iceberg-Go, filtering on a single integer value works correctly, but filtering on multiple values returns an empty result, even when matching records exist in the table.

The issue appears to be related to how the IsIn predicate is constructed in Go, as it differs from the expected format observed in PyIceberg.

---

Observations:

✅ Case 1: Passing a Single Value (Works Correctly)

scannertable, err := c.LoadTable(ctx, tableIdentifier, nil)
if err != nil {
	return nil, fmt.Errorf("failed to load table: %v", err)
}

docID := []int64{10} // Single value

userFilter := iceberg.EqualTo(iceberg.Reference("user"), "alice")
fmt.Println("userFilter: ", userFilter)

docIDFilter := iceberg.IsIn(iceberg.Reference("doc_id"), docID...)
fmt.Println("docIDFilter: ", docIDFilter)

combinedFilter := iceberg.NewAnd(userFilter, docIDFilter)
fmt.Println("combinedFilter: ", combinedFilter)

scanner := scannertable.Scan(
    table.WithRowFilter(combinedFilter),
    table.WithSelectedFields("doc_id"),
)

Output:

userFilter: Equal(term=Reference(name='user'), literal=alice)
docIDFilter: Equal(term=Reference(name='doc_id'), literal=10)
combinedFilter: And(left=Equal(term=Reference(name='user'), literal=alice), right=Equal(term=Reference(name='doc_id'), literal=10))
Collected doc_ids: [10] ✅

❌ Case 2: Passing Multiple Values (Not Working)

docID := []int64{10, 20} // Multiple values

userFilter := iceberg.EqualTo(iceberg.Reference("user"), "alice")
fmt.Println("userFilter: ", userFilter)

docIDFilter := iceberg.IsIn(iceberg.Reference("doc_id"), docID...)
fmt.Println("docIDFilter: ", docIDFilter)

combinedFilter := iceberg.NewAnd(userFilter, docIDFilter)
fmt.Println("combinedFilter: ", combinedFilter)

scanner := scannertable.Scan(
    table.WithRowFilter(combinedFilter),
    table.WithSelectedFields("doc_id"),
)

Output:

userFilter: Equal(term=Reference(name='user'), literal=alice)
docIDFilter: In(term=Reference(name='doc_id'), {[10 20]})
combinedFilter: And(left=Equal(term=Reference(name='user'), literal=alice), right=In(term=Reference(name='doc_id'), {[10 20]}))
Collected doc_ids: [] ❌

🚨 Issue: Even though doc_id values 10 and 20 exist in the table, the filter returns an empty result instead of the expected rows.

Comparison with PyIceberg (Works as Expected):

Using the same filter logic in PyIceberg, multiple values work correctly:

def filter_data(table):
    user_filter = EqualTo("user", "alice")

    doc_id=[10, 20,30,60]
    doc_id_filter = In("doc_id", doc_id)
    print("doc_id_filter",doc_id_filter)

    combined_filter = And(user_filter, doc_id_filter)
    print("combined_filter",combined_filter)

    filtered_data = table.scan(row_filter=combined_filter,selected_fields=["doc_id"]).to_pandas()
    doc_id_list = filtered_data["doc_id"].tolist()
    print(doc_id_list)

The above python code give the results like this

doc_id_filter: In(Reference(name='doc_id'), {10, 20, 30, 60})
combined_filter: And(left=EqualTo(term=Reference(name='user'), literal=literal('alice')), right=In(Reference(name='doc_id'), {10, 20, 30, 60}))
Collected doc_ids: [10, 20, 60]

Suspected Issue:

The IsIn predicate in Iceberg-Go appears to be formatted incorrectly:

• Iceberg-Go: In(term=Reference(name='doc_id'), {[10 20]}) (using a slice [])

• PyIceberg: In(Reference(name='doc_id'), {10, 20, 30, 60}) (using a set {})

This mismatch likely causes the filter to fail, leading to an empty result.

Expected Behavior:

• IsIn should correctly format the values in a set-like structure (similar to PyIceberg).
• Queries should return matching rows instead of an empty result when valid data exists in the table.

Question:

Is this the correct way to filter data in Iceberg-Go, or is there a different approach I should be using? If this is a bug, what would be the recommended fix?
Would appreciate any guidance on this! Thanks in advance.

[zeroshade]

So the bug here is that it's not returning an ErrNotImplemented 😄

Currently, since this library uses arrow-go to delegate the filtering we have the limitation that I haven't yet implemented the is_in function in the Go Arrow library. If you provide only a single value to IsIn it gets converted into an EqualTo predicate which is why that works.

I guess I finally need to put the work in and actually go implement the is_in functionality in the Arrow Go library, that, in turn, will get this up and running for you.

In the meantime, I need to look into why it's not correctly surfacing the computation error.

[zeroshade]

Once http://github.com/apache/arrow-go/pull/319 gets merged, I'll be able to update iceberg-go to properly utilize the is_in predicate. So far it works in local testing