idea: Make `list` required to return `content-length`

[Xuanwo]

OpenDAL previously handled returning content-length in the list response as a best-effort approach. We encouraged users to call stat outside of list. However, this approach has placed a significant burden on our users. Perhaps we should make it mandatory for list to return content-length and update our implementations accordingly. This way, our users won't be confused by this behavior anymore.

[PiyushXCoder]

I have added a PR for Fs service, should I proceed making it for other services too?

[meteorgan]

I'm not sure this is the best approach. For services that don't include content-length in the list response, this could cause a noticeable performance regression. Would it make sense to have content_length() return Option<u64> in Metadata instead?

[Xuanwo]

Would it make sense to have content_length() return Option<u64> in Metadata instead?

I don't like this idea since users will need to check every content_length even when they are sure that the size exists. Sometimes they need to call stat on themselves.

I'm not sure this is the best approach. For services that don't include content-length in the list response, this could cause a noticeable performance regression.

I double check all the services with the help codex. Nearly all services support or can be configured to return the size during listing, with only the following exceptions.

• redis (need extra call)
• cloudflare-kv (nobody use it in fact, I prefer to remove it if it's the blocker)
• ipfs (nobody use it in fact, I prefer to remove it if it's the blocker)
• fs, compfs, opfs

The only thing that matters to us is the local fs and redis.

The questions on my mind are:

• When users call list, what do they actually want? Can we assume that in 90% of cases, they only want to know the size?
• Is returning Option<u64> a good tradeoff just to satisfy two services?
• Is an extra stat call to fs or redis a high cost?

What's your answers to them?

[meteorgan]

Thanks for the detailed and in-depth response.

The only thing that matters to us is the local fs and redis.

We might also have etcd, but even so, there are only three services need to consider.

The questions on my mind are:

• When users call list, what do they actually want? Can we assume that in 90% of cases, they only want to know the size?

Since I'm not a typical user, i may not have a very reasonable answer here.

• Is returning Option<u64> a good tradeoff just to satisfy two services?

No. One of our principles is Object Storage First, so could we just simply document these two(or three) services don't return object size during listing and enforce this behavior with test case ?

• Is an extra stat call to fs or redis a high cost?

Although it depends on the use case. this isn't really what I'd expect as a user.

[chitralverma]

adding to the discussion here:

Is returning Option a good tradeoff just to satisfy two services?

IMO, this seems to be a good idea as services like fs return entry.metadata.content_length == 0 right now which is correct (for directories and files actually with 0 length) and incorrect (for files not with 0 length) at the same time.

changing the signature to return Option<u64> might be a overhead + breaking change considering only 3 services, but for the end user, its a consistent api in longer run and the control is with the user to locally check on what the value of Option is and what to do with it. if all other services always return a value, then content_length is always Some(...) for them.

When users call list, what do they actually want?

as a typical user, consistent {relative path, size, and type (dir/ file), timings (last_modified), version and id (md5/ etag) } is what I use on a day to day basis.

side note for redis: not sure if its possible, but we can look into a lua script transaction to merge that extra call into 1.

side note for http: from what i remember, content_length for http service, depends on the server and is not deterministic see #6289

[aawsome]

To give my 2ct as maintainer of rustic:

• we need the size in almost all cases when we do a list - as this provides information we'll use to do some cheap extra checks. Having list always return a valid content-length would be perfect!
• for us, we strongly depend on consistency between all used services - needing to treat some services differently is a no-go and would most likely cause us to not support that services (or actually get unexpectedly
• getting Option<u64> is actually much better than the current behavior, which IMO is also bad documented (just came over to the topic as I wanted to open a bug that fs service always returns a 0 content-length...)