S3-compatible endpoint consistency causing 409 Upload-Offset errors

[chendo]

We're using an S3 API 'compatible' storage which seems to occasionally take a second or two to actually persist the update. This seems to be causing our 409 Upload-Offset header doesn't match current offset errors as when subsequent chunks are uploaded, the change to the info hasn't updated yet.

I'm thinking of subclassing the S3 storage adapter to add a delay for subsequent chunks, or potentially adding some kind of automatic retry mechanism on the client side. Is there a better solution?

[janko]

Thanks for the report. The approach that tus-ruby-server uses is that it stores the state of the upload in a separate S3 object (in JSON format). When a PATCH request arrives and a new multipart part is uploaded, tus-ruby-server will write the new Upload-Offset to that S3 object. Then on next request it will retrieve that S3 object and compare the Upload-Offset.

The reason why I mention this is because tusd, the official golang tus server implementation, uses a different approach. Instead of storing this state in the S3 object, on each PATCH request it fetches the parts of the multipart upload and sums up their sizes, and uses that as the current offset, comparing it to the Upload-Offset provided by the client.

Since I don't see any 409 errors reported on their issue tracker, I'm wondering if that approach is generally more reliable. Some time ago I asked the maintainers which approach is better, and we came to a conclusion that both probably "suffer" from eventual consistency (even when you look at AWS S3 itself). However, because tusd is more battle tested and has more contributors, I'm guessing it would be better if tus-ruby-server just copies their approach. Though it would require some rewriting.

If you see these errors consistently, it would be helpful if you could try running tusd as the tus server, and tell me if these errors are gone. That might tell us whether multipart uploads are "more consistent" than regular uploads, and whether tus-ruby-server should use that as the source of truth.

[chendo]

Interesting, I will give tusd a go. Thanks!