Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,39 @@ For comprehensive guides, examples, and API reference:
- 🤝 Contributing guidelines and development setup


## ☁️ Cloud files contract

`SW.PrimitiveTypes` defines the object-storage contract that the
[SimplyWorks.CloudFiles](https://github.com/simplify9/SW-CloudFiles) providers implement
(S3 and S3-compatible stores, Azure Blob, Google Cloud Storage, Oracle Object Storage, and a local store for tests):

| Type | Role |
|------|------|
| `ICloudFilesService` | write, read, list, metadata, delete, public and signed URLs, streamed upload (S3 only) |
| `ICloudFilesLifecycle` | reads the bucket's age-based deletion rules (`temp1/`, `temp7/`, `temp30/`, `temp365/` and any others) |
| `WriteFileSettings`, `CloudFileInfo`, `RemoteBlob`, `WriteWrapper` | inputs and results |
| `CloudFilesOptions` | connection settings, bound from the `CloudFiles` configuration section |

The members are the same on every provider, but some behaviour differs: metadata keys, the result of
deleting a missing key, which exception a missing key throws, and URL shapes. The XML docs note the main
differences; the full table is the CloudFiles
[provider behaviour reference](https://github.com/simplify9/SW-CloudFiles/blob/main/docs/provider-behavior.md).

**Changing the contract.** Applications implement `ICloudFilesService` too (test fakes, stubs,
decorators), so a new member breaks them even though it looks additive. Never add, change or remove
members of `ICloudFilesService` or the shared models. Add a capability as a new interface that providers
implement and register in DI beside `ICloudFilesService`, as `ICloudFilesLifecycle` is; add new model
fields as optional properties. Release PrimitiveTypes first, then the providers, then consumers. Rules,
roadmap and what stays out of scope:
[Evolving the contract](https://github.com/simplify9/SW-CloudFiles/blob/main/docs/evolving-the-contract.md).

## 🏛️ Related Packages

Part of the Simplify9 ecosystem:
- **[SimplyWorks.Bus](https://github.com/simplify9/Bus)** - Message bus implementation
- **[SimplyWorks.CqApi](https://github.com/simplify9/CqApi)** - Dynamic API generation
- **[SimplyWorks.Searchy](https://github.com/simplify9/Searchy)** - Search abstraction
- **[SimplyWorks.CloudFiles](https://github.com/simplify9/CloudFiles)** - Cloud storage
- **[SimplyWorks.CloudFiles](https://github.com/simplify9/SW-CloudFiles)** - Cloud storage

## 📄 License

Expand Down
11 changes: 10 additions & 1 deletion SW.PrimitiveTypes/Contracts/CloudFiles/CloudFileInfo.cs
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
using System;
using System;
using System.Collections.Generic;
using System.Text;

namespace SW.PrimitiveTypes
{
/// <summary>One object returned by <see cref="ICloudFilesService.ListAsync"/>.</summary>
public class CloudFileInfo
{
/// <summary>The object's full key, including the listed prefix.</summary>
public string Key { get; set; }

/// <summary>Size in bytes. Zero-byte keys ending in <c>/</c> are folder placeholders.</summary>
public long Size { get; set; }

/// <summary>
/// An opaque change marker: the ETag on S3 and Azure, the base64 MD5 on Google, <c>null</c> on
/// Oracle and the local store. Compare it for equality only; it is not always a checksum.
/// </summary>
public string Signature { get; set; }
}
}
14 changes: 13 additions & 1 deletion SW.PrimitiveTypes/Contracts/CloudFiles/CloudFilesOptions.cs
Original file line number Diff line number Diff line change
@@ -1,16 +1,28 @@
using System;
using System;

namespace SW.PrimitiveTypes
{
/// <summary>
/// Connection settings shared by the storage providers, bound from the <c>CloudFiles</c>
/// configuration section. Each provider adds its own options in a subclass.
/// </summary>
public class CloudFilesOptions
{

public const string ConfigurationSection = "CloudFiles";

/// <summary>Access key (S3) or account name (Azure shared key).</summary>
public string AccessKeyId { get; set; }
/// <summary>Secret key (S3) or account key (Azure shared key).</summary>
public string SecretAccessKey { get; set; }
/// <summary>The bucket, or Azure container, every key is relative to.</summary>
public string BucketName { get; set; }
/// <summary>The store's endpoint, such as an S3-compatible endpoint or an Azure blob service URL.</summary>
public string ServiceUrl { get; set; }
/// <summary>
/// Not read by the SimplyWorks.CloudFiles providers: the provider is the one whose
/// <c>Add*CloudFiles()</c> the application calls. Applications may use it to choose one.
/// </summary>
public string Driver { get; set; } = "S3";
}
}
127 changes: 93 additions & 34 deletions SW.PrimitiveTypes/Contracts/CloudFiles/ICloudFilesService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,76 +5,135 @@
using System.Threading.Tasks;

namespace SW.PrimitiveTypes
{
{
/// <summary>
/// Object storage behind one interface: one bucket (an Azure container), addressed by string keys.
/// Implemented by the SimplyWorks.CloudFiles providers (S3 and S3-compatible stores, Azure Blob,
/// Google Cloud Storage, Oracle Object Storage, and a local filesystem store for tests).
/// </summary>
/// <remarks>
/// <para>
/// The signatures are the same on every provider; the behaviour is not always. Metadata keys,
/// what <see cref="DeleteAsync"/> returns for a missing key, the exception thrown for a missing
/// key, URL shapes and the cost of a signed URL all differ. The per-provider reference is
/// docs/provider-behavior.md in https://github.com/simplify9/SW-CloudFiles.
/// </para>
/// <para>
/// Applications also implement this interface (fakes, stubs, decorators), so adding a member
/// breaks them. New capabilities are added as separate interfaces registered next to this one,
/// as <see cref="ICloudFilesLifecycle"/> is. Never add, change or remove members here.
/// </para>
/// </remarks>
public interface ICloudFilesService
{
/// <summary>
/// Method to write a file asynchronously (privately or publicly)
/// Uploads <paramref name="inputStream"/> to <see cref="WriteFileSettings.Key"/>, replacing any
/// object already stored under that key.
/// </summary>
/// <param name="inputStream"></param>
/// <param name="settings"></param>
/// <returns></returns>
/// <param name="inputStream">
/// The content. Use a seekable stream (memory or file): the S3 and Azure providers read its
/// <see cref="Stream.Length"/> and throw for streams that can't report one.
/// </param>
/// <param name="settings">
/// Key, content type, metadata and visibility. <see cref="WriteFileSettings.Public"/> and
/// <see cref="WriteFileSettings.Metadata"/> are not honoured by every provider.
/// </param>
/// <returns>
/// The stored blob. <see cref="RemoteBlob.Location"/> may be a signed URL that expires after an
/// hour, so store the key, not the location.
/// </returns>
Task<RemoteBlob> WriteAsync(Stream inputStream, WriteFileSettings settings);

/// <summary>
/// Method to write a text file asynchronously (private or public)
/// Uploads <paramref name="text"/> as UTF-8 to <see cref="WriteFileSettings.Key"/>, replacing any
/// object already stored under that key. The content type defaults to <c>text/plain</c>.
/// </summary>
/// <param name="text"></param>
/// <param name="settings"></param>
/// <returns></returns>
/// <param name="text">The content.</param>
/// <param name="settings">
/// As for <see cref="WriteAsync"/>; the S3 provider ignores
/// <see cref="WriteFileSettings.Metadata"/> here.
/// </param>
/// <returns>The stored blob; see <see cref="WriteAsync"/>.</returns>
Task<RemoteBlob> WriteTextAsync(string text, WriteFileSettings settings);

/// <summary>
/// Get a public url for the a file with an a specific expiry
/// A read-only URL for <paramref name="key"/> that works whether or not the object is public,
/// until <paramref name="expiry"/> has passed.
/// </summary>
/// <param name="key"></param>
/// <param name="expiry"></param>
/// <returns></returns>
/// <remarks>
/// Computed locally on S3, Google and Azure with a shared key. Azure with a managed identity makes
/// a network call per URL, and Oracle creates a server-side pre-authenticated request per URL,
/// so don't generate them in loops on those providers.
/// </remarks>
/// <param name="key">The object's key.</param>
/// <param name="expiry">How long the URL stays valid, from now.</param>
string GetSignedUrl(string key, TimeSpan expiry);

/// <summary>
/// Get a url that only works for public files
/// The permanent, unsigned URL of <paramref name="key"/>. It is built without calling the store
/// and works only when the object or its bucket is publicly readable.
/// </summary>
/// <param name="key"></param>
/// <returns></returns>
/// <param name="key">The object's key.</param>
string GetUrl(string key);

/// <summary>
/// Method to open and update a file
/// Starts a streamed upload to <see cref="WriteFileSettings.Key"/> through a single HTTP PUT:
/// write to <see cref="WriteWrapper.GetStreamAsync"/>, then call
/// <see cref="WriteWrapper.CompleteRequestAsync"/>.
/// </summary>
/// <param name="key"></param>
/// <returns></returns>
/// <remarks>
/// S3 only. Other providers throw <see cref="NotImplementedException"/> or don't work. Prefer
/// <see cref="WriteAsync"/>.
/// </remarks>
/// <param name="settings">Key, content type and visibility of the object to write.</param>
WriteWrapper OpenWrite(WriteFileSettings settings);

/// <summary>
/// Method to read a file
/// Opens <paramref name="key"/> for reading. The caller must dispose the stream.
/// </summary>
/// <param name="key"></param>
/// <returns></returns>
/// <remarks>
/// Most providers stream from the network; the Google provider buffers the whole object in
/// memory first. A missing key throws, and the exception type is the provider SDK's own
/// (a 404 status in every network provider, <see cref="FileNotFoundException"/> locally).
/// </remarks>
/// <param name="key">The object's key.</param>
Task<Stream> OpenReadAsync(string key);

/// <summary>
///
/// Every object whose key starts with <paramref name="prefix"/>, including objects in nested
/// "folders". The whole listing is returned, one request per 1,000 keys, and held in memory.
/// </summary>
/// <param name="prefix"></param>
/// <returns></returns>
/// <remarks>
/// The S3 and Oracle providers returned only the first 1,000 keys before SimplyWorks.CloudFiles
/// 10.0.4. The match is by prefix: to test whether one key exists, compare
/// <see cref="CloudFileInfo.Key"/> exactly instead of calling <c>Any()</c> on the result.
/// </remarks>
/// <param name="prefix">Key prefix, such as <c>"invoices/2026/"</c>. Empty lists the bucket.</param>
Task<IEnumerable<CloudFileInfo>> ListAsync(string prefix);

/// <summary>
///
/// The user metadata stored with <paramref name="key"/>, plus some system values on some
/// providers.
/// </summary>
/// <param name="key"></param>
/// <returns></returns>
/// <remarks>
/// Only user metadata is returned everywhere. <c>Hash</c>, <c>ContentType</c> and
/// <c>ContentLength</c> are added by S3 and Azure; Oracle adds <c>Hash</c> and
/// <c>ContentType</c>; Google adds none. Read keys case-insensitively: S3 lowercases them.
/// A missing key throws, as for <see cref="OpenReadAsync"/>.
/// </remarks>
/// <param name="key">The object's key.</param>
Task<IReadOnlyDictionary<string, string>> GetMetadataAsync(string key);

/// <summary>
/// Deletes File.
/// Deletes <paramref name="key"/>.
/// </summary>
/// <param name="key"></param>
/// <returns>True if file is successfully deleted, false if not successfully deleted</returns>
/// <remarks>
/// The result for a key that doesn't exist depends on the provider: S3 and local return
/// <c>true</c>, Google and Oracle return <c>false</c>, Azure throws. Oracle also returns
/// <c>false</c> for any other failure. Check first if you need to know whether it existed.
/// </remarks>
/// <param name="key">The object's key.</param>
/// <returns><c>true</c> when the store accepted the delete; see remarks.</returns>
Task<bool> DeleteAsync(string key);



}
}
24 changes: 18 additions & 6 deletions SW.PrimitiveTypes/Contracts/CloudFiles/WriteFileSettings.cs
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
using System;
using System;
using System.Collections.Generic;
using System.Text;

namespace SW.PrimitiveTypes
{
/// <summary>What to write and how: passed to <see cref="ICloudFilesService.WriteAsync"/>,
/// <see cref="ICloudFilesService.WriteTextAsync"/> and <see cref="ICloudFilesService.OpenWrite"/>.</summary>
public class WriteFileSettings
{
public WriteFileSettings()
Expand All @@ -13,27 +15,37 @@ public WriteFileSettings()


/// <summary>
/// It consists of the path and filename
/// The object's key inside the bucket: path and file name, separated by <c>/</c>, with no leading
/// slash (<c>"invoices/2026/INV-1.pdf"</c>). Keys under <c>temp1/</c>, <c>temp7/</c>, <c>temp30/</c>
/// and <c>temp365/</c> are deleted by the store after that many days where the provider set up
/// its lifecycle rules (see <see cref="ICloudFilesLifecycle"/>).
/// </summary>
public string Key { get; set; }

/// <summary>
/// Content type of the file
/// The MIME type stored with the object. When null, most providers store
/// <c>application/octet-stream</c> (<c>text/plain</c> for text writes).
/// </summary>
public string ContentType { get; set; }

/// <summary>
/// Flag to set the file to be public viewed
/// Asks for the object to be publicly readable. Only the S3 provider acts on it, by sending the
/// <c>public-read</c> canned ACL, and stores without object ACLs ignore that. Azure, Google and
/// Oracle decide visibility per bucket or container. Don't rely on it for privacy either way:
/// configure the bucket.
/// </summary>
public bool Public { get; set; }

/// <summary>
/// Flag to close the input stream, default: true
/// Close the input stream once uploaded (default <c>true</c>). Only the S3 provider honours it;
/// the others leave the stream open, so dispose it yourself.
/// </summary>
public bool CloseInputStream { get; set; } = true;

/// <summary>
/// File metadata dictionary to be saved
/// User metadata to store with the object, read back by
/// <see cref="ICloudFilesService.GetMetadataAsync"/>. Ignored by the Google provider, and by the S3
/// provider on text writes. Use lowercase ASCII keys: S3 lowercases them.
/// </summary>
public IDictionary<string, string> Metadata { get; set; }
}
Expand Down
8 changes: 7 additions & 1 deletion SW.PrimitiveTypes/Contracts/CloudFiles/WriteWrapper.cs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
using System;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
Expand All @@ -7,6 +7,12 @@

namespace SW.PrimitiveTypes
{
/// <summary>
/// A streamed upload started by <see cref="ICloudFilesService.OpenWrite"/>: write the body to
/// <see cref="GetStreamAsync"/>, then call <see cref="CompleteRequestAsync"/>. Supported by the S3
/// provider only. Built on the obsolete <see cref="HttpWebRequest"/>; prefer
/// <see cref="ICloudFilesService.WriteAsync"/> in new code.
/// </summary>
public class WriteWrapper : IDisposable
{
private readonly HttpWebRequest httpWebRequest;
Expand Down
13 changes: 12 additions & 1 deletion SW.PrimitiveTypes/Patterns/ValueObjects/RemoteBlob.cs
Original file line number Diff line number Diff line change
@@ -1,14 +1,25 @@
using System;
using System;
using System.Collections.Generic;
using System.Text;

namespace SW.PrimitiveTypes
{
/// <summary>The result of writing an object to storage.</summary>
public class RemoteBlob
{
/// <summary>
/// A URL for the object. Depending on the provider it is the public URL, a signed URL valid for one
/// hour, or an unsigned URL that a private bucket refuses. Store the key, not this.
/// </summary>
public string Location { get; set; }
/// <summary>
/// Size in bytes, where the provider reports it (not every provider does, and the S3 provider
/// reports characters for text writes). An <c>int</c>, so it overflows above 2 GiB.
/// </summary>
public int Size { get; set; }
/// <summary>The content type that was requested for the object.</summary>
public string MimeType { get; set; }
/// <summary>The object's key (not set by every provider).</summary>
public string Name { get; set; }

public override string ToString()
Expand Down
Loading