Add additional method to parse raw call data bytes into an unsigned payload so it can be signed and submitted later on

[paulormart]

Hi @jsdw,

Opening this issue to share what I'm currently doing since I haven't found a default option on the latest version of subxt and I think it might be good to have something similar available.

I often find it useful to keep the raw bytes exported from here tx().call_data(&proxy_call)? around, and only sign and submit on a second stage. I couldn't find a method on the latest version that could easily decode these bytes, the option tx().from_bytes(encoded) requires the encoded data to be already signed. I have looked to the other methods on how to create partial transaction, but they don't seem to apply here.

Do you have any suggestions?

Sharing the steps and code blocks to demonstrate my case:

1. Export call data:

pub fn wrap_call_into_proxy(
    api: &ClientAtBlock<SubstrateConfig, OnlineClientAtBlockImpl<SubstrateConfig>>,
    call: RuntimeCall,
    proxied_account: &AccountId32,
) -> Result<Vec<u8>, Error> {
    let proxy_call = node_runtime::tx().proxy().proxy(
        (*proxied_account).into(),
        Some(ProxyType::NonTransfer),
        call,
    );

    let payload = api.tx().call_data(&proxy_call).boxed()?;

    Ok(payload)
}

2. With the chain Metadata available, decode the original call data from previous method wrap_call_into_proxy, sign it with a valid Keypair and submit:

    async fn sign_and_submit_call_data(
        &self,
        api: &OnlineClient<SubstrateConfig>,
        proxy_signer: &Keypair,
        call_data: &[u8],
    ) -> Result<Response, Error> {
        let at_block = api.at_current_block().await.boxed()?;
        let metadata = at_block.metadata();
        let payload = RawPayload::from_bytes(&metadata, call_data).boxed()?;
        let response = api
            .tx()
            .await
            .boxed()?
            .sign_and_submit_then_watch_default(&payload, proxy_signer)
            .await
            .boxed()?;

        Ok(Response::transaction_submitted(response))
    }

3. RawPayload implementation:

use scale_encode::{EncodeAsFields, Error as EncodeError, FieldIter, TypeResolver};
use subxt::{
    transactions::{Payload, ValidationDetails},
    Metadata,
};

pub type Bytes = Vec<u8>;

pub struct RawPayload {
    pallet_name: String,
    call_name: String,
    field_bytes: RawFields,
}

pub struct RawFields(Bytes);

impl EncodeAsFields for RawFields {
    fn encode_as_fields_to<R: TypeResolver>(
        &self,
        _fields: &mut dyn FieldIter<'_, R::TypeId>,
        _types: &R,
        out: &mut Bytes,
    ) -> Result<(), EncodeError> {
        out.extend_from_slice(&self.0);
        Ok(())
    }
}

impl Payload for RawPayload {
    type CallData = RawFields;

    fn pallet_name(&self) -> &str {
        &self.pallet_name
    }
    fn call_name(&self) -> &str {
        &self.call_name
    }
    fn call_data(&self) -> &RawFields {
        &self.field_bytes
    }
    fn validation_details(&self) -> Option<ValidationDetails<'_>> {
        None
    }
}

impl RawPayload {
    pub fn from_bytes(metadata: &Metadata, bytes: &[u8]) -> Result<Self, Error> {
        let pallet = metadata
            .pallet_by_call_index(bytes[0])
            .ok_or(Error::PalletNotFound)?;
        let call_variant = pallet
            .call_variant_by_index(bytes[1])
            .ok_or(Error::CallNotFound)?;

        Ok(Self {
            pallet_name: pallet.name().to_string(),
            call_name: call_variant.name.clone(),
            field_bytes: RawFields(bytes[2..].to_vec()),
        })
    }
}

#[derive(thiserror::Error, Debug)]
pub enum Error {
    #[error("Pallet not found")]
    PalletNotFound,
    #[error("Call not found")]
    CallNotFound,
}

[jsdw]

Mmm, something like your RawPayload is currently the only way to do this. I thnk this is actually the best approach for solving the problem of turning raw call data bytes into something Subxt can use, because once you have your RawPayload (which is just the call data and target basically) you can then decide what sort of transaction to build from it (unsigned, signed, v4, v5, or just use the defaults) using all of the existing Subxt methods.

If we want to move some of this into Subxt, the question is then; why do you want to store the call data bytes to use later? is it just because you want to store something that represents the transaction, and the current Payload impls don't work for that (since they are not object safe and each call is a different type)? Or do you want to do something with the actual bytes?

1. If you just want to be able to store it, then we could add something like client_at_block.tx().storable(call: &Call) -> Result<RawPayload> for instance, which takes some arbitrary Payload impl and turns it into something like your RawPayload type, which can easily be stored away and used later.
2. If you actually want to store the call data bytes themselves, then your current approach is basically the way we'd go I think, possibly with adding a helper like client_at_block.tx().payload_from_call_data_bytes(bytes: &[u8]) -> Result<RawPayload>. This has the advantage of being more generic since it's just working with bytes, but also requires the extra step of getting the call data bytes, and is more prone to being used wrong (ie giving the wrong bytes).

I'd personally lean slightly in favour of (1) because you can't really misuse it; method name tbc though!

What do you think?

[paulormart]

Or do you want to do something with the actual bytes?

Yes, on my current use case, I don't really need to store the raw call data bytes, but I need to keep them around to be able to hex encode the bytes for logging, present them to the user or just share a direct link to PJS here for example where it can be decoded.

As for the output of Result<RawPayload>, I don't really need it, is just an intermediary step so it can be used in sign_and_submit_then_watch_default because at that stage, I already have all the information needed to build (raw call_data bytes), sign (Keypair) and submit a transaction.

From your suggestion I see my self using client_at_block.tx().payload_from_call_data_bytes(bytes: &[u8]) -> Result<RawPayload> rather than client_at_block.tx().storable(call: &Call) -> Result<RawPayload>.

From a library user perspective, I think this step could be abstracted even further and built in, Metadata is also available internally at this stage, so something like this client_at_block.tx().sign_and_submit_then_watch_raw(bytes: &[u8], signer: &S) -> Result<TransactionProgress<T, Client>, ExtrinsicError> could work just nicely.

[paulormart]

Just a side note, I have opened this #2196 so that the RawPayload as shown earlier could work (I'm currently using a fork)

[jsdw]

@paulormart thank you for the feedback!

From a library user perspective, I think this step could be abstracted even further and built in, Metadata is also available internally at this stage, so something like this client_at_block.tx().sign_and_submit_then_watch_raw(bytes: &[u8], signer: &S) -> Result<TransactionProgress<T, Client>, ExtrinsicError> could work just nicely.

My issue with this is that there are many ways to use the call data bytes; constructing a v4 or v5 unsigned transaction, a v4 or v5 signed transaction, letting Subxt pick, creating a signable payload to sign etc.

Given what you said I think I'd lean towards something like:

• Adding RawPayload, which is a simple type which is easy to clone/store etc and impls Payload. Expose a payload.call_data() -> &[u8] maybe for convenience (though can use the existing method to get the call data bytes too).
• Adding something like client_at_block.tx().to_raw_payload(call: &Call) -> Result<RawPayload, Error>.
• And then consider also something to take the bytes directly like client_at_block.tx().to_raw_payload_from_call_data_bytes() but that is less "safe" (easy to pass the wrong bytes) and more inconvenient naming so slightly less keen on this one :)

Then you can convert any Payload impl into a RawPayload type which you can store easily as well as get the call data bytes from to print or whatever. RawPayload can also be

[paulormart]

My issue with this is that there are many ways to use the call data bytes; constructing a v4 or v5 unsigned transaction, a v4 or v5 signed transaction, letting Subxt pick, creating a signable payload to sign etc.

I agree.

But let me revisit the initial pain point and the reason why RawPayload was created. Imagine the following pseudo code below:

        let at_block = api.at_current_block().await?;
        let inner_call = RuntimeCall::System(SystemCall::remark_with_event {
            remark: "something".into(),
        });
        let proxy_call = node_runtime::tx().proxy().proxy(
            (*proxied_account).into(),
            Some(ProxyType::Staking),
            inner_call,
        );
        let bytes = at_block.tx().call_data(&proxy_call)?;
        
        
        //  --> Do wthatever with those bytes <--
        
        
        //  Question: how to easily bring those bytes back in again, sign and submit?
        
        // ------------------------------------------------>
        // The following doesn't work, the extrinsic cannot be decoded.
        // `from_bytes` expect a previous signed transaction

        let call_data_vec = bytes.to_vec();
        let b = at_block.tx().from_bytes(call_data_vec);
        let response = b.submit_and_watch().await?;

        // <------------------------------------------------


        // ------------------------------------------------>
        // What is currently possible with the `RawPayload` trait is the following

        let metadata = at_block.metadata();
        let payload = RawPayload::from_bytes(&metadata, bytes)?;
        let response = at_block
            .tx()
            .await?
            .sign_and_submit_then_watch_default(&payload, proxy_signer)
            .await?;

        // <------------------------------------------------
         
        // ------------------------------------------------>
        // Possible solution on Subxt, something that could easily flow and might work?

        let payload = at_block.tx().from_call_data_bytes(&bytes);
        let response = at_block
            .tx()
            .await?
            .sign_and_submit_then_watch_default(&payload, proxy_signer)
            .await?;

        // <------------------------------------------------

[paulormart]

Adding RawPayload, which is a simple type which is easy to clone/store etc and impls Payload. Expose a payload.call_data() -> &[u8] maybe for convenience (though can use the existing method to get the call data bytes too).
Adding something like client_at_block.tx().to_raw_payload(call: &Call) -> Result<RawPayload, Error>.

Do you mean we will get the following:

        let at_block = api.at_current_block().await?;
        let inner_call = RuntimeCall::System(SystemCall::remark_with_event {
            remark: "something".into(),
        });
        let proxy_call = node_runtime::tx().proxy().proxy(
            (*proxied_account).into(),
            Some(ProxyType::Staking),
            inner_call,
        );
        let raw_payload = at_block.tx().to_raw_payload(&proxy_call)?;
        let bytes = raw_payload.call_data();

       //  --> Do wthatever with those bytes <--

The pain point still remains, how to easily decode those bytes into Subxt using the defaults? Would you keep it like this:

        let metadata = at_block.metadata();
        let raw_payload = RawPayload::from_bytes(&metadata, bytes)?;
        let response = at_block
            .tx()
            .await?
            .sign_and_submit_then_watch_default(&raw_payload, proxy_signer)
            .await?;

[jsdw]

My thinking was that you could potentially store the RawPayloads instead of storing the raw call data bytes; the RawPayloads just being a better typed "opaque" reference to some call you'd like to make. You can still access and print the bytes etc as you wish, and when ready you pass the RawPayload to subxt to construct whatever you'd like.

If you'd rather store the bytes, then I would also add client_at_block.tx().to_raw_payload_from_call_data_bytes() which is a bit uglier but lets you just pass the call data bytes in directly.

So the "easy" API lets you do (1) below, and if you actually want to work more directly with the call data bytes then we could also add (2):

// 1. Using/storing RawPayloads:
//

let at_block = api.at_current_block().await?;
let proxy_call = node_runtime::tx().proxy().proxy(...);
let raw_payload = at_block.tx().to_raw_payload(&proxy_call)?;

//  --> Do wthatever with the raw payload, storing it away etc,
//  --> or getting `raw_payload.call_data()` or pallet name or whatever from it.
//  --> and then bring it back into subxt:

let response = at_block
    .tx()
    .await?
    .sign_and_submit_then_watch_default(&raw_payload, proxy_signer)
    .await?;

// 2. Using/storing raw call data bytes
//

let at_block = api.at_current_block().await?;
let proxy_call = node_runtime::tx().proxy().proxy(...);
let call_data_bytes = at_block.tx().call_data(&proxy_call)?;

//  --> Do wthatever with the call data bytes, storing them away etc.
//  --> and then bring it back into subxt:

let raw_payload = at_block.tx().to_raw_payload_from_call_data_bytes(&call_data_bytes)?;
let response = at_block
    .tx()
    .await?
    .sign_and_submit_then_watch_default(&raw_payload, proxy_signer)
    .await?;

[paulormart]

Ok sounds good. The issue was raised just to tackle point (2) and there is not really a need to keep the RawPayload unless strictly necessary (that is my currently workaround). If you add a new method to return a StaticPayload or DynamicPayload will work just fine for me, and no need for inconvenient method names, so renaming to_raw_payload_from_call_data_bytes to to_payload_from_call_data_bytes or just from_call_data_bytes .. up to you, naming is hard :)

// 2. Using/storing raw call data bytes
//

let at_block = api.at_current_block().await?;
let proxy_call = node_runtime::tx().proxy().proxy(...);
let call_data_bytes = at_block.tx().call_data(&proxy_call)?;

//  --> Do wthatever with the call data bytes, storing them away etc.
//  --> and then bring it back into subxt:

let payload = at_block.tx().from_call_data_bytes(&call_data_bytes)?;
let response = at_block
    .tx()
    .await?
    .sign_and_submit_then_watch_default(&payload, proxy_signer)
    .await?;