Add support for MarshalAs(UnmanagedType.BSTR) on UNIX systems

[odhanson]

Although Marshal.StringToBStr et al. works fine on Mac. The following code

delegate void Callback([MarshalAs(UnmanagedType.BStr)]string s);

        [DllImport("libTestDynLib.dylib")]
        static extern void MyTest([MarshalAs(UnmanagedType.BStr)]string s, Callback c);

        static void Main(string[] args)
        {
            MyTest("This is a test", s =>
            {
                Console.WriteLine(s);
            });
        }

Is failing on Mac:
System.Runtime.InteropServices.MarshalDirectiveException: "Cannot marshal 'parameter dotnet/coreclr#1': Invalid managed/unmanaged type combination (String parameters and return types must be paired with LPStr, LPWStr, or LPTStr)."
at TestBSTR.Program.MyTest(String s, Callback c)\n at TestBSTR.Program.Main(String[] args) in /Users/odhanson/Projects/MyGuidTest/TestBSTR/Program.cs:15

Seems like (for looking at the code) this has only been implemented for windows (as part of COM support).

We are currently in the process of porting a FX application to Core, that has extensive interop with existing native libraries (that are cross platform). The BSTR is used extensively and we would definitely need support for this type of marshaling.

[odhanson]

@jkotas , @danmosemsft , @richlander

[luqunl]

@odhanson, do you also need Ansi BSTR support?

[odhanson]

@luqunl we only use UnmanagedType.BSTR in our project

[AaronRobinsonMSFT]

@odhanson What @luqunl is asking is, do you use ANSI characters (8-bit) in your BSTRs or do you use wchar_t (16-bit/32-bit) characters only? I guess a better way of asking this is, what encoding(s) do you use in you BSTRs?

[odhanson]

@AaronRobinsonMSFT I got what he meant. But I assumed that all UnmanagedType.BSTR means wchar_t and that UnmanagedType.AnsiBStr means ANSI 8bit. Am I wrong ?

Anyhow we are using wchar_t

[AaronRobinsonMSFT]

Doh. I completely forgot about the ‘AnsiBStr’ enum value, you are absolutely right. Sorry about the noise. 🙂

[odhanson]

@luqunl any estimate to when this might be incorporated into 2.1 ? We hade a workaround using our own custom marshaller for BSTR but now we have found out that the custom marshaller doesn't work for structures that contain BSTR strings. So having this change incorporated will solve us a lot of time

[luqunl]

@odhanson , the product part is done and the testing part should be finished in one or two weeks

[danmoseley]

Reactivating for 2.2 port. (When buddy tested)

[danmoseley]

Was ported