Adds correct UTF-8 encoding to Net::BER::BerIdentifiedString

[andibachmann]

Dear Net-LDAP maintainers

Currently, net-ldap returns values containing umlauts (e.g. 'Müller') with an encoding as 'ASCII-8BIT'.
This is wrong. LDAP in version 3 should encode all data in 'UTF-8' and therefore when
casting returned ''string'' data into a Net::BER::BerIdentifiedString the encoding should be 'UTF-8'.

The current code tries to '#encode('UTF-8')' which results in an
Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8. This error is trapped
and the binary string is returned instead.

If we assume that the data coming from our LDAP/AD-Server is correctly encoded in 'UTF-8', there is
no need to use #encode. The only thing is to set correctly the encoding of the (string) data, i.e. to use force_encoding.

Example:

bin_str = "Müller".b
p [:bin_str, bin_str.encoding, bin_str.bytes, bin_str]
# => [:bin_str, #<Encoding:ASCII-8BIT>, [77, 195, 188, 108, 108, 101, 114], "M\xC3\xBCller"]

# Now try to 'encode' it:
bin_str.encode('UTF-8')
# =>Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8

# Better:
bin_str.force_encoding('UTF-8')
p [:bin_str, bin_str.encoding, bin_str.bytes, bin_str]
# => [:bin_str, #<Encoding:UTF-8>, [77, 195, 188, 108, 108, 101, 114], "Müller"]

I must admit that I have only checked the code (and added one test case) for Umlauts (and characters more or less covered by ISO-8859-1). I have no idea how to test against
Korean, Japanese, Russian, or Chinese encodings.

Nevertheless, I am pretty sure that the current code is bogus for any 'non-ASCII' characters.

Please let me know, if you need further details and I'd be happy to help you out.

regards
andi

[jch]

Thanks for fixing this.

[jch]

@andibachmann thank you for taking the time to open this pull request and including helpful comments. I think this should be fine and compatible with the past changes in #212 because we're changing the internal string object. Is there anything else you would like me to review specifically before I merge?

[andibachmann]

@jch, besides special Encodings like Japanese, Chinese and Korean, I'm pretty sure the code is OK.
thanks for the review!

[satoryu]

besides special Encodings like Japanese, Chinese and Korean, I'm pretty sure the code is OK.

Yes, I have a monkey patch as well as your changes does :)

[jch]

Thanks all for chiming in. Merging.