No output for sites with https

[snipem]

Does not work for some websites like psprices.com or google.com (for testing):

When downloading the website with curl and uploading it to a private webspace I get "decoded_content" as output.

[vzvu3k6k]

I confirmed these two pages only contains <script type="text/javascript" charset="utf-8" src="/js/inspector.js"></script>.

• http://xpathfeed.com/frame?url=http://google.com/search?q=test
• http://xpathfeed.com/frame?url=http://psprices.com/region-us/index

I think one reason is that LWP::Protocol::https is not included in dependency.

http://google.com/search?q=test redirects me https://www.google.co.jp/search?q=test&gws_rd=ssl when I set Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) as User-Agent (as XPathFeed::UserAgent does).

http://psprices.com/ redirects me https://psprices.com/region-us/index.

Without LWP::Protocol::https, XPathFeed::UserAgent can't access URLs with https scheme.

[snipem]

Did you deactivate the SSL Support from LWP::Protocol::https on purpose?

[vzvu3k6k]

As I'm not the original author, I'm not sure... But probably it has been just forgotten.

I've pushed a commit to add LWP::Protocol::https (thanks, @onishi!), but I can't update XPathFeed.com. If you want to get xpathfeed with https support, the easiest way is to deploy XPathFeed for yourself.

[snipem]

I've added LWP::Protocol::https by myself to XPathFeed.pm with no positive outcome. Instead the request fails with the following output as for my $res = $self->_get($url); print($res->as_string);. Where $url is a HTTPS site.

500 Can't verify SSL peers without knowing which Certificate Authorities to trust
Content-Type: text/plain
Client-Date: Sun, 05 Apr 2015 16:16:15 GMT
Client-Warning: Internal response

Can't verify SSL peers without knowing which Certificate Authorities to trust

This problem can be fixed by either setting the PERL_LWP_SSL_CA_FILE
envirionment variable or by installing the Mozilla::CA module.

To disable verification of SSL peers set the PERL_LWP_SSL_VERIFY_HOSTNAME
envirionment variable to 0.  If you do this you can't be sure that you
communicate with the expected peer.

[snipem]

I pushed a solution in #6

[vzvu3k6k]

Judging from the error message, the reason the request failed is that Mozilla::CA is not installed and CA files are not given.

If hostname verification is requested by LWP::UserAgent's ssl_opts, and neither SSL_ca_file nor SSL_ca_path is set, then SSL_ca_file is implied to be the one provided by Mozilla::CA. If the Mozilla::CA module isn't available SSL requests will fail. Either install this module, set up an alternative SSL_ca_file or disable hostname verification.
LWP::Protocol::https - Provide https support for LWP::UserAgent - metacpan.org

If you install dependencies by cpanm --installdeps . (to install cpanm, see https://cpanmin.us) or carton install, Mozilla::CA will be installed and a request to https://psprices.com/ will success. It is safer than not verifying hostnames.

(Now I'm thinking of adding instructions to install this app to local environments to README.)

[snipem]

I had success fixing this by using Mozilla::CA with #7. Works fine and I second your security concerns.

(For me I've installed cpanm and Plack via cpanm Task::Plack. I took me some time to discover the installdeps option of cpanm but after changing the folder /var/log/app/access_log in app.psi I got the application running by using plackup app.psgi. It was my first time after years coding Perl and using an OSGI framework in the context. So providing a small setup guide could help in the process. I've also missed logging outputs from the application itself. Currently there is only an access log, which is not very helpful when debugging backend problems like the one with HTTPS.)