Link to External Data is not responding

Gurkan · December 30, 2021, 11:55pm

Calc 7.2.4.1

Hello All,

Trying to get data from a web page. When I create a new sheet, and use the Sheet > Link to External Data and then enter the following link: “IBM | International Business Machines Corp. Stock Price & News - WSJ” into the URL of External Data Source and then hit Enter, nothing happens. No error messages, simply no nothing.

When I try the same with the following link “IBM Stock Price Today | IBM Live Ticker - Investing.com” it at least tells me there is nothing to return, as attached.

Might I be doing something wrong? Or is this something related to the webpage (wsj) I am trying to get data from and cannot find a way to do so?

Thanks in advance and kind regards,
Link to External Data

erAck · December 31, 2021, 12:55am

That first site is broken. It does not deliver anything useful if the user agent is not one it expects and accepts. Just try with curl -i and you’ll see some 403 loaded with theming. Hooray to CloudFront.
Calc maybe could display a message for such cases. If you created an enhancement bug (RFE, request for enhancement).

The second site delivers the page but data is only in div span elements that is not in a table layout Calc would process.

Gurkan · December 31, 2021, 7:25am

Thanks for your reply but the link works

Any browser does display this page successfully, as attached. Might there be a timeout parameter for loading external links in the calc? Maybe for this site, I need to increase that one?

erAck · December 31, 2021, 6:46pm

You did not understand. It works for browsers, but some other client applications get rejected. Though it’s not the underlying cause here, that’s yet something different, see below.
Again, try
curl -I https://www.wsj.com/market-data/quotes/IBM
to fetch the headers and you’ll see a 403 error response.

More detailed,

curl -H 'User-Agent: curl' -I https://www.wsj.com/market-data/quotes/IBM

gives HTTP/2 403

curl -H 'User-Agent: foo' -I https://www.wsj.com/market-data/quotes/IBM

gives HTTP/2 200

curl -H 'User-Agent: LibreOffice' -I https://www.wsj.com/market-data/quotes/IBM

gives HTTP/2 200

curl -H 'User-Agent: LibreOffice curl' -I https://www.wsj.com/market-data/quotes/IBM

gives HTTP/2 403

So in these examples they decide on the presence of the string curl in the User-Agent header whether to deliver the page or not. Already that is broken.

But worse, they refuse the OPTIONS request method completely.

curl -H 'User-Agent: LibreOffice' -X OPTIONS -I https://www.wsj.com/market-data/quotes/IBM

gives HTTP/2 500
And that hits here as it is what LibreOffice tries first to determine if the URI is request-able at all.
That site is broken.