Calling Python urllib functions in Basic

English
,
#1

Microsoft Windows Version 22H2 (Win10)

Version: 25.2.5.2 (X86_64) / LibreOffice Community
Build ID: 03d19516eb2e1dd5d4ccd751a0d6f35f35e08022
CPU threads: 2; OS: Windows 10 X86_64 (10.0 build 19045); UI render: Skia/Raster; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: threaded

I’m setting up an app that makes http HEAD requests. The http client that comes with Windows 10 and the http client in the OAuth2OOo extension are blocked or banned at some URLs. Curl gets around these blocks & bans somehow, but has to be called in Shell(), which creates other problems. I’ve tried changing the User-Agent header in the Win & OAuth2OOo clients to “curl/8.13.0” but that doesn’t create any improvement.

So I’m trying to test the Python urllib library. According to the comments in the request.py file, “The simplest way to use this module is to call the urlopen function, which accepts a string containing a URL or a Request object…” Since I know nothing about Python, I’m interested in simple things.

So…

Sub TestUrllibUrlopenFunction
	Dim objScriptProvider	As Object
	Dim objRequest			As Object
	Dim objResponse			As Object
	Dim strUri				As String
	
	objScriptProvider = ThisComponent.getScriptProvider
	strUri = "vnd.sun.star.script:urllib/request.py$urlopen?language=Python&location=user"
	objRequest = objScriptProvider.getScript(strUri)
	objResponse = objRequest.invoke(array("https://atun-rsia.org/resources"), array(), array())
End Sub

This results in

BASIC runtime error.
An exception occurred 
Type: com.sun.star.uno.RuntimeException
Message: Error during invoking function urlopen in module file:///C:/Users/user/AppData/Roaming/LibreOffice/4/user/Scripts/python/urllib/request.py (<class 'AttributeError'>: module 'http.client' has no attribute '_create_https_context'
  File "C:\Program Files\LibreOffice\program\pythonscript.py", line 921, in invoke
    ret = self.func( *args )
  File "C:\Users\user\AppData\Roaming\LibreOffice\4\user\Scripts\python\urllib\request.py", line 184, in urlopen
    _opener = opener = build_opener()
  File "C:\Users\user\AppData\Roaming\LibreOffice\4\user\Scripts\python\urllib\request.py", line 565, in build_opener
    opener.add_handler(klass())
  File "C:\Users\user\AppData\Roaming\LibreOffice\4\user\Scripts\python\urllib\request.py", line 1363, in __init__
    context = http.client._create_https_context(http_version)
).

OK, try the other simple thing.

Sub TestUrllibRequestObject
	Dim objScriptProvider	As Object
	Dim objRequest			As Object
	Dim objResponse			As Object
	Dim strHeaders			As String
	Dim strUri				As String
	
	objScriptProvider = ThisComponent.getScriptProvider
	strUri = "vnd.sun.star.script:urllib/request.py$Request?language=Python&location=user"
	objRequest = objScriptProvider.getScript(strUri)
	strHeaders = "If-Modified-Since: Mon, 01 Sep 2025 00:00:00 GMT"
	objResponse = objRequest.invoke(array("https://atun-rsia.org/resources"), _
		array("headers={" & strHeaders & "}"), array("method=HEAD"))
End Sub

Result:

BASIC runtime error.
An exception occurred 
Type: com.sun.star.uno.RuntimeException
Message: Couldn't convert <ooo_script_framework.Request object at 0x000002D5C93CAEF0> to a UNO type; caught exception: <class 'AttributeError'>: 'Request' object has no attribute 'getTypes', traceback follows
no traceback available
.

What am I doing wrong?

#3
  • the whole urllib is part of the the python-standard-libraries, why do you try to copy&paste the whole folder-tree into …/Scripts/python?
  • why do you »sophisticate« your workflow with basic -calls via ScriptProvider.invoke(…) ?
#2

sounds like, YOUR »request.py« try to import http.client… … but there is some »http.py« nearby YOUR request.py which shadows the original which should be somewhere in …/lib/python…/http/ ? sorry, in your case propably …/libreoffice/program/pythonx.y/http/
So find the wrong »http.py« and remove or rename it.

#4

Partial solution: wrong version. The Python core that is part of LO is 3.10. I installed the 3.13 version of urllib. When I installed 3.10, TestUrllibUrlopenFunction stopped producing errors. So that part is fixed. But TestUrllibRequestObject is still producing the same error.

It’s 4:30 AM. I’m fine but my computer is getting sleepy.

#5

I propose a slightly more complex training setup.
Cell B3 of the attached file contains the request URI.
Cell B4 contains text in JSON format.
When you click the Run Macro button, a macro is launched that performs a POST request of the “application/json” type and places the result (a JSON format string) in cell B5.
Basic:

Sub HttpPost()
  Dim oSheet as Object, pyScript As Object
  Dim retval As String, url As String, jsonData As String
  pyScript = ThisComponent.getScriptProvider().getScript("vnd.sun.star.script:" & _
             "utils.py$http_post?language=Python&location=document") 
  oSheet = ThisComponent.Sheets(0)
  url = oSheet.getCellbyPosition(1, 2).String
  jsonData = oSheet.getCellbyPosition(1, 3).String            
  
  retval=pyScript.invoke(Array(url, jsonData), Array(), Array())
  oSheet.getCellbyPosition(1, 4).String = retval
End Sub

Python:

# coding: utf-8
import urllib.request
import urllib.parse
import json

def http_post(url, str_json):

    try:
        data_json = json.loads(str_json)
    except Exception as e:
        return f"** Error: {e}"
    data_encoded = json.dumps(data_json).encode("utf-8")
    req = urllib.request.Request(url, data=data_encoded, method="POST")
    req.add_header("Content-Type", "application/json")

    try:
        with urllib.request.urlopen(req) as response:
            return response.read().decode("utf-8")
    except Exception as e:
            return f"** Error: {e}"

HttpPostJson.ods (16.3 KB)

1 Like
#6

why do you so?

1 Like
#7

Fixed the error (confused strings and JSON objects), restored the message.

#9

@Sokol92 Is this utils.py script documented anywhere?

#10

yes, its just the python-snippet from @sokol92 above! its also embedded in his example:

HttpPostJson.ods (16.3 KB)

#11

@Sokol92 I didn’t realize that the python code was the function called in the Basic code. So I pulled the utils file out of the .ods file & …oh. OK, got it.

I rewrote your python function thusly:

import urllib.request

def http_head(url, mod_date):

    req = urllib.request.Request(url, method="HEAD")
    req.add_header('If-Modified-Since', mod_date)

    try:
        with urllib.request.urlopen(req) as response:
            return response.read()
    except Exception as e:
            return f"** Error: {e}"

I rewrote my Basic sub thusly:

Sub TestUrllibUrlopenFunction
	Dim objScriptProvider	As Object
	Dim objRequest			As Object
	Dim strUri				As String
	Dim strResponse			As String
	
	objScriptProvider = ThisComponent.getScriptProvider
	strUri = "vnd.sun.star.script:httpHead.py$http_head?language=Python&location=user"
	objRequest = objScriptProvider.getScript( strUri )
	strResponse = objRequest.invoke( array( "https://wiki.documentfoundation.org/Macros/Python_Guide/Introduction", "Mon, 01 Sep 2025 00:00:00 GMT" ), array(), array() )
MsgBox strResponse
End Sub

It runs w/out errors, but it isn’t returning anything. I should be getting a “304 Not Modified” status. What am I missing?

Edit: Never mind. I changed a couple of double quotes to single quotes & got the 304. Not bad for hacking in notepad.
I’m crediting my own post w/ the solution, because that’s the post w/ the solution, but the the credit should go to Sokol92 for supplying a python function that someone who knows nothing about python could easily modify.

2 Likes
#12

For anybody looking to make http HEAD requests, the Python script I posted above contains an error. It calls the response object’s read method. What is wanted are the response object’s status, reason, and headers properties. Plus you have to search through the headers property for the specific headers you’re looking for. Here’s the updated code, heavily annotated:

import urllib.request
from re import search

# Function for sending http HEAD requests & using the response headers to determine
# if a web page has changed.

def http_head( url, mod_date="none", user_agent="none" ):

	# Setting up a request module Request object.
	req = urllib.request.Request( url, method='HEAD' )

	# If mod_date contains a value other than "none" then add the If-Modified-Since header.
	if mod_date:
		req.add_header( 'If-Modified-Since', mod_date )

	# If user_agent contains a value other than "none" then add the User-Agent header.
	if user_agent:
		req.add_header( 'User-Agent', user_agent )

	try:
		# Run the urlopen function w/ the Request object as an argument.
		# Output goes to the response data structure.
		with urllib.request.urlopen( req ) as response:

			# response.headers returns a list. Convert it to a string.
			all_headers = str(response.headers)

			# search() returns a Match object.
			# "(?i)" = case insensitive search.
			header = search( "(?i)Last-Modified:.+", all_headers )

			# If there is no Last-Modified header, then search returned a None object.
			# Test for that w/ "is None".
			if header is None:
				header = search( "(?i)Content-Length:.+", all_headers )

			# If there is no Content-Length header, then search returned a None object.
			if header is None:
				header = search( "(?i)Etag:.+", all_headers )

			# status is an integer. reason is a string. Convert status to a
			# string & join it to reason, seperated by a space.
			result = str( response.status ) + " " + response.reason

			# Add header to result only if it contains a Match object & not a None object.
			if header is not None:
				# Join result to header, seperated by a new line character.
				# The Match object's group() method returns the substring that matched.
				result = result + "\n" + header.group()

			return result

	except Exception as e:
		return f"** Error: {e}"

If I put a date in the If-Modified-Since header, some web sites will tell me whether the web page has been modified since that date. Most web sites won’t do that, but it’s helpful when they do.
Some websites block Python http clients, so I pass in whatever version of curl is on my computer & the code sets the User-Agent header to that. Sometimes that gets around the block & sometimes it doesn’t.

Impressum (Legal Info) | Datenschutzerklärung (Privacy Policy)   Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.