The conversion itself is fairly simple in Python:
def float_to_hex(f):
return hex(struct.unpack('<I', struct.pack('<f', f))[0])
def double_to_hex(f):
return hex(struct.unpack('<Q', struct.pack('<d', f))[0])
(taken from python - How to convert a float into hex - Stack Overflow).
You’d need to either create a Calc Add-In extension of that, or glue Python calls into a BASIC user defined function (UDF) callable from Calc. You might get some ideas from Basic to Python.
See also Macros/Python Guide/Calling Python from Basic - The Document Foundation Wiki.