As discovered in tdf#148662, @VkPatidar used some “3D .prt” file format (likely Unigraphics / Siemens NX), and renamed the files to XLS and XLSX, then tried to open these files and/or convert to PDF. Without describing the final result (which likely was garbage), the question was basically “why there is a difference, when the same garbage (from the program point of view) is processed without errors, when using one extension, but gives errors and is not processed using another”.
The technical reason it that there is a special condition for XLS extension in the code, caused by wide use of CSV files renamed to have XLS extension: every file with XLS extension, after usual detection fails, is opened using CSV import filter. Since CSV is a plain text format, it may open any file, just the result would be garbage.
On the other hand, there was no such abuse of XLSX extension, hence LibreOffice does not try to open such files as CSV. Usual detection fails (compound file is detected, but it’s not XLS), no CSV fallback happens, and the error is emitted.
Anyway, the behavior is OK - as @Regina told, “Why do you expect that it works?”. This is clear “garbage in, garbage out” case.