Base - Export data of a stored query in a csv file within a macro

Hello,

I would like to export data of a stored query to a csv (or any character delimited) file. I know how to do by graphically copy or drag and drop the query content to a Calc spreadsheet and then save it in csv.
But what I need is to handle all these actions within a macro, that only requires from the end-user to just click on a button in the form.

I’ve note found how to do in my searches (but may have not searched well…?), and would need some advices to guide me.

Would this need for example to use the SaveDataToFile(FilePath as String, DataList()) LibreOffice function (Tools > UCB)? But then, how to get the query data (several rows and several columns) matching the DataList() type?

Note: I’m using an embedded Firebird-LibreOffice database, on Debian 10, with LibreOffice 7.0.1.2 (but need that the macro works well in a Windows environment).

Here is an example file with table, query, form (with button) prepared, but with empty macro, if it is useful: export_csv.odb

Thanks in advance for any help, and please ask if any more detail required!


[EDIT] - 2021-01-24

A potential solution would be to use a SQL query, and LibreOffice open and write function.

As an example, if it can be useful for some (even if I’m not sure this code is the best code than can be found):

Sub ExportCsv
GlobalScope.BasicLibraries.loadLibrary("Tools")
Dim iCount As Integer
Dim sValue As String
Dim oDoc as Object
Dim sUrl_Source as String
Dim sFolder as String

' Initialize the file (??)
iCount = Freefile

' Get the odb document
oDoc = ThisComponent.Parent

' Get the url (folder + filename) of the current database file
sUrl_Source = oDoc.URL

' Extract the folder of the current database file
sFolder = DirectoryNameoutofPath(ThisComponent.Parent.getUrl(),"/")
sFolder = ConvertFromUrl(sFolder)

' Open the text file
Open sFolder & "/test.txt" For Output As iCount

' Specific SQL handling declaration and preparation
Dim oDatasource as Object
Dim oConnection as Object
Dim oSQL_Command as Object
' Activate the SQL connection
oDatasource = thisDatabaseDocument.CurrentController
If Not (oDatasource.isConnected()) Then
	oDatasource.connect()
End if
oConnection = oDatasource.ActiveConnection()
oSQL_Command = oConnection.createStatement()

' Check if file already indexed in the application	
stSql = "SELECT * FROM ""Query1"""
oResult = oSQL_Command.executeQuery(stSql)
While oResult.next
	sValue = oResult.getString(1) & "," & oResult.getString(2)
	Write #iCount,sValue
Wend

Close #iCount
End Sub

Inspired from write page and this thread.

See as well for eg. the attached file

However, on my case (not exactly the same as the attached file), I have some cell in my table where there are more than 1 line (I deliberately input newline characters). In those case, only the first line in the cell is reproduce during the write process.

Of course a function on type “write2csv” could be very useful, but at least, this can do some part of the job for some, hopefully.


[EDIT] - 2021-02-02

I’ve been through the different options and files submitted below in the answer by Ratslinger.

Eventually, I wrote a macro who doesn’t need to “go out” of the user form within the ODB (ie. it does not need for the user to open a specific ODS file but s/he stays within the Base application), nor to pre-create an ODS file, nor attach macro to it.
It uses in addition the macro and form built by Ratslinger of the below QueryToCSV.odb file below which allow to load the embedded queries of the ODB file in a list box.

In the following is what the updated macro does:

  • get the name of the selected query in the form list box.
  • get the name of the ODB file and check if registered in LibreOffice using this post (if not, the sub is ended and the user explained how to register it)
  • load a new blank ODS file (source post, create a range (source doc here and import the query result in it
  • export the file with arguments for CSV saving
  • close (ie. delete because it was not saved) the ODS file

Here is the single file: QueryToCSV.odb
The only requirement from the user is to register the file as a database source in LibreOffice.

I’ve been testing it on my computer (Debian 10 - LibreOffice 7.0.4.2 - english), and on a virtual machine (Mageia 7 - LibreOffice 6.4.7.2 - french).

In addition, I could fix the mentioned problem of the printing of the first line only of multilines values (which is not in the example files, but on a personal working file).
These ones are originally computed thanks to a LIST() Firebird SQL function, using an ASCII_CHAR(13) (newline) separator.
It could be solved by converting the LIST() result in a VARCHAR() type, eg:

CAST(LIST("my_table"."my_field", ASCII_CHAR(13)) AS VARCHAR(400)) AS "my_modified_field"

(400 has been taken arbitrarily).

Hoping this can be useful for someones.


[EDIT] - 2021-02-03

Following what Ratslinger spotted out in his answer comment, here is the updated file: QueryToCSV.odb.

In order to deal with the registration issue (ie. base checking on URL matching better than registered name), I’ve attached a function (inspired by answer here), which is called within the main macro and simply produces as a result either a blank string if the file is not registered (then shows the message to explain how to register, and ends sub), or the name the ODB file is registered with (this may be different from the ODB file name, indeed).

Tested with registration, no registration, and registration with a different name from the ODB file. All worked.

@greb,

Have working model in Ubuntu 20 with LO v7.0.4.2 but it still uses Calc to process (user need not touch). Base file needs to be registered. Is this of interest?

@Ratslinger,
Thanks for offering. Indeed, I would have preferred a “direct” way, but still, this can be useful.
I will just need to prepare a clear procedure document for the end-user (but there will be mainly only one single person in charge of this, so it could be an option).

@greb,

Unfortunately I may have been too quick with my offer. Because of your concern, I tested today in Win 10 and there was a problem. Have not been able to yet resolve and will continue looking for an answer to your situation. Do not want to publish something which doesn’t completely work. If not working in Win 10 the problem may also be hiding in Linux.

@Ratslinger,
It’s ok, no problem. Anyway, thanks for the updated information.
Eventually, I will opt for the “easy” graphical solution (copy query, paste in a Calc spreadsheet and save as CSV), which is actually not so tricky to explain in the application documentation.

Hi,

Not familiar with Firebird, however would assume the db supports text tables.
If it does you might want to try the following

save your query as a View
then embed Firebird flavor SQL of HSQL code as below in a macro

adapted from [Solved] Exporting Queries from Base to CSV (View topic) • Apache OpenOffice Community Forum

Create Text Table "vReport_txt" AS  (Select * FROM "tblOnline") WITH NO DATA;
SET TABLE "vReport_txt" SOURCE "vReport_txt.csv";  --  could also add other specs if needed
INSERT INTO "vReport_txt" (Select * From "tblOnline" );
SET TABLE  "vReport_txt" SOURCE OFF;
DROP TABLE "vReport_txt" IF EXISTS;

Tested with WIN 10 LO6.4.7 HSQL 2.51 (split db) In a split db HSQL saves the csv in the database folder -
you may need to check out the firebird doco.

@gkick,

Text tables are not available in Firebird.

@gkick and @Ratslinger,
Sorry for the delay of my answer, and thanks for your suggestion @gkick and for the Firebird precision @Ratslinger.

Hello,

Edit 2021-01-27:

Have replaced all the samples. Changed the csv file names to be the Registered DB name plus the query name. In the standalone (Calc only) added log in when needed for DB’s. This will be asked for twice - once to fill the list box (any time list box is to be filled) and once on the first creation after a list box selection. Subsequent runs do not require further sign-in’s.
Eliminated some unnecessary code.

End Edit

Have found the Windows problem noted in my comment. It is more than Windows. It is a bug which can be avoided and no further concern here. Keep adding to this project but feel it is time for you to test. Can be a nice tool for generating csv files through Base.

Have attach two files below. Place both of these in the same directory. The macro in the Base file will look for the Calc file in the same directory it is in. The name of the Calc file needs to be (based on macro code used) QueryToCSV.ods. The Base file can be named whatever you want. It must be registered with the name QueryToCsv (based upon current macro code).

Operation is simple. Open the form QueryToCSV. There is a button on the form Load Query List. Select and a macro will fill the list box with all queries in the Base file. Using the list box, as soon as an item is selected, the macro is fired off opening the Calc file (seen for a moment & then minimized), then calls the macro in the Calc file with parameters (query and registered DB names) to load the data for the query and create the csv file (same directory as the Calc file). The Calc file is cleared of needing to be saved and the file is closed.

Another csv can then be run. If the same is to be run, reload the list using the push button. Files with the same name are overwritten with each new execution.

Samples (replaced 2021-01-27:

Calc ------ QueryToCSV.ods

Base ----- QueryToCSV.odb

For use in your setup, the Calc file, as already noted, is to be in the same directory as the Base file. The name stays as QueryToCSV.ods. No other changes are needed. You can change the file to not Visible on opening instead of minimized. That is in the macro runMinimised. This sub is attached to the Open Document event of the file. Just remark out the item(s) you wish not to run.

For the Base file, Copy and paste the form QueryToCsv. The external name is not important. Register you Base file. Create a basic module under Standard named Export. Copy the macros in the Base sample and paste in your file. In line #25 of the code, replace QueryToCSV with your registered name. This is a string. Insure the quotes are there.

Save & test.

Have tested this (not extensively) using LO v7.0.4.2 on Ubuntu 20.04. Also tested on Windows 10. Used both Firebird embedded (your sample) and a PostgreSQL database. Others should also work.

Most tested files were generated in very short order. Did run from the PostgreSQL file an 81,000+ record table (20+ fields). The file was created in less than 25 seconds. During that time the Calc file was visible but blank. However all processed without a problem.

Would appreciate your feedback on this as it seems to be a fairly useful tool to create csv files from a database. Have made one of the table records last names a multi-line entry. Seems to work.

Have used bits and pieces of my code and code from others to create this. Can’t even say where every line came from.

Here is a link for csv arguments (calc macro) → Filter Options

A couple of other links used:

Have used this before but just forgot. This helped jog my memory:

Edit 2021-01-26:

Here is a single Calc file to process queries to csv. When the document is opened the registered DB names are retrieved and fill the list box. When an item is selected, it accesses the registered DB gets the available queries and fills the Query list box. The push button runs the process and the csv file is created in the same directry as where the Calc file resides.

Replaced 2021-01-27

Sample ----- RegisteredToCSV.ods

It may need more in the way of error checking but seems to work well.

1 Like

@Ratslinger,
Thank you very much for your time and this much documented solution!
I’ll have a deeper look and test it during the week, and give feedback.

@Ratslinger, indeed very useful, thanks for the research, will test with HSQL a little later and put into my Lo KB. Apropos github,(probably should be a separate post) Does git handle odbs OK for versioning frontends?

@gkick,

Don’t have answer for your question.

Had a new thought which should not affect this answer but may make the processing easier in the future. Will start looking today at placing all functionality into the Calc file. To be able to a get list of all registered databases. Then once one is selected, list queries available within the selected DB. Once that is selected create the csv file. May take some time finding access to required data.

Latest post uses just one file - Calc. It does not accommodate Base files attached to databases requiring user name and/or password. Will look into that if this seems to be a worth while item.

@Ratslinger,
Thank you very much for all this review and solution work!
Both solution are working very well.
On my side, not yet sure, but I think I will stay on the Calc (2nd) solution. Indeed, the first solution (form in Base + automatic Calc spreadsheet opening and saving) requires to provide the registered database name within the query. Which may be done at the very end of my work (when database will be finalized and deployed to the partner), but which may be a bugging point during all the working process before.
Or maybe is there a way to automatically check/register the database in local user LibreOffice directly within the macro ? I’ve seen that maybe the com::sun::star::container Module may be involved in the database registration (but I may be wrong), but this is still far too far from my Basic knowledge to handle this…

@Ratslinger,
Another point, but which I think is very specific to my query: two columns of my query result in N/A values.
One of them is the one with multiple lines values, but as this was working well with your attached files, I think it’s something else.
And the characteristic in common is that these two columns have been generated with a LIST Firebird SQL function.
Note: when I select the query (Queries tab in LibreOffice Base interface), right click, copy, and paste in a Calc spreadsheet, everything is pasting correctly (ie. no N/A values and correct multilines cells).

@greb,

Auto registration is problematic not because of code but rather the many circumstances it presents. Will look as possibly using URL but not clear this is viable.

As for the LIST situation, will try to look at this and see where there may be a problem. The difference is your copy & paste vs using internal connection provided through LO. Not much control in that.

@greb,

Don’t know what the answer is for your List field situation. In my test the new line is accomplished on a form with Ctrl + Enter. Yours is most likely using ASCII_CHAR(10). Don’t know how to get around that. There is no method to have Ctrl in the SQL - not an ASCII character. Just using a new spreadsheet and datasources, the query with List returns N/A.

One thought is that it is a bug. Since this works in a Base query and can display on a form and can copy/paste query to Calc OK, possibly the internal LO connection is not translating correctly.

@Ratslinger,
Ok, noted for auto registration. Alternative may so just be to add a small text to explain how to register if the target database is not in the list. End user can do it quickly, easily.
For the LIST issue, indeed, I’m using ASCII_CHAR(10). Ok, noted for the bug, I’ll try to see how to work around.
Anyway, thank you very much for all your time, solution and comments!

@Ratslinger,
Have solved the LIST field issue, using a CAST function, see my question post.

As well, have sorted out the CSV export using only ODB macro, mainly using what you submitted in the 2 options of your answer.
Thank you very much again, it was a great guide!