I have some CSV files with a lot of rows. Too many rows that I cannot copy and paste from Calc into a Base table.
I know I can create a DB against a CSV file or a Calc spreadsheet but that won’t let me do advanced things like queries with join.
So how can I “import” all of the rows from the CSV files into a Base DB?
Hello,
You do not state just how many rows is Too many. Just did almost 82,000 but you may be talking of millions.
If you are talking of importing into HSQLDB embedded and you have a great amount of data, you need to consider another database. HSQLDB embedded is prone to problems with large amounts of data - possible loss of data.
There are no built in methods within Base for importing via csv files. Even using Calc, large file imports can take a long time. See my answer in this post → How to import a big amount of data into LibreOffice Base.
If you go through the results you will see much in the way of variations depending upon the DB. Best (from tests done) was PostgreSQL. Using pgAdmin III was able to, using pre-defined table, import the 81,000+ record csv file directly. This complete import (table defined already) seemed like less time than opening the csv file in Calc (did not actually time it).
I tried just 100K rows and waited an hour and a half. I am beginning to think Base is not capable of doing what I need.
I have a website getting 800K to 1.1million hits per month. I am trying to use a database to get unique IP addresses from the first five fields of the log file. I had been using Access and someone recommended I move over to Base instead. I can import Excel or Calc files but can never add to them. Which makes it a problem going from year to year and getting the total unique IPs.
This does not answer the question yet you post as an answer.
Your assumption is incorrect. Have given you information via links in a previous question of yours here → Unable to Insert in Base after importing.
The link gave information about the type of database being used making a significant difference in the time needed to transfer this large amount of data. However it appears you are STILL taking an ill advised approach. You did NOT respond to that last comment but make this post here. Base is NOT the problem. Base is not even a database but only a front end to a database. YOU the user chose which database to use.
Use a better database like PostgreSQL and the waiting will be greatly reduced!
Yes I have tried everything my 8 hours of research into this as well as multiple links and resources provided. None of them are working. Firebird does not work either.
Obviously I don’t know where to respond to what and all of these rules that must be followed just so to make something work. Why doesn’t it just work? I never had these issues with Access. It just worked without having to jump through all these hoops.
I have my answer - Base is not user friendly and it does not work. I have to go buy an updated version of Access now since my old version won’t work on the latest Windows. I have give Microsoft some more money which I hate doing. But I don’t have any more time to waste on getting this working. It already put me behind on multiple critical projects.