We will be migrating from Ask to Discourse on the first week of August, read the details here

Ask Your Question
0

How can I merge over 1,500 tsv files? [closed]

asked 2021-01-02 02:30:19 +0200

ClubCrypto gravatar image

updated 2021-06-01 06:52:20 +0200

Alex Kemp gravatar image

They all have the same format - column names in row 1.

Obviously I can't do it manually, and I'm sure someone has had this issue before.

I'm new to Calc, having moved from Office a few months ago. Where should I begin?

Operating system: Windows 10. Merging these to import into a database. Around 400,000 lines total, various within each file from 1 to over 1,000.

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by Alex Kemp
close date 2021-05-27 11:16:38.247624

Comments

What operating system are you on? How many lines are in each of the files? (there a limit to the number of rows in calc. A database may be a better solution for your project. We need details to answers this! Idle prying minds wanna know! ;)

AdmFubar gravatar imageAdmFubar ( 2021-01-02 02:38:49 +0200 )edit

Thank you - I updated the question.

ClubCrypto gravatar imageClubCrypto ( 2021-01-02 02:49:48 +0200 )edit

1 Answer

Sort by » oldest newest most voted
0

answered 2021-01-02 03:59:26 +0200

Ratslinger gravatar image

Hello,

Sort of a guess here that each file has the same columns - names and number of.

One item is to combine csv files. Tested in Linux without a problem using:

head -n 1 file1.csv > combined.out && tail -n+2 -q *.csv >> combined.out

This was found on this post -> Merging contents of multiple .csv files into single .csv file. See comment under answer by zsha.

For Windows 10 here is a post -> How to Merge Multiple CSV Files Using Command Prompt. Have not tested this.

Now determine where you want the data. If it is to go to a database, you need to choose which database to use. Importing this amount of data is critical in the amount of time needed depending upon this choice. For an example using 81K+ records, see this post -> How to import a big amount of data into LibreOffice Base.

So depending upon the choice of residence of the data, you may want to have multiple files so 400K+ records is not a very, very, very long import task.

edit flag offensive delete link more

Comments

It was as easy as copy *.tsv complete.tsv in Windows 10.

I was in help desk for 25 years. Never knew you could do that! Thank you.

ClubCrypto gravatar imageClubCrypto ( 2021-01-02 04:05:39 +0200 )edit

Question Tools

1 follower

Stats

Asked: 2021-01-02 02:30:19 +0200

Seen: 101 times

Last updated: Jan 02