Displaying weather data in libreoffice

[HinzundKunz]

I have a weather station that connects to the internet. On the internet, I can get a csv file for each month. So I have around 40 csv files now, each one having around 4000 to 5000 lines.

I would like to import these csv files into libreoffice (calc or base), and create a visual display of some of the data (rain for now, which is one of the columns).

How would I go about this? I tried importing it into calc, but manually importing 40 files is cumbersome. Concatenating them into one file worked, I then have one big csv file (over 20MB) that imports fine. I can sort it, remove the duplicate header lines. Ok, that works.
Create a chart with selecting only the two columns “date” and “rain” works as well. But the display is crammed, and libreoffice takes over 6.5 GB of memory and gets really slow. Also the display doesn’t really look nice. If I choose a line, its ugly, if I choose columns, they turn out tiny and barely visible.

In base I had trouble importing the files, and then I don’t even know how to create a visual report. Is that even possible?

[Villeroy]

csv is not a file format. csv is a convention to store database data in plain text. It should always be handled by some database.
Base is not a database. Base is a tool to work with databases. You can connect a Base document to a wide variety of databases using an even wider variety of database drivers (the actual software reading from and writing to a database).
Yes, it is possible to import csv into a Base connected database easily. However, it is not so easy to create a database with the required tools and helpers.
[Basic, SQL] CSV import into some database

Most office users give up on databases and try fumbling around with spreadsheets.
Merge sheets of spreadsheet documents

[HinzundKunz]

Well, I can create a database in Base (don’t know which standard format they use right now, but that’s irrelevant for this case), and import the data. No problem with that.
But how can I make a visual representation of the data? I haven’t figured out that one yet…

I also managed to import it into calc, but the chart function doesn’t look too good with 141601 lines of data… I don’t know whether you can do some kind of summarizing the data (like summarize monthly rains instead of rain data every 5 minutes), so I’m stuck there as well. Didn’t find anything online yet.

[Villeroy]

The type of database is indicated in the status bar of the main Base window. It might be embedded HSQL or embedded Firebird.
Spreadsheets don’t play well with many thousands of records. A well designed database handles millions of them easily. The major problem however is the csv because csv is not standardized. Unless you post some typical lines of csv, nobody can help you with the database.

Have a look at the csv file attached to Apache OpenOffice Community Forum - [HSQL, Base, Basic] Macro Driven CSV Import - (View topic)

ACCOUNT,DATE,TIME,NAME,AMOUNT

00420815,2/29/2020,221132,"Adams, Mary",-431
00420815,2/2/2020,221132,"Fuentes, Pablo",975
00420815,12/11/2019,104559,"Doe, John","1,234.98"

It uses commas as column separators, it has one header row, a blank row, a sequence of numbers to be imported as text, US-American date, 6-digit times, double-quoted strings with commas, and numbers which may be doubel-quoted when they include a comma as thousands-separator. This type of csv is folly but not unusual. Any software importing such weird data into a database table with strictly typed columns needs to be well prepared. Calc can import this file properly if you are familiar with all the options in the import dialog. For a database you need an SQL script transforming the weird text snippets into valid dates, times and numbers.

A csv file which could be imported into any database without trouble would look like this:

"00420815",2020-02.29,22:11:32,"Adams, Mary",-1234.95

Quoted strings, ISO date, ISO time, and plain decimals with point as decimal separator.

[Wanderer]

@HinzundKunz

I don’t know whether you can do some kind of summarizing the data (like summarize monthly rains instead of rain data every 5 minutes), so I’m stuck there as well. Didn’t find anything online yet.

You have to know the magic words: google for Pivot-tables.

I’d also suggest to import in a database first. You can the reduce the number of data for your charts by constructing a SQL-query. There are aggregating functions to count/sum for over partitions like month or year using for example GROUP BY in SQL.
.
Then you can either drag the result to Calc quite easily and create charts from there or try to master charts in base-reports.
.
Read more in the guide to Base found in the help-section on LibreOffice.org
English documentation | LibreOffice Documentation - LibreOffice User Guides

@HinzundKunz

then I don’t even know how to create a visual report. Is that even possible?

Charts can be integrated in Reports, but it seems Sometimes Tricky

BASE: Charts in Reports

I have been working on a pair of macros to try to get around the current weakness (failure) of Report Builder to generate Charts in Base Reports, for SQL databases in particular. I prefer to embed charts in Reports, rather than in Forms, because it is so much easier to add commentary and description to reports intended for printing: A) Create (or re-create) a Chart report in Report Builder. To allow quick regeneration of Chart reports each time LO is upgraded - and Base chart reports get broken…

[Villeroy]

Charts in reports are broken. However, you can link database data to Calc and then build the chart based on the linked data. And yes, you can create pivot tables from database data and then pivot charts.
Constantly growing sets of meteorological data can not be stored in spreadsheets. You want to compare different time intervals, different locations, different sets of measurements. And later you may want to use the same data on a web site, with advanced types of analysis software beyond office software. Calc is a data trap, good enough for ad-hoc analysis of small data sets, preferably pulled from some database.