My MariaDB suddenly not working and it is looking a table that does not exist when uploading my csv
For 4 years my MariaDB has been working but this month it suddenly became less responsive so I did a check, optimize, and repair and do a optimization on MariaDB-server.cnf. I used this script to upload my csv file to the mariadb server: I call this bash script "process_all_files.sh"
or file in /home/devdb/incoming/*.csv;
do
file_name=${file##*/}
file_path=${file%/*}
file_path_incoming="${file_path}/${file_name}"
file_path_process="${file_path}/processing/${file_name}"
file_path_archive="${file_path}/archive/${file_name}"
#echo $file_path;
#echo $file_path_incoming;
#echo $file_path_process;
#echo $file_path_archive;
#check csv file has an header
header=$(head -n 1 "$file_path_incoming")
header_string="IMO_Number,SerialNo,ChannelNo,ChannelDescription,TimeStamp,Value,Unit"
# Add header to csv file if not found
if [[ $header != $header_string ]]
then
sed -i '1s/^/$header_string\n/' $file_path_incoming
fi
# Move file from incoming folder to processing folder.
echo "Move $file_name to processing to folder"
mv -f $file_path_incoming $file_path_process
# Import csv file to database. Header line will skip.
echo "Import $file into database"
sql="
LOAD DATA LOCAL INFILE '$file_path_process'
INTO TABLE Monitoring
FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(IMO_No,SerialNo,ChannelNo,ChannelDescription,TimeStampOriginal,@Value,Unit)
set TimeStamp=STR_TO_DATE(TimeStampOriginal, '%Y-%m-%dT%H:%i:%s+00:00'),Value = @Value,FileName='${file_name}';
"
echo $sql > "${file_path}/import.sql"
mysql -u root -ppassword01 drums < "${file_path}/import.sql" && mv -f $file_path_process $file_path_archive
echo "";
done
And my import.sql is this:
LOAD DATA LOCAL INFILE '/home/devdb/incoming/processing/*.csv' INTO TABLE Monitoring FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES (IMO_No,SerialNo,ChannelNo,ChannelDescription,TimeStampOriginal,@Value,Unit) set TimeStamp=STR_TO_DATE(TimeStampOriginal, '%Y-%m-%dT%H:%i:%s+00:00'),Value = @Value,FileName='*.csv';
These codes run in a cron job every 5 minutes when I inspect the logs there is an error:
Move MODmQueenie-2023-12-30-12-40.csv to processing to folder Import /home/devdb/incoming/MODmQueenie-2023-12-30-12-40.csv into database ERROR 1146 (42S02) at line 1: Table 'drums.a' doesn't exist
I tried a different approach using python with the print(load_sql) to see if the script looks for "drums.a" but it does not. This is my python code:
import os
import pandas as pd
import pymysql
import shutil
from datetime import datetime
# Database credentials and configuration
db_config = {
'user': 'root',
'password': 'password01',
'host': 'localhost',
'database': 'drums',
'local_infile': True
}
# Directories
incoming_dir = '/home/devdb/incoming'
processing_dir = os.path.join(incoming_dir, 'processing')
archive_dir = os.path.join(incoming_dir, 'archive')
# Header for CSV files
header_string = ["IMO_No", "SerialNo", "ChannelNo", "ChannelDescription", "TimeStampOriginal", "Value", "Unit"]
# Function to convert TimeStampOriginal to TimeStamp
def convert_timestamp(original_timestamp):
return datetime.strptime(original_timestamp, '%Y-%m-%dT%H:%M:%S+00:00').strftime('%Y-%m-%d %H:%M:%S')
# Process each CSV file in the incoming directory
for file_name in os.listdir(incoming_dir):
if file_name.endswith('.csv'):
file_path_incoming = os.path.join(incoming_dir, file_name)
file_path_process = os.path.join(processing_dir, file_name)
file_path_archive = os.path.join(archive_dir, file_name)
# Read CSV
df = pd.read_csv(file_path_incoming, names=header_string, header=None)
# Convert TimeStampOriginal to TimeStamp
df['TimeStamp'] = df['TimeStampOriginal'].apply(convert_timestamp)
# Save the modified CSV file to processing directory
df.to_csv(file_path_process, index=False, header=True)
# Connect to the database and load the data
try:
conn = pymysql.connect(**db_config)
cursor = conn.cursor()
# Construct the SQL query
load_sql = f"""
LOAD DATA LOCAL INFILE '{file_path_process}'
INTO TABLE monitoring
FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
LINES TERMINATED BY '\\n'
IGNORE 1 LINES
(IMO_No, SerialNo, ChannelNo, ChannelDescription, TimeStamp, Value, Unit)
"""
print(load_sql)
cursor.execute(load_sql)
conn.commit()
except pymysql.MySQLError as error:
print(f"Error: {error}")
finally:
if conn and conn.open:
cursor.close()
conn.close()
# Move file to archive directory
shutil.move(file_path_process, file_path_archive)
print(f"Processed and archived {file_name}")
This is the output of the python script which shows it is looking for "a" table:
Processed and archived MODmQueenie-2023-12-30-14-13.csv
LOAD DATA LOCAL INFILE '/home/devdb/incoming/processing/MODmQueenie-2023-12-30-14-16.csv'
INTO TABLE monitoring
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(IMO_No, SerialNo, ChannelNo, ChannelDescription, TimeStamp, Value, Unit)
Error: (1146, "Table 'drums.a' doesn't exist")
I inspect my database, and there is no "a" tables in my schema but my code error tells me it is looking for 'drums.a"
I dump all data which is about 40GB and put it on the cloud server hosted in Ubuntu but still the same response looking for "drums.a"
Please help me what is this causing my problem that my script looks for "a" table which is not existing on my codes that working for 4 years.
Thanks and Regards,
Romel