Comments - Identifying and removing duplicates

There is another alternative that should work on versions before MariaDB 10.3. This works by setting a unique index on all the identical fields, and by making use of the old ALTER TABLE copying behaviour, ignoring errors when it copies the data back, basically silently dropping duplicates. This requires the old_alter_table system variable to be temporarily set. Here's a sample:

CREATE TABLE t (id INT, f1 VARCHAR(2));

INSERT INTO t VALUES (1,'a'), (2,'a'), (3,'b'), (4,'a');

SELECT * FROM t;
+------+------+
| id   | f1   |
+------+------+
|    1 | a    |
|    2 | a    |
|    3 | b    |
|    4 | a    |
+------+------+

SET SESSION old_alter_table=1;

ALTER IGNORE TABLE t ADD UNIQUE INDEX u(f1);
Query OK, 4 rows affected (0.82 sec)               
Records: 4  Duplicates: 2  Warnings: 0

SELECT * FROM t;
+------+------+
| id   | f1   |
+------+------+
|    1 | a    |
|    3 | b    |
+------+------+

SET SESSION old_alter_table=0;

With your data, you would create the unique index on all the duplicated fields, so something like:

ALTER TABLE t ADD UNIQUE INDEX u(f1, f2, f3...);

Thanks Ian for the reply.

If I apply the coding shown in 'Removing Duplicates' to my problem as

DELETE FROM jobtransactionactuallabour WHERE I_TransactionId IN (
  SELECT t2.I_TransactionId  FROM jobtransactionactuallabour t1, jobtransactionactuallabour t2 
	WHERE t1.SI_JobYearNumber=t2.SI_JobYearNumber 
	AND t1.I_JobNumber=t2.I_JobNumber
	AND t1.DT_ActualDate=t2.DT_ActualDate
	AND t1.I_Hours1=t2.I_Hours1
	AND t1.C_OperationCode=t2.C_OperationCode
	AND t1.C_EmployeeCode=t2.C_EmployeeCode
	AND t1.I_TransactionId <>t2.I_TransactionId  AND t1.I_TransactionId =(
    SELECT MAX(I_TransactionId ) FROM jobtransactionactuallabour tab 
		WHERE tab SI_JobYearNumber=t1.SI_JobYearNumber 
		AND tab.I_JobNumber=t1.I_JobNumber
		AND tab.DT_ActualDate=t1.DT_ActualDate
		AND tab.I_Hours1=t1.I_Hours1
		AND tab.C_OperationCode=t1.C_OperationCode
		AND tab.C_EmployeeCode=t1.C_EmployeeCode
  )
);

I get the error message-

SQL Erroe (1093): Table 'jobtransactionactuallabour' is specified twice, both as a target for 'DELETE' and as a separate source for the data.

I had thought that as the table was identified with aliases this should not be a problem but this doesn't appear to be the case

Please paste your CREATE TABLE statement so that it's easier to replicate your structure.

Hi Ian. This is the file structure dumped using HeidiSQL. The actual file size is 181.2Mb.

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET NAMES utf8 */;
/*!50503 SET NAMES utf8mb4 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;


-- Dumping database structure for antareso
CREATE DATABASE IF NOT EXISTS `antareso` /*!40100 DEFAULT CHARACTER SET utf8 */;
USE `antareso`;

-- Dumping structure for table antareso.jobtransactionactuallabour
CREATE TABLE IF NOT EXISTS `jobtransactionactuallabour` (
  `I_TransactionId` int(11) NOT NULL AUTO_INCREMENT,
  `I_JobNumber` int(11) DEFAULT NULL,
  `SI_JobYearNumber` smallint(6) DEFAULT NULL,
  `SI_FinancialPeriod` smallint(6) DEFAULT NULL,
  `BIT_Flags1` tinyint(3) unsigned DEFAULT NULL,
  `SI_FinancialYear` smallint(6) DEFAULT NULL,
  `SI_OvertimeRate` smallint(6) DEFAULT NULL,
  `C_WhereAmI` char(64) DEFAULT NULL,
  `DT_PostingDate` datetime DEFAULT NULL,
  `DT_ActualDate` datetime DEFAULT NULL,
  `I_Hours1` int(11) DEFAULT NULL,
  `I_Cost` int(11) DEFAULT NULL,
  `I_ProductionUnits` int(11) DEFAULT NULL,
  `C_OperationCode` char(16) DEFAULT NULL,
  `C_EmployeeCode` char(16) DEFAULT NULL,
  `C_Comments1` char(255) DEFAULT NULL,
  `B_IsAuthorsCorrection` tinyint(4) DEFAULT NULL,
  `B_IsHouseCorrection` tinyint(4) DEFAULT NULL,
  `B_IsIndirect` tinyint(4) DEFAULT NULL,
  `B_IsFinished` tinyint(4) DEFAULT NULL,
  `I_Rate` int(11) DEFAULT NULL,
  `DT_DateOn` datetime DEFAULT NULL,
  `DT_DateOff` datetime DEFAULT NULL,
  `C_TerminalOn` char(16) DEFAULT NULL,
  `C_TerminalOff` char(16) DEFAULT NULL,
  `C_Comments2` char(255) DEFAULT NULL,
  `C_Comments3` char(255) DEFAULT NULL,
  `C_Comments4` char(255) DEFAULT NULL,
  `I_Hours2` int(11) DEFAULT NULL,
  `I_Hours3` int(11) DEFAULT NULL,
  `I_Hours4` int(11) DEFAULT NULL,
  `BIT_Flags2` tinyint(3) unsigned DEFAULT NULL,
  `duplicate` tinyint(1) NOT NULL DEFAULT 0,
  PRIMARY KEY (`I_TransactionId`),
  KEY `IDX_JobTransactionActualLabour1` (`SI_JobYearNumber`,`I_JobNumber`,`I_TransactionId`),
  KEY `IDX_JobTransactionActualLabour2` (`DT_PostingDate`,`C_EmployeeCode`,`I_TransactionId`),
  KEY `IDX_JobTransactionActualLabour3` (`C_OperationCode`)
) ENGINE=InnoDB AUTO_INCREMENT=303159 DEFAULT CHARSET=utf8;

-- Data exporting was unselected.
/*!40101 SET SQL_MODE=IFNULL(@OLD_SQL_MODE, '') */;
/*!40014 SET FOREIGN_KEY_CHECKS=IF(@OLD_FOREIGN_KEY_CHECKS IS NULL, 1, @OLD_FOREIGN_KEY_CHECKS) */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;

Apologies, I was leading you to syntax that only works in MariaDB 10.3 - MDEV-12137.

Thanks for the info. I'll dowwnload 10.3 and give it a go.

Note that MariaDB 10.3 is still alpha. So you can use it just to convert your data and then re-import it back to your current version, or you can try the alternative version I've added above. I wouldn't suggest using 10.3 in production yet.

Many thanks for the solution Ian. I have tried it using 10.3 and it has worked.

I now have six other major files on the system that have similarly been corrupted with duplicates to work my wat through.

The corruption occured not, I hasten to add, in MariaDB but in a D-ISAM based system which I have converted to MariaDB to solve the problem.

I really appreciate the time and trouble you have spent in helping me solve the issue.