Comments - Identifying and removing duplicates

Please paste your CREATE TABLE statement so that it's easier to replicate your structure.

Hi Ian. This is the file structure dumped using HeidiSQL. The actual file size is 181.2Mb.

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET NAMES utf8 */;
/*!50503 SET NAMES utf8mb4 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;


-- Dumping database structure for antareso
CREATE DATABASE IF NOT EXISTS `antareso` /*!40100 DEFAULT CHARACTER SET utf8 */;
USE `antareso`;

-- Dumping structure for table antareso.jobtransactionactuallabour
CREATE TABLE IF NOT EXISTS `jobtransactionactuallabour` (
  `I_TransactionId` int(11) NOT NULL AUTO_INCREMENT,
  `I_JobNumber` int(11) DEFAULT NULL,
  `SI_JobYearNumber` smallint(6) DEFAULT NULL,
  `SI_FinancialPeriod` smallint(6) DEFAULT NULL,
  `BIT_Flags1` tinyint(3) unsigned DEFAULT NULL,
  `SI_FinancialYear` smallint(6) DEFAULT NULL,
  `SI_OvertimeRate` smallint(6) DEFAULT NULL,
  `C_WhereAmI` char(64) DEFAULT NULL,
  `DT_PostingDate` datetime DEFAULT NULL,
  `DT_ActualDate` datetime DEFAULT NULL,
  `I_Hours1` int(11) DEFAULT NULL,
  `I_Cost` int(11) DEFAULT NULL,
  `I_ProductionUnits` int(11) DEFAULT NULL,
  `C_OperationCode` char(16) DEFAULT NULL,
  `C_EmployeeCode` char(16) DEFAULT NULL,
  `C_Comments1` char(255) DEFAULT NULL,
  `B_IsAuthorsCorrection` tinyint(4) DEFAULT NULL,
  `B_IsHouseCorrection` tinyint(4) DEFAULT NULL,
  `B_IsIndirect` tinyint(4) DEFAULT NULL,
  `B_IsFinished` tinyint(4) DEFAULT NULL,
  `I_Rate` int(11) DEFAULT NULL,
  `DT_DateOn` datetime DEFAULT NULL,
  `DT_DateOff` datetime DEFAULT NULL,
  `C_TerminalOn` char(16) DEFAULT NULL,
  `C_TerminalOff` char(16) DEFAULT NULL,
  `C_Comments2` char(255) DEFAULT NULL,
  `C_Comments3` char(255) DEFAULT NULL,
  `C_Comments4` char(255) DEFAULT NULL,
  `I_Hours2` int(11) DEFAULT NULL,
  `I_Hours3` int(11) DEFAULT NULL,
  `I_Hours4` int(11) DEFAULT NULL,
  `BIT_Flags2` tinyint(3) unsigned DEFAULT NULL,
  `duplicate` tinyint(1) NOT NULL DEFAULT 0,
  PRIMARY KEY (`I_TransactionId`),
  KEY `IDX_JobTransactionActualLabour1` (`SI_JobYearNumber`,`I_JobNumber`,`I_TransactionId`),
  KEY `IDX_JobTransactionActualLabour2` (`DT_PostingDate`,`C_EmployeeCode`,`I_TransactionId`),
  KEY `IDX_JobTransactionActualLabour3` (`C_OperationCode`)
) ENGINE=InnoDB AUTO_INCREMENT=303159 DEFAULT CHARSET=utf8;

-- Data exporting was unselected.
/*!40101 SET SQL_MODE=IFNULL(@OLD_SQL_MODE, '') */;
/*!40014 SET FOREIGN_KEY_CHECKS=IF(@OLD_FOREIGN_KEY_CHECKS IS NULL, 1, @OLD_FOREIGN_KEY_CHECKS) */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;

Apologies, I was leading you to syntax that only works in MariaDB 10.3 - MDEV-12137.

Thanks for the info. I'll dowwnload 10.3 and give it a go.

Note that MariaDB 10.3 is still alpha. So you can use it just to convert your data and then re-import it back to your current version, or you can try the alternative version I've added above. I wouldn't suggest using 10.3 in production yet.

Many thanks for the solution Ian. I have tried it using 10.3 and it has worked.

I now have six other major files on the system that have similarly been corrupted with duplicates to work my wat through.

The corruption occured not, I hasten to add, in MariaDB but in a D-ISAM based system which I have converted to MariaDB to solve the problem.

I really appreciate the time and trouble you have spent in helping me solve the issue.