hsheng

Data duplication is not a problem that is specific to any particular industry. Data often gets duplicated for a variety of reasons: manual entry errors, redundant entries in different source systems, missing data constraints, etc. However, before data can be successfully de-duplicated, you must first define the criteria which identify which 2 records are considered duplicates. For example, do you see any duplicate records in the following table?

Some may say yes, while others will say no. It all depends on how the criteria for identifying duplicate records are defined.

If the requirement is that every field value must match, then there are no duplicate records in the table. However, if the requirements are that the fields item, size, and color must match, and we want to keep the most recent entry, then the resulting table would look like this:

In this example, we removed 3 records because they contained duplicate entries for item, size, and color:

Of course this is a very simple exercise with only 11 records. What happens when you are dealing with 11,000 or even 11 million records? Naturally not something you would want to do in Excel or by hand! This is where expressor shines and makes data de-duplication an easy task.

To accomplish this logic using expressor, all you need to do is use the Unique operator in your dataflow:

The unique operator can be easily configured by
specifying the fields which would constitute a duplicate record. In this example, we would check item, size, and color as show to the right.

The mode setting identifies which record should be kept when duplicates are encountered. The method setting allows you to specify how the processing should occur. If there’s enough memory available and your data volumes are not too large, the In Memory setting will yield the best performance.

Once configured properly, the Unique operator will identify and eliminate duplicate records at runtime.
This simple approach works regardless of the total number of records being processed. So the next
time you encounter duplicate data that shouldn’t be there, just remember how easy it is to use
expressor to eliminate those duplicates.