@flatfile/plugin-dedupe

Dedupe records in a sheet via a sheet level custom action.

When embedding Flatfile, this plugin should be deployed in a server-side listener. Learn more

@flatfile/plugin-dedupe

This plugin dedupes records in a sheet via a sheet level custom action.

Event Type:

listener.on('job:ready')

Parameters

jobOperation · string · required

The jobOperation parameter specifies the name of the job operation to match on.

opt.on · string

The on parameter specifieds which field key to match on.

opt.keep · 'first' | 'last'

The keep option lets you choose whether to keep the first or last duplicate record.

opt.custom · function

The custom parameter accepts a custom dedupe function. This will override the keep parameter.

opt.debug · boolean

The debug parameter lets you toggle on/off helpful debugging messages for development purposes.

API Calls

• api.records.get
• api.jobs.ack
• api.records.delete
• api.jobs.fail
• api.jobs.complete

Imported NPM Packages

• @flatfile/api@1.5.14+
• @flatfile/hooks
• @flatfile/listener@0.3.15+
• remeda@1.14.0+ offers a set of utility functions for functional programming and data manipulation in JavaScript, providing a convenient way to work with arrays and objects.
• ts-pattern@5.0.4+ The exhaustive Pattern Matching library for TypeScript with smart type inference.

Usage

An action with the operation name of “dedupe-email” must be configured on a Sheet in order to the plugin to be triggered.

install

npm i @flatfile/plugin-dedupe

import

import { dedupePlugin } from "@flatfile/plugin-dedupe";

// ... inside the Sheet configuration
"actions": [
  {
    "operation": "dedupe-email",
    "mode": "background",
    "label": "Dedupe emails",
    "description": "Remove duplicate emails"
  }
]
// ...

listener.js

// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet level action specified with the operation name `dedupe-email`
listener.use(
  dedupePlugin("dedupe-email", {
    on: "email",
    keep: "last",
  })
);

// user specified dedupe function using RemedaJS
// must return a list a record id's for deletion
listener.use(
  dedupePlugin("dedupe-email", {
    custom: (records) => {
      let uniques = new Set();
      return R.pipe(
        records,
        R.reduce((acc, record) => {
          const { value } = record.values["email"];

          if (uniques.has(value)) {
            return [...acc, record.id];
          } else {
            uniques.add(value);

            return acc;
          }
        }, [] as Array<string>)
      );
    },
  })
);

// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet level action specified with the operation name `dedupe-email`
listener.use(
  dedupePlugin("dedupe-email", {
    on: "email",
    keep: "last",
  })
);

// user specified dedupe function using RemedaJS
// must return a list a record id's for deletion
listener.use(
  dedupePlugin("dedupe-email", {
    custom: (records: Flatfile.RecordsWithLinks) => {
      let uniques = new Set();

      return R.pipe(
        records,
        R.reduce((acc, record) => {
          const { value } = record.values["email"];

          if (uniques.has(value)) {
            return [...acc, record.id];
          } else {
            uniques.add(value);

            return acc;
          }
        }, [] as Array<string>)
      );
    },
  })
);

See the code

@flatfile/plugin-dedupe