> ## Documentation Index
> Fetch the complete documentation index at: https://flatfileinc-feat-updatereactquickstart.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# @flatfile/plugin-dedupe

> Dedupe records in a sheet via a sheet level custom action.

<Warning>
  When embedding Flatfile, this plugin should be deployed in a server-side
  listener. [Learn more](/orchestration/listeners#listener-types)
</Warning>

<CardGroup cols={1}>
  <Card title="@flatfile/plugin-dedupe" href="https://github.com/FlatFilers/flatfile-plugins/tree/main/plugins/dedupe" icon="github">
    <br />

    This plugin dedupes records in a sheet via a sheet level custom action.

    <br />

    <br />

    **Event Type:**

    <br />

    `listener.on('job:ready')`
  </Card>
</CardGroup>

## Parameters

<ParamField path="jobOperation" type="string" required>
  The `jobOperation` parameter specifies the name of the job operation to match
  on.
</ParamField>

<ParamField path="opt.on" type="string">
  The `on` parameter specifieds which field key to match on.
</ParamField>

<ParamField path="opt.keep" type="'first' | 'last'">
  The `keep` option lets you choose whether to keep the first or last duplicate
  record.
</ParamField>

<ParamField path="opt.custom" type="function">
  The `custom` parameter accepts a custom dedupe function. This will override
  the `keep` parameter.
</ParamField>

<ParamField path="opt.debug" type="boolean">
  The `debug` parameter lets you toggle on/off helpful debugging messages for
  development purposes.
</ParamField>

## API Calls

* `api.records.get`
* `api.jobs.ack`
* `api.records.delete`
* `api.jobs.fail`
* `api.jobs.complete`

## Imported NPM Packages

* [`@flatfile/api@1.5.14+`](https://www.npmjs.com/package/@flatfile/api)
* [`@flatfile/hooks`](https://www.npmjs.com/package/@flatfile/hooks)
* [`@flatfile/listener@0.3.15+`](https://www.npmjs.com/package/@flatfile/listener)
* [`remeda@1.14.0+`](https://www.npmjs.com/package/remeda) offers a set of utility functions for functional programming and data manipulation in JavaScript, providing a convenient way to work with arrays and objects.
* [`ts-pattern@5.0.4+`](https://www.npmjs.com/package/ts-pattern) The exhaustive Pattern Matching library for TypeScript with smart type inference.

## Usage

An action with the operation name of "dedupe-email" must be configured on a Sheet in order to the plugin to be triggered.

```bash install
npm i @flatfile/plugin-dedupe
```

```js import
import { dedupePlugin } from "@flatfile/plugin-dedupe";
```

```ts
// ... inside the Sheet configuration
"actions": [
  {
    "operation": "dedupe-email",
    "mode": "background",
    "label": "Dedupe emails",
    "description": "Remove duplicate emails"
  }
]
// ...
```

<CodeGroup>
  ```js listener.js
  // common usage
  // Keep the last record encountered (from top to bottom) based on the`email` field key.
  // Must have a Sheet level action specified with the operation name `dedupe-email`
  listener.use(
    dedupePlugin("dedupe-email", {
      on: "email",
      keep: "last",
    })
  );

  // user specified dedupe function using RemedaJS
  // must return a list a record id's for deletion
  listener.use(
    dedupePlugin("dedupe-email", {
      custom: (records) => {
        let uniques = new Set();
        return R.pipe(
          records,
          R.reduce((acc, record) => {
            const { value } = record.values["email"];

            if (uniques.has(value)) {
              return [...acc, record.id];
            } else {
              uniques.add(value);

              return acc;
            }
          }, [] as Array<string>)
        );
      },
    })
  );
  ```

  ```ts listener.ts
  // common usage
  // Keep the last record encountered (from top to bottom) based on the`email` field key.
  // Must have a Sheet level action specified with the operation name `dedupe-email`
  listener.use(
    dedupePlugin("dedupe-email", {
      on: "email",
      keep: "last",
    })
  );

  // user specified dedupe function using RemedaJS
  // must return a list a record id's for deletion
  listener.use(
    dedupePlugin("dedupe-email", {
      custom: (records: Flatfile.RecordsWithLinks) => {
        let uniques = new Set();

        return R.pipe(
          records,
          R.reduce((acc, record) => {
            const { value } = record.values["email"];

            if (uniques.has(value)) {
              return [...acc, record.id];
            } else {
              uniques.add(value);

              return acc;
            }
          }, [] as Array<string>)
        );
      },
    })
  );
  ```
</CodeGroup>

## See the code

<CardGroup cols={2}>
  <Card title="@flatfile/plugin-dedupe" href="https://github.com/FlatFilers/flatfile-plugins/tree/main/plugins/dedupe" icon="github" />
</CardGroup>