Jobs

The anatomy of Jobs

In Flatfile, a Job represents a large unit of work performed asynchronously on a resource such as a file, Workbook, or Sheet. The Jobs workflow provides visibility into the status and progress of your Jobs, allowing you to monitor and troubleshoot the data processing pipeline.

Types of Jobs

Jobs can be triggered in a number of ways, most commonly in response to user activity. Jobs are then managed via Listeners, which receive Events published by Flatfile in response to activity.

There are three types of Jobs on the Flatfile Platform:

Action Jobs

Attached to custom actions

Custom Jobs

Created dynamically in your listener

System Jobs

Flatfile system jobs

Action Based Jobs

Actions are developer-defined operations that can be mounted on a number of domains (including Sheets, Workbooks, Documents, and Files). Mounting an Action means attaching a custom operation to that domain. That operation can then be triggered by a user event (clicking a button or selecting a menu item).

When an Action is triggered a job:ready Event for a Job named [domain]:[operation] is published. Your Listener can then be configured to respond to that Action via it’s Event.

To run an Action based job, two configurations are necessary.

First, create an Action on a domain. Here’s an example of a Workbook containing an Action:

javascript

api.workbook.create({
  name: "October Report Workbook",
  actions: [
    {
      label: "Export Data",
      description: "Send data to destination system",
      operation: "export",
      type: "file",
    },
  ],
});

Then, create a Listener to respond to the Action:

javascript

listener.on(
  "job:ready",
  { job: "workbook:export" },
  ({ context: { jobId } }, payload) => {
    const { jobId } = event.context;

    try {
        await api.jobs.ack(jobId, {
          info: "Starting submit job...",
          progress: 10,
        });

        // Custom code here

        await api.jobs.complete(jobId, {
          outcome: {
            message: "Submit Job was completed succesfully.",
          },
        });

    } catch (error) {
        await api.jobs.fail(jobId, {
          outcome: {
            message: "This Job failed.",
          },
        });
    }
  }
);

Note that the Listener is listening for the job:ready event, for the workbook:export Job, which was defined in our Workbook.

Custom Jobs

Another trigger option is to create a Custom Job via SDK/API. In the SDK, Jobs are created by calling the api.jobs.create() method.

Creating a custom Job in your Listener enables any Event to trigger a Job.

Here’s an example of creating a custom Job in a Listener:

javascript

listener.on(
  "commit:created",
  { sheet: "contacts" },
  async ({ context: { workbookId, sheetId } }) => {
    const { data } = await api.jobs.create({
      type: "workbook",
      operation: "myCustomOperation",
      trigger: "immediate",
      source: workbookId,
      config: {
        sheet: sheetId,
        filter: "all",
      },
    });
  }
);

Note that the trigger for this Listener is set to immediate, which means that the Job will be created and executed immediately upon the Event firing.

Therefore, we should have our Listener ready to respond to this Job:

javascript

listener.on(
  "job:ready",
  { job: "workbook:myCustomOperation" },
  async ({ context: { jobId, workbookId }, payload }) => {
    try {
      await api.jobs.ack(jobId, {
        info: "Starting my custom operation.",
        progress: 10,
      });

      // Custom code here.

      await api.jobs.complete(jobId, {
        outcome: {
          message: "Sucessfully completed my custom operation.",
        },
      });
    } catch {
      await api.jobs.fail(jobId, {
        outcome: {
          message: "Custom operation failed.",
        },
      });
    }
  }
);

Please note that Flatfile does not support Job plans for custom Actions.

System Jobs

Internally, Flatfile uses Jobs to power many of the features of the Flatfile Platform, such as extraction, record mutation, and AI Assist. Here are some examples of Jobs that the Flatfile Platform creates and manages on your behalf:

Job Name	Description
Extract	Extracts data from the specified source.
Map	Maps data from it’s ingress format to Blueprint fields.
DeleteRecords	Deletes records from a dataset based on specified criteria.
Export	Exports data to a specified format or destination.
MutateRecords	Alters records in a dataset according to defined rules.
Configure	Sets up or modifies the configuration of a Space.
AiAssist	Utilizes AI to assist with tasks such as data categorization.
FindReplace	Searches for specific values and replaces them.

Thespace:configure Job is the only system-level Job published for developer consumption. This is a special Job that allows developers to configure their Spaces dynamically. See our Space Configuration documentation for more information.

The Anatomy of a Job

Lifecycle Events

Jobs fire the following Events during their lifecycle. In chronological order, the Job Events are:

Event	Description
job:created	Fires when a Job is created, but before it does anything.
job:ready	Fires when a Job is ready to move into the execution stage, but before it does anything.
job:updated	Fires when there is an update to a Job while it is executing.
job:completed OR job:failed	job:completed fires when a Job is completed successfully, job:failed fires if a Job is completed but fails. One of these events will fire upon Job completion, but never both.
job:outcome-acknowledged	Fires when a user acknowledges the completion of a Job through a UI popup.

You can listen on any of these events in your Listener, but the most common event to listen for is job:ready. For more on managing your Job see Working with Jobs.

Required Parameters

type · string · required

Workbook, File, Sheet, Space

operation · string · required

export, extract, map, delete, etc

source · string · required

The id of the data source (FileId, WorkbookId, or SheetId)

Optional Parameters

trigger · string

manual or immediate

destination · string

The id of the data target (if any)

status · string

created, planning, scheduled, ready, executing, complete, failed, cancelled

progress · number

A numerical or percentage value indicating the completion status of the Job.

estimatedCompletionAt · date

An estimated completion time. The UI will display the estimated processing time in the foreground Job overlay.

formAction · object

label	A user-friendly name for the action.
description	A string describing the action.
tooltip	Additional info on hover.
schedule	weekly, daily, hourly
operation	The operation to perform.
mode	foreground, background, toolbarBlocking
primary	Whether this action is considered the primary or default action.
confirm	Whether user confirmation is required before the action is executed.
icon	Icon representative of the action.
inputForm	{ type: simple, fields: { key, label, description, type, *config, constraints }
*config	{ options: { value, label, description, color, icon, meta } }

input · string

Input parameters for the Job type.

subject · object

resource	Identifier for the subject resource.
type	resource, collection
query	The query used to locate or identify the subject resource.
params	Parameters that detail or modify the subject query.

outcome · object

heading	The heading text summarizing the outcome of the action.
acknowledge	Indicates if the outcome requires user acknowledgment.
message	Detailed message describing the outcome.
buttonText	Label for the button to acknowledge the outcome.
next	{ id, url, download, wait, snapshot, retry }

info · string

Additional information regarding the Job’s current status.

managed · string

Indicates whether the Job is managed by the Flatfile platform or not.

mode · string

foreground, background, toolbarBlocking

Please see our API Reference for details on all possible values.

Working with Jobs

Jobs can be managed via SDK/API. For a complete list of ways to interact with Jobs please see our API Reference . Commonly, Jobs are acknowledged, progressed, and then completed, or failed. Here’s look at those steps.

You can also use the JobHandler Plugin which simplifies the handling of Flatfile Jobs. This plugin works by listening to the job:ready event and executing the handler callback. There is an optional tick function which updates the Job’s progress.

Jobs.Ack

First, acknowledge a Job. This will update the Job’s status to executing.

javascript

await api.jobs.ack(jobId, {
  info: "Starting submit job...",
  progress: 10,
});

Jobs.Update

Once a Job is acknowledged, you can begin running your custom operation. Jobs were designed to handle large processing loads, but you can easily update your user by updating the Job with a progress value.

javascript

await api.jobs.update(jobId, {
  progress: 50,
  estimatedCompletionAt: new Date("Tue Aug 23 2023 16:19:42 GMT-0700"),
});

Progress is a numerical or percentage value indicating the completion status of the work. You may also provide an estimatedCompletionAt value which will display your estimate of the remaining processing time in the foreground Job overlay. Additionally, the Jobs Panel will share visibility into the estimated remaining time for acknowledged jobs.

Finally, when your Job is complete, update the Job’s status to complete. This status update includes an optional outcome message which will be displayed to the user. Use this to provide detail on the outcome of the Job.

Jobs.Complete

Once a job is complete, you can display an alert to the end user using outcome.

javascript

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
  },
});

Next Links

Optionally, you can add a button to the outcome dialog to control the next step in the workflow once job has finished.

Internal Link

Add a button to the dialog that will redirect the user somewhere within a Space using next > Id.

In this code below, we will create a button that says “See all downloads” with this path: space/us_sp_1234/files?mode=export

javascript

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
    //Reference: https://spaces.flatfile.com/space/{$id}/{path}?{$query}
    next: {
      type: "id",
      id: "dev_sp_1234",
      path: "files",
      query: "mode=export",
      label: "See all downloads",
    },
  },
});

External Url

Add a button to the dialog that will redirect the user to an external link using next > Url.

In this code below, we will create a button that says “Go to Google”. It will open in a new tab.

javascript

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "url",
      url: "http://www.google.com",
      label: "Go to Google",
    },
  },
});

Download

Add a button to the dialog that will redirect the user to an external link using next > Url.

In this code below, we will create a button that says “Download this file”.

javascript

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "download",
      fileName: "DownloadedFromFlatfile.csv",
      url: "source_of_file.csv",
      label: "Download this file",
    },
  },
});

Retry

Often in the event of a failure, you may want to add a button to the dialog that will retry the Job using next > Retry.

In this code below, we will create a button that says “Retry”.

javascript

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was not completed succesfully. No records were processed.`,
    acknowledge: true,
    next: {
      type: "retry",
      label: "Try again",
    },
  },
});