Last updated: December 8, 2022 (Strapi v4 era)
25 min read

How to create an import content plugin (part 3/4)

Pouya Miralayi

December 12, 2019

how-to-create-an-import-content-plugin-part-3-4

This tutorial is part of the « How to create your own plugin »:

Please note: Since we initially published this blog post, we released new versions of Strapi and tutorials may be outdated. Sorry for the inconvenience if it's the case. Please help us by reporting it here.

Table of contents

Upload From file (part 1)
Upload from url and raw input (part 2)
Services (part 3) - current
History Page (part 4)

You can reach the code for this tutorial at this link

Service: utils

We are going to setup our “analyzing endpoint” so our UI actually receives a proper response when it sends an “analyze request”.

Go to services directory which is located in plugin’s root directory and create a new directory called utils with a file named utils.js inside it:
services/utils/utils.js
Paste the below code in that file:

"use strict";
const request = require("request");
const contentTypeParser = require("content-type-parser");
const RssParser = require("rss-parser");
const CsvParser = require("csv-parse/lib/sync");
const urlRegEx = /((([A-Za-z]{3,9}:(?:\/\/)?)(?:[\-;:&=\+\$,\w]+@)?[A-Za-z0-9\.\-]+|(?:www\.|[\- ;:&=\+\$,\w]+@)[A-Za-z0-9\.\-]+)((?:\/[\+~%\/\.\w\-]*)?\??(?:[\-\+=&;%@\.\w]*)#?(?:[\.\!\/\\\w]*))?)/g;
const URL_REGEXP = new RegExp(urlRegEx);
const validateUrl = url => {
  URL_REGEXP.lastIndex = 0;
  return URL_REGEXP.test(url);
};
const EMAIL_REGEXP = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
const stringIsEmail = data => {
  EMAIL_REGEXP.lastIndex = 0;
  return EMAIL_REGEXP.test(data);
};
const getDataFromUrl = url => {
  return new Promise((resolve, reject) => {
    if (!validateUrl(url)) return reject("invalid URL");
    request(url, null, async (err, res, body) => {
      if (err) {
        reject(err);
      }
      resolve({ dataType: res.headers["content-type"], body });
    });
  });
};
const resolveDataFromRequest = async ctx => {
  const { source, type, options, data } = ctx.request.body;
  switch (source) {
    case "upload":
      return { dataType: type, body: data, options };
    case "url":
      const { dataType, body } = await getDataFromUrl(options.url);
      return { dataType, body, options };
    case "raw":
      return {
        dataType: type,
        body: options.rawText,
        options
      };
  }
};

We are about to use above utility functions across our services. resolveDataFromRequest will receive the “analyze request” and detect the type of data alongside its content. For instance, if we are providing “external url” for our source of import, it will use getDataFromUrl function to fetch that url in order to extract the data and its type. Now that we have the data, we must parse that data to extract its items;

Append the following function to the utils.js:

const getItemsFromData = ({ dataType, body, options }) =>
  new Promise(async (resolve, reject) => {
    const parsedContentType = contentTypeParser(dataType);
    if (parsedContentType.isXML()) {
      const parser = new RssParser();
      const feed = await parser.parseString(body);
      return resolve({ sourceType: "rss", items: feed.items });
    }
    if (dataType === "text/csv" || dataType === "application/vnd.ms-excel") {
      const items = CsvParser(body, {
        ...options,
        columns: true
      });
      return resolve({ sourceType: "csv", items });
    }
    reject({
      contentType: parsedContentType.toString()
    });
  });

As you can see, we are using different parsers for different types of data. If the data type is xml we are using rss-parser and in case it is csv, we are using csv-parser.

For the last function, we want to see if a “url” contains any sort of media or not. Append the following code to the utils.js file:

const urlIsMedia = url => {
  try {
    const parsed = new URL(url);
    const extension = parsed.pathname
      .split(".")
      .pop()
      .toLowerCase();
    switch (extension) {
      case "png":
      case "gif":
      case "jpg":
      case "jpeg":
      case "svg":
      case "bmp":
      case "tif":
      case "tiff":
        return true;
      case "mp3":
      case "wav":
      case "ogg":
        return true;
      case "mp4":
      case "avi":
        return true;
      default:
        return false;
    }
  } catch (error) {
    return false;
  }
};

Well done! The only thing left, is to actually analyze the extracted items.

Before leaving this file, append the below exports to the bottom of the file:

module.exports = {
  resolveDataFromRequest,
  getItemsFromData,
  getDataFromUrl,
  stringIsEmail,
  urlIsMedia
};

Service: fieldUtils

Create a new file called fieldUtils.js inside utils directory:
services/utils/fieldUtils.js
Paste the below code inside that file:

const getUrls = require("get-urls");
const { urlIsMedia, stringIsEmail } = require("./utils");
const striptags = require("striptags");
const detectStringFieldFormat = data => {
  if (new Date(data).toString() !== "Invalid Date") return "date";
  if (stringIsEmail(data)) return "email";
  if (data.length !== striptags(data).length) {
    return "xml";
  }
  return "string";
};
const detectFieldFormat = data => {
  switch (typeof data) {
    case "number":
      return "number";
    case "boolean":
      return "boolean";
    case "object":
      return "object";
    case "string":
      return detectStringFieldFormat(data);
  }
};

The important factor in our analyzing process is that each item is consisted of several fields and for each field we want to know what exactly is that field’s format. For instance, the above detectFieldFormat function which is doing the format detection for us, when has an encounter with a string field, will do further investigations to find out if that field is a date or a url or even an xml as well.

Beside the field’s format, we want to learn more information such as if the field has a “media url” or what is the “length” of that field;

Append the following function to the file:

const compileStatsForFieldData = fieldData => {
  const stats = {};
  switch (typeof fieldData) {
    case "string":
      try {
        const urls = Array.from(getUrls(fieldData));
        const l = urls.length;
        for (let i = 0; i < l; ++i) {
          if (urlIsMedia(urls[i])) {
            stats.hasMediaUrls = true;
            break;
          }
        }
      } catch (e) {
        console.log(e);
      }
      stats.length = fieldData.length;
      break;
    case "object":
      if (urlIsMedia(fieldData.url)) {
        stats.hasMediaUrls = true;
      }
      stats.length = JSON.stringify(fieldData).length;
      break;
    default:
      console.log(typeof fieldData, fieldData);
  }
  stats.format = detectFieldFormat(fieldData);
  return stats;
};

For the last function in this file, we want to extract the “media urls” from the field:

const getMediaUrlsFromFieldData = fieldData => {
  switch (typeof fieldData) {
    case "string":
      return Array.from(getUrls(fieldData)).filter(urlIsMedia);
    case "object":
      return urlIsMedia(fieldData.url) ? [fieldData.url] : [];
  }
};

We are all set for the analyzing process!

Just before leaving the file append the exports at the bottom:

module.exports = {
  detectStringFieldFormat,
  detectFieldFormat,
  compileStatsForFieldData,
  getMediaUrlsFromFieldData
};

Service: analyzer

Create a new file named analyzer.js inside utils directory:
services/utils/analyzer.js
Paste the below code inside that file:

"use strict";
const _ = require("lodash");
var ss = require("simple-statistics");
const { compileStatsForFieldData } = require("./fieldUtils");
const getFieldNameSet = items => {
  const fieldNames = new Set();
  items.forEach(item => {
    try {
      Object.keys(item).forEach(fieldName => fieldNames.add(fieldName));
    } catch (e) {
      console.log(e);
    }
  });
  return fieldNames;
};
const analyze = (sourceType, items) => {
  const fieldNames = getFieldNameSet(items);
  const fieldAnalyses = {};
  fieldNames.forEach(fieldName => (fieldAnalyses[fieldName] = []));
  items.forEach(item => {
    fieldNames.forEach(fieldName => {
      const fieldData = item[fieldName];
      const fieldStats = compileStatsForFieldData(fieldData);
      fieldAnalyses[fieldName].push(fieldStats);
    });
  });
  const fieldStats = Object.keys(fieldAnalyses).map(fieldName => {
    const fieldAnalysis = fieldAnalyses[fieldName];
    const fieldStat = { fieldName, count: fieldAnalysis.length };
    try {
      fieldStat.format = _.chain(fieldAnalysis)
        .countBy("format")
        .map((value, key) => ({ count: value, type: key }))
        .sortBy("count")
        .reverse()
        .head()
        .get("type")
        .value();
    } catch (e) {
      console.log(e);
    }
    fieldStat.hasMediaUrls = fieldAnalysis.some(fa => Boolean(fa.hasMediaUrls));
    const lengths = _.map(fieldAnalysis, "length");
    fieldStat.minLength = ss.min(lengths);
    fieldStat.maxLength = ss.max(lengths);
    fieldStat.meanLength = ss.mean(lengths).toFixed(2);
    return fieldStat;
  });
  return { itemCount: items.length, fieldStats };
};
module.exports = { getFieldNameSet, analyze };

The getFieldNameSet will create a Set of unique field names available in every item. This way we will have all the fields across all the items without repeating ourselves. Then we receive the information about every field by calling compileStatsForFieldData function and based on that information, we will make an educated analytics for each item such as the number of fields per item, the format of fields for every item, existence of any “media url” on the item fields and the calculation of 3 different type of length based on the length of all the fields per item.

Alright! it’s time to actually use our “analyze function”;

Open ImportContent.js file inside services directory and change its content to the following:

"use strict";
/** * ImportContent.js service
 * * @description: A set of functions similar to controller's actions to avoid code duplication. */
const { resolveDataFromRequest, getItemsFromData } = require("./utils/utils");
const analyzer = require("./utils/analyzer");
module.exports = {
  preAnalyzeImportFile: async ctx => {
    const { dataType, body, options } = await resolveDataFromRequest(ctx);
    const { sourceType, items } = await getItemsFromData({
      dataType,
      body,
      options
    });
    const analysis = analyzer.analyze(sourceType, items);
    return { sourceType, ...analysis };
  }
};

Now that we have our service setup, let’s use it inside our controller.

Open ImportContent.js file inside controllers directory, remove the default content and paste the following:

"use strict";
module.exports = {
  preAnalyzeImportFile: async ctx => {
    const services = strapi.plugins["import-content"].services;
    try {
      const data = await services["importcontent"].preAnalyzeImportFile(ctx);
      ctx.send(data);
    } catch (error) {
      console.log(error);
      ctx.response.status = 406;
      ctx.response.message = "could not parse: " + error;
    }
  }
};

Awesome! With an action for the “analyzing process” Let’s setup an end point for this action as well.

Open routes.json inside config directory and paste the following:

{
  "routes": [
    {
      "method": "POST",
      "path": "/preAnalyzeImportFile",
      "handler": "ImportContent.preAnalyzeImportFile",
      "config": {
        "policies": []
      }
    }
  ]
}

Congratulations, we are a hell of an analyzer!

Allow public access on our new endpoint, try to upload a “csv file” and watch the console logs when you press the “Analyze” button, as you can see we are receiving the analytics for that file, Awesome!

Now the big question! How do we fill the content type with our imported data? Of course we can restrict the imported data to have the exact fields of our target content type, but this would be a huge disappointment for our plugin; because most of the times, our data will have different field counts and different field names in comparison to our content types; that being said, it is not always the field value that we are looking for, it might be a url and we may be interested in the media that the url is referring to, not the url itself. So what is the solution here? Yes, your guess is correct! We have to provide our users with a way to relate their imported data fields with the content type fields. And that is what we are going to do next

MappingTable Component

Go to the components directory and create a new directory named MappingTable with an index.js file inside:
admin/src/components/MappingTable/index.js
Open the index.js file and paste the following:

import React, { Component } from "react";
import PropTypes from "prop-types";
class MappingTable extends Component {
  state = { mapping: {} };
}
MappingTable.propTypes = {
  analysis: PropTypes.object.isRequired,
  targetModel: PropTypes.object,
  onChange: PropTypes.func
};
export default MappingTable;

Before we write our MappingTable UI, there are 2 components that we want to use inside this component.

Create a new file named TargetFieldSelect.js inside MappingTable directory:
admin/src/components/MappingTable/TargetFieldSelect.js
Paste the following inside that file:

import React, { Component } from "react";
import { Select } from "@buffetjs/core";
import { get } from "lodash";

class TargetFieldSelect extends Component {
  state = {
    selectedTarget: ""
  };
  componentDidMount() {
    const options = this.fillOptions();
    this.setState({ selectedTarget: options && options[0] });
  }
  onChange(selectedTarget) {
    this.props.onChange(selectedTarget);
    this.setState({ selectedTarget });
  }
  fillOptions() {
    const { targetModel } = this.props;
    const schemaAttributes = get(targetModel, ["schema", "attributes"], {});
    const options = Object.keys(schemaAttributes)
      .map(fieldName => {
        const attribute = get(schemaAttributes, [fieldName], {});

        return attribute.type && { label: fieldName, value: fieldName };
      })
      .filter(obj => obj !== undefined);

    return [{ label: "None", value: "none" }, ...options];
  }
  render() {
    return (
      <Select
        name={"targetField"}
        value={this.state.selectedTarget}
        options={this.fillOptions()}
        onChange={({ target: { value } }) => this.onChange(value)}
      />
    );
  }
}
export default TargetFieldSelect;

This component will show a “Select” element for choosing the desired field among “target content type" fields. This way our users can finally map fields of imported data with fields of their “target content type”. While this is sufficient for most cases, there are scenarios that we want to map certain qualities of a field to our “target content type” fields and not the direct value itself; for instance, if a field is pointing to a media url, we do not want the value of the “url”, instead we want the referenced media itself; or if the field contains any xml, do we want the contents with xml tags or without them (stripe tags).

So we have to show our TargetFieldSelect once again for covering these scenarios. We consider these scenarios as “mapping options”.

Create a new file named MappingOptions.js inside MappingTable directory:
admin/src/components/MappingTable/MappingOptions.js
Paste the below code inside:

import React, { Component } from "react";
import TargetFieldSelect from "./TargetFieldSelect";
import { Label } from "@buffetjs/core";
const MappingOptions = ({ stat, onChange, targetModel }) => {
  return (
    <div>
      {stat.format === "xml" && (
        <div>
          <Label htmlFor={"stripCheckbox"} message={"Strip Tags"} />
          <input
            name={"stripCheckbox"}
            type="checkbox"
            onChange={e => onChange({ stripTags: e.target.checked })}
          />
        </div>
      )}
      {stat.hasMediaUrls && (
        <div style={{ paddingTop: 8, paddingBottom: 8 }}>
          <Label
            htmlFor={"mediaTargetSelect"}
            message={"Import Media to Field"}
          />
          <TargetFieldSelect
            name={"mediaTargetSelect"}
            targetModel={targetModel}
            onChange={targetField =>
              onChange({ importMediaToField: targetField })
            }
          />
        </div>
      )}
    </div>
  );
};
export default MappingOptions;

Go back to index.js file inside MappingTable directory to use our newly created components. As you can see we have a mapping object defined in our state that we expect to get filled based on user specified field mappings.

We also expect our HomePage to provide this component with the analysis object which we received from our backend in the previous steps, the “target content type” which has been selected by the user and an onChange function which we will call whenever the user changes our “field mappings”.

Add the following imports to the index.js file:

...
import MappingOptions from "./MappingOptions";
import TargetFieldSelect from "./TargetFieldSelect";
import _ from "lodash";
import Row from "../Row";
import { Table } from "@buffetjs/core";
import {
  Bool as BoolIcon,
  Json as JsonIcon,
  Text as TextIcon,
  NumberIcon,
  Email as EmailIcon,
  Calendar as DateIcon,
  RichText as XmlIcon
} from "@buffetjs/icons";
...

For our table, we are going to use “Buffet js” table with a custom row which is filled with the analysis object handed down from HomePage component.

Paste the following inside MappingTable class:

CustomRow = ({ row }) => {
    const { fieldName, count, format, minLength, maxLength, meanLength } = row;
    return (
      <tr style={{ paddingTop: 18 }}>
        <td>{fieldName}</td>
        <td>
          <p>{count}</p>
        </td>
        <td>
          {format === "string" && <TextIcon fill="#fdd835" />}
          {format === "number" && <NumberIcon fill="#fdd835" />}
          {format === "boolean" && <BoolIcon fill="#fdd835" />}
          {format === "object" && <JsonIcon fill="#fdd835" />}
          {format === "email" && <EmailIcon fill="#fdd835" />}
          {format === "date" && <DateIcon fill="#fdd835" />}
          {format === "xml" && <XmlIcon fill="#fdd835" />} <p>{format}</p>
        </td>
        <td>
          <span>{minLength}</span>
        </td>
        <td>
          <p>{maxLength}</p>
        </td>
        <td>
          <p>{meanLength}</p>
        </td>
        <td>
          <MappingOptions
            targetModel={this.props.targetModel}
            stat={row}
            onChange={this.changeMappingOptions(row)}
          />
        </td>
        <td>
          {this.props.targetModel && (
            <TargetFieldSelect
              targetModel={this.props.targetModel}
              onChange={targetField => this.setMapping(fieldName, targetField)}
            />
          )}
        </td>
      </tr>
    );
  };

As you can see, we are using 2 methods named setMapping and changeMappingOptions which are as below:

  changeMappingOptions = stat => options => {
    let newState = _.cloneDeep(this.state);
    for (let key in options) {
      _.set(newState, `mapping[${stat.fieldName}][${key}]`, options[key]);
    }
    this.setState(newState, () => this.props.onChange(this.state.mapping));
  };
  setMapping = (source, targetField) => {
    const state = _.set(
      this.state,
      `mapping[${source}]['targetField']`,
      targetField
    );
    this.setState(state, () => this.props.onChange(this.state.mapping));
    console.log(this.state.mapping);
  };

What we expect from above methods, is to fill our mapping object based on user specified relations. changeMappingOptions is going to fill the mapping object with the specified options like this:

{
  sourceField1: {
    importMediaToField: targetField1
  },
  sourceField2: {
    stripTags: true
  }
}

and setMapping is going to fill the mapping object like below:

{
  sourceField1: {
    targetField: targetField1
  },
  sourceField2: {
    targetField: targetField2
  }
}

Awesome! for the final piece, let’s write our render method:

  render() {
    const { analysis } = this.props;
    const props = {
      title: "Field Mapping",
      subtitle:
        "Configure the Relationship between CSV Fields and Content type Fields"
    };
    const headers = [
      { name: "Field", value: "fieldName" },
      { name: "Count", value: "count" },
      { name: "Format", value: "format" },
      { name: "Min Length", value: "minLength" },
      { name: "Max Length", value: "maxLength" },
      { name: "Mean Length", value: "meanLength" },
      { name: "Options", value: "options" },
      { name: "Destination", value: "destination" }
    ];
    const items = [...analysis.fieldStats];
    return (
      <Table
        {...props}
        headers={headers}
        rows={items}
        customRow={this.CustomRow}
      />
    );
  }

We are all set! Let’s use our MappingTable inside HomePage component.

Open index.js file at HomePage directory and append the following to the render method output as the last child of parent div (parent div is the tag with container-fluid as its className):

       {this.state.analysis && (
          <Row className="row">
            <MappingTable
              analysis={this.state.analysis}
              targetModel={this.getTargetModel()}
              onChange={this.setFieldMapping}
            />
          </Row>
        )}

Import MappingTable by adding the following line:
import MappingTable from "../../components/MappingTable";
Append the following to index.js file as well:

  state = {
    loading: true,
    modelOptions: [],
    models: [],
    importSource: "upload",
    analyzing: false,
    analysis: null,
    selectedContentType: "",
    fieldMapping: {} // <---
  };

  getTargetModel = () => { // <---
    const { models } = this.state;
    if (!models) return null;
    return models.find(model => model.uid === this.state.selectedContentType);
  };
  
  setFieldMapping = fieldMapping => { // <---
    this.setState({ fieldMapping });
  };

Your index.js file should look like this:

import React, { memo, Component } from "react";
import {request} from "strapi-helper-plugin";
import PropTypes from "prop-types";
import pluginId from "../../pluginId";
import UploadFileForm from "../../components/UploadFileForm";
import ExternalUrlForm from "../../components/ExternalUrlForm";
import RawInputForm from "../../components/RawInputForm";
import MappingTable from "../../components/MappingTable";

import {
  HeaderNav,
  LoadingIndicator,
  PluginHeader
} from "strapi-helper-plugin";
import Row from "../../components/Row";
import Block from "../../components/Block";
import { Select, Label } from "@buffetjs/core";
import { get, has, isEmpty, pickBy, set } from "lodash";

const getUrl = to =>
  to ? `/plugins/${pluginId}/${to}` : `/plugins/${pluginId}`;

class HomePage extends Component {
  importSources = [
    { label: "External URL ", value: "url" },
    { label: "Upload file", value: "upload" },
    { label: "Raw text", value: "raw" }
  ];

  state = {
    loading: true,
    modelOptions: [],
    models: [],
    importSource: "upload",
    analyzing: false,
    analysis: null,
    selectedContentType: "",
    fieldMapping: {}
  };

  getTargetModel = () => {
    const { models } = this.state;
    if (!models) return null;
    return models.find(model => model.uid === this.state.selectedContentType);
  };
  
  setFieldMapping = fieldMapping => {
    this.setState({ fieldMapping });
  };

  selectImportDest = selectedContentType => {
    this.setState({ selectedContentType });
  };

  componentDidMount() {
    this.getModels().then((res) => {
      const {models, modelOptions} = res
      this.setState({
        models,
        modelOptions,
        selectedContentType: modelOptions ? modelOptions[0].value : ""
      });
    });
  }

  getModels = async () => {
    this.setState({ loading: true });
    try {
      const response = await request("/content-type-builder/content-types", {
        method: "GET"
      });

      // Remove content types from models
      const models = get(response, ["data"], []).filter(
        obj => !has(obj, "plugin")
      );
      const modelOptions = models.map(model => {
        return {
          label: get(model, ["schema", "name"], ""),
          value: model.uid
        };
      });

      this.setState({ loading: false });

      return { models, modelOptions };
    } catch (e) {
      this.setState({ loading: false }, () => {
        strapi.notification.error(`${e}`);
      });
    }
    return [];
  };

  selectImportSource = importSource => {
    this.setState({ importSource });
  };

  onRequestAnalysis = async analysisConfig => {
    this.analysisConfig = analysisConfig;
    this.setState({ analyzing: true }, async () => {
      try {
        const response = await request("/import-content/preAnalyzeImportFile", {
          method: "POST",
          body: analysisConfig
        });

        this.setState({ analysis: response, analyzing: false }, () => {
          strapi.notification.success(`Analyzed Successfully`);
        });
      } catch (e) {
        this.setState({ analyzing: false }, () => {
          strapi.notification.error(`${e}`);
        });
      }
    });
  };

  render() {
    return (
      <div className={"container-fluid"} style={{ padding: "18px 30px" }}>
        <PluginHeader
          title={"Import Content"}
          description={"Import CSV and RSS-Feed into your Content Types"}
        />
        <HeaderNav
          links={[
            {
              name: "Import Data",
              to: getUrl("")
            },
            {
              name: "Import History",
              to: getUrl("history")
            }
          ]}
          style={{ marginTop: "4.4rem" }}
        />
        <div className="row">
          <Block
            title="General"
            description="Configure the Import Source & Destination"
            style={{ marginBottom: 12 }}
          >
            <Row className={"row"}>
              <div className={"col-4"}>
                <Label htmlFor="importSource">Import Source</Label>
                <Select
                  name="importSource"
                  options={this.importSources}
                  value={this.state.importSource}
                  onChange={({ target: { value } }) =>
                    this.selectImportSource(value)
                  }
                />
              </div>
              <div className={"col-4"}>
                <Label htmlFor="importDest">Import Destination</Label>
                <Select
                  value={this.state.selectedContentType}
                  name="importDest"
                  options={this.state.modelOptions}
                  onChange={({ target: { value } }) =>
                    this.selectImportDest(value)
                  }
                />
              </div>
            </Row>
            <Row>
              {this.state.importSource === "upload" && (
                <UploadFileForm
                  onRequestAnalysis={this.onRequestAnalysis}
                  loadingAnalysis={this.state.analyzing}
                />
              )}
              {this.state.importSource === "url" && (
                <ExternalUrlForm
                  onRequestAnalysis={this.onRequestAnalysis}
                  loadingAnalysis={this.state.analyzing}
                />
              )}
              {this.state.importSource === "raw" && (
                <RawInputForm
                  onRequestAnalysis={this.onRequestAnalysis}
                  loadingAnalysis={this.state.analyzing}
                />
              )}
            </Row>
          </Block>
        </div>
        {this.state.analysis && (
          <Row className="row">
            <MappingTable
              analysis={this.state.analysis}
              targetModel={this.getTargetModel()}
              onChange={this.setFieldMapping}
            />
          </Row>
        )}
      </div>
    );
  }
}
export default memo(HomePage);

Import something, analyze and see the result yourself!

Congratulations! the last thing is to add an “Import Button” at the bottom of our MappingTable component to actually “Run the Import“ for us

Add the Button just under your MappingTable component tag inside HomePage/index.js file as in below code and don't forget to import the Button component at the top of the file:

import { Button } from "@buffetjs/core";

...

<Button
  style={{ marginTop: 12 }}
  label={"Run the Import"}
  onClick={this.onSaveImport}
/>
...

Add the onSaveImport method as below just under your state:

onSaveImport = async () => {
    const { selectedContentType, fieldMapping } = this.state;
    const { analysisConfig } = this;
    const importConfig = {
      ...analysisConfig,
      contentType: selectedContentType,
      fieldMapping
    };
    try {
      await request("/import-content", { method: "POST", body: importConfig });
      this.setState({ saving: false }, () => {
      strapi.notification.info("Import started");
      });
    } catch (e) {
      strapi.notification.error(`${e}`);
    }
  };

Awesome! Whenever the user clicks on “Run the Import” button, we send the “analysis result” alongside “target content type” and user provided “field mapping” to our backend API, and then our import will be initiated

Service: importItems

Let’s setup our endpoint.

Go to services directory and replace ImportContent.js file with the following:

"use strict";
/** * ImportContent.js service
 * * @description: A set of functions similar to controller's actions to avoid code duplication. */
const { resolveDataFromRequest, getItemsFromData } = require("./utils/utils");
const analyzer = require("./utils/analyzer");
const _ = require("lodash");
const importFields = require("./utils/importFields");
const importMediaFiles = require("./utils/importMediaFiles");

const import_queue = {};

module.exports = {
  preAnalyzeImportFile: async ctx => {
    const { dataType, body, options } = await resolveDataFromRequest(ctx);
    const { sourceType, items } = await getItemsFromData({
      dataType,
      body,
      options
    });
    const analysis = analyzer.analyze(sourceType, items);
    return { sourceType, ...analysis };
  },
  importItems: (importConfig, ctx) =>
    new Promise(async (resolve, reject) => {
      const { dataType, body } = await resolveDataFromRequest(ctx);
      console.log("importitems", importConfig);
      try {
        const { items } = await getItemsFromData({
          dataType,
          body,
          options: importConfig.options
        });
        import_queue[importConfig.id] = items;
      } catch (error) {
        reject(error);
      }
      resolve({
        status: "import started",
        importConfigId: importConfig.id
      });
      importNextItem(importConfig);
    }),
};

Similar to what we did in “analyzing”, we are extracting the data and parsing its items once again. Then we will queue those parsed items for further processing. What we need here is a function that iterates on that queue and stores every item.

Add the following function before the module.exports:

const importNextItem = async importConfig => {
  const sourceItem = import_queue[importConfig.id].shift();
  if (!sourceItem) {
    console.log("import complete");
    await strapi
      .query("importconfig", "import-content")
      .update({ id: importConfig.id }, { ongoing: false });
    return;
  }
  try {
    const importedItem = await importFields(
      sourceItem,
      importConfig.fieldMapping
    );
    const savedContent = await strapi
      .query(importConfig.contentType)
      .create(importedItem);
    const uploadedFiles = await importMediaFiles(
      savedContent,
      sourceItem,
      importConfig
    );
    const fileIds = _.map(_.flatten(uploadedFiles), "id");
    await strapi.query("importeditem", "import-content").create({
      importconfig: importConfig.id,
      ContentId: savedContent.id,
      ContentType: importConfig.contentType,
      importedFiles: { fileIds }
    });
  } catch (e) {
    console.log(e);
  }
  const { IMPORT_THROTTLE } = strapi.plugins["import-content"].config;
  setTimeout(() => importNextItem(importConfig), IMPORT_THROTTLE);
};

Our importing process at above code is consisted of creating the content type, importing the media files, and creating an “imported item” entry that will keep track of our created content types for the future; for instance, if we ever want to undo our imports, we must know which content types entries are related to this specific import.

Service: importFields

Before we continue on to create our action, there are 2 functions that we are calling here: importFields and importMediaFiles.

Go to utils directory and create a new file named importFields.js:
services/utils/importFields.js
Paste the below code inside the new file:

const striptags = require("striptags");
const importFields = async (sourceItem, fieldMapping) => {
  const importedItem = {};
  Object.keys(fieldMapping).forEach(async sourceField => {
    const { targetField, stripTags } = fieldMapping[sourceField];
    if (!targetField || targetField === "none") {
      return;
    }
    const originalValue = sourceItem[sourceField];
    importedItem[targetField] = stripTags
      ? striptags(originalValue)
      : originalValue;
  });
  return importedItem;
};
module.exports = importFields;

the “field mapping” object is a relation mapping between “imported data item” fields and “target content type” fields. The above function extracts the value for the target fields and generate the candidate object for our “content type” entry on top of those fields.

Service: importMediaFiles

Create a new file named importMediaFiles.js in utils directory:
services/utils/importMediaFiles.js
Paste the following:

const _ = require("lodash");
const request = require("request");
const fileFromBuffer = require("./fileFromBuffer");
const { getMediaUrlsFromFieldData } = require("../utils/fieldUtils");

const fetchFiles = url =>
  new Promise((resolve, reject) => {
    request({ url, method: "GET", encoding: null }, async (err, res, body) => {
      if (err) {
        reject(err);
      }
      const mimeType = res.headers["content-type"].split(";").shift();
      const parsed = new URL(url);
      const extension = parsed.pathname
        .split(".")
        .pop()
        .toLowerCase();
      resolve(fileFromBuffer(mimeType, extension, body));
    });
  });
const storeFiles = async file => {
  const uploadProviderConfig = await strapi
    .store({
      environment: strapi.config.environment,
      type: "plugin",
      name: "upload"
    })
    .get({ key: "provider" });
  return await strapi.plugins["upload"].services["upload"].upload(
    [file],
    uploadProviderConfig
  );
};
const relateFileToContent = ({
  contentType,
  contentId,
  targetField,
  fileBuffer
}) => {
  fileBuffer.related = [
    {
      refId: contentId,
      ref: contentType,
      source: "content-manager",
      field: targetField
    }
  ];
  return fileBuffer;
};
const importMediaFiles = async (savedContent, sourceItem, importConfig) => {
  const { fieldMapping, contentType } = importConfig;
  const uploadedFileDescriptors = _.mapValues(
    fieldMapping,
    async (mapping, sourceField) => {
      if (mapping.importMediaToField) {
        const urls = getMediaUrlsFromFieldData(sourceItem[sourceField]);
        const fetchPromises = _.uniq(urls).map(fetchFiles);
        const fileBuffers = await Promise.all(fetchPromises);
        const relatedContents = fileBuffers.map(fileBuffer =>
          relateFileToContent({
            contentType,
            contentId: savedContent.id,
            targetField: mapping.importMediaToField,
            fileBuffer
          })
        );
        const storePromises = relatedContents.map(storeFiles);
        const storedFiles = await Promise.all(storePromises);
        console.log(_.flatten(storedFiles));
        return storedFiles;
      }
    }
  );
  return await Promise.all(_.values(uploadedFileDescriptors));
};
module.exports = importMediaFiles;

What we are doing here is to check if we have importMediaToField specified on the mapping or not, and in case we have, we first extract the urls, then fetch the files for those urls, relate them to the target content type and store them as well.

Service: fileFromBuffer

Create a new file named fileFromBuffer.js in utils directory:
services/utils/fileFromBuffer.js
Paste the following code inside that file:

const crypto = require('crypto')
const uuid = require("uuid/v4");

function niceHash(buffer) {
  return crypto
    .createHash("sha256")
    .update(buffer)
    .digest("base64")
    .replace(/=/g, "")
    .replace(/\//g, "-")
    .replace(/\+/, "_");
}
const fileFromBuffer = (mimeType, extension, buffer) => {
  const fid = uuid();
  return {
    buffer,
    sha256: niceHash(buffer),
    hash: fid.replace(/-/g, ""),
    name: `${fid}.${extension}`,
    ext: `.${extension}`,
    mime: mimeType,
    size: (buffer.length / 1000).toFixed(2)
  };
};
module.exports = fileFromBuffer;

our service is ready!

Open ImportContent.js file inside controllers directory and add the following function:

create: async ctx => {
    const services = strapi.plugins["import-content"].services;
    const importConfig = ctx.request.body;
    importConfig.ongoing = true;
    const record = await strapi
      .query("importconfig", "import-content")
      .create(importConfig);
    console.log("create", record);
    await services["importcontent"].importItems(record, ctx);
    ctx.send(record);
}

For each import, we are storing a specific config entry called “import config”. This will help us keep details of our import like the date of this import and the identification of all the content types entries that are created because of this import.

Open routes.json inside config directory and define our new route:

{
      "method": "POST",
      "path": "/",
      "handler": "ImportContent.create",
      "config": {
        "policies": []
      }
}

Awesome! Our import endpoint is implemented; The main purpose of this plugin is accomplished. The rest is to show our initiated imports to the users. Also we are going to allow our users to “undo” the initiated imports as well as “removing” their related configs.

Add the following function to ImportContent.js file inside controllers directory:

index: async ctx => {
    const entries = await strapi.query("importconfig", "import-content").find();
    const withCounts = entries.map(entry => ({
      ...entry,
      importedCount: entry.importeditems.length,
      importeditems: []
    }));
    const withName = withCounts.map(entry =>
      ({
        ...entry,
        contentType: strapi.contentTypes[entry.contentType].info.name || 
        entry.contentType
      }))
    ctx.send(withName);
}

it will retrieve the stored configs for user initiated imports. This will allow us to provide our users with a history of their imports and show some details about each import. Beside retrieving the import configs, it would be nice to have a way to delete them as well.

Append the following function to the file:

  delete: async ctx => {
    const importId = ctx.params.importId;
    const res = await strapi.query("importconfig", "import-content").delete({
      id: importId
    });
    if (res && res.id) {
      ctx.send(res.id);
    } else {
      ctx.response.status = 400;
      ctx.response.message = "could not delete: the provided id might be wrong";
    }
  }

The only thing left is a way to undo our imports. For this we are about to implement a service similar to what we did for importing data.

Service: undoImports

Open ImportContent.js file in services directory, and add the following before the module.exports:

const undo_queue = {};
const removeImportedFiles = async (fileIds, uploadConfig) => {
  const removePromises = fileIds.map(id =>
    strapi.plugins["upload"].services.upload.remove({ id }, uploadConfig)
  );
  return await Promise.all(removePromises);
};
const undoNextItem = async (importConfig, uploadConfig) => {
  const item = undo_queue[importConfig.id].shift();
  if (!item) {
    console.log("undo complete");
    await strapi
      .query("importconfig", "import-content")
      .update({ id: importConfig.id }, { ongoing: false });
    return;
  }
  try {
    await strapi.query(importConfig.contentType).delete({ id: item.ContentId });
  } catch (e) {
    console.log(e);
  }
  try {
    const importedFileIds = _.compact(item.importedFiles.fileIds);
    await removeImportedFiles(importedFileIds, uploadConfig);
  } catch (e) {
    console.log(e);
  }
  try {
    await strapi.query("importeditem", "import-content").delete({
      id: item.id
    });
  } catch (e) {
    console.log(e);
  }
  const { UNDO_THROTTLE } = strapi.plugins["import-content"].config;
  setTimeout(() => undoNextItem(importConfig, uploadConfig), UNDO_THROTTLE);
};

The undoNextItem function is iterating on the queue for removing every imported item. This undo process is consisted of removing the content type entry, removing the related files and the imported item entry as well.

Now add the below function to module.exports:

undoItems: importConfig =>
    new Promise(async (resolve, reject) => {
      try {
        undo_queue[importConfig.id] = importConfig.importeditems;
      } catch (error) {
        reject(error);
      }
      await strapi
        .query("importconfig", "import-content")
        .update({ id: importConfig.id }, { ongoing: true });
      resolve({
        status: "undo started",
        importConfigId: importConfig.id
      });
      const uploadConfig = await strapi
        .store({
          environment: strapi.config.environment,
          type: "plugin",
          name: "upload"
        })
        .get({ key: "provider" });
      undoNextItem(importConfig, uploadConfig);
}),

well done!

Go back to ImportContent.js file inside controllers directory and add the following function:

undo: async ctx => {
    const services = strapi.plugins["import-content"].services;
    const importId = ctx.params.importId;
    const importConfig = await strapi
      .query("importconfig", "import-content")
      .findOne({ id: importId });
    console.log("undo", importId);
    await services["importcontent"].undoItems(importConfig);
    ctx.send(importConfig);
}

Open routes.json file at config directory and append the following routes:

    {
      "method": "GET",
      "path": "/",
      "handler": "ImportContent.index",
      "config": {
        "policies": []
      }
    },
    {
      "method": "DELETE",
      "path": "/:importId",
      "handler": "ImportContent.delete",
      "config": {
        "policies": []
      }
    },
    {
      "method": "POST",
      "path": "/:importId/undo",
      "handler": "ImportContent.undo",
      "config": {
        "policies": []
      }
    }

Now that we have all the endpoints we need, let’s go back to our admin directory and implement the HistoryPage. This page will show a history of all the user initiated imports, their status and 2 buttons for undo and delete actions.

Warning! while we are OK to continue to the next section, there is one point we have to mention here: Until now we have only provided endpoints for the functionalities that we already have like "analyzing", "importing" and running an "undo" operation. The point is that if we want to be consistent with other plugins like "graphql", we must define a set of controllers for each one of our content types which in our case are importconfig & importeditem. These controllers are the famous set of find, findOne, count, update and delete which i bet you are already familiar with them. So in case that you want this plugin to be efficient in production environments, you must make it consistent. Before jumping to the next section, please do the followings:

Create a new file named ImportConfig in controllers directory:

controllers/ImportConfig.js

Paste the following code inside that file:

module.exports = {
  count: async ctx => {
    const entries = await strapi.query("importconfig", "import-content").count(ctx.request.query)
    ctx.send(entries)
  },
  findOne: async ctx => {
    const entries = await strapi.query("importconfig", "import-content").findOne({id: ctx.params.importId})
    ctx.send(entries)
  },
  find: async ctx => {
    const entries = await strapi.query("importconfig", "import-content").find(ctx.request.query)
    ctx.send(entries)
  },
  delete: async ctx => {
    return strapi.query("importconfig", "import-content").delete({id: ctx.params.importId})
  },
  update: async ctx => {
    return strapi.query("importconfig", "import-content").update({id: ctx.params.importId}, ctx.request.body)
  }
};

Create a new file named ImportedItem in controllers directory:

controllers/ImportedItem.js

Paste the following code inside that file:

module.exports = {
  count: async ctx => {
    const entries = await strapi.query("importeditem", "import-content").count(ctx.request.query)
    ctx.send(entries)
  },
  findOne: async ctx => {
    const entries = await strapi.query("importeditem", "import-content").findOne({id: ctx.params.importId})
    ctx.send(entries)
  },
  find: async ctx => {
    const entries = await strapi.query("importeditem", "import-content").find(ctx.request.query)
    ctx.send(entries)
  },
  delete: async ctx => {
    return strapi.query("importeditem", "import-content").delete({id: ctx.params.importId})
  },
  update: async ctx => {
    return strapi.query("importeditem", "import-content").update({id: ctx.params.importId}, ctx.request.body)
  }
};

For the last step, add the below routes to the routes.json:

config/routes.json

{
  "method": "GET",
  "path": "/importconfig",
  "handler": "ImportConfig.find",
  "config": {
    "policies": []
  }
},
{
  "method": "GET",
  "path": "/importconfig/:importId",
  "handler": "ImportConfig.findOne",
  "config": {
    "policies": []
  }
},
{
  "method": "GET",
  "path": "/importconfig/count",
  "handler": "ImportConfig.count",
  "config": {
    "policies": []
  }
},
{
  "method": "PUT",
  "path": "/importconfig/:importId",
  "handler": "ImportConfig.update",
  "config": {
    "policies": []
  }
},
{
  "method": "DELETE",
  "path": "/importconfig/:importId",
  "handler": "ImportConfig.delete",
  "config": {
    "policies": []
  }
},
{
  "method": "GET",
  "path": "/importeditem",
  "handler": "ImportedItem.find",
  "config": {
    "policies": []
  }
},
{
  "method": "GET",
  "path": "/importeditem/:importId",
  "handler": "ImportedItem.findOne",
  "config": {
    "policies": []
  }
},
{
  "method": "GET",
  "path": "/importeditem/count",
  "handler": "ImportedItem.count",
  "config": {
    "policies": []
  }
},
{
  "method": "PUT",
  "path": "/importeditem/:importId",
  "handler": "ImportedItem.update",
  "config": {
    "policies": []
  }
},
{
  "method": "DELETE",
  "path": "/importeditem/:importId",
  "handler": "ImportedItem.delete",
  "config": {
    "policies": []
  }
}

Great you finished the third part of this tutorial! Let's create the History page! https://strapi.io/blog/how-to-create-an-import-content-plugin-part-4-4

Pouya Miralayi

Senior Software Engineer @ Ertebat Gostar Khavarmiane

Pouya is an active member of the Strapi community, who has been contributing actively with contributions in the core and plugins.